A target speaker extraction method, device, equipment and storage medium

Abstract

The application is suitable for the field of speech extraction, and provides a target speaker extraction method, device and equipment and a storage medium. The method comprises the following steps: based on mixed speech data and multi-modal clues, generating a plurality of first attribute level features and speech feature evidence of a target speaker; based on the speech feature evidence, global semantic features and the first attribute level features, generating a plurality of second attribute level features; based on the global semantic features and the second attribute level features, generating adaptive weights of all the second attribute level features through a routing unit of a mixed expert model, and generating adaptive features corresponding to each second attribute level feature according to the adaptive weights; generating a low-level feature group and a high-level feature group based on all the adaptive features, and extracting target speaker speech from the mixed speech data according to the low-level feature group and the high-level feature group, so as to realize accurate extraction of target speaker speech under the condition of a plurality of unreliable auxiliary clues.

Description

Technical Field

[0001] This invention belongs to the field of speech extraction technology, and particularly relates to a method, apparatus, device and storage medium for extracting a target speaker. Background Technology

[0002] Target-Speaker Extraction (TSE) is an important research direction in the field of speech separation. Its goal is to extract the speech signal of a specific target speaker from mixed speech containing multiple speakers. This technology has wide application value in scenarios such as voice communication, remote conferencing, voice assistants, human-computer interaction, and hearing-assisted devices.

[0003] In recent years, with the development of deep learning technology, target speaker extraction methods have gradually evolved from traditional single-modal methods to multi-modal methods. Existing target speaker extraction methods improve the model's ability to distinguish target speakers by introducing auxiliary cues, such as lip movement video, speaker registration speech, text information, emotion descriptions, and speaking style information. However, in real-world applications, these auxiliary cues are often not always reliable. For example, lip movement information may be difficult to recognize due to occlusion or blurring; speaker registration speech may be contaminated by noise or inconsistent with the target speaker; text cues may contain errors or not match the target speech content; and emotion or style descriptions may be too coarse or even incorrect. These problems mean that target speaker extraction not only needs to integrate information from different modalities and judge the reliability of different modal cues, but also needs to filter the correct cues when there are conflicts or errors.

[0004] Therefore, how to accurately extract the speech of the target speaker under various unreliable auxiliary cues has become an important problem facing current technology. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, device, and storage medium for extracting target speakers, aiming to solve the problem that existing technologies cannot accurately extract target speaker speech under various unreliable auxiliary cues.

[0006] In a first aspect, the present invention provides a method for extracting a target speaker, the method comprising the following steps: Receive mixed speech data with a target speaker, generate a first feature map of the mixed speech data, and obtain global semantic features of the first feature map and multimodal cues for assisting in the extraction of the target speaker; Based on the mixed speech data and the multimodal cues, several first attribute-level features and speech feature evidence of the target speaker are generated; Based on the speech feature evidence, the global semantic features, and the first attribute-level features, several second attribute-level features are generated. Based on the global semantic features and the second attribute-level features, an adaptive weight for all the second attribute-level features is generated through a hybrid expert model. Based on the adaptive weight, an adaptive feature corresponding to each second attribute-level feature is generated. Based on all the adaptive features, a low-level feature set and a high-level feature set are generated, and the target speaker's speech is extracted from the mixed speech data according to the low-level feature set and the high-level feature set.

[0007] In some embodiments, a number of first attribute-level features and speech feature evidence of the target speaker are generated by a feature refinement module, wherein the feature refinement module includes a first encoder, a projection head, a coarse extractor, and a second encoder. The step of generating several first-attribute-level features and speech feature evidence of the target speaker based on the mixed speech data and the multimodal cues includes: Based on the multimodal cues, several cue features are generated by the first encoder, and each cue feature is decomposed into several first attribute-level features by the projection head; Based on all the first attribute-level features with the same attribute, coarse-grained features corresponding to the first attribute-level features of each attribute are generated respectively. Based on all the coarse-grained features and the first feature map, the coarse extractor generates the coarse estimated speech of the target speaker. Based on the coarsely estimated speech, several speech feature evidences of the target speaker are obtained through the second encoder.

[0008] In some embodiments, the multimodal cues include several global cues and dynamic cues, and the cue features include global features and dynamic features; The step of generating several cue features based on the multimodal cues using the first encoder includes: The first encoder generates global features corresponding to each global cue. Obtain the visual feature sequence of the dynamic cue, and based on the visual feature sequence, generate the dynamic features corresponding to the dynamic cue through the first encoder.

[0009] In some embodiments, the step of generating a low-level feature set and a high-level feature set based on all the adaptive features, and extracting the target speaker's speech from the mixed speech data according to the low-level feature set and the high-level feature set includes: A first-dimensional feature is generated based on the first feature map and the low-level feature group; a second-dimensional feature is generated based on the visual feature sequence; and a second feature map is generated based on the first-dimensional feature and the second-dimensional feature. The global semantic features, the low-level feature group, and the high-level feature group are input into the SepFormer network to generate the first input weight and the second input weight of each SepFormer module in the SepFormer network. The low-level feature group is input into the internal block of each SepFormer module, and a first conditional feature is generated based on the first input weight and the low-level feature group. The high-level feature group is input into the inter-layer block of the SepFormer module starting from the second-level SepFormer module, and a second conditional feature is generated based on the second input weight and the high-level feature group. A target mask is generated based on the second feature map, the first conditional feature, and the second conditional feature. A third feature map is generated based on the first feature map and the target mask. The third feature map is then decoded to generate the target speaker's speech.

[0010] In some embodiments, the SepFormer network includes a temporal visual fusion module, which generates a first-dimensional feature based on the first feature map and the low-level feature group, generates a second-dimensional feature based on the visual feature sequence, and generates a second feature map based on the first-dimensional feature and the second-dimensional feature, including: Based on the temporal dimension of the first feature map, the low-level feature group is fused with the first feature map through the temporal visual fusion module to obtain the first dimension feature. The second dimension feature is generated based on the channel dimension of the SepFormer network and the visual feature sequence; The second feature map is generated based on the first dimension feature and the second dimension feature.

[0011] In some embodiments, the step of generating several second attribute-level features based on the speech feature evidence, the global semantic features, and the first attribute-level features includes: Based on the speech feature evidence, the global semantic features, and the first attribute-level features, a third attribute-level feature is generated; Based on the speech feature evidence, the first attribute-level feature, and the third attribute-level feature, a fusion weight is generated, and a fusion feature of the first attribute-level feature and the third attribute-level feature is generated according to the fusion weight. Based on the speech feature evidence, the global semantic features, and the fused features, a reliability probability is generated; based on the reliability probability and the fused features, a fourth attribute-level feature is generated. Based on the speech feature evidence, the global semantic features, and the fourth attribute-level features, competitive weights are generated. Based on the competitive weights, the fourth attribute-level features, and the attributes of the fourth attribute-level features, the second attribute-level features are generated.

[0012] In some embodiments, the hybrid expert model includes several expert networks and routing units. The step of generating adaptive weights for all second attribute-level features based on the global semantic features and the second attribute-level features through the routing units of the hybrid expert model, and generating adaptive features corresponding to each second attribute-level feature according to the adaptive weights, includes: Based on the global semantic features and the second attribute-level features, the routing score of each expert network is calculated through the routing unit of the hybrid expert model, and the adaptive weights are generated based on the routing scores; Based on the adaptive weights, the adaptive features corresponding to the second attribute-level features are generated through the expert network.

[0013] In a second aspect, the present invention also includes a target speaker extraction device, the device comprising: The data acquisition unit is used to receive mixed speech data with a target speaker, generate a first feature map of the mixed speech data, and acquire global semantic features of the first feature map and multimodal cues for assisting in the extraction of the target speaker. The first feature generation unit is used to generate several first attribute-level features and speech feature evidence of the target speaker based on the mixed speech data and the multimodal cues. The second feature generation unit is used to generate several second attribute-level features based on the speech feature evidence, the global semantic features, and the first attribute-level features. The third feature generation unit is used to generate adaptive weights for all second attribute-level features based on the global semantic features, and to generate adaptive features corresponding to each second attribute-level feature according to the adaptive weights. The speech extraction unit is used to generate a low-level feature group and a high-level feature group based on all the adaptive features, and to extract the target speaker's speech from the mixed speech data according to the low-level feature group and the high-level feature group.

[0014] Thirdly, the present invention also provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0015] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0016] In the target speaker extraction process of this invention, several second-attribute-level features are generated based on speech feature evidence, global semantic features, and first-attribute-level features. Based on the global semantic features and second-attribute-level features, an adaptive weight for all second-attribute-level features is generated through a hybrid expert model. According to the adaptive weight, an adaptive feature corresponding to each second-attribute-level feature is generated. Then, a low-level feature group and a high-level feature group are generated based on all adaptive features. The target speaker's speech is extracted from the mixed speech data based on the low-level feature group and the high-level feature group, thereby achieving accurate extraction of the target speaker's speech under various unreliable auxiliary cues. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the method for extracting a target speaker provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating step S102 in the target speaker extraction method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart illustrating step S105 in the target speaker extraction method provided in Embodiment 3 of the present invention; Figure 4 This is a flowchart illustrating step S103 in the target speaker extraction method provided in Embodiment 4 of the present invention; Figure 5 This is a flowchart illustrating step S104 in the target speaker extraction method provided in Embodiment 5 of the present invention; Figure 6 This is a schematic diagram of the target speaker extraction device provided in Embodiment Six of the present invention; Figure 7 This is a schematic diagram of the structure of the computing device provided in Embodiment 7 of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. Furthermore, the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. The terms "first," "second," and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Above," "below," "left," "right," etc., are used only to indicate relative positional relationships, which may change accordingly when the absolute position of the described object changes.

[0020] To keep the following description of the embodiments of the present invention clear and concise, detailed descriptions of some known functions and known components are omitted in this specification.

[0021] The specific implementation of the present invention will be described in detail below with reference to specific embodiments: Example 1 This embodiment provides a method for extracting a target speaker, which can be used on mobile terminals such as mobile phones and tablets. Figure 1 The flowchart of a target speaker extraction method provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: In step S101, mixed speech data with the target speaker is received, a first feature map of the mixed speech data is generated, and global semantic features of the first feature map and multimodal cues for assisting in the extraction of the target speaker are obtained.

[0022] In this embodiment of the invention, the mixed speech data refers to speech data containing multiple speakers. Specifically, a first feature map can be generated by performing mixed encoding on the mixed speech data, and global semantic features can be obtained by performing global average pooling on the first feature map. Multimodal cues are data in various formats used to determine the target speaker. Specifically, multimodal cues may include lip movement video sequences, speaker registration speech, transcribed text, emotion tags, and style tags. The lip movement video sequence is a series of lip movement images of the speaker speaking; the speaker registration speech is a pre-recorded speech sample of the speaker; the transcribed text is the text content transcribed from the speaker's speech; and the emotion tags and style tags are the emotional and stylistic features of the speaker's speech.

[0023] In step S102, based on the mixed speech data and multimodal cues, several first attribute-level features and speech feature evidence of the target speaker are generated.

[0024] In this embodiment of the invention, each multimodal cue can be decomposed into several attribute-level features, and the resulting features are called first attribute-level features. Specifically, each multimodal cue can be decomposed into four first attribute-level features, each corresponding to a different attribute. The attributes of the first attribute-level features can be four types: identity, pronunciation, content, and prosody.

[0025] In step S103, several second attribute-level features are generated based on speech feature evidence, global semantic features, and first attribute-level features.

[0026] In this embodiment of the invention, when generating several second attribute-level features, specifically, based on speech feature evidence and global semantic features, the first attribute-level features are sequentially modified, gated, and subjected to competition processing to obtain the second attribute-level features.

[0027] In step S104, based on the global semantic features and the second attribute-level features, an adaptive weight for all second attribute-level features is generated through a hybrid expert model. Based on the adaptive weight, an adaptive feature corresponding to each second attribute-level feature is generated.

[0028] In this embodiment of the invention, when generating the adaptive feature corresponding to each second attribute-level feature, the second attribute-level feature and the global semantic feature are input into the hybrid expert model to generate the adaptive weight corresponding to the second attribute-level feature. The adaptive weight is then multiplied by the corresponding second attribute-level feature to obtain the adaptive feature corresponding to the second attribute-level feature.

[0029] In this embodiment of the invention, the Mixture of Experts (MoE) model includes E expert networks. In some embodiments, as an example, each expert network can be configured as a multilayer perceptron (MLP), the value of E can be set to 4, and the routing unit can adopt a top-2 routing strategy.

[0030] In step S105, a low-level feature group and a high-level feature group are generated based on all adaptive features, and the target speaker's speech is extracted from the mixed speech data according to the low-level feature group and the high-level feature group.

[0031] In this embodiment of the invention, when generating low-level feature groups and high-level feature groups based on all adaptive features, the adaptive features can optionally be divided into two groups to obtain low-level feature groups and high-level feature groups. The low-level feature groups correspond to two attributes: identity and pronunciation, while the high-level feature groups correspond to two attributes: content and prosody. Based on the low-level feature groups and high-level feature groups, the target speaker's speech is extracted from the mixed speech data.

[0032] This invention generates several second-attribute-level features based on speech feature evidence, global semantic features, and first-attribute-level features, thereby improving the reliability of the second-attribute-level features. Then, based on all adaptive features, low-level feature groups and high-level feature groups are generated. The target speaker's speech is extracted from mixed speech data according to the low-level feature groups and high-level feature groups. Finally, the target speaker's speech can be accurately extracted under the condition of multiple unreliable multimodal cues.

[0033] Example 2 Figure 2 The implementation flow of step S102 in the target speaker extraction method provided in Embodiment 1 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below: In step S201, based on multimodal cues, several cue features are generated by the first encoder, and each cue feature is decomposed into several first attribute-level features by the projection head.

[0034] In some embodiments of the present invention, based on mixed speech data and multimodal cues, a feature refinement module generates several first attribute-level features and speech feature evidence of the target speaker. As an example, the feature refinement module may include a first encoder, a projection head, a coarse extractor, and a second encoder, wherein the first encoder encodes each multimodal cue to obtain cue features corresponding to each multimodal cue. Specifically, the multimodal cue may include several global cues and dynamic cues, and the cue features may include global features and dynamic features. In this case, when generating several cue features based on multimodal cues through the first encoder, it can be achieved in the following way: (1) Generate global features corresponding to each global cue in the multimodal cue through the first encoder.

[0035] In this embodiment of the invention, the multimodal cues include lip-sync video sequences, speaker registration speech, transcribed text, sentiment tags, and style tags. When generating global features for each global cue, the speaker registration speech is input into a speech encoder to extract its embedding vector, which is then set as the registration speech cue representation. The transcribed text is input into a text encoder to extract its semantic features, which are then set as the text cue representation. The sentiment tags and style tags are input into a joint model (Contrastive Language-Audio Pretraining, CLAP) composed of the sentiment encoder and style encoder, respectively, to encode the sentiment tags and style tags, obtaining sentiment cue representations and style cue representations. In other words, the global features corresponding to each global cue include the registration speech cue representation, the text cue representation, the sentiment cue representation, and the style cue representation. For example, the sentiment tag could be "neutral," "happy," etc., and the style tag could be "She is a female speaker with a high pitch," etc.

[0036] (2) Obtain the visual feature sequence of the dynamic cue, and generate the dynamic features corresponding to the dynamic cue through the first encoder based on the visual feature sequence.

[0037] In some embodiments, when generating dynamic features corresponding to dynamic cues, the lip video sequence, which serves as a dynamic cue, is input into a lip encoder to extract time-varying lip visual features from the lip video sequence. These time-varying lip visual features are then averaged along the temporal dimension to obtain a global lip cue representation.

[0038] In this embodiment of the invention, based on the obtained global lip cue representation, registered speech cue representation, text cue representation, emotion cue representation, and style cue representation, a unified mapping is performed through at least one set of learnable projection matrices to obtain cue features with consistent dimensions.

[0039] In an embodiment of the present invention, as an example, the obtained clue features can be represented as follows: (j=1,…,5) The mapping space represents the feature of the clues, where d represents the dimension of the space, which can be 256, 512, etc.

[0040] In this embodiment of the invention, when decomposing each clue feature into several first attribute-level features using a projection head, each clue feature is decomposed using a set of projection heads with specific attributes to obtain four attribute-level features. As an example, the decomposition of each clue feature can be represented as follows: ; in, This represents the first-level attribute feature. Indicates the projection head. Indicate clue characteristics, This represents an attribute, where id represents identity. Indicates pronunciation, Represents content, To represent rhythm, the corresponding first-attribute level features include the first-attribute level features of identity attributes, pronunciation attributes, content attributes, and rhythm attributes.

[0041] In some embodiments, when decomposing multimodal cues to obtain first attribute-level features, if any first attribute-level feature of identity, pronunciation, content, or prosody is missing, a null token can be set as a substitute. For example, the null token is represented as: .

[0042] In step S202, based on all first attribute-level features of the same attribute, coarse-grained features corresponding to the first attribute-level features of each attribute are generated respectively. Based on all coarse-grained features and the first feature map, a coarse estimate of the target speaker's speech is generated by a coarse extractor.

[0043] In this embodiment of the invention, when generating coarse-grained features corresponding to the first attribute-level features of each attribute based on all first attribute-level features of the same attribute, average pooling can be performed on all first attribute-level features of the same attribute to obtain a coarse-grained feature corresponding to each attribute. As an example, coarse-grained features can be generated using the following formula: , ; Where J is the number of clue features, This represents coarse-grained features. For example, J can be set to 5. There can be four coarse-grained features: coarse-grained features of identity attributes, coarse-grained features of pronunciation attributes, coarse-grained features of content attributes, and coarse-grained features of prosody attributes.

[0044] In this embodiment of the invention, when generating a coarse estimate of the target speaker's speech using a coarse extractor based on all coarse-grained features and the first feature map, the coarse-grained features and the first feature map can be... Figure 1 A lightweight, shallow, dual-path extraction network with the same input coarse extractor generates a coarse estimate of the target speaker's speech. As an example, the coarse estimate can be generated using the following formula: ; in, This indicates a rough estimate of the speech. Indicates the crude extractor. Represents the first feature map. This represents the time dimension of the mixed speech data. This represents a feature group consisting of coarse-grained features with all different attributes.

[0045] In step S203, based on the coarsely estimated speech, several speech feature evidences of the target speaker are obtained through the second encoder.

[0046] In this embodiment of the invention, speech feature evidence is extracted from the coarsely estimated speech using a second encoder. Specifically, the second encoder may include a one-dimensional convolutional backbone network and an output head with four specific attributes. As an example, speech feature evidence can be generated using the following formula: , attr∈{id, art, cont, pros}; in, Indicates evidence of speech features, Indicates the output header. This represents a one-dimensional convolutional backbone network.

[0047] As an example, the number of output heads can be set to four, namely the output head for identity attributes, the output head for pronunciation attributes, the output head for content attributes, and the output head for prosody attributes. In this case, the speech feature evidence can include four types, namely the speech feature evidence for identity attributes, the speech feature evidence for pronunciation attributes, the speech feature evidence for content attributes, and the speech feature evidence for prosody attributes.

[0048] In this embodiment of the invention, based on all first attribute-level features of the same attribute, coarse-grained features corresponding to the first attribute-level features of each attribute are generated respectively. Based on all coarse-grained features and the first feature map, a coarse estimate of the target speaker's speech is generated by a coarse extractor. Based on the coarse estimate of the speech, several speech feature evidences of the target speaker are obtained by a second encoder, thereby improving the accuracy of subsequent extraction of the target speaker's speech.

[0049] Example 3 Figure 3 The implementation flow of step S105 in the target speaker extraction method provided in Embodiment 2 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below: In step S301, a first-dimensional feature is generated based on the first feature map and the low-level feature group, a second-dimensional feature is generated based on the visual feature sequence, and a second feature map is generated based on the first-dimensional feature and the second-dimensional feature.

[0050] In some embodiments, step S301 can be implemented in the following ways: (1) Based on the temporal dimension of the first feature map, the low-level feature group is fused with the first feature map through the temporal visual fusion module to obtain the first dimension feature.

[0051] In this embodiment of the invention, the low-level feature group is fused with the first feature map element-by-element by an affine transformation operation using a temporal-visual fusion module (TVFuse) to obtain the first-dimensional feature. As an example, the first-dimensional feature can be generated using the following formula: , ; in, This represents the first dimension of features. Represents the first feature map. Denotes the affine transformation matrix. Denotes the dimension of the affine transformation matrix, where, The channel dimension represents the first-dimensional feature. This indicates the dimension of the low-level feature group.

[0052] (2) Generate second-dimensional features based on the channel dimension and visual feature sequence of SepFormer network.

[0053] In this embodiment of the invention, based on the channel dimension of the SepFormer network (Separation Transformer), time-varying lip visual features are mapped through a projection head with lip feature attributes, and temporal dimension alignment is performed to obtain the second dimension features.

[0054] (3) Generate a second feature map based on the first dimension features and the second dimension features.

[0055] In this embodiment of the invention, the first-dimensional features and the second-dimensional features are fused using a temporal-visual fusion module to obtain a second feature map. As an example, the second feature map can be generated using the following formula: , ; in, This represents the second feature map. Indicates the second dimension feature. This represents the tensor dimension of the second feature map.

[0056] In step S302, the global semantic features, low-level feature groups, and high-level feature groups are input into the SepFormer network to generate the first input weight and second input weight of each SepFormer module in the SepFormer network.

[0057] In this embodiment of the invention, global semantic features, low-level feature groups, and high-level feature groups are input into each layer of the SepFormer network's SepFormer module. The first and second input weights of each layer's SepFormer module are generated through the Adaptive Hierarchical module in the SepFormer network. As an example, the first and second input weights can be generated using the following formula: , ; in, Indicates the first input weight. Indicates the second input weight. This indicates an adaptive hierarchical module. Represents low-level feature groups. Indicates high-level feature groups. This indicates the number of layers in the two-path block of the SepFormer network. For example, the two-path block of the SepFormer network has L layers, and the first input weight and the second input weight are respectively the layers of the Lth and Lth layers. Input weight vectors for internal blocks and inter-layer blocks of the SepFormer module.

[0058] In some embodiments, as an example, the adaptive hierarchical module can be a lightweight multilayer perceptron network, and L can be set to 2.

[0059] In step S303, the low-level feature group is input into the internal block of each SepFormer module, and the first conditional feature is generated based on the first input weight and the low-level feature group. The high-level feature group is input into the inter-layer block of the SepFormer module starting from the second-level SepFormer module, and the second conditional feature is generated based on the second input weight and the high-level feature group.

[0060] In this embodiment of the invention, low-level feature groups are input into the internal blocks of each SepFormer module. Based on the first input weight and the low-level feature groups, the low-level feature groups are weighted element-wise and fused to generate the first conditional feature. High-level feature groups are input into the inter-layer blocks of the SepFormer modules starting from the second-layer SepFormer module. Based on the second input weight and the high-level feature groups, the high-level feature groups are weighted element-wise and fused to generate the second conditional feature. As an example, the first and second conditional features can be generated using the following formula: ; ; in, Indicates the first The first conditional feature of an internal block Indicates the first The second conditional feature of each interlayer block This indicates element-wise gating, i.e., weighted fusion operation.

[0061] In step S304, a target mask is generated based on the second feature map, the first conditional feature, and the second conditional feature. A third feature map is generated based on the first feature map and the target mask. The third feature map is then decoded to generate the target speaker's speech.

[0062] Specifically, step S304 above can be implemented in the following way: (1) Generate a target mask based on the second feature map, the first conditional feature and the second conditional feature.

[0063] In this embodiment of the invention, the second feature map, the first conditional feature, and the second conditional feature are input into the SepFormer network to generate a target mask. As an example, the target mask M can be generated using the following formula: , .

[0064] (2) Generate a third feature map based on the first feature map and the target mask.

[0065] As an example, the third feature map can be generated using the following formula. : ; (3) Decode the third feature map to generate the target speaker's speech.

[0066] In this embodiment of the invention, the third feature map is decoded by the decoder of the SepFormer network to obtain a time-domain waveform signal. The decoder can be set as a one-dimensional transposed convolutional network, and the time-domain waveform signal is the target speaker's speech. As an example, the third feature map can be decoded using the following formula: , ; in, Indicates the target speaker's voice. This indicates the decoder.

[0067] In this embodiment of the invention, low-level feature groups are input into the internal blocks of each SepFormer module, and first conditional features are generated based on the first input weights and the low-level feature groups. High-level feature groups are input into the inter-layer blocks of the SepFormer module starting from the second-level SepFormer module. By dividing the adaptive features into low-level feature groups and high-level feature groups and then inputting them into the SepFormer module, the target speaker's speech can be accurately extracted by utilizing the attribute characteristics of the adaptive features.

[0068] Example 4 Figure 4 The implementation flow of step S103 in the target speaker extraction method provided in Embodiment 2 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below: In step S401, a third attribute-level feature is generated based on speech feature evidence, global semantic features, and the first attribute-level feature.

[0069] In this embodiment of the invention, when generating the third attribute-level feature, specifically, based on speech feature evidence and global semantic features, the correction direction of the first attribute-level feature is predicted using a residual corrector for a specific attribute. Then, the first attribute-level feature is corrected to obtain the corrected first attribute-level feature, which is the third attribute-level feature. As an example, the third attribute-level feature can be generated using the following formula: ; in, This represents a third-level attribute feature. This represents the first-level attribute feature. Indicates residual corrector, [·;·] represents global semantic features, and [·;·] represents vector concatenation operations. This means that the first attribute-level features, speech feature evidence, and global semantic features are concatenated as vectors. In some embodiments, the residual corrector can be set as a multilayer perceptron.

[0070] In step S402, based on speech feature evidence, first attribute-level features, and third attribute-level features, fusion weights are generated, and fusion features of the first attribute-level features and third attribute-level features are generated according to the fusion weights.

[0071] In this embodiment of the invention, specifically, the fusion weights of the first attribute-level features and the third attribute-level features can be calculated using a dual-path hybrid multilayer perceptron with specific attributes. Based on these fusion weights, the first attribute-level features and the third attribute-level features are weighted and summed to obtain the fused features. As an example, the fused features can be generated using the following formula: ; , attr∈{id, art, cont, pros}; in, Represents the fusion weight, where, =1, This indicates a normalization operation. This indicates a dual-path hybrid multilayer sensor. This indicates the fusion feature.

[0072] In step S403, a reliability probability is generated based on speech feature evidence, global semantic features, and fused features. Based on the reliability probability and fused features, a fourth attribute-level feature is generated.

[0073] In specific embodiments of the present invention, reliability probabilities can be generated based on speech feature evidence, global semantic features, and fused features, using a multilayer perceptron with specific attributes and a sigmoid activation function. Then, based on the reliability probabilities and fused features, fourth attribute-level features are generated. As an example, the fourth attribute-level features can be generated using the following formula: ; , attr∈{id, art, cont, pros}; in, Represents the reliability probability. This represents the Sigmoid activation function. This represents a multilayer perceptron. This represents the fourth attribute level feature. When the reliability probability... When the probability approaches 1, it indicates that the fourth attribute-level feature is unreliable for the current attribute. When the reliability probability approaches 0, it indicates that the fourth attribute-level feature is reliable.

[0074] In step S404, competitive weights are generated based on speech feature evidence, global semantic features, and fourth attribute-level features. Second attribute-level features are generated based on the competitive weights, fourth attribute-level features, and the attributes of the fourth attribute-level features.

[0075] In some embodiments, step S404 above can be implemented in the following ways: (1) Based on speech feature evidence, global semantic features and fourth attribute level features, the score of each fourth attribute level feature is calculated through a scoring network of specific attributes. Based on the temperature scaling scoring mechanism, the calculated score is scaled according to the temperature coefficient. The competitive weight of each fourth attribute level feature is obtained by processing through the Softmax function.

[0076] Specifically, the competition weight can be obtained using the following formula: ; in, Indicates the competition weight. Represents the normalization function. This represents the scoring network, where k represents the temperature coefficient. The smaller k is, the greater the weight of the higher-scoring clue.

[0077] (2) Based on the competition weights and the fourth attribute level features, the second attribute level features are obtained.

[0078] In this embodiment of the invention, specifically, the fourth attribute-level features are weighted according to the competition weights to obtain the second attribute-level features. As an example, the second attribute-level features can be obtained through the following formula: , attr∈{id, art, cont, pros}; in, This represents a second-level attribute feature. .

[0079] This invention generates a third attribute-level feature based on speech feature evidence, global semantic features, and a first attribute-level feature. It then generates a fusion weight based on the speech feature evidence, the first attribute-level feature, and the third attribute-level feature. A fused feature of the first and third attribute-level features is generated based on the fusion weight. A reliability probability is generated based on the speech feature evidence, global semantic features, and the fused feature. A fourth attribute-level feature is generated based on the reliability probability and the fused feature. A competition weight is generated based on the speech feature evidence, global semantic features, and the fourth attribute-level feature. Finally, a second attribute-level feature is generated based on the competition weight, the fourth attribute-level feature, and the attributes of the fourth attribute-level feature, thereby improving the reliability of the second attribute-level feature.

[0080] Example 5 Figure 5 The implementation flow of step S104 in the target speaker extraction method provided in Embodiment 1 of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: In step S501, the routing score of each expert network is calculated by the routing unit based on the global semantic features and the second attribute-level features, and adaptive weights are generated based on the routing scores.

[0081] In this embodiment of the invention, specifically, the second attribute-level features can be concatenated to obtain a second attribute-level feature group. This second attribute-level feature group and the global semantic features are then input into a hybrid expert model. The routing unit calculates the routing score for each expert network. Based on the routing scores, adaptive weights are generated using a routing policy and a Softmax function. As an example, the adaptive weights are generated using the following formula: , ; in, This represents the second-level attribute feature group, where π represents the adaptive weight. , This represents the routing operation corresponding to the routing policy, as an example. The corresponding routing operation can be set to -2.

[0082] In step S502, adaptive features corresponding to the second attribute-level features are generated through an expert network based on adaptive weights and second attribute-level features.

[0083] In this embodiment of the invention, specifically, adaptive features corresponding to the second attribute-level features can be generated through an expert network based on adaptive weights and second attribute-level features. As an example, adaptive features can be generated using the following formula: ; in, Represents an adaptive feature set. Indicates the first Adaptive weights of an expert network Indicates the first A network of experts.

[0084] In this embodiment of the invention, the routing score of each expert network is calculated by the routing unit based on global semantic features and second attribute-level features. Adaptive weights are generated based on the routing scores, and adaptive features corresponding to the second attribute-level features are generated by the expert network based on the adaptive weights, thereby improving the accuracy of subsequent target speaker speech extraction.

[0085] Example 6 Figure 6 The structure of the target speaker extraction device provided in Embodiment Six of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, including: The data acquisition unit 61 is used to receive mixed speech data with the target speaker, generate a first feature map of the mixed speech data, and acquire the global semantic features of the first feature map and multimodal cues for assisting in the extraction of the target speaker. The first feature generation unit 62 is used to generate several first attribute-level features and speech feature evidence of the target speaker based on mixed speech data and multimodal cues. The second feature generation unit 63 is used to generate several second attribute-level features based on speech feature evidence, global semantic features and first attribute-level features. The third feature generation unit 64 is used to generate adaptive weights for all second attribute level features based on global semantic features and second attribute level features through a hybrid expert model, and generate adaptive features corresponding to each second attribute level feature according to the adaptive weights. The speech extraction unit 65 is used to generate low-level feature groups and high-level feature groups based on all adaptive features, and extract the target speaker's speech from the mixed speech data according to the low-level feature groups and high-level feature groups.

[0086] In this embodiment of the invention, for the sake of convenience and brevity, only the division of the above-described functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to achieve all or part of the functions described above. Each unit and module of the device can be implemented by corresponding hardware or software units. Each unit and module can be an independent hardware or software unit, or it can be integrated into a single hardware or software unit, which is not intended to limit the invention. In addition, the specific names of each functional unit and module are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the device can be referred to the corresponding description in the foregoing method embodiments, and will not be repeated here.

[0087] Example 7 Figure 7 The structure of the computing device provided in Embodiment 7 of the present invention is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0088] The computing device 7 of this embodiment includes a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, it implements the steps described in the various speech extraction method embodiments above, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each unit in the above-described device embodiment, for example... Figure 6 The functions of each unit are shown.

[0089] When the processor 70 in the computing device 7 executes the computer program 72 to implement the speech extraction method, the steps implemented can be referred to the description of the aforementioned method embodiment, and will not be repeated here.

[0090] Example 8 In this embodiment of the invention, a computer-readable storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the steps in the above-described speech extraction method embodiments. For example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the computer program is executed by a processor, it implements the functions of each unit in the above-described apparatus embodiment, for example... Figure 6 The functions of each unit are shown.

[0091] The computer-readable storage medium of this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0092] The above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the scope of disclosure involved in the above embodiments is not limited to technical solutions formed by specific combinations of the above technical features, but should also cover other technical solutions formed by arbitrary combinations of the above technical features or their equivalent features without departing from the above-disclosed concept. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0093] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A method for extracting a target speaker, characterized in that, The method includes the following steps: Receive mixed speech data with a target speaker, generate a first feature map of the mixed speech data, and obtain global semantic features of the first feature map and multimodal cues for assisting in the extraction of the target speaker; Based on the mixed speech data and the multimodal cues, several first attribute-level features and speech feature evidence of the target speaker are generated; Based on the speech feature evidence, the global semantic features, and the first attribute-level features, several second attribute-level features are generated. Based on the global semantic features and the second attribute-level features, an adaptive weight for all the second attribute-level features is generated through a hybrid expert model. Based on the adaptive weight, an adaptive feature corresponding to each second attribute-level feature is generated. Based on all the adaptive features, a low-level feature set and a high-level feature set are generated, and the target speaker's speech is extracted from the mixed speech data according to the low-level feature set and the high-level feature set.

2. The method as described in claim 1, characterized in that, The feature refinement module generates several first attribute-level features and speech feature evidence of the target speaker. The feature refinement module includes a first encoder, a projection head, a coarse extractor, and a second encoder. The steps of generating several first-attribute-level features and speech feature evidence of the target speaker based on the mixed speech data and the multimodal cues include: Based on the multimodal cues, several cue features are generated by the first encoder, and each cue feature is decomposed into several first attribute-level features by the projection head; Based on all the first attribute-level features with the same attribute, coarse-grained features corresponding to the first attribute-level features of each attribute are generated respectively. Based on all the coarse-grained features and the first feature map, the coarse extractor generates the coarse estimated speech of the target speaker. Based on the coarsely estimated speech, several speech feature evidences of the target speaker are obtained through the second encoder.

3. The method as described in claim 2, characterized in that, The multimodal cues include several global cues and dynamic cues, and the cue features include global features and dynamic features; The step of generating several cue features based on the multimodal cues using the first encoder includes: The first encoder generates global features corresponding to each global cue. Obtain the visual feature sequence of the dynamic cue, and based on the visual feature sequence, generate the dynamic features corresponding to the dynamic cue through the first encoder.

4. The method as described in claim 3, characterized in that, The step of generating a low-level feature set and a high-level feature set based on all the adaptive features, and extracting the target speaker's speech from the mixed speech data according to the low-level feature set and the high-level feature set includes: A first-dimensional feature is generated based on the first feature map and the low-level feature group; a second-dimensional feature is generated based on the visual feature sequence; and a second feature map is generated based on the first-dimensional feature and the second-dimensional feature. The global semantic features, the low-level feature group, and the high-level feature group are input into the SepFormer network to generate the first input weight and the second input weight of each SepFormer module in the SepFormer network. The low-level feature group is input into the internal block of each SepFormer module, and a first conditional feature is generated based on the first input weight and the low-level feature group. The high-level feature group is input into the inter-layer block of the SepFormer module starting from the second-level SepFormer module, and a second conditional feature is generated based on the second input weight and the high-level feature group. A target mask is generated based on the second feature map, the first conditional feature, and the second conditional feature. A third feature map is generated based on the first feature map and the target mask. The third feature map is then decoded to generate the target speaker's speech.

5. The method as described in claim 4, characterized in that, The SepFormer network includes a temporal visual fusion module, which generates a first-dimensional feature based on the first feature map and the low-level feature group, generates a second-dimensional feature based on the visual feature sequence, and generates a second feature map based on the first-dimensional feature and the second-dimensional feature, including the following steps: Based on the temporal dimension of the first feature map, the low-level feature group is fused with the first feature map through the temporal visual fusion module to obtain the first dimension feature. The second dimension feature is generated based on the channel dimension of the SepFormer network and the visual feature sequence; The second feature map is generated based on the first dimension feature and the second dimension feature.

6. The method as described in claim 2, characterized in that, The step of generating several second attribute-level features based on the speech feature evidence, the global semantic features, and the first attribute-level features includes: Based on the speech feature evidence, the global semantic features, and the first attribute-level features, a third attribute-level feature is generated; Based on the speech feature evidence, the first attribute-level feature, and the third attribute-level feature, a fusion weight is generated, and a fusion feature of the first attribute-level feature and the third attribute-level feature is generated according to the fusion weight. Based on the speech feature evidence, the global semantic features, and the fused features, a reliability probability is generated; based on the reliability probability and the fused features, a fourth attribute-level feature is generated. Based on the speech feature evidence, the global semantic features, and the fourth attribute-level features, competitive weights are generated. Based on the competitive weights, the fourth attribute-level features, and the attributes of the fourth attribute-level features, the second attribute-level features are generated.

7. The method as described in claim 1, characterized in that, The hybrid expert model includes several expert networks and routing units. Based on the global semantic features and the second attribute-level features, the hybrid expert model generates adaptive weights for all the second attribute-level features. Based on these adaptive weights, the step of generating an adaptive feature corresponding to each second attribute-level feature includes: Based on the global semantic features and the second attribute-level features, the routing score of each expert network is calculated through the routing unit of the hybrid expert model, and the adaptive weights are generated based on the routing scores; Based on the adaptive weights, the adaptive features corresponding to the second attribute-level features are generated through the expert network.

8. A target speaker extraction device, characterized in that, The device includes: The data acquisition unit is used to receive mixed speech data with a target speaker, generate a first feature map of the mixed speech data, and acquire global semantic features of the first feature map and multimodal cues for assisting in the extraction of the target speaker. The first feature generation unit is used to generate several first attribute-level features and speech feature evidence of the target speaker based on the mixed speech data and the multimodal cues. The second feature generation unit is used to generate several second attribute-level features based on the speech feature evidence, the global semantic features, and the first attribute-level features. The third feature generation unit is used to generate adaptive weights for all second attribute-level features based on the global semantic features and the second attribute-level features through the hybrid expert model, and generate adaptive features corresponding to each second attribute-level feature according to the adaptive weights. The speech extraction unit is used to generate a low-level feature group and a high-level feature group based on all the adaptive features, and to extract the target speaker's speech from the mixed speech data according to the low-level feature group and the high-level feature group.

9. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images