Target speaker voice extraction method and device

By introducing a multi-headed self-attention layer and a gated loop unit in the target speaker speech extraction method, combining the fusion of the first and second speech extraction models, the multi-speaker, absence and low in the prior art is solved. The problem of poor extraction of overlap rate mixed voice signal is achieved, and efficient and accurate target speaker voice extraction is achieved.

CN119007728AActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
CN202311626816.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-11-29
Publication Date
2024-11-22
Estimated Expiration
2043-11-29

AI Technical Summary

Technical Problem

The existing target speaker extraction method is not effective when dealing with mixed voice signals with multiple speakers, target speaker absences, and low overlap rate of target speakers.

Method used

Using a target speaker speech extraction method, a first speech extraction model based on a multi-head self-attention layer and a gated loop unit and a second speech extraction model for extracting a target speaker mask vector are used, by Generate the target voice signal of the target speaker and combine it with the characteristics of different models to improve extraction performance.

Benefits of technology

Accurate and efficient extraction of multiple speakers, absent speakers and low overlap rate mixed voice signals is achieved, and the quality and performance of target speaker voice extraction is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007728A_ABST
    Figure CN119007728A_ABST
Patent Text Reader

Abstract

The invention provides a target speaker voice extraction method, a target speaker voice extraction device and equipment using an artificial intelligence technology, and a computer readable storage medium. The target speaker voice extraction method comprises the following steps: generating a target speaker feature vector based on a reference voice signal from a target speaker; at least based on the mixed voice signal and the target speaker feature vector, a first voice signal of the target speaker is generated by using a first voice extraction model, a second voice signal of the target speaker is generated by using a second voice extraction model, the first voice extraction model comprises at least one converter block, and the converter block comprises at least one converter block; at least one converter block is provided with a multi-head self-attention layer and a gated circulation unit layer; and generating a target voice signal of the target speaker based on at least one of the first voice signal and the second voice signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a target speaker voice extraction method, a target speaker voice extraction device and equipment, and a computer-readable storage medium. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0003] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics and other technologies. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0004] In today's life, speech technology has been widely used. The key technologies of speech technology include automatic speech recognition technology (ASR), text to speech (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future. In voiceprint recognition, target speaker extraction (TSE) is to extract the target speaker's voice from a mixed speech signal with noise and interference speech with the help of the target speaker's registered voice information. The existing target speaker extraction method has poor extraction effect on mixed speech signals with many speakers, the absence of the target speaker and low target speaker overlap rate. Therefore, a target speaker extraction method that can effectively process such mixed speech signals is needed. Summary of the invention

[0005] The present disclosure provides a target speaker speech extraction method, a target speaker speech extraction device and equipment, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of an embodiment of the present disclosure, a method for extracting speech of a target speaker is provided, comprising: generating a target speaker feature vector based on a reference speech signal from a target speaker; generating a first speech signal of the target speaker using a first speech extraction model and generating a second speech signal of the target speaker using a second speech extraction model based at least on a mixed speech signal and the target speaker feature vector, wherein the first speech extraction model comprises at least one transformer block having a multi-head self-attention layer and a gated recurrent unit layer; and generating a target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal.

[0007] According to an example of an embodiment of the present disclosure, generating the first speech signal of the target speaker using a first speech extraction model includes: generating a first mixed feature vector based on the mixed speech signal; generating a target speaker activity probability based on the target speaker feature vector and the first mixed feature vector using the first speech extraction model, wherein the target speaker activity probability represents the probability that each time frame in the mixed speech signal is a speech signal of the target speaker; and generating the first speech signal based on the target speaker activity probability and the mixed speech signal.

[0008] According to an example of an embodiment of the present disclosure, generating the first speech signal based on the target speaker activity probability and the mixed speech signal includes: converting the target speaker activity probability into a target speaker activity label in binary form based on a predetermined threshold; and filtering the mixed speech signal using the target speaker activity label to generate the first speech signal.

[0009] According to an example of an embodiment of the present disclosure, generating the second speech signal of the target speaker using a second speech extraction model includes: generating the second speech signal based on the target speaker feature vector and the first speech signal using the second speech extraction model, wherein generating the target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal includes: determining the second speech signal as the target speech signal.

[0010] According to an example of an embodiment of the present disclosure, generating the target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal includes: generating the target speech signal of the target speaker by multiplying the first speech signal and the second speech signal.

[0011] According to an example of an embodiment of the present disclosure, generating the second speech signal of the target speaker using a second speech extraction model includes: generating a target speaker mask vector based on the target speaker feature vector and the mixed speech signal using the second speech extraction model, wherein the target speaker mask vector indicates the position of the target speaker speech signal in the mixed speech signal; generating the second speech signal based on the target speaker mask vector and the mixed speech signal.

[0012] According to an example of an embodiment of the present disclosure, generating the first speech signal of the target speaker using a first speech extraction model includes: generating a second mixed feature vector based on the second speech signal; generating a target speaker activity probability based on the target speaker feature vector and the second mixed feature vector, wherein the target speaker activity probability represents the probability that each time frame in the mixed speech signal is a speech signal of the target speaker; generating the first speech signal based on the target speaker activity probability and the mixed speech signal, wherein generating the target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal includes: determining the first speech signal as the target speech signal.

[0013] According to an example of an embodiment of the present disclosure, the first speech extraction model also includes at least one convolution block, and the at least one transformer block also includes a linear layer and at least one residual connection and normalization layer.

[0014] According to an example of an embodiment of the present disclosure, the first speech extraction model is a target speaker speech activity detection model for detecting target speaker activity probability, and the second speech extraction model is a target speaker mask extraction model for extracting a target speaker mask vector, the target speaker activity probability represents the probability that each time frame in the mixed speech signal is the speech signal of the target speaker, and the target speaker mask vector indicates the position of the target speaker speech signal in the mixed speech signal.

[0015] According to an example of an embodiment of the present disclosure, the mixed speech signal includes speech signal segments in which multiple speakers exist and speech signal segments in which the target speaker is absent, and wherein the ratio of the speech duration of the target speaker to the total speech duration of the mixed speech signal varies.

[0016] According to another aspect of an embodiment of the present disclosure, a target speaker speech extraction device is provided, the device comprising: a feature vector extraction unit, configured to generate a target speaker feature vector based on a reference speech signal from a target speaker; a target speech signal generation unit, configured to generate a first speech signal of the target speaker using a first speech extraction model and a second speech signal of the target speaker using a second speech extraction model based at least on a mixed speech signal and the target speaker feature vector, and to generate a target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal, wherein the first speech extraction model comprises at least one transformer block, and the at least one transformer block has a multi-head self-attention layer and a gated recurrent unit layer.

[0017] According to an example of an embodiment of the present disclosure, the first speech extraction model is a target speaker speech activity detection model for detecting target speaker activity probability, and the second speech extraction model is a target speaker mask extraction model for extracting a target speaker mask vector, the target speaker activity probability represents the probability that each time frame in the mixed speech signal is the speech signal of the target speaker, and the target speaker mask vector indicates the position of the target speaker speech signal in the mixed speech signal.

[0018] According to another aspect of an embodiment of the present disclosure, a target speaker speech extraction device is provided, comprising: one or more processors; and one or more memories, wherein the memories store computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the one or more processors execute the methods described in the above-mentioned various aspects.

[0019] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor executes a method as described in any one of the above-mentioned aspects of the present disclosure.

[0020] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, the processor executes a method as described in any one of the above aspects of the present disclosure.

[0021] By using the target speaker speech extraction method, target speaker speech extraction device and apparatus, computer-readable storage medium and computer program product of the above-mentioned various aspects of the present disclosure, by introducing at least one transformer block including a multi-head self-attention sublayer and a gated recurrent sublayer in a first speech extraction model such as a TSVAD model, the quality of the target speaker speech extraction can be improved compared with the traditional target speaker speech extraction method; in addition, by fusing the first speech extraction model and the second speech extraction model in different ways to perform the target speaker speech extraction, the characteristics of different speech extraction models can be effectively utilized to further improve the performance of the target speaker speech extraction. The target speaker speech extraction method according to the embodiment of the present disclosure is particularly suitable for processing mixed speech signals with multiple speakers, the absence of the target speaker, and a low overlap rate or a variable overlap rate of the target speaker, so as to achieve accurate and efficient target speaker speech extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other purposes, features and advantages of the embodiments of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 An exemplary scene diagram of a target speaker speech extraction system according to an embodiment of the present disclosure is shown.

[0024] Figure 2 A flowchart of a target speaker speech extraction method according to an embodiment of the present disclosure is shown.

[0025] Figure 3 A schematic diagram of the structure of a first speech extraction model according to an example of an embodiment of the present disclosure is shown.

[0026] Figure 4A A first fusion method of the first speech extraction model and the second speech extraction model according to an example of an embodiment of the present disclosure is shown.

[0027] Figure 4B A second fusion method of the first speech extraction model and the second speech extraction model according to an example of an embodiment of the present disclosure is shown.

[0028] Figure 4C A third fusion method of the first speech extraction model and the second speech extraction model according to an example of an embodiment of the present disclosure is shown.

[0029] Figure 5A schematic structural diagram of a target speaker voice extraction device according to an embodiment of the present disclosure is shown.

[0030] Figure 6 A schematic diagram showing the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] As shown in the embodiments and claims of the present disclosure, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" and the like are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0034] In addition, flow charts are used in the present disclosure to illustrate the operations performed by the system according to the embodiments of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be superimposed on these processes, or one or more operations may be removed from these processes.

[0035] In voice technology, voiceprint recognition is to achieve the purpose of identifying unknown voices by analyzing the characteristics of one or more voice signals. Target speaker extraction (TSE) is one of its categories. Target speaker extraction is also called personalized speech enhancement or personalized noise suppression. It uses the registered voice information of the target speaker to extract the target speaker's voice from a mixed voice signal with noise and interference. Here, the target speaker refers to the speaker of interest or the designated speaker.

[0036] Typically, the target speaker extraction model can use a pre-trained speaker feature extraction model to generate a target speaker feature vector based on a reference speech signal from the target speaker, and further extract the target speaker's speech signal from the mixed speech signal based on the target speaker feature vector. However, most target speaker extraction models are designed for mixed speech with a limited number of speakers (e.g., less than or equal to 3) and the presence of a target speaker, and the ratio of the target speaker's speech duration to the total speech duration in the mixed speech (which can be called the overlap rate) is high. When processing mixed speech signals with a large number of speakers, or the absence of a target speaker (which can be called target speaker absence), or a low target speaker overlap rate, the effect is not good.

[0037] In order to solve the above problems, the present disclosure proposes a target speaker speech extraction model, which can effectively extract a target speaker's speech signal with higher quality from a mixed speech signal with multiple speakers, the absence of the target speaker, and low target speaker overlap.

[0038] Figure 1 An exemplary scene diagram of a target speaker voice extraction system according to an embodiment of the present disclosure is shown. Figure 1 As shown, the target speaker speech extraction system 100 may include a user terminal 110 , a network 120 , a server 130 , and a database 140 .

[0039] The user terminal 110 may be, for example, Figure 1 It is understood that, in fact, the user terminal 110 may be any other type of electronic device capable of performing data processing, which may include but is not limited to fixed terminals such as desktop computers, smart TVs, etc., mobile terminals such as smart phones, tablet computers, portable computers, handheld devices, etc., or any combination thereof, and the embodiments of the present disclosure do not impose specific restrictions on this.

[0040] The user terminal 110 according to an embodiment of the present disclosure can be used to receive a mixed speech signal, and use the target speaker speech extraction method provided by the present disclosure to generate a speech signal of the target speaker. In some embodiments, the target speaker speech extraction method provided by the present disclosure can be executed by a processing unit of the user terminal 110. In some implementations, the user terminal 110 can execute the target speaker speech extraction method provided by the present disclosure using an application built into the user terminal. In other implementations, the user terminal 110 can execute the target speaker speech extraction method provided by the present disclosure by calling an application stored externally of the user terminal.

[0041] In other embodiments, the user terminal 110 sends the received mixed speech signal to be processed to the server 130 via the network 120, and the server 130 executes the target speaker voice extraction method. In some implementations, the server 130 may execute the target speaker voice extraction method using an application built into the server. In other implementations, the server 130 may execute the target speaker voice extraction method by calling an application stored outside the server.

[0042] The network 120 may be a single network, or a combination of at least two different networks. For example, the network 120 may include, but is not limited to, a local area network, a wide area network, a public network, a private network, or a combination of one or more of the above. The server 130 may be an independent server, or a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, positioning services, and big data and artificial intelligence platforms, and the embodiments of the present disclosure do not impose specific restrictions on this.

[0043] The database 140 may generally refer to a device with a storage function. The database 140 is mainly used to store various data used, generated and output during the operation of the user terminal 110 and the server 130. The database 140 may be local or remote. The database 140 may include various memories, such as random access memory (RAM), read only memory (ROM), etc. The storage devices mentioned above are only some examples, and the storage devices that can be used in the system are not limited to these. The database 140 may be interconnected or communicated with the server 130 or a part thereof via the network 120, or directly interconnected or communicated with the server 130, or a combination of the above two methods.

[0044] Refer to the following Figure 2 A target speaker speech extraction method according to an embodiment of the present disclosure is described. Figure 2 The flowchart of the target speaker voice extraction method 200 according to the embodiment of the present disclosure is shown. As described above, the target speaker voice extraction method 200 can be executed by a user terminal or a server, and the embodiment of the present disclosure does not make any specific limitation on this.

[0045] In step S210, a target speaker feature vector is generated based on a reference speech signal from the target speaker. The reference speech signal is a signal that only includes the speech of the target speaker, for example, it can be a speech signal of 10 seconds from the target speaker, which is used to provide clues for extracting the target speaker's speech from the mixed speech signal. The embodiment of the present disclosure does not impose any specific restrictions on the length of the reference speech signal, and it can be a speech signal of any length according to actual needs. The target speaker feature vector, or target speaker embedding vector, is a vector that can characterize the audio features of the target speaker. In the embodiment of the present disclosure, a speaker encoder can be constructed and trained to extract feature vectors from speech signals, or an existing speaker encoder model can be used to generate feature vectors. For example, ResNet34 based on a residual neural network can be used, and the embodiment of the present disclosure does not impose any specific restrictions on this.

[0046] In step S220, the first speech signal of the target speaker can be extracted using a first speech extraction model, and the second speech signal of the target speaker can be extracted using a second speech extraction model based at least on the target speaker feature vector and the mixed speech signal. Here, the mixed speech signal refers to a mixed audio including a target speaker signal and an interfering speaker signal, a background noise signal, etc. In the embodiment of the present disclosure, the mixed speech signal includes speech signal segments with multiple speakers and speech signal segments in which the target speaker is absent, and wherein the ratio of the target speaker's speech duration to the total speech duration of the mixed speech signal (i.e., the overlap rate) is variable, that is, speech signal segments with low, medium, and high overlap rates of the target speaker can be included at the same time, thereby adapting to various mixed speech signal acquisition scenarios. For example, the mixed speech signal Y can be expressed as:

[0047] Y=S+I+N (1)

[0048] Wherein, Y represents a mixed speech signal; S represents a target speaker signal; I represents an interfering speaker signal; and N represents a background noise signal. Generally speaking, the target speaker speech extraction method according to an embodiment of the present disclosure is dedicated to solving single-channel target speaker speech extraction, that is, the mixed speech signal Y comes from a single audio acquisition channel. However, the target speaker speech extraction method according to an embodiment of the present disclosure can also be applied to multi-channel target speaker speech extraction with appropriate modifications, as long as there is no contradiction therein. In the following, the embodiment of the present disclosure is explained by taking single-channel target speaker speech extraction as an example.

[0049] In the embodiment of the present disclosure, the first speech extraction model and the second speech extraction model based on different principles can be used to obtain the first speech signal and the second speech signal of the target speaker respectively, and then in step S230, the target speech signal of the target speaker is generated based on at least one of the first speech signal and the second speech signal. The first speech extraction model and the second speech extraction model can be combined with each other in various forms to generate the first speech signal, the second speech signal and the final target speech signal of the target speaker, as will be further described in detail below.

[0050] In an embodiment of the present disclosure, the first speech extraction model can be implemented based on a target speaker voice activity detection (TSVAD) model for detecting the target speaker activity probability, which can predict the probability of the target speaker appearing in each time frame in a mixed speech signal or a processed mixed speech signal with the help of a feature vector of the target speaker, which is called the target speaker activity probability. The processed mixed speech signal, for example, refers to a mixed speech signal that has been processed by other target speaker voice extraction. Specifically, the first speech extraction model according to an embodiment of the present disclosure may include at least one transformer block, which may have a neural network layer such as a multi-head self-attention layer, a gated recurrent unit (GRU) layer, etc. Among them, the multi-head self-attention layer is configured to enable the first speech extraction model to simultaneously process information at different positions in the audio sequence of the mixed speech signal or the processed mixed speech signal, thereby improving the spatial expression ability of the model and increasing the training speed of the model; the gated recurrent unit layer is configured to enable the first speech extraction model to better capture the dependency between time frames with large time step distances in the time frame sequence of the mixed speech signal or the processed mixed speech signal, thereby effectively solving the problems of gradient explosion and gradient decay in long-term memory and back propagation in neural networks.

[0051] Figure 3 FIG. 2 shows a schematic diagram of the structure of a first speech extraction model according to an example of an embodiment of the present disclosure, wherein TSVAD is used as an example of the first speech extraction model for illustration. Figure 3 As shown, in addition to the transformer block, the first speech extraction model may also include at least one convolution block, a smoothing block for post-processing, etc.; and in addition to the multi-head self-attention layer and the GRU layer, each transformer block may also include a linear layer and at least one residual connection and normalization layer, wherein the linear layer may be used to introduce a linear component, and the residual connection and normalization layer may be used to integrate information, enhance model stability, and accelerate model convergence. Figure 3In the example, the first speech extraction model has two transformer blocks, but the embodiment of the present disclosure is not limited thereto. The first speech extraction model may also include any number of transformer blocks according to actual needs. Similarly, Figure 3 The numbers of the various blocks and layers shown in the figure are examples and do not constitute any limitation to the embodiments of the present disclosure.

[0052] The mixed speech signal Y or the processed mixed speech signal Y' can be divided into frame vectors [y1, y2, ..., y t ] (where t represents the number of frames). Then, Figure 3 As shown, a speaker encoder can be used to extract a mixed feature vector E from a mixed speech signal divided into frame vectors or a processed mixed speech signal. Y =[E y1 , E y2 , ..., E yt ]. Mixed eigenvector E Y The target speaker feature vector E extracted from the target speaker's reference speech signal S are connected, are input into the TSVAD model, and after being processed by at least one convolution block, at least one transformer block and a smoothing block, the target speaker activity probability P = [P y1 , P y2 , ..., P yt ]. This process can be expressed as follows:

[0053] P = TSVAD (E Y , E S ) (2)

[0054] Among them, P represents the target speaker activity probability, E Y represents the mixed eigenvector, E S represents the target speaker feature vector, TSVAD(·) represents the target speaker activity detection function, which is based on the mixed feature vector E Y and the target speaker feature vector E S Generate target speaker activity probabilities.

[0055] After obtaining the target speaker activity probability, the target speaker activity probability can be converted into a target speaker activity label in binary form based on a predetermined threshold, such as a target speaker activity label with a value of 0 or 1. For example, when the target speaker activity label is 1, it can be indicated that the current time frame is a speech signal of the target speaker; otherwise, when the target speaker activity label is 0, it can be indicated that the current time frame is not a speech signal of the target speaker. Here, the predetermined threshold can be obtained based on model training, determined based on empirical parameters, or determined based on probability distribution characteristics, and the embodiments of the present disclosure do not impose specific restrictions on this. Afterwards, the target speaker activity label can be used to filter the mixed speech signal or the processed mixed speech signal, for example, filtering out the time frame with the target speaker activity label of 0, and only retaining the time frame with the target speaker activity label of 1, so that the speech signal of the target speaker, that is, the first speech signal, can be obtained.

[0056] The first speech extraction model according to the embodiment of the present disclosure can improve the performance of the target speaker's speech extraction by introducing at least one transformer block including a multi-head self-attention layer and a gated recurrent unit layer. Taking the TSVAD model as an example of the first speech extraction model for explanation, the following Table 1 shows the results of the ablation experiment on the TSVAD model, where TSVAD represents the TSVAD model with at least one transformer block according to the embodiment of the present disclosure, and TSVAD* represents the removal of the transformer block from the TSVAD model according to the embodiment of the present disclosure, or the replacement of the transformer block with other neural network modules such as long short memory networks, and other experimental conditions are the same. In Table 1, DER represents the speaker segmentation clustering error rate, INT represents the energy difference between the mixed speech signal and the extracted target speech signal of the target speaker, and the smaller the value of DER and the larger the value of INT, the better the target speaker's speech extraction performance. It can be seen from Table 1 that compared with the TSVAD* model with the transformer block removed or replaced, the TSVAD model according to the embodiment of the present disclosure has a smaller DER and a larger INT, indicating that its target speaker speech extraction performance is better, which shows that the introduction of at least one transformer block plays an important role in improving the target speaker speech extraction performance.

[0057]

[0058] Table 1 Ablation experiment results of TSVAD model

[0059] In an embodiment of the present disclosure, the second speech extraction model may be a target speaker mask extraction model (TSE) for extracting a target speaker mask vector from a mixed speech signal or a processed mixed speech signal, wherein the processed mixed speech signal, for example, refers to a mixed speech signal that has been subjected to other target speaker speech extraction processing, and the target speaker mask vector may indicate the position of the target speaker speech signal in the mixed speech signal or the processed mixed speech signal. For example, the target speaker mask vector may be a vector composed of 0 and 1 having the same dimension as the vector of the mixed speech signal, wherein a certain element value of 1 indicates that the element at the same position of the mixed speech signal vector is the target speaker speech signal, and a certain element value of 0 indicates that the element at the same position of the mixed speech signal vector is not the target speaker speech signal. The process of extracting a target speaker mask vector from a mixed speech signal or a processed mixed speech signal may be expressed as:

[0060] M=TSE(Y,E S ) (3)

[0061] Where M represents the target speaker mask vector, Y represents the mixed speech signal or the processed mixed speech signal, and E S represents the target speaker feature vector, TSE(·) represents the target speaker mask extraction function, which is based on the mixed speech signal and the target speaker feature vector E S Generate a target speaker mask vector. Based on the target speaker mask vector and the mixed speech signal or the processed mixed speech signal, a second speech signal that retains only the speech signal of the target speaker can be generated. For example, the second speech signal can be generated by multiplying the target speaker mask vector with the mixed speech signal or the processed mixed speech signal.

[0062] In the embodiments of the present disclosure, the first speech extraction model and the second speech extraction model may be fused in different ways to generate a target speech signal of a target speaker. The fusion process may be expressed as follows, for example:

[0063] S'=F(Y,M,P) (4)

[0064] Wherein, S' represents the extracted target speech signal of the target speaker, Y represents the mixed speech signal, M represents the target speaker mask vector, P represents the target speaker activity probability, and F(·) represents the fusion function, which can generate the target speech signal based on the mixed speech signal Y, the target speaker mask vector M and the target speaker activity probability P.

[0065] Refer to the following FIG. 4A to FIG. 4C Describe different fusion methods of the first speech extraction model and the second speech extraction model according to an embodiment of the present disclosure. Figure 4AThe first fusion mode of the first speech extraction model and the second speech extraction model according to the example of the embodiment of the present disclosure is shown. Figure 4B A second fusion method of the first speech extraction model and the second speech extraction model according to an example of an embodiment of the present disclosure is shown, and Figure 4C A third fusion method of the first speech extraction model and the second speech extraction model according to an example of an embodiment of the present disclosure is shown.

[0066] In one example, the output of the first speech extraction model can be input into the second speech extraction model, and the target speech signal of the final target speaker can be generated by the second speech extraction model. Figure 4A As shown, the first mixed feature vector can be generated based on the mixed speech signal Y, for example, using Figure 4A The speaker encoder shown generates a first mixed feature vector E based on the mixed speech signal Y Y Afterwards, the first speech extraction model is based on the first mixed feature vector E Y and the target speaker feature vector E generated in the above step S210 based on the reference speech signal from the target speaker S , to generate the target speaker activity probability P. Then, the first speech signal S1 of the target speaker can be generated based on the target speaker activity probability P and the mixed speech signal Y. For example, the target speaker activity probability can be converted into a target speaker activity label in a binary form based on a predetermined threshold, and the mixed speech signal can be filtered using the target speaker activity label to generate the first speech signal S1.

[0067] At this point, the first speech extraction model has been used to perform preliminary target speaker speech extraction on the mixed speech signal. Therefore, the first speech signal S1 can also be called a processed mixed speech signal. In order to perform more accurate target speaker speech extraction, the first speech signal S1 can be input into the second speech extraction model. The second speech extraction model can be based on the first speech signal S1 and the target speaker feature vector E S A second speech signal is generated as a target speech signal of the extracted target speaker.

[0068] In this example, a lower predetermined threshold can be used to generate a target speaker activity label for filtering the mixed speech signal, thereby minimizing the risk of erroneously filtering out the target speaker's speech signal; by combining the second speech extraction model to perform further target speaker speech extraction, the target speaker's speech signal with higher quality can be effectively extracted.

[0069] In another example, the output of the second speech extraction model can be input into the first speech extraction model, and the target speech signal of the final target speaker can be generated by the first speech extraction model. Figure 4B As shown, the second speech extraction model can be used based on the mixed speech signal Y and the target speaker feature vector E generated based on the reference speech signal from the target speaker in the above step S210. S , to generate a target speaker mask vector M, and then generate a second speech signal S2 based on the target speaker mask vector M and the mixed speech signal Y.

[0070] At this point, the second speech extraction model has been used to perform preliminary target speaker speech extraction on the mixed speech signal. Therefore, the second speech signal S2 can also be called a processed mixed speech signal. In order to perform more accurate target speaker speech extraction, a second mixed feature vector can be generated based on the second speech signal S2, for example, using Figure 4B The speaker encoder shown generates a second mixed feature vector E based on the second speech signal S2 Y’ Afterwards, the first speech extraction model can be based on the second mixed feature vector E Y’ and the target speaker feature vector E S , generate a target speaker activity probability P, and then generate a first speech signal based on the target speaker activity probability P and the mixed speech signal Y as the target speech signal of the extracted target speaker.

[0071] In this example, the second speech extraction model is first used to extract an initial speech signal with a low overlap rate from a mixed speech signal, and then the second speech extraction model is used to achieve further accurate extraction of a target speech signal of a target speaker.

[0072] In another example, the first speech extraction model and the second speech extraction model may be connected in parallel. Figure 4C As shown, the first mixed feature vector can be generated based on the mixed speech signal Y, for example, using Figure 4A The speaker encoder shown generates a first mixed feature vector E based on the mixed speech signal Y Y Afterwards, the first speech extraction model is based on the first mixed feature vector E Y and the target speaker feature vector E generated in the above step S210 based on the reference speech signal from the target speaker S, to generate the target speaker activity probability P. Then, the first speech signal S1 of the target speaker can be generated based on the target speaker activity probability P and the mixed speech signal Y. On the other hand, the second speech extraction model can be based on the mixed speech signal Y and the target speaker feature vector E generated based on the reference speech signal from the target speaker in the above step S210. S , to generate the target speaker mask vector M, and then generate the second speech signal S2 based on the target speaker mask vector M and the mixed speech signal Y. Afterwards, the target speech signal of the target speaker can be generated based on the first speech signal S1 and the second speech signal S2, for example Figure 4C As shown, the extracted target speech signal S' of the target speaker can be generated by multiplying the first speech signal S1 and the second speech signal S2.

[0073] In this example, the first speech extraction model and the second speech extraction model are connected in parallel with each other, and the interaction between the two models is small, thereby reducing the dependence on auxiliary information and improving the efficiency of speech extraction of the target speaker.

[0074] Combination of the above FIG. 4A to FIG. 4C Different fusion methods of the first speech extraction model and the second speech extraction model are described. It should be noted that FIG. 4A to FIG. 4C The fusion method and specific modules, input parameters and output parameters shown in are only used as examples or for the purpose of illustration, and do not mean that the embodiments of the present disclosure are limited thereto. For example, the first speech extraction model and the second speech extraction model can also be fused in other appropriate ways.

[0075] By using the target speaker speech extraction method according to the embodiment of the present disclosure, by introducing at least one transformer block including a multi-head self-attention sublayer and a gated recurrent sublayer in a first speech extraction model such as a TSVAD model, the quality of the target speaker speech extraction can be improved compared with the traditional target speaker speech extraction method; in addition, by fusing the first speech extraction model and the second speech extraction model in different ways to perform the target speaker speech extraction, the characteristics of different speech extraction models can be effectively utilized to further improve the performance of the target speaker speech extraction. The target speaker speech extraction method according to the embodiment of the present disclosure is particularly suitable for processing mixed speech signals with multiple speakers, the absence of the target speaker, and a low overlap rate or a variable overlap rate of the target speaker, thereby achieving accurate and efficient target speaker speech extraction.

[0076] Refer to the following Figure 5 A target speaker voice extraction device according to an embodiment of the present disclosure is described. Figure 5 FIG. 5 shows a schematic diagram of the structure of a target speaker voice extraction device 500 according to an embodiment of the present disclosure. Figure 5As shown, the target speaker speech extraction device 500 includes a feature vector extraction unit 510 and a target speech signal generation unit 520. In addition to these two units, the device 500 may also include other related components, but since these components are not related to the present disclosure, detailed descriptions of their specific contents are omitted here. In addition, since the details of some functions of the device 500 are similar to those of the reference Figure 2 The details of the steps of the method 200 described are similar, so for the sake of brevity, the repeated description of some contents is omitted here. The device 500 according to the embodiment of the present disclosure can be implemented as a terminal or a server, as described above with reference to Figure 1 As described.

[0077] The feature vector extraction unit 510 is configured to generate a target speaker feature vector based on a reference speech signal from the target speaker. The reference speech signal is a signal that includes only the speech of the target speaker, for example, it can be a speech signal of 10 seconds from the target speaker, which is used to provide clues for extracting the target speaker's speech from the mixed speech signal. The embodiment of the present disclosure does not impose any specific restrictions on the length of the reference speech signal, and it can be a speech signal of any length according to actual needs. The target speaker feature vector, or the target speaker embedding vector, is a vector that can characterize the audio features of the target speaker. In the embodiment of the present disclosure, the feature vector extraction unit 510 can construct and train a speaker encoder to extract feature vectors from speech signals, or it can use an existing speaker encoder model to generate feature vectors, for example, it can use ResNet34 based on a residual neural network, etc. The embodiment of the present disclosure does not impose any specific restrictions on this.

[0078] The target speech signal generating unit 520 is configured to extract the first speech signal of the target speaker using the first speech extraction model based on at least the target speaker feature vector and the mixed speech signal, and extract the second speech signal of the target speaker using the second speech extraction model. Here, the mixed speech signal refers to a mixed audio including the target speaker signal and the interfering speaker signal, the background noise signal, etc. In the embodiment of the present disclosure, the mixed speech signal includes speech signal segments with multiple speakers and speech signal segments without the target speaker, and the ratio of the speech duration of the target speaker to the total speech duration of the mixed speech signal (i.e., the overlap rate) is variable, that is, the speech signal segments with low, medium, and high overlap rates of the target speaker can be included at the same time, so as to adapt to various mixed speech signal acquisition scenarios.

[0079] In the embodiment of the present disclosure, the target speech signal generating unit 520 can respectively obtain the first speech signal and the second speech signal of the target speaker using the first speech extraction model and the second speech extraction model based on different principles, and then generate the target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal. The first speech extraction model and the second speech extraction model can be combined with each other in various forms to generate the first speech signal, the second speech signal and the final target speech signal of the target speaker, as will be further described in detail below.

[0080] In an embodiment of the present disclosure, the first speech extraction model can be implemented based on a target speaker voice activity detection (TSVAD) model for detecting the target speaker activity probability, which can predict the probability of the target speaker appearing in each time frame in the mixed speech signal or the processed mixed speech signal with the help of the feature vector of the target speaker, which is called the target speaker activity probability. Wherein, the processed mixed speech signal, for example, refers to the mixed speech signal after other target speaker voice extraction processing. Specifically, the first speech extraction model according to the embodiment of the present disclosure may include at least one transformer block, and the at least one transformer block may have a neural network layer such as a multi-head self-attention layer, a gated recurrent unit (GRU) layer, etc. Wherein, the multi-head self-attention layer is configured to enable the first speech extraction model to simultaneously process information at different positions in the audio sequence of the mixed speech signal or the processed mixed speech signal, thereby improving the spatial expression ability of the model and improving the training speed of the model; the gated recurrent unit layer is configured to enable the first speech extraction model to better capture the dependency between time frames with large time step distances in the time frame sequence of the mixed speech signal or the processed mixed speech signal, thereby effectively solving the problems of gradient explosion and gradient decay in long-term memory and back propagation in the neural network.

[0081] like Figure 3 As shown, in addition to the transformer block, the first speech extraction model can also include at least one convolution block, a smoothing block for post-processing, etc.; and in addition to the multi-head self-attention layer and the GRU layer, each transformer block can also include a linear layer and at least one residual connection and normalization layer, wherein the linear layer can be used to introduce linear components, and the residual connection and normalization layer can be used to integrate information, enhance model stability and accelerate model convergence.

[0082] After obtaining the target speaker activity probability, the target speaker activity probability can be converted into a target speaker activity label in binary form based on a predetermined threshold, such as a target speaker activity label with a value of 0 or 1. For example, when the target speaker activity label is 1, it can be indicated that the current time frame is the speech signal of the target speaker; otherwise, when the target speaker activity label is 0, it can be indicated that the current time frame is not the speech signal of the target speaker. Here, the predetermined threshold can be obtained based on model training, determined based on empirical parameters, or determined based on probability distribution characteristics, and the embodiments of the present disclosure do not impose specific restrictions on this. Afterwards, the target speaker activity label can be used to filter the mixed speech signal or the processed mixed speech signal, for example, the time frame with the target speaker activity label of 0 is removed, and only the time frame with the target speaker activity label of 1 is retained, so that the speech signal of the target speaker, that is, the first speech signal, can be obtained.

[0083] The first speech extraction model according to the embodiment of the present disclosure can improve the performance of target speaker speech extraction by introducing at least one transformer block including a multi-head self-attention layer and a gated recurrent unit layer, as specifically described above with reference to Table 1.

[0084] In an embodiment of the present disclosure, the second speech extraction model may be a target speaker mask extraction model (TSE) for extracting a target speaker mask vector from a mixed speech signal or a processed mixed speech signal, wherein the processed mixed speech signal, for example, refers to a mixed speech signal that has been subjected to other target speaker speech extraction processes, and the target speaker mask vector may indicate the position of the target speaker speech signal in the mixed speech signal or the processed mixed speech signal. For example, the target speaker mask vector may be a vector composed of 0 and 1 having the same dimension as the vector of the mixed speech signal, wherein a certain element value of 1 indicates that the element at the same position of the mixed speech signal vector is the target speaker speech signal, and a certain element value of 0 indicates that the element at the same position of the mixed speech signal vector is not the target speaker speech signal.

[0085] In the embodiment of the present disclosure, the first speech extraction model and the second speech extraction model can be fused in different ways to generate the target speech signal of the target speaker, as described above with reference to Figure 4A-4C In one example, the output of the first speech extraction model can be input into the second speech extraction model, and the target speech signal of the final target speaker can be generated by the second speech extraction model, such as Figure 4A In another example, the output of the second speech extraction model can be input into the first speech extraction model, and the target speech signal of the final target speaker can be generated by the first speech extraction model, such as Figure 4BIn another example, the first speech extraction model and the second speech extraction model can be connected in parallel, and the target speech signal of the extracted target speaker can be generated by, for example, multiplying the first speech signal and the second speech signal. Figure 4C shown.

[0086] By using the target speaker speech extraction device according to the embodiment of the present disclosure, by introducing at least one transformer block including a multi-head self-attention sublayer and a gated recurrent sublayer in a first speech extraction model such as a TSVAD model, the quality of the target speaker speech extraction can be improved compared with the traditional target speaker speech extraction method; in addition, by fusing the first speech extraction model and the second speech extraction model in different ways to perform the target speaker speech extraction, the characteristics of different speech extraction models can be effectively utilized to further improve the performance of the target speaker speech extraction. The target speaker speech extraction device according to the embodiment of the present disclosure is particularly suitable for processing mixed speech signals with multiple speakers, the absence of the target speaker, and a low overlap rate or a variable overlap rate of the target speaker, thereby achieving accurate and efficient target speaker speech extraction.

[0087] In addition, the device according to the embodiment of the present disclosure (for example, the target speaker voice extraction device, etc.) can also be used by means of Figure 6 The exemplary computing device architecture shown is implemented. Figure 6 A schematic diagram showing the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. Figure 6 As shown, the computing device 600 may include a bus 610, one or more CPUs 620, a read-only memory (ROM) 630, a random access memory (RAM) 640, a communication port 650 connected to a network, an input / output component 660, a hard disk 670, etc. The storage device in the computing device 600, such as the ROM 630 or the hard disk 670, may store various data or files used for computer processing and / or communication and program instructions executed by the CPU. The computing device 600 may also include a user interface 680. Of course, Figure 6 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 6 One or more components in the computing device shown. The device according to the embodiment of the present disclosure may be configured to execute the target speaker voice extraction method according to the above-mentioned various embodiments of the present disclosure, or to implement the target speaker voice extraction device according to the above-mentioned various embodiments of the present disclosure.

[0088] The embodiments of the present disclosure may also be implemented as a computer-readable storage medium. Computer-readable instructions are stored on a computer-readable storage medium according to an embodiment of the present disclosure. When the computer-readable instructions are executed by a processor, the target speaker speech extraction method according to the embodiment of the present disclosure described with reference to the above figures may be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0089] According to an embodiment of the present disclosure, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer-readable instructions, and the computer-readable instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer-readable instructions from the computer-readable storage medium, and the processor executes the computer-readable instructions, so that the computer device executes the target speaker voice extraction method described in the above various embodiments.

[0090] The program part of the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.

[0091] All software or parts of it may sometimes be communicated over a network, such as the Internet or other communications network. Such communications can load software from one computer device or processor to another. Therefore, another medium capable of transmitting software elements may also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, can also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media that participate in the process of executing any instructions by the processor.

[0092] The present application uses specific words to describe the embodiments of the present application. For example, "first / second embodiment", "one embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or multiple times in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0093] In addition, it will be appreciated by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present application may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0094] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0095] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A method for extracting speech of a target speaker, comprising: generating a target speaker feature vector based on a reference speech signal from the target speaker; Based at least on the mixed speech signal and the target speaker feature vector, using a first speech extraction model to generate a first speech signal of the target speaker and using a second speech extraction model to generate a second speech signal of the target speaker, wherein the first speech extraction model includes at least one transformer block, and the at least one transformer block has a multi-head self-attention layer and a gated recurrent unit layer; as well as A target speech signal of the target speaker is generated based on at least one of the first speech signal and the second speech signal.

2. The method according to claim 1, wherein: Generating the first speech signal of the target speaker by using the first speech extraction model includes: generating a first mixed feature vector based on the mixed speech signal; Generate a target speaker activity probability based on the target speaker feature vector and the first mixed feature vector using the first speech extraction model, wherein the target speaker activity probability represents the probability that each time frame in the mixed speech signal is a speech signal of the target speaker; The first speech signal is generated based on the target speaker activity probability and the mixed speech signal.

3. The method according to claim 2, wherein: Generating the first speech signal based on the target speaker activity probability and the mixed speech signal includes: Converting the target speaker activity probability into a target speaker activity label in a binary form based on a predetermined threshold; The mixed speech signal is filtered using the target speaker activity label to generate the first speech signal.

4. The method according to claim 2, wherein: Generating the second speech signal of the target speaker by using the second speech extraction model includes: generating the second speech signal based on the target speaker feature vector and the first speech signal using the second speech extraction model, The step of generating a target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal comprises: The second speech signal is determined as the target speech signal.

5. The method according to claim 2, wherein: Generating a target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal includes: A target speech signal of the target speaker is generated by multiplying the first speech signal and the second speech signal.

6. The method according to claim 1, wherein: Generating the second speech signal of the target speaker by using the second speech extraction model includes: Generate a target speaker mask vector based on the target speaker feature vector and the mixed speech signal using a second speech extraction model, wherein the target speaker mask vector indicates a position of the target speaker speech signal in the mixed speech signal; The second speech signal is generated based on the target speaker mask vector and the mixed speech signal.

7. The method according to claim 6, wherein: Generating the first speech signal of the target speaker by using the first speech extraction model includes: generating a second mixed feature vector based on the second speech signal; generating a target speaker activity probability based on the target speaker feature vector and the second mixed feature vector, wherein the target speaker activity probability represents a probability that each time frame in the mixed speech signal is a speech signal of the target speaker; generating the first speech signal based on the target speaker activity probability and the mixed speech signal, The step of generating a target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal comprises: The first speech signal is determined as the target speech signal.

8. The method according to claim 1, wherein: The first speech extraction model further comprises at least one convolution block, and the at least one transformer block further comprises a linear layer and at least one residual connection and normalization layer.

9. The method according to claim 1, wherein: The first speech extraction model is a target speaker speech activity detection model for detecting a target speaker activity probability, and the second speech extraction model is a target speaker mask extraction model for extracting a target speaker mask vector, The target speaker activity probability represents the probability that each time frame in the mixed speech signal is a speech signal of the target speaker. The target speaker mask vector indicates a position of the target speaker's speech signal in the mixed speech signal.

10. The method according to claim 1, wherein: The mixed speech signal includes speech signal segments with multiple speakers and speech signal segments without the target speaker, and the ratio of the speech duration of the target speaker to the total speech duration of the mixed speech signal varies.

11. A target speaker voice extraction device, the device comprising: a feature vector extraction unit configured to generate a target speaker feature vector based on a reference speech signal from the target speaker; A target speech signal generating unit is configured to generate a first speech signal of the target speaker using a first speech extraction model and generate a second speech signal of the target speaker using a second speech extraction model based at least on a mixed speech signal and a target speaker feature vector, and to generate a target speech signal of the target speaker based on at least one of the first speech signal and the second speech signal, wherein the first speech extraction model includes at least one transformer block, and the at least one transformer block has a multi-head self-attention layer and a gated recurrent unit layer.

12. The device according to claim 11, wherein The first speech extraction model is a target speaker speech activity detection model for detecting a target speaker activity probability, and the second speech extraction model is a target speaker mask extraction model for extracting a target speaker mask vector, The target speaker activity probability represents the probability that each time frame in the mixed speech signal is a speech signal of the target speaker. The target speaker mask vector indicates a position of the target speaker's speech signal in the mixed speech signal.

13. A target speaker voice extraction device, comprising: one or more processors; as well as One or more memories, wherein the memories store computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 10.

14. A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 10.

15. A computer program product comprising computer readable instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Voice separation method and device

    CN114898756A

  • Speech processing method and related equipment thereof

    CN116259311A

  • Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks

    US20200043467A1

  • Target speaker mode

    WO2023049407A1

Cited By

  • Target voice extraction method and device based on masking beam forming

    CN119943087A

  • A method and device for extracting target speech based on masked beamforming

    CN119943087B

  • Voice recognition processing method and device applied to smart power application

    CN120600034A