Method for separating human voice, electronic device and readable storage medium

Abstract

The present application proposes a method for human voice separation, which is applied to an execution device, and includes: receiving the voice of a user in a designated space; the designated space is a limited space with multiple members; determining a main speaker using the user's voice based on the registered voiceprints of the multiple members; determining that multiple members other than the main speaker are noise speakers; obtaining a mixed spectrum corresponding to mixed audio in the designated space, where the mixed audio is voice information aliased by the audio of the main speaker and the audio of at least one noise speaker; processing the mixed spectrum and the registered voiceprints of the multiple members through a voiceprint filtering network to obtain the spectrum of the main speaker in the mixed audio; wherein the voiceprint filtering network is used to use the voiceprint of the main speaker in the registered voiceprints as positive excitation and the voiceprints of the multiple noise speakers in the registered voiceprints as negative excitation to separate the spectrum of the main speaker from the mixed spectrum; and obtaining the audio of the main speaker based on the spectrum of the main speaker.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a method, electronic device, and readable storage medium for separating human voices. Background Art

[0002] Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use computers or computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theory.

[0003] Automatic speech recognition (ASR) technology is an important technology for human-computer interaction, and recognition accuracy is a key indicator of the interactive experience. The current development of speech recognition technology has a relatively high recognition accuracy in quiet scenes. However, in noisy scenes, especially crowd noise scenes, the recognition accuracy is easily reduced by interference from other voices.

[0004] Speech recognition technology is the primary interaction method for smart devices like smart TVs and smart speakers. In indoor locations with a large number of people, such as homes, residences, and offices, noise primarily comes from interference from other people. Therefore, how to suppress other people's voices to improve the accuracy of speech recognition for the primary speaker is a pressing issue. Summary of the Invention

[0005] In order to solve the above problems, embodiments of the present application provide a method for separating human voice, an electronic device, and a readable storage medium.

[0006] In a first aspect, an embodiment of the present application provides a method for human voice separation, applied to an execution device, the method comprising: receiving a user's voice in a designated space; the designated space being a limited space with multiple members; determining a main speaker using the user's voice based on the registered voiceprints of the multiple members; determining that multiple members other than the main speaker are noise speakers; obtaining a mixed spectrum corresponding to mixed audio in the designated space, the mixed audio being speech information of a mixture of the main speaker's audio and at least one noise speaker's audio; processing the mixed spectrum and the registered voiceprints of the multiple members through a voiceprint filtering network to obtain a spectrum of the main speaker in the mixed audio; wherein the voiceprint filtering network is configured to use the voiceprint of the main speaker in the registered voiceprints as a positive stimulus and the voiceprints of multiple noise speakers in the registered voiceprints as negative stimulus to separate the spectrum of the main speaker from the mixed spectrum; and obtaining audio of the main speaker based on the spectrum of the main speaker. In this way, the voiceprint information of multiple members can be combined to perform human voice separation when different members use smart devices, thereby obtaining clean human voice separation audio of the members.

[0007] In one possible implementation, determining the primary speaker using the user's voice based on the registered voiceprints of the multiple members includes: extracting the user's voice print through a voiceprint network; the voiceprint network being a deep neural network model or a long short-term memory network; calculating the similarity between the user's voice print and the registered voiceprints of each of the multiple members; sorting the similarity values; and, based on the sorting results, determining the speaker corresponding to the registered voiceprint with the highest similarity to the user's voice print as the primary speaker. In this way, the identities of speakers can be distinguished when different members use smart devices, and registered voiceprints consistent with their identities can be obtained.

[0008] In one possible implementation, the voiceprint filtering network includes a first subnetwork, a second subnetwork, and a third subnetwork; the first subnetwork is configured to embed a first code into the voiceprint of the main speaker to obtain a positively enhanced voiceprint, and to embed a second code into each of the at least one noise speaker voiceprints to obtain at least one negatively suppressed voiceprint; the second subnetwork is configured to positively excite the mixed spectrum based on the positively enhanced voiceprint to obtain a corresponding positively enhanced spectrum, and to negatively excite the mixed spectrum based on the at least one negatively suppressed voiceprint to obtain a corresponding at least one negatively suppressed spectrum; and the third subnetwork is configured to fuse the positively enhanced spectrum and the at least one negatively suppressed spectrum to obtain a fused spectrum mask, so that the fused spectrum mask and the mixed spectrum can be point-by-point multiplied to obtain the spectrum of the main speaker. In this way, the voiceprint filtering network can process the voiceprint of the main speaker, the voiceprint of the noise speaker, and the spectrum of the mixed audio data of multiple voices obtained by the smart device to enhance the voice of the main speaker, suppress the voice of the noise speaker, and separate the audio information of the main speaker from the mixed speech audio of multiple voices.

[0009] In one possible implementation, processing the mixed spectrum and the registered voiceprints of the multiple members through the voiceprint filtering network to obtain the spectrum of the primary speaker in the mixed audio includes: inputting the registered voiceprints of the multiple members into the first sub-network, embedding the first code into the voiceprint of the primary speaker among the registered voiceprints of the multiple members to obtain a positive enhancement voiceprint; and embedding the second code into the voiceprint of each of the at least one noise speaker voiceprint to obtain at least one negative suppression voiceprint. In this way, the first sub-network can distinguish between processing the voiceprint of the primary speaker and the voiceprint of the noise speaker.

[0010] In one possible implementation, the voiceprint filtering network processes the mixed spectrum and the registered voiceprints of the multiple members to obtain the spectrum of the main speaker in the mixed audio, including: inputting the mixed spectrum into a second subnetwork for convolution and outputting the convolution result; splicing the convolution result with the positive enhancement voiceprint frame by frame through a splicing module, and sequentially passing the splicing result through a long short-term memory network, a fully connected network, and a soft mask prediction network for positive excitation to obtain a positive enhancement spectrum; splicing the convolution result with the at least one negative suppression voiceprint frame by frame through a splicing module, and sequentially passing the splicing result through a long short-term memory network, a fully connected network, and a soft mask prediction network for negative excitation to obtain at least one negative suppression spectrum. In this way, the second subnetwork can enhance the spectrum of the main speaker in the mixed spectrum and suppress the spectrum of the noise speaker in the mixed spectrum.

[0011] In one possible implementation, processing the mixed spectrum and the registered voiceprints of the multiple members through the voiceprint filtering network to obtain the spectrum of the primary speaker in the mixed audio includes: inputting the positive enhancement spectrum and the at least one negative suppression spectrum into the third subnetwork for fusion to obtain a fused spectrum mask; and multiplying the fused spectrum mask by the mixed spectrum point by point to calculate the spectrum of the primary speaker. In this way, the third subnetwork can separate the spectrum of the primary speaker from the mixed spectrum.

[0012] In a possible implementation, the first code and the second code are generated by a positive and negative code generator, so that different codes can be obtained to be embedded in the voiceprint of the main speaker and the voiceprint of the noise speaker respectively.

[0013] In one possible implementation, the third sub-network uses a Sigmoid function to perform mask calculation processing on the output of the fully connected network to obtain a fused spectrum mask. In this way, a fused spectrum mask that suppresses the noisy speaker's voiceprint spectrum can be obtained from the mixed spectrum.

[0014] In one possible implementation, before receiving the voices of users in the designated space, the method includes pre-registering the voiceprints of at least two users using a voiceprint network, wherein the voiceprint network is a deep neural network model or a long short-term memory network model. In this way, registered voiceprints of multiple users in the designated space can be obtained.

[0015] In a possible implementation, after receiving the voice of the user in the designated space, the method further includes: determining that the voice is a wake-up audio based on the presence of a wake-up word in the user's voice. In this way, the wake-up audio of the smart device in the designated space can be obtained.

[0016] In one possible implementation, determining the primary speaker using the user's voice based on the registered voiceprints of the multiple members includes: extracting the voiceprint of the wake-up audio through a voiceprint network; the voiceprint network being a deep neural network model or a long short-term memory network; calculating the similarity between the voiceprint of the wake-up audio and the registered voiceprint of each of the multiple members; sorting the similarity values; and determining, based on the sorting results, the speaker corresponding to the registered voiceprint with the highest similarity to the voiceprint of the wake-up audio as the primary speaker. In this way, the speaker's identity can be determined based on the voiceprint of the wake-up audio.

[0017] In the second aspect, an embodiment of the present application provides a method for training a voiceprint filtering network, which is applied to a training device. The voiceprint filtering network includes a first sub-network, a second sub-network, and a third sub-network; the target person's voiceprint and the first and second codes are used as inputs to the first sub-network; the output of the first sub-network and the mixed spectrum corresponding to the mixed audio of the target person's audio, other people's audio, and noise are used as inputs to the second sub-network; the positive enhancement spectrum and the negative suppression spectrum are used as outputs of the second sub-network, and the first sub-network and the second sub-network are trained using a joint training method; the output of the second sub-network is used as input to the third sub-network, and the masked spectrum is used as output to train the third sub-network; and the trained voiceprint filtering network is obtained. Its beneficial effects are as described above and will not be repeated here.

[0018] In a possible embodiment, according to the training method of the voiceprint filtering network of claim 12, the training of the first subnetwork and the second subnetwork using a joint training method includes: calculating a first loss function between the spectrum corresponding to the target person's audio and the positive enhancement spectrum; calculating a second loss function between the spectrum corresponding to the other person's audio and the negative suppression spectrum; and iterative training is performed with the goal of converging the first loss function and the second loss function to a certain threshold, to obtain the trained first subnetwork and the second subnetwork.

[0019] In one possible implementation, the method of using the output of the second sub-network as the input of the third sub-network and the masked spectrum as the output for training the third sub-network includes: calculating a third error loss function between the masked spectrum and the target human voice spectrum; and performing iterative training with the goal of converging the third error loss function to a certain threshold to obtain a trained third sub-network.

[0020] In one possible implementation, training the third subnetwork using the output of the second subnetwork as input and the masked spectrum as output includes: freezing the parameters of the first subnetwork and the second subnetwork and updating the parameters of the third subnetwork at each iteration; or simultaneously updating the parameters of the first subnetwork, the second subnetwork, and the third subnetwork at each iteration.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the program or instruction is executed by the processor, the electronic device executes any one of the above-mentioned methods for separating human voices; or, when the program or instruction is executed by the processor, the electronic device executes any one of the above-mentioned methods for training a voiceprint filtering network.

[0022] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code is executed by a computer, the computer implements any of the above-mentioned methods for separating human voices; or, when the computer program code is executed by a computer, the computer implements any of the above-mentioned methods for training a voiceprint filtering network.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program or instruction, which, when executed, enables the computer to execute any one of the above-mentioned methods for separating human voices; or, when executed, enables the computer to execute any one of the above-mentioned methods for training a voiceprint filtering network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0026] Figure 1 A schematic diagram of an artificial intelligence main framework proposed in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the system architecture of a method for voice separation proposed in an embodiment of the present application;

[0028] Figure 3 A flow chart of a method for voice separation proposed in an embodiment of the present application;

[0029] Figure 4A This is a mobile phone UI interface interaction diagram for registering voiceprints for a method for voice separation proposed in an embodiment of the present application;

[0030] Figure 4B A schematic diagram of a voiceprint registration process for multiple members of a method for voice separation proposed in an embodiment of the present application;

[0031] Figure 5 This is a flow chart of the logic function of voiceprint recognition of wake-up speech in a method for voice separation proposed in an embodiment of the present application;

[0032] Figure 6 This is a functional schematic diagram of a voiceprint filtering network for a method of voice separation proposed in an embodiment of the present application;

[0033] Figure 7 A flow chart of a method for voice separation proposed in an embodiment of the present application using a voiceprint filtering network for voiceprint filtering;

[0034] Figure 8 A schematic diagram of the joint training of a positive and negative voiceprint embedding network and a voiceprint filtering network for a method for voice separation proposed in an embodiment of the present application;

[0035] Figure 9 A schematic diagram of a mask fusion network training for a method for voice separation proposed in an embodiment of the present application;

[0036] Figure 10 A schematic diagram of the deployment of a positive and negative voiceprint embedding network and a voiceprint filtering network for a method for voice separation proposed in an embodiment of the present application;

[0037] Figure 11 A schematic diagram of an application scenario of a method for voice separation proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third, etc." or module A, module B, module C, etc. are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that the specific order or sequence can be interchanged where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In the following description, the numbers representing the steps, such as S110, S120, etc., do not necessarily mean that the steps must be executed in this manner. If permitted, the order of the steps can be interchanged or they can be executed simultaneously.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0042] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0043] One possible approach uses a voice separation method based on the voiceprint features of a single target person. This method uses the target person's voiceprint features to filter through common multi-person noise audio and isolate the target person's voice. The specific steps are: calculating the power spectrum of the target person's audio; inputting the target person's audio power spectrum into a convolutional neural network (CNN) to obtain the target person's corresponding voiceprint features; calculating the power spectrum of the mixed noise audio containing the target person's voice using a short-time Fourier transform (STFT); and inputting the power spectrum into an audio filtering neural network consisting of eight convolutional layers, one LSTM layer, and two fully connected layers. The voiceprint features are repeated in the time dimension and then concatenated with the output of the convolutional layers. The concatenated result is fed into a long short-term memory (LSTM) layer, which uses a sigmoid activation function to output a soft mask. The soft mask is multiplied by the target person's audio power spectrum to calculate the enhanced power spectrum of the target person. Finally, the target person's audio is recovered using an inverse STFT.

[0044] Since the above scheme only uses the voiceprint features of the target person, only the enhanced mask of the target person is obtained when obtaining the soft mask. Therefore, when enhancing the audio of multi-person noise, a small amount of information of other voices will be retained, and the noise of other voices cannot be completely removed.

[0045] When multiple people are speaking at once, members generally want their smart devices to identify the main speaker's voice and eliminate the noise of other voices. Therefore, in a fixed indoor location, it is necessary to separate the voices of different target speakers. This embodiment of the present application proposes a method for voice separation that combines the voiceprint information of multiple members and performs voice separation when different members use smart devices, thereby obtaining clean voice separation audio of the members.

[0046] Figure 1 The following is a schematic diagram of an artificial intelligence framework, which describes the overall workflow of the artificial intelligence system and is applicable to the general needs of the artificial intelligence field. Figure 1 The artificial intelligence main framework shown explains a method for voice separation provided by this application from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0047] The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom."

[0048] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0049] (1) Infrastructure 10:

[0050] Infrastructure 10 provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, and other hardware acceleration chips); the foundational platform includes distributed computing frameworks and networks, along with other related platform guarantees and support, including cloud storage and computing, and interconnected networks.

[0051] (2) Data 11

[0052] The data 11 on the upper layer of the infrastructure 10 is used to represent the data source in the field of artificial intelligence.

[0053] In a method, device, and electronic device for separating human voices proposed in an embodiment of the present application, data 11 on a higher layer of infrastructure 10 is derived from mixed audio data of multiple human voices in an indoor location with a fixed number of people.

[0054] (3) Data processing 12

[0055] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0056] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0057] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0058] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0059] In a method for voice separation proposed in an embodiment of the present application, the data processing process for voice separation includes at least data training of mixed audio data, machine learning, deep learning, etc.

[0060] (4) General ability 13

[0061] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system.

[0062] In a method for voice separation proposed in an embodiment of the present application, after the mixed audio data undergoes the above-mentioned data processing, voice separation can be performed in a multi-person scenario based on the results of the data processing, and the voice of the main speaker can be directionally separated by referring to the voiceprints of multiple target persons.

[0063] (5) Smart products and industry applications 14

[0064] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They encapsulate the overall artificial intelligence solution, productize intelligent information decision-making, and realize practical application. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, smart healthcare, smart security, autonomous driving, safe cities, smart terminals, etc. The embodiment of this application proposes a method for voice separation, which can be applied to human-computer interaction technology of intelligent devices in fields such as smart terminals, smart homes, smart security, and autonomous driving. The main application scenario is in indoor places with fixed members, when multiple people are speaking at the same time, the intelligent device can separate the voice of a specific target person.

[0065] For example, the member's fixed indoor location includes home, residence, office or cab, etc.

[0066] Exemplarily, smart devices include smart TVs, smart speakers, robots, smart air conditioners, smart smoke alarms, smart fire extinguishers, smart vehicle-mounted terminals (electronic control units, ECUs), mobile phones, tablets, laptops, desktop computers, all-in-one computers, etc.

[0067] The users of the above-mentioned smart devices are basically fixed, and the voiceprint feature database composed of the members' voiceprints is relatively stable. At the same time, the sound sources in the multi-voice mixed scene collected by the smart devices are within a limited space. Under these conditions, a method for voice separation provided in an embodiment of the present application can enhance the voice of the main speaker, suppress the voice of the noise speaker, and separate the audio information of the main speaker from the mixed speech audio of the multi-voice mixed scene.

[0068] Figure 2 The following is a schematic diagram showing the system architecture of a method for voice separation proposed in an embodiment of the present application. Figure 2 As shown in system diagram 200, data collection device 260 is used to collect voiceprint feature data of multiple members in a fixed indoor location and store it in database 230. Training device 220 trains voiceprint filtering network 201 based on the voiceprint feature data of multiple members maintained in database 230. Voiceprint filtering network 201 can perform voice separation in a scenario where multiple members are speaking in an indoor location, and separate the voice of each member based on the voiceprint feature of each member.

[0069] The following will describe in more detail how the training device 220 trains the voiceprint filtering network 201 based on the voiceprint feature data of multiple members.

[0070] Because it is hoped that the audio output by the voiceprint filtering network 201 is as close as possible to the audio of the real speaker, the weight vector of each layer of the network can be updated based on the difference between the audio output by the current voiceprint filtering network 201 and the audio of the real speaker. Of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the network. For example, if the audio value output by the network is high, the weight vector is adjusted to make its output lower, and the adjustment is continued until the network can output the audio of the real speaker. Therefore, it is necessary to pre-define "how to compare the difference between the output audio and the audio of the real speaker", which is the loss function. They are important equations for measuring the difference between the output audio and the audio of the real speaker. Among them, taking the mean square error loss (mse loss) function as an example, the higher the output value of the mean square error loss function, the greater the difference, so the training of the network becomes a process of minimizing the mse loss as much as possible.

[0071] The member voiceprint filtering network 201 obtained by the training device 220 can be applied to different systems or devices.

[0072] exist Figure 2 In the embodiment, the execution device 210 is configured with an I / O interface 212 for exchanging voice data with an external device. A member can input voice data into the I / O interface 212 through a client device 240. The voice data can be mixed audio data of multiple voices simultaneously spoken by multiple members in an indoor place.

[0073] Exemplarily, the client device 240 is used to collect the user's voice data and can be a sound receiving device, including an integrated microphone, a microphone or microphone array connected to a smart device, or a microphone or microphone array connected to the smart device via short-range connection technology.

[0074] The execution device 210 can call data, code, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0075] The voiceprint extraction module 213 processes the input voice data, extracts the speaker's voiceprint from the voice data, calculates the similarity between the voiceprint and the registered voiceprints, and sorts them. The speaker corresponding to the registered voiceprint with the highest similarity is the main speaker, and the speakers corresponding to the other registered voiceprints are noise speakers.

[0076] The calculation module 211 is used to calculate the spectrum of the mixed audio data using the short-time Fourier transform (STFT). This spectrum is then combined with the main speaker and noise speaker determined by the voiceprint extraction module 213 and passed through the voiceprint filtering network 201 to obtain the clean spectrum of the main speaker. Finally, the inverse short-time Fourier transform (ISTFT) is used to obtain the clean audio of the main speaker.

[0077] Finally, the I / O interface 212 returns the processed speech recognition results to the client device 240 for providing to the members.

[0078] More deeply, the training device 220 takes the output of the audio of multiple members as the training target, and generates a corresponding voiceprint filtering network 201 based on the audio training data of multiple members to provide better results for the members.

[0079] In the attached Figure 2 In the case shown in , members can manually specify the mixed audio data of multiple voices mixed into the execution device 210, for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input the mixed audio data of multiple voices mixed into the I / O interface 212 and obtain the result. If the client device 240 automatically inputs the mixed audio data of multiple voices mixed, it needs to obtain the authorization of the member, the member can set the corresponding permission in the client device 240. The member can view the result output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also serve as a voice data acquisition terminal to store the collected voice data in the database 230.

[0080] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by the embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the attached Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0081] An embodiment of the present application proposes a method for voice separation. In an indoor location where the members are fixed, the registered voiceprints of multiple members are sorted based on the speaker audio obtained by a smart device, the voiceprint of the main speaker and the voiceprint of the noise speaker are determined, the frequency spectrum of the voiceprint of the main speaker and the voiceprint of the noise speaker, as well as the mixed audio data of multiple voices obtained by the smart device are processed, the voice of the main speaker is enhanced, the voice of the noise speaker is suppressed, and the audio information of the main speaker is separated from the mixed speech audio of multiple voices.

[0082] Figure 3 This is a flow chart of a method for voice separation proposed in an embodiment of the present application. Figure 3 As shown, based on the voiceprint information of each registered member, the execution device 210 performs the following steps to implement a method for voice separation proposed in an embodiment of the present application:

[0083] S300, register each member’s voiceprint.

[0084] In indoor places such as home, residence, office or cab, since its members are usually fixed, the voiceprint information of each member can be registered on the smart device first.

[0085] Figure 4A This is a diagram of the mobile phone UI interface for voiceprint registration. Figure 4A As shown, the smart home APP installed on the mobile phone reminds the user to enter the voiceprints of family members in turn. The user follows the operating instructions and enters four voice segments of father, mother, sister and himself in turn. After the entry is completed, the APP reminds the user that the voiceprints of family members have been extracted.

[0086] After the audio recording is completed, the mobile phone sends the audio to the cloud execution device 210. The execution device 210 calls the voiceprint network 41, processes the audio to obtain a voiceprint, and then stores the voiceprint in the execution device 210, and / or sends the voiceprint to the mobile phone, and / or sends an identification of the successful voiceprint to the mobile phone.

[0087] Figure 4B Schematic diagram of the voiceprint registration process for multiple members. Figure 4B As shown, the execution device 210 inputs the voice audio of each member, such as member A, member B, ... member N, into the voiceprint network 41 respectively. The voiceprint network 41 extracts the voiceprint features (d-vector) from the voice audio of each member and registers the voiceprint of each member.

[0088] For example, there are multiple members in a family. When registering their own voiceprint information, each member can Figure 2The client devices 240 in the group input their respective voice signals to the I / O interface 212. In the voiceprint extraction module 213, after the voiceprint network 41 obtains the voice signal of each member input from the I / O interface 212, it extracts the voiceprint features of each member. The execution device 210 can store the voiceprint features of each member in the data storage system 250 to complete the registration.

[0089] For example, the voiceprint network 41 can be a background model based on a deep neural network (DNN), which is used to directly model the speaker's speech audio and extract the voiceprint. The filter bank energy features of each frame of speech are extracted as the input of the trained DNN. The speech features are extracted from the last hidden layer of the trained DNN, regularized, and then accumulated to obtain the voiceprint features, which are also called deep vectors or "d-vectors".

[0090] Exemplarily, the voiceprint network 41 can also use a voiceprint recognition encoder to extract the voiceprint features of the target speaker based on the background model of the long short-term memory network. Specifically, the voiceprint recognition encoder converts the speaker's speech audio into frames of a certain width (e.g., 25ms) and a certain step length (e.g., 10ms), filters each frame through a Mel filter (Mel), and extracts an energy spectrum of a certain size (e.g., 40) from the result as network input. A sliding window of fixed length is constructed on these frames, and a long short-term memory network is run on each window. The final frame output of the long short-term memory network is then used as the voiceprint feature of the sliding window. The voiceprint feature can also be represented by a deep vector or "d-vector".

[0091] S301, obtaining user's voice information, and extracting the voiceprint of the user's voice information.

[0092] In one possible implementation, Figure 2 The client device 240 connected to the smart device collects the voice information of the speaking user and inputs the voice information to the I / O interface 212. After receiving the voice information input from the I / O interface 212, the voiceprint network 41 in the voiceprint extraction module 213 extracts the voiceprint of the voice information. The user's voice information may be a mixed audio that is mixed with the voices of other members.

[0093] The client device 240 may be any audio receiving device, including an integrated microphone, a microphone or microphone array connected to a smart device, or a microphone or microphone array connected to a smart device via a short-range connection technology.

[0094] Optionally, the voiceprint extraction module 213 may also include a voice wake-up module 51. The wake-up module 51 obtains the input voice information, performs voice wake-up processing through wake-up algorithms such as acoustic feature extraction, wake-up word detection, and wake-up word confirmation, and outputs the processing result. Under the condition that the voice is the wake-up audio corresponding to the wake-up word, the execution device 210 wakes up the smart device.

[0095] The wake-up module 51 is a common technology in the industry and will not be described in detail here.

[0096] S302: Determine a main speaker using the user's voice according to the registered voiceprints of multiple members; and determine that multiple members other than the main speaker are noise speakers.

[0097] In one possible implementation, the voiceprint of the user's speech information can be similarly matched and sorted with registered voiceprints to determine the voiceprint of the main speaker and the voiceprint of the noise speaker, thereby determining the main speaker and the noise speaker.

[0098] Specifically, the execution device 210 calculates the similarity between the voiceprint of the speech information and the registered voiceprints and sorts them, and determines, based on the sorting result, that the speaker corresponding to the registered voiceprint with the highest similarity to the voiceprint of the speech information is the main speaker, and the speakers corresponding to the other registered voiceprints are noise speakers.

[0099] Figure 5 Provides a logic flow chart of voiceprint recognition of user's voice information, such as Figure 5 As shown, the wake-up module 51 can first determine whether the received voice information is the wake-up audio corresponding to the wake-up word of the device. If the voice signal is the wake-up audio, the device is started and the voice information is input into the voiceprint network 41. The voiceprint network 41 extracts and outputs the voiceprint of the voice information. The voiceprint of the wake-up audio is matched and sorted with the registered voiceprints for similarity to determine the voiceprint of the main speaker and the voiceprint of the noise speaker.

[0100] S303 : Acquire a mixed audio of the main speaker audio and the noise speaker audio, and perform short-time Fourier transform processing on the mixed audio to obtain a mixed spectrum.

[0101] In one possible implementation, combining Figure 2 When other members are speaking, the main speaker can input voice to the I / O interface 212 through the client device 240. Due to the mixing of the voices of other members, the voice obtained by the execution device 210 is mixed audio data of multiple voices. The calculation module 211 performs short-time Fourier transform on the mixed audio data of multiple voices to obtain a mixed spectrum.

[0102] S304: The mixed spectrum, the voiceprint of the main speaker, and the voiceprints of multiple noise speakers are input into the voiceprint filtering network 201 for processing to obtain the spectrum of the main speaker.

[0103] The voiceprint filtering network 201 is used to use the voiceprint of the main speaker in the registered voiceprints as positive excitation and the voiceprints of multiple noise speakers in the registered voiceprints as negative excitation to separate the spectrum of the main speaker from the mixed spectrum.

[0104] In one possible implementation, the voiceprint filtering network 201 includes: a positive and negative voiceprint embedding network 61, a multi-channel mask filtering network 62, and a mask fusion network 63. The positive and negative voiceprint embedding network 61 is denoted as the first sub-network, and is used to embed a main speaker's voiceprint into a positive encoding process to obtain a positively enhanced voiceprint, and to embed multiple noise speakers' voiceprints into a negative encoding process to obtain multiple negatively suppressed voiceprints. The positively enhanced voiceprint, the negatively suppressed voiceprint, and the mixed spectrum are input into the multi-channel mask filtering network 62. The multi-channel mask filtering network 62 is denoted as the second sub-network, and is used to perform feature extraction, concatenation, long short-term memory, full connection, and masking to obtain a positively enhanced spectrum and multiple negatively suppressed spectrums. The mask fusion network 63 is denoted as the third sub-network, and is used to fuse the positively enhanced spectrum and multiple negatively suppressed spectrums to obtain a fused spectrum mask. The fused spectrum mask is point-wise multiplied with the original mixed spectrum to obtain a filtered clean spectrum, which is the spectrum of the main speaker.

[0105] S305 , performing an inverse short-time Fourier transform calculation on the frequency spectrum of the main speaker to obtain the audio of the main speaker.

[0106] The following combination Figure 6 and Figure 7 The voiceprint filtering network 201 is described in detail.

[0107] Figure 6 This is a functional principle diagram of the voiceprint filtering network in a method for voice separation proposed in an embodiment of the present application, such as Figure 6 As shown, the voiceprint filtering network 201 includes the following subnetworks: a positive and negative voiceprint embedding network 61, a multi-channel mask filtering network 62, and a mask fusion network 63. The input to the voiceprint filtering network 201 is a mixed spectrum corresponding to a mixed audio stream of a positive voiceprint, multiple negative voiceprints, and multiple voices; the output is a filtered clean spectrum. The positive voiceprint is the voiceprint of the primary speaker. For example, when the primary speaker is member A, the positive voiceprint is member A's voiceprint. The negative voiceprint includes the voiceprints of noise speakers. For example, when the noise speakers include members B, C, and N, the negative voiceprints are members B's voiceprint, C's voiceprint, and N's voiceprint.

[0108] Specifically, the execution device 210 inputs a main speaker's voiceprint and multiple noise speaker's voiceprints into the positive and negative voiceprint embedding network 61 respectively.

[0109] The positive and negative voiceprint embedding network 61 embeds the positive voiceprint by positive coding, and embeds the multiple negative voiceprints by negative coding respectively, and outputs a corresponding positive enhanced voiceprint and multiple negative suppressed voiceprints.

[0110] For example, when the code embedded in the positive voiceprint is [1,0], [1,0] is the positive code; at this time, the code embedded in the negative voiceprint can be [0,1], [0,1] is the negative code. Positive coding can enhance the strength of the first time point in the positive voiceprint and weaken the strength of the second time point; negative coding can weaken the strength of the negative voiceprint corresponding to the first time point and enhance the strength of the negative voiceprint corresponding to the second time point.

[0111] The multi-channel mask filtering network 62 extracts the spectral features of the mixed spectrum, concatenates the spectral features with a positive enhancement voiceprint, performs long and short-term memory, fully connects and mask processing on the spectral features, and outputs a corresponding positive enhancement spectrum; concatenates the spectral features with multiple negative suppression voiceprints, performs long and short-term memory, fully connects and mask processing on the spectral features, and outputs multiple corresponding negative suppression spectra.

[0112] The mask fusion network 63 fuses a positive enhancement spectrum and multiple negative suppression spectra to obtain a fused spectrum mask. The fused spectrum mask and the mixed spectrum are multiplied point by point to obtain a filtered clean spectrum, which is the main speaker spectrum.

[0113] Figure 7 Flowchart of voiceprint filtering using voiceprint filtering network 201. Figure 7 As shown, the execution device 210 performs the following steps to perform voiceprint filtering:

[0114] S701: embed a main speaker's voiceprint into positive coding and embed multiple noise speaker's voiceprints into negative coding respectively, to obtain a corresponding positive enhancement voiceprint and multiple negative suppression voiceprints.

[0115] In one possible implementation, a positive voiceprint is embedded into a positive code [1,0], and a corresponding positive enhanced voiceprint is output; multiple negative voiceprints are respectively embedded into negative codes [0,1], and multiple corresponding negative suppressed voiceprints are output.

[0116] In a possible implementation, the positive and negative code generator 87 may output [1, 0] as the positive code; and the positive and negative code generator 87 may output [0, 1] as the negative code.

[0117] S702: Combine one positive enhancement voiceprint and multiple negative suppression voiceprints with the spectrum characteristics of the mixed spectrum, perform multi-channel mask filtering, and output corresponding one positive enhancement spectrum and multiple negative suppression spectrums.

[0118] Exemplarily, the spectral features of the mixed spectrum are extracted through the multi-channel mask filtering network 62, the spectral features are spliced ​​with a positive enhancement voiceprint, long short-term memory, full connection and mask processing are performed, and a corresponding positive enhancement spectrum is output; the spectral features are spliced ​​with multiple negative suppression voiceprints, long short-term memory, full connection and mask processing are performed, and multiple corresponding negative suppression spectra are output.

[0119] S703 : Perform mask fusion processing on a positive enhancement spectrum and multiple negative suppression spectrums to obtain a fused spectrum mask.

[0120] S704: Multiply the fused spectrum mask and the input mixed spectrum point by point to obtain the spectrum of the main speaker.

[0121] The embodiment of the present application proposes a method for training a voiceprint filtering network 201. The training process is as follows: Figure 2 The method is implemented in the training device 220 shown. The method uses the target person's voiceprint and the first and second codes as inputs to the positive and negative voiceprint embedding network 61; uses the output of the positive and negative voiceprint embedding network 61 and the mixed spectrum corresponding to the mixed audio of the target person's audio, other people's audio, and noise as inputs to the multi-channel mask filtering network 62; uses the positive enhancement spectrum and the negative suppression spectrum as the output of the multi-channel mask filtering network 62, and uses a joint training method to train the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62; uses the output of the multi-channel mask filtering network 62 as input to the mask fusion network 63, and uses the masked spectrum as output to train the mask fusion network 63; after iterative training, the trained voiceprint filtering network is obtained.

[0122] Figure 8 This is a schematic diagram of the joint training of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62 in the training method of the voiceprint filtering network 201 proposed in an embodiment of the present application.

[0123] like Figure 8 As shown, the positive and negative voiceprint embedding network 61 adopts a network structure transformer 81. Transformer 81 is used to construct the features of each element in the target person's voiceprint vector using an attention structure, thereby finding the importance of all other elements in the voiceprint to each of the above elements.

[0124] The essence of attentional structure stems from the human visual attention mechanism. When people perceive something, they typically don't look at the entire scene from beginning to end. Instead, they focus on specific parts based on their needs. When they notice that a certain part of a scene frequently contains something they're interested in, they learn to focus on that part in similar scenes in the future.

[0125] Therefore, the attention structure is often used to calculate the degree of relevance. The principle is to regard the query in the source data as a mapping of a series of <key, value> data pairs according to the degree of dependence. By calculating the similarity or correlation between the query and each key, the weight coefficient of each key corresponding to the value is obtained, and then the values ​​are weighted and summed to obtain the final attention output value.

[0126] For example, when calculating attention, the query and each key are first similarly calculated to obtain weights. Commonly used similarity functions include dot product, concatenation, and perceptron. These weights are then normalized using a softmax function. Finally, the weights and the corresponding key values ​​are weighted and summed to obtain the final attention.

[0127] Transformer 81 can also adopt a self-attention structure, where query = key = value. For example, when a sentence is input, each word in the sentence must be subjected to attention calculation with all the words in the sentence. The purpose is to learn the dependency relationship between the words within the sentence and capture the internal structure of the sentence.

[0128] In one possible implementation, the positive and negative voiceprint embedding network 61 uses a self-attention structure, taking the target person's voiceprint input vector and positive and negative encodings as input, and outputting positively enhanced voiceprints and negatively suppressed voiceprints as targets for training. During training, the target person's voiceprint is positively and negatively embedded and re-encoded. The encoded positive and negative encoding vectors of the target person's voiceprint serve as the query in the source data, and each element in the target person's voiceprint input vector serves as a key. The query and each key are similarly calculated to obtain weights, which are then normalized using a softmax function. Finally, the weights are weighted and summed with the corresponding key values ​​to obtain the final attention. The dimension of the output target feature always remains consistent with the dimension of the target person's voiceprint.

[0129] Exemplarily, positive and negative encoding can use one-hot encoding with a length of 2. When the encoding is [1,0], it indicates positive enhancement, which increases the intensity of the target person's voiceprint at the first time point and weakens the intensity of the target person's voiceprint at the second time point. The training goal of the positive and negative voiceprint embedding network 61 is to output positively enhanced voiceprints. When the encoding is [0,1], it indicates negative inhibition, which has the opposite effect of the encoding rule of positive enhancement, weakening the intensity of the target person's voiceprint at the first time point and increasing the intensity of the target person's voiceprint at the second time point. The training goal of the positive and negative voiceprint embedding network 61 is to output negatively inhibited voiceprints. The purpose of positive and negative encoding is to filter out the target person's voiceprint from the mixed spectrum. [0,0] and [1,1] are illegal inputs.

[0130] Among them, One-Hot encoding, also known as one-bit effective encoding, mainly uses an N-bit state register to encode N states. Each state has its own independent register bit, and only one bit is valid at any time.

[0131] Continue as Figure 8 As shown, the positive and negative voiceprint embedding network 61 inputs the positive enhancement voiceprint and negative suppression voiceprint output in the training state into the multi-channel mask filtering network 62.

[0132] The multi-channel mask filtering network 62 adopts a cascade structure of convolutional neural network 82 (convolutional neural network, CNN) - splicing module 83 (concat) - long short-term memory network 84 (long short-term memory, LSTM) - fully connected network 85 (full connected, FC) - soft mask prediction 86 to train positive enhancement voiceprints and negative suppression voiceprints.

[0133] The convolutional neural network 82 is a deep neural network with a convolutional structure, which is a deep learning architecture. A deep learning architecture uses machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, the convolutional neural network is a feed-forward artificial neural network in which individual neurons respond to overlapping regions in a mixed spectrum input to the network.

[0134] During training, a mixed spectrum containing the target voice, other voices, noise, and other multi-voice mixture is used as the input of the convolutional neural network 82. Each neuron in the convolutional neural network 82 performs a convolution operation on the overlapping area in the input mixed spectrum, and the output result is a feature matrix, which is recorded as the first matrix.

[0135] The feature matrix output by the convolutional neural network 82 is concatenated frame by frame with the positive enhancement voiceprint and negative suppression voiceprint output by the positive and negative voiceprint embedding network 61 through the concatenation module 83 (concat) to obtain a concatenated matrix. The concatenated matrix is ​​then input into the long short-term memory network 84. The concatenated matrix is ​​denoted as the second matrix.

[0136] Long Short-Term Memory (LSTM) networks 84 are commonly used for processing time series. The computational units within LSTM networks 84, such as neurons, can remember previous information and use it as input for subsequent processing. Due to their temporal recursive nature, LSTM networks 84 are often used to process sequences with temporal characteristics, such as text (ordered by characters or words) and audio (time-domain signals), which are sequential sequences of behaviors.

[0137] In a feasible implementation, the long short-term memory network 84 is first trained based on the initial weights, and the parameter information obtained from this training is memorized in the neurons, and used as the parameter information for subsequent training, and the weights of the self-loop are adjusted each time training is performed.

[0138] In one feasible implementation, the splicing matrix output by the splicing module 83 is used as input, and the long short-term memory network 84 is run on each frame of the splicing matrix. The spectrum matrix output by the last frame of the long short-term memory network 84 is then fed into the fully connected network 85. The dimension of the spectrum matrix is ​​the same as the neuron dimension of the long short-term memory network 84. The spectrum matrix is ​​denoted as the third matrix.

[0139] The fully connected network 85 is used for computation from input to output. The parameters of the fully connected network 85 are trained to establish a certain correspondence between the output and input, for example, if the input is 1, 1, the output is 1, 0, implementing a model similar to an adder. During training, data normalization is required because the calculation results of the previous layer of the network are likely not between 0 and 1, and the data needs to be within a uniform range. Therefore, an activation function is often used to limit the data range.

[0140] In a feasible implementation, the fully connected network 85 reduces the dimension of the spectrum matrix to the dimension of the target human voice spectrum, recorded as the fourth matrix, and outputs the fourth matrix to the soft mask prediction 86.

[0141] Soft mask prediction 86 is a procedure that can correct errors and can be modified during training or field trials.

[0142] A mask is a binary code used to perform bitwise operations on the target field, masking the current input bit. A common approach involves applying bitwise or logical operations, such as OR and AND, to the source field to produce a new operand. For example, a mask can be used to convert uppercase letters in ASCII code to lowercase.

[0143] In the embodiment of the present application, the "mask" in the soft mask prediction 86 is variable, or "soft", and the value of the mask can be changed through program code.

[0144] In a feasible implementation, the soft mask prediction uses a Sigmoid function to perform a mask operation on each element of the fourth matrix. The output of the soft mask prediction 86 is a spectrum mask with the same dimension as the fourth matrix, including a positive enhancement spectrum and a negative suppression spectrum.

[0145] Finally, the spectrum mask is multiplied point by point with the input mixed spectrum to obtain the masked spectrum, and the masked spectrum is compared with the true value to calculate the mean square error loss function (MSE loss). Depending on the positive and negative encoding values, different true values ​​are used to calculate the mean square error loss function.

[0146] In one possible implementation, the target human voice spectrum is trained using a forward training method, and the mean square error loss function between the spectrum corresponding to the target human audio and the forward enhanced spectrum is calculated, which is recorded as the first loss function mse loss1; the value of mse loss1 should become smaller and smaller during the iterative training process until a preset threshold or a preset number of iterations is reached, and then the training is stopped.

[0147] The negative training method is used to train the non-target human voice spectrum, that is, the aliased spectrum of other human voices and noise, and the mean square error loss function between the spectrum corresponding to other people's audio and the negative suppression spectrum is calculated, which is recorded as the second loss function mseloss2; the value of mseloss2 should become smaller and smaller during the iterative training process until it reaches a preset threshold or a preset number of iterations, and then the training is stopped.

[0148] It should be noted that when the encoded value of the positive and negative voiceprint embedding network 61 input is [1,0], the joint training process of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62 is called forward training, the input of the multi-channel mask filtering network 62 is the positive enhanced voiceprint, the corresponding mask is called the positive enhanced spectrum, the masked spectrum is called the positive enhanced spectrum, the training target is the target human voice spectrum, and the loss function is the mean square error loss function between the target human voice spectrum and the positive enhanced spectrum.

[0149] When the encoded value of the positive and negative voiceprint embedding network 61 input is [0,1], the joint training process of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62 is called negative training, the input of the multi-channel mask filtering network 62 is the negative suppression voiceprint, the corresponding mask is called the negative suppression spectrum, the masked spectrum is called the negative suppression spectrum, the training target is the non-target human voice spectrum, that is, the aliased spectrum of other human voices and noise, and the loss function is the value of the mean square error loss function between the non-target human voice spectrum and the negative suppression spectrum.

[0150] The "loss function" that evaluates the magnitude of the error is also called the "cost function." A smaller loss function indicates a better model fit. A larger loss function indicates a significant impact on the error, and a larger stride is required for weight adjustments. Conversely, a smaller stride and slightly smaller weight adjustments are sufficient.

[0151] It's important to understand that when weights need to be changed during training, a pre-set learning rate (learning rate) can be used to determine how much the weights need to change during each training session, i.e., the size of the pre-set step. Gradient descent is used to estimate the change in overall error caused by each weight change. If the learning rate is set too high, it's possible that no amount of training will lead to optimal values, preventing the loss function from converging. If the learning rate is set too low, training will be very slow, and it's easy to get stuck in a local optimum and miss the optimal / better solution.

[0152] It can be understood that the training of the above-mentioned neural network is supervised learning. That is, the input X has a corresponding true value Y, and the loss function value between the neural network's output Y' and the true value Y is the object of network backpropagation. The entire network training process is the process of continuously reducing the loss function value. Through continuous iteration, the degree of each weight update becomes smaller and smaller, and the loss function value also becomes smaller and smaller. Training stops when the loss function value reaches a preset threshold or the number of training iterations reaches a preset number of iterations. The trained positive and negative voiceprint embedding network 61 and multi-channel mask filtering network 62 are obtained.

[0153] In one possible implementation, assuming that the training dataset includes a clean human voice corpus dataset and a noise dataset, the human voice corpus dataset contains n speakers, each speaker has m corpora, and n and m are arbitrary natural numbers, an example of the joint training process of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62 is as follows:

[0154] S801: Randomly select two speakers, a and b, from n speakers. Take speaker a as the target speaker for this training. Randomly select two corpora, wav_a1 and wav_a2, from speaker a's corpus; randomly select one corpus, wav_b, from speaker b's corpus; and randomly select one corpus, wav_n, from the noise dataset.

[0155] S802, calculate the voiceprint of the corpus wav_a1 to obtain the target person's voiceprint a1. The dimension of the target person's voiceprint a1 is [1, C], where 1 indicates that [1, C] is a one-dimensional vector, C is the number of elements in the one-dimensional vector, and its algebraic form is 1×C; for example, the value of C is 128, which means that the one-dimensional voiceprint includes 128 feature data, and its algebraic form is 1×128.

[0156] S8031, align the lengths of corpus wav_a2 of a, corpus wav_b of b, and noise corpus wav_n. The aligned corpus audios are represented by wav_a2_align, wav_b_align, and wav_n_align respectively.

[0157] For example, zero padding can be used to align the lengths of wav_a2, wav_b, and wav_n. For example, if the lengths of wav_a2, wav_b, and wav_n are 2, 3, and 5 frames, respectively, to align them to the longest wav_n, zero padding is used. Three frames of zeros are added after wav_a2 to obtain the aligned target vocal wav_a2_align, and two frames of zeros are added after wav_b to obtain the aligned other vocal wav_b_align. This aligns the lengths of wav_a2, wav_b, and wav_n, and makes wav_n_align the same as wav_n.

[0158] Optionally, you can align the lengths of wav_a2, wav_b, and wav_n by truncating them. For example, if the lengths of wav_a2, wav_b, and wav_n are 2, 3, and 5 frames, respectively, they can be aligned to the shortest wav_a2. Randomly truncating two frames from wav_b and wav_n to obtain wav_b_align and wav_n_align, aligning the lengths of wav_a2, wav_b, and wav_n. The lengths of wav_a2, wav_b, and wav_n are aligned, and wav_a2_align is the same as wav_a2. When truncating, select audio frames with sound.

[0159] S8032: directly add wav_a2_align, wav_b_align, and wav_n_align or mix them by reverberation to form an audio wav_mix, recorded as the first aliased audio. Similarly, directly add wav_b_align and wav_n_align or mix them by reverberation to form an audio wav_b_n_align, recorded as the second aliased audio.

[0160] S8033 calculates the spectrum matrix spec_mix for wav_mix, the spectrum matrix spec_a2 for wav_a2_align, and the spectrum matrix spec_b_n for wav_b_n_align. The dimensions of the spectrum matrices spec_mix, spec_a2, and spec_b_n are all [1, T, F], where 1 indicates that [1, T, F] is a one-dimensional matrix, T is the number of audio spectrum frames, and F is the spectral feature data for each frame. The corresponding algebraic expression is 1 × T × F. For example, a matrix dimension of 1 indicates a one-dimensional matrix; a T value of 10 indicates a spectrum length of 10 frames; and an F value of 256 indicates that the spectral features of each frame are represented by 256 numbers. Therefore, [1, 10, 256] indicates that the spectrum matrix is ​​a one-dimensional matrix, the spectrum length is 10 frames, and each frame has 256 spectral feature data. The algebraic expression is 1 × 10 × 256.

[0161] S804, copy the voiceprint of the corpus wav_a1 in S802, and use the voiceprint of wav_a1 and the copied voiceprint together as the target person's voiceprint input vector. The dimension of the target person's voiceprint input vector is [2, C], 2 means [2, C] is a two-dimensional vector, C is the number of elements in each dimension of the two-dimensional vector, and the algebraic form is 2×C; for example, the value of C is 128, which means that each dimension of the two-dimensional voiceprint vector includes 128 feature data.

[0162] The corresponding positive and negative encoding inputs are also two-dimensional vectors: [[1,0], [0,1]], where [1,0] corresponds to positive enhancement and [0,1] corresponds to negative inhibition.

[0163] The target person's voiceprint input vector and the positive and negative encoding inputs are fed into the positive and negative voiceprint embedding network 61 for training. The output of the positive and negative voiceprint embedding network 61 is the embedded voiceprint embedding_a1, which also has a 2D dimension: [2, C], representing two voiceprints: the positive enhancement voiceprint and the negative suppression voiceprint. The positive enhancement voiceprint has a dimension of [1, C], with an algebraic form of 1 × C; the negative suppression voiceprint has a dimension of [1, C], with an algebraic form of 1 × C; these dimensions are consistent with the voiceprint dimension of wav_a1.

[0164] S805. Copy the spectrum matrix spec_mix obtained in S8033 to obtain a spectrum matrix input with a dimension of [2, T, F], corresponding to the algebraic expression 2×T×F. Send the spectrum matrix input to the convolutional neural network 82 part of the multi-channel mask filter network 62. After training, the output result is represented by the feature matrix feature_cnn with a dimension of [2, T, F_cnn], corresponding to the algebraic expression 2×T×F_cnn; where F_cnn is the dimension after training the convolutional neural network 82.

[0165] S806: The embedded voiceprint embedding_a1 obtained in S804 is expanded by one dimension, from a two-dimensional vector [2, C] to a two-dimensional matrix [2, 1, C]. This is then replicated T times, resulting in a dimension of [2, T, C]. This means the embedded voiceprint is expanded into a two-dimensional matrix with a length of T frames, C elements per frame, and a corresponding algebraic expression of 2 × T × C. This is then concatenated with the feature matrix feature_cnn [2, T, F_cnn] obtained in S805 through concatenation module 83, resulting in a concatenated matrix with dimensions of [2, T, C + F_cnn]. This means the concatenated matrix is ​​a two-dimensional matrix with a length of T frames, C + F_cnn elements per frame, and a corresponding algebraic expression of 2 × T × C + F_cnn. This matrix is ​​then fed into the subsequent long short-term memory network 84.

[0166] The long short-term memory network 84 takes the concatenated matrix [2, T, C+F_cnn] as input, first performs training based on the initial weights, memorizes the parameter information obtained from this training, and uses it as the parameter information for the next training, adjusts the weights of the self-loop, and outputs a spectrum matrix. The dimension of the spectrum matrix is ​​the same as the neuron dimension of the long short-term memory network 84.

[0167] For example, the neuron dimension of the long short-term memory network 84 is [2, T, F_lstm], and the corresponding algebraic expression is 2×T×F_lstm. The spectrum matrix [2, T, F_lstm] is added to the fully connected network 85.

[0168] The fully connected network 85 reduces the dimension of the spectrum matrix from [2, T, F_lstm] to the dimension [2, T, F] of the target human voice spectrum spec_a2, and outputs it to the soft mask prediction 86.

[0169] The output of soft mask prediction 86 is a mask with a dimension of [2, T, F].

[0170] S807: Multiply the mask obtained in S806 by the spec_mix obtained in S803 point by point to obtain a masked spectrum spec_masked with a dimension of [2, T, F]. Spec_masked[0] is the positive enhancement spectrum, which represents the output result corresponding to the positive and negative encoding [1, 0]. The first loss function mse loss1 is calculated based on the positive enhancement spectrum spec_masked[0] and spec_a2 obtained in S803. Spec_masked[1] is the negative suppression spectrum, which represents the output result corresponding to the positive and negative encoding [0, 1]. The second loss function mse loss2 is calculated based on the negative suppression spectrum spec_masked[1] and spec_b_n obtained in S803.

[0171] S808: Perform gradient calculation and backpropagation based on the mse loss1 and mse loss2 obtained in S807, and update the entire model parameters. The values ​​of mse loss1 and mse loss2 should decrease during the iterative training process until a preset threshold or a preset number of iterations is reached, at which point training is stopped.

[0172] Alternatively, a dataset containing only human voice data and no noise data may be used to train the joint model. In this case, in S803, only wav_a2_align and wav_b_align are mixed into wav_mix, and the spectrum of wav_b_align is used to replace the spectrum of wav_b_n_align.

[0173] The above training method trains positive reinforcement and negative suppression in pairs, and the ratio of positive reinforcement to negative suppression can be adjusted as needed. In this case, there is no need to copy the input voiceprint in S804; the positive and negative encodings are set as needed; there is no need to copy spec_mix in S805; and the true value of the mean square error loss function calculated in S807 is determined by the positive and negative encodings of the input.

[0174] Figure 9 This is a training flow chart for the mask fusion network 63 in the training method for the voiceprint filtering network 201 provided in an embodiment of the present application. The mask fusion network 63 can be trained independently, freezing the parameters of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62. When the parameters are updated, only the parameters of the mask fusion network 63 are updated.

[0175] Optionally, the mask fusion network 63 can also be jointly trained with the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62, and all parameters of the positive and negative voiceprint embedding network 61, the multi-channel mask filtering network 62 and the mask fusion network 63 are updated simultaneously.

[0176] like Figure 9As shown, the network structure of the mask fusion network 63 uses a transformer 92 (transformer), and the attention structure therein uses a common attention structure. The positive enhancement spectrum is used as the query of the attention. The positive enhancement spectrum and all negative suppression spectra are concatenated in the splicer 91. The concatenation result is used as the keys and values ​​of the attention. The transformer 92 is trained, and the mask output by the splicer 91 is multiplied point by point with the mixed spectrum to obtain the masked spectrum. The mean square error loss function between the masked spectrum and the target human voice spectrum is calculated, which is recorded as the third error loss function mse loss3; iterative training is performed with mse loss3 converging to a preset threshold or as a target to obtain a trained third sub-network.

[0177] Assuming that the training dataset includes a clean human voice corpus dataset and a noise dataset, the human voice corpus dataset contains n speakers, and each speaker has m corpora, an example of the training process of the mask fusion network 63 is as follows.

[0178] S901: Randomly select multiple speakers a, b, ... from n speakers. The number of speakers selected is controlled to 2 to 5 according to the average family size. Speaker a is used as the target speaker for this training.

[0179] S902: Based on the training process of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62, the mask fusion network 63 is trained with the target person's voiceprint, the voiceprints of other speakers, the mixed spectrum of multiple voices, and the clean spectrum of the target person as input.

[0180] S903: Assume that the positive and negative codes of the target person are [1, 0] and the positive and negative codes of other speakers are [0, 1]. Use the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62 to obtain the positive enhancement spectrum and the negative suppression spectrum. The dimension of each mask is [1, T, F]. The training process is the same as the training process of the multi-channel mask filtering network 62.

[0181] In step S904, the positive enhancement spectrum is used as the query input to transformer 92. The positive enhancement spectrum and all negative suppression spectra are concatenated in concatenator 91 (concat). The concatenated results are used as the keys and values ​​input to transformer 92, and transformer 92 is trained. The network output is a mask with the dimension [1, T, F].

[0182] S905 , multiply the mask obtained in S904 by the mixed spectrum in S902 point by point to obtain a masked spectrum, and then calculate the third loss function mse loss3 with the clean spectrum of the target person obtained in S902.

[0183] S906: Perform gradient calculation and backpropagation based on the mse loss 3 obtained in S905, and update the model parameters. The value of mse loss 3 should decrease during the iterative training process until it reaches a preset threshold or a preset number of iterations, at which point training is stopped.

[0184] Figure 10 Schematic diagram of the joint deployment of the positive and negative voiceprint embedding network 61 and the multi-channel mask filtering network 62. Figure 10 As shown, the deployed multi-channel mask filtering network 62 is the same as the trained multi-channel mask filtering network 62; a positive and negative encoding generator 87 is added to the deployed positive and negative voiceprint embedding network 61, and the positive and negative encoding generator 87 is used to generate positive encoding and negative encoding.

[0185] In one possible implementation, if the currently input registration voiceprint is the voiceprint of the main speaker, the positive and negative encoding generator 87 outputs [1, 0], and the mask output by the multi-channel mask filtering network 62 is a positive enhancement spectrum; if the currently input registration voiceprint is the voiceprint of a noise speaker, the positive and negative encoding generator 87 outputs [0, 1], and the mask output by the multi-channel mask filtering network 62 is a negative suppression spectrum.

[0186] In one possible implementation, after voiceprint registration is complete, the batch processing capacity of the multi-channel masked filtering network 62 can be set based on the number of registered voiceprints. This refers to the number of input voiceprints that the multi-channel masked filtering network 62 can process in one run. The multi-channel masked filtering network 62 can process multiple voiceprints simultaneously.

[0187] When there is only one registered voiceprint, the registered voiceprint is the main speaker's voiceprint; when there are multiple registered voiceprints, the main speaker's voiceprint and the noise speaker's voiceprint are determined based on the wake-up audio.

[0188] In one possible implementation, after the voiceprint registration phase is completed, the number of positive and negative suppression spectra to be spliced ​​can be determined based on the number of registered voiceprints. During operation, the final spectrum mask is generated based on the output of the multi-channel mask filtering network 62.

[0189] Deployment status of mask fusion network 63 Figure 6 The relevant parts are the same as in , so they are not repeated here.

[0190] The embodiments of the present application can determine the main speaker based on wake-up, and treat other speakers as noise speakers, to achieve directional voice separation of multiple target persons within a limited range. In addition, based on the voiceprint features of multiple speakers, the main speaker's voice audio is enhanced and the noise speaker's voice audio is suppressed, thereby obtaining clean audio of the main speaker.

[0191] An embodiment of the present application provides an electronic device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the program or instruction is executed by the processor, the electronic device executes any of the above-described methods for training a voiceprint filtering network; or, when the program or instruction is executed by the processor, the electronic device executes the method for human voice separation described in any of the above-described embodiments.

[0192] See attached Figure 11 , an embodiment of the present application provides an electronic device 300. The execution device 210 is implemented by one or more processors. Optionally, the processor is coupled with other computing devices, such as data storage, routers, load balancers and other devices; the execution device 210 can be arranged at one physical site, or distributed across multiple physical sites. The execution device 210 can use the data in the data storage system 250, and / or call the program code in the data storage system 250 to implement a method for voice separation proposed in an embodiment of the present application. For voice separation of a specific target person in a multi-person speaking scenario, calling the program code in the data storage system 250 can implement the method of any of the above embodiments.

[0193] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0194] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0195] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, the local device 301 may provide local data or feedback calculation results to the execution device 210 .

[0196] It should be noted that all functions of the execution device 210 can also be implemented by local devices. For example, the local device 301 implements the functions of the execution device 210 and provides services to its own users, or provides services to the users of the local device 302.

[0197] An embodiment of the present application provides a computer-readable medium, which is used to store a computer program. When the computer program runs on a computer, it enables the computer to execute the method executed by the training device in the above method embodiment; or, when the computer program runs on a computer, it enables the computer to execute the method executed by the execution device in the above method embodiment.

[0198] An embodiment of the present application provides a chip including a processor and a communication interface, such as an input / output interface, a pin, or a circuit. The processor is configured to read instructions to execute the method performed by the training device in the above method embodiment; alternatively, the processor is configured to read instructions to execute the method performed by the execution device in the above method embodiment.

[0199] An embodiment of the present application provides a computer program product, which, when executed by a computer, implements the method performed by the training device in the above method embodiment; or, when executed by a computer, implements the method performed by the execution device in the above method embodiment.

[0200] Optionally, the processor mentioned in any of the above is a general-purpose central processing unit (CPU), a microprocessor, or an application-specific integrated circuit (ASIC).

[0201] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0202] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0203] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0204] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0206] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0207] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0208] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A method for separating human voices, applied to an execution device, characterized in that: The method comprises: Receiving a voice of a user in a designated space; the designated space is a limited space having a plurality of members; Determining a main speaker using the user's voice based on the registered voiceprints of the multiple members; and determining that multiple members other than the main speaker are all noise speakers; Obtaining a mixed spectrum corresponding to mixed audio in the specified space, where the mixed audio is speech information of a mixture of audio of a main speaker and audio of at least one noise speaker; The mixed spectrum and the registered voiceprints of the multiple members are processed by a voiceprint filtering network to obtain the spectrum of the main speaker in the mixed audio; wherein the input of the voiceprint filtering network is a mixed spectrum corresponding to a positive voiceprint, multiple negative voiceprints and a mixed audio of multiple voices; the output is a filtered clean spectrum; wherein the positive voiceprint is the voiceprint of the main speaker, and the negative voiceprint includes the voiceprint of the noise speaker; the voiceprint filtering network uses the voiceprint of the main speaker in the registered voiceprints as positive excitation, and uses the voiceprints of the multiple noise speakers in the registered voiceprints as negative excitation, to separate the spectrum of the main speaker from the mixed spectrum; and obtains the audio of the main speaker based on the spectrum of the main speaker.

2. The method for separating human voice according to claim 1, wherein: The determining the main speaker by using the user's voice according to the registered voiceprints of the multiple members includes: Extracting a voiceprint of the user's voice through a voiceprint network; the voiceprint network is a deep neural network model including a long short-term memory network; Calculating a similarity value between the voiceprint of the user's voice and the registered voiceprint of each member among the registered voiceprints of the multiple members; sorting the similarity values; According to the ranking result, the speaker corresponding to the registered voiceprint having the highest similarity to the voiceprint of the user's voice is determined as the main speaker.

3. The method for separating human voice according to claim 1, wherein: The voiceprint filtering network includes a first sub-network, a second sub-network and a third sub-network; The first subnetwork is used to embed a first code in the voiceprint of the main speaker to obtain a positively enhanced voiceprint, and embed a second code in each of the at least one noise speaker voiceprint to obtain at least one negatively suppressed voiceprint; the second subnetwork is used to positively excite the mixed spectrum according to the positively enhanced voiceprint to obtain a corresponding positively enhanced spectrum, and negatively excite the mixed spectrum according to the at least one negatively suppressed voiceprint to obtain a corresponding at least one negatively suppressed spectrum; the third subnetwork is used to fuse the one positively enhanced spectrum and the at least one negatively suppressed spectrum to obtain a fused spectrum mask, so that the fused spectrum mask and the mixed spectrum are multiplied point by point to obtain the spectrum of the main speaker.

4. The method for separating human voice according to claim 3, wherein: The processing of the mixed spectrum and the registered voiceprints of the plurality of members through a voiceprint filtering network to obtain the spectrum of the main speaker in the mixed audio includes: The registered voiceprints of the multiple members are input into the first sub-network, and the first code is embedded in the voiceprint of the main speaker among the registered voiceprints of the multiple members to obtain a positive enhancement voiceprint; and the second code is embedded in the voiceprint of each noise speaker among the at least one noise speaker voiceprint to obtain at least one negative suppression voiceprint.

5. The method for separating human voice according to claim 3, wherein: The processing of the mixed spectrum and the registered voiceprints of the plurality of members through a voiceprint filtering network to obtain the spectrum of the main speaker in the mixed audio includes: Inputting the mixed spectrum into the second sub-network for convolution, and outputting the convolution result; The convolution result and the forward-enhanced voiceprint are spliced ​​frame by frame through a splicing module. The spliced ​​result is sequentially forward-excited through a long short-term memory network, a fully connected network, and a soft mask prediction network to obtain a forward-enhanced spectrum. The convolution result and the at least one negative suppression voiceprint are spliced ​​frame by frame through a splicing module, and the spliced ​​result is negatively excited in turn through a long short-term memory network, a fully connected network and a soft mask prediction network to obtain at least one negative suppression spectrum.

6. The method for separating human voice according to claim 3 or 4, characterized in that: The first code and the second code are generated by a positive and negative code generator.

7. The method for separating human voice according to claim 5, characterized in that: The third sub-network uses the Sigmoid function to perform mask calculation processing on the result output by the fully connected network to obtain a fused spectrum mask.

8. The method for separating human voice according to claim 1, wherein: Before receiving the voice of the user in the designated space, the method includes: pre-acquiring the voiceprints of at least two members through a voiceprint network for registration; the voiceprint network is a deep neural network model, including a long short-term memory network model.

9. The method for separating human voice according to claim 1, wherein: After receiving the voice of the user in the designated space, the method further includes: The voice is determined to be a wake-up audio according to the wake-up word contained in the user's voice.

10. The method for separating human voice according to claim 9, characterized in that: The determining the main speaker by using the user's voice according to the registered voiceprints of the multiple members includes: Extracting the voiceprint of the wake-up audio through a voiceprint network; the voiceprint network is a deep neural network model including a long short-term memory network; Calculating a similarity value between the voiceprint of the wake-up audio and the registered voiceprint of each member among the registered voiceprints of the multiple members; sorting the similarity values; According to the sorting result, the speaker corresponding to the registered voiceprint having the highest similarity to the voiceprint of the wake-up audio is determined as the main speaker.

11. A method for training a voiceprint filtering network, applied to a training device, characterized in that: The voiceprint filtering network input is a mixed spectrum corresponding to a positive voiceprint, multiple negative voiceprints, and a mixed audio of multiple voices; the output is a filtered clean spectrum; wherein the positive voiceprint is the voiceprint of the main speaker, and the negative voiceprint includes the voiceprint of the noise speaker; The voiceprint filtering network includes a first sub-network, a second sub-network and a third sub-network; The first sub-network is configured to embed a first code into the voiceprint of the main speaker to obtain a positive enhancement voiceprint, and embed a second code into each of the at least one noise speaker voiceprint to obtain at least one negative suppression voiceprint; the first code and the second code are generated by a positive and negative code generator; The second sub-network is configured to perform positive excitation on the mixed spectrum according to the positive enhancement voiceprint to obtain a corresponding positive enhancement spectrum, and perform negative excitation on the mixed spectrum according to the at least one negative suppression voiceprint to obtain a corresponding at least one negative suppression spectrum; The third sub-network is used to fuse the one positive enhancement spectrum and the at least one negative suppression spectrum to obtain a fused spectrum mask, so as to multiply the fused spectrum mask and the mixed spectrum point by point to obtain a spectrum of the main speaker; The target person's voiceprint, the first code, and the second code are used as inputs of the first sub-network, and the positive enhancement voiceprint and the negative suppression voiceprint are used as outputs of the first sub-network; The output of the first sub-network and the mixed spectrum corresponding to the mixed audio of the target person's audio, other people's audio and noise are used as the input of the second sub-network; The first sub-network and the second sub-network are trained in a joint training manner, using the positive enhancement spectrum and the negative suppression spectrum as outputs of the second sub-network; The output of the second sub-network is used as the input of the third sub-network, and the masked spectrum is used as the output to train the third sub-network; Obtain the trained voiceprint filtering network.

12. The method for training a voiceprint filtering network according to claim 11, characterized in that: The method of training the first sub-network and the second sub-network by joint training includes: Calculating a first loss function between the spectrum corresponding to the target person audio and the forward enhancement spectrum; Calculating a second loss function between the spectrum corresponding to the other person's audio and the negative suppression spectrum; Iterative training is performed with the goal of respectively converging the first loss function and the second loss function to a certain threshold value to obtain the trained first sub-network and the second sub-network.

13. The method for training a voiceprint filtering network according to claim 11 or 12, characterized in that: The method of using the output of the second sub-network as the input of the third sub-network and using the masked spectrum as the output to train the third sub-network includes: Calculating a third error loss function between the masked spectrum and the target human voice spectrum; Iterative training is performed with the goal of converging the third error loss function to a certain threshold to obtain a trained third sub-network.

14. The method for training a voiceprint filtering network according to claim 13, wherein: The step of using the output of the second sub-network as the input of the third sub-network and using the masked spectrum as the output to train the third sub-network includes: Freeze the parameters of the first sub-network and the second sub-network at each iteration, and update the parameters of the third sub-network; or The parameters of the first sub-network, the second sub-network and the third sub-network are updated simultaneously in each iteration.

15. An electronic device, characterized in that: include: A processor, the processor being coupled to a memory, the memory being used to store programs or instructions, which, when executed by the processor, causes the electronic device to perform the method for separating human voices as described in any one of claims 1 to 10; or, when executed by the processor, causes the electronic device to perform the method for training a voiceprint filtering network as described in any one of claims 11 to 14.

16. A computer program product, comprising computer program code, characterized in that: When the computer program code is executed by a computer, the computer implements the method for separating human voices as described in any one of claims 1 to 10; or, when the computer program code is executed by a computer, the computer implements the method for training a voiceprint filtering network as described in any one of claims 11 to 14.

17. A computer-readable storage medium, characterized in that Used to store computer programs or instructions, which, when executed, enable the computer to execute the method for separating human voices as described in any one of claims 1 to 10; or, when executed, enable the computer to execute the method for training a voiceprint filtering network as described in any one of claims 11 to 14.

Images