A model training method, device and equipment based on speech extraction

Abstract

The embodiments of this specification provide a model training method, device and equipment based on speech extraction. The method includes: obtaining mixed speech sample data; the mixed speech sample data is a single-channel speech signal; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, a reverberation signal and a target speech signal; constructing an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; inputting the mixed speech sample data into the initial speech extraction model to obtain a predicted target speech; optimizing the initial speech extraction model based on the predicted target speech and a preset loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal from the single-channel speech signal. The above method accurately and effectively extracts speech from single-channel speech, meeting the relevant requirements of subsequent speech processing.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of speech signal processing technology, and in particular to a model training method, device and equipment based on speech extraction. Background Art

[0002] With the development of technologies such as computers and artificial intelligence, automatic speech recognition based on smart devices has also been widely used. In actual applications, while smart devices collect the speech of the target object, they often also collect interference signals such as the speech of other objects and noise in the environment. Therefore, before performing speech recognition, the speech signal corresponding to the target object must first be extracted from the acquired speech signal.

[0003] At present, when processing multi-channel speech signals, speech extraction can be performed by comparing speech signals of different channels. However, when processing single-channel speech signals, it is more difficult to directly extract the corresponding sound source from a noisy and reverberant environment. The current mainstream method of processing multi-channel speech signals does not have a good processing effect if it is directly applied to processing single-channel speech signals. In the absence of a highly accurate and applicable technology for extracting single-channel speech signals, it is impossible to effectively deal with such problems in practical applications, and thus it is impossible to guarantee the user experience of specific users. Therefore, there is an urgent need for a technical solution that can accurately and effectively extract single-channel speech. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a model training method, device and equipment based on speech extraction to solve the problem of how to accurately and effectively extract single-channel speech.

[0005] To solve the above technical problems, an embodiment of the present specification provides a model training method based on speech extraction, comprising: obtaining mixed speech sample data; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, a reverberation signal and a target speech signal; constructing an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; inputting the mixed speech sample data into the initial speech extraction model to obtain a predicted target speech; optimizing the initial speech extraction model based on the predicted target speech and a preset loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in a single-channel speech signal.

[0006] The embodiment of this specification also proposes a model training device based on speech extraction, including: a mixed speech sample data acquisition module, used to acquire mixed speech sample data; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, a reverberation signal and a target speech signal; an initial speech extraction model construction module, used to construct an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; a mixed speech sample data input module, used to input the mixed speech sample data into the initial speech extraction model to obtain a predicted target speech; an initial speech extraction model optimization module, used to optimize the initial speech extraction model based on the predicted target speech and a loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in a single-channel speech signal.

[0007] The embodiment of this specification also proposes a model training device based on speech extraction, including a memory and a processor; the memory is used to store computer program instructions; the processor is used to execute the computer program instructions to implement the following steps: obtaining mixed speech sample data; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, a reverberation signal and a target speech signal; constructing an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; inputting the mixed speech sample data into the initial speech extraction model to obtain a predicted target speech; optimizing the initial speech extraction model based on the predicted target speech and a loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in a single-channel speech signal.

[0008] It can be seen from the technical solutions provided in the above embodiments of this specification that the embodiments of this specification pre-build a corresponding initial speech extraction model based on a deep neural network architecture, and then input the mixed speech sample data into the initial speech extraction model, and optimize the model using a loss function in combination with the obtained prediction results, thereby obtaining a final speech extraction model, so that the target speech signal in the single-channel speech signal can be extracted using the speech extraction model. The above method ensures accurate and effective extraction of single-channel speech signals, and then the extracted speech can be used for speech recognition and other purposes in subsequent processes, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 This is a flow chart of a model training method based on speech extraction according to an embodiment of this specification;

[0011] Figure 2 This is a schematic diagram of the structure of an initial speech extraction model according to an embodiment of this specification;

[0012] Figure 3 This is a schematic diagram of the structure of an extractor according to an embodiment of this specification;

[0013] Figure 4 This is a schematic diagram of the structure of a multi-ratio separator according to an embodiment of this specification;

[0014] Figure 5 This is a schematic diagram of the structure of a multi-scale convolution unit according to an embodiment of this specification;

[0015] Figure 6 This is a schematic diagram of the result of a reconstructor according to an embodiment of this specification;

[0016] Figure 7 This is a module diagram of a model training device based on speech extraction according to an embodiment of this specification;

[0017] Figure 8 This is a structural diagram of a model training device based on speech extraction according to an embodiment of this specification. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0019] In order to solve the above technical problems, a model training method based on speech extraction is introduced in the embodiment of this specification. The execution subject of the model training method based on speech extraction is a model training device based on speech extraction, and the model training device based on speech extraction includes but is not limited to a server, an industrial computer, a PC, etc. Figure 1 As shown, the model training method based on speech extraction may include the following specific implementation steps.

[0020] S110: Acquire mixed speech sample data; the mixed speech sample data is a single-channel speech signal; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, and a reverberation signal and a target speech signal.

[0021] In this embodiment, in order to achieve the purpose of extracting the speech of one or more objects from the single-channel speech, it is necessary to build and train a corresponding model, and finally the model can be used to achieve the purpose of speech extraction.

[0022] The mixed speech sample data is the data used when training the model. In order to achieve the purpose of separating the speech of the target object from the speech signal, the mixed speech sample data may include at least one of a noise signal, an interfering speech signal, and a reverberation signal and a target speech signal.

[0023] The noise signal may be a signal that interferes with the original voice during the voice collection process due to the incomplete voice signal collected by the microphone, the loss during the signal transmission process, etc. The interfering voice signal may be the voice produced by other objects other than the target object. For example, the voice collection area contains multiple objects that make sounds, and only one of them is the target object required for voice extraction this time. Then the collected voice signals corresponding to other objects are the interfering voice signals. The reverberation signal may be the signal received by the sound collection device after the sound emitted by the target object itself is reflected by surrounding obstacles, barriers and other objects. Since these sounds have a certain delay when collected compared to the sounds emitted directly by the target object, they also cause certain interference to the voice extraction.

[0024] The target speech signal may be a signal corresponding to the speech generated by the target object. The number of the target objects may be one or more. In order to perform speech recognition and other processing on the target speech signal in a subsequent process, the target speech signal needs to be separated from the mixed speech sample data.

[0025] Specifically, in order to meet the purpose of performing speech extraction on single-channel speech in the embodiment of this specification, the mixed speech sample data may also be a single-channel speech signal. The single-channel speech signal may be a sound signal collected by only one microphone.

[0026] Specifically, when the training model is implemented based on supervised learning, the mixed speech sample data may also correspond to corresponding tags for identifying the target speech signal therein. The specific identification method may be set based on the needs of the actual application and is not limited thereto.

[0027] In some embodiments, the mixed voice sample data can be prepared in the following manner: first, at least two human voice signals are mixed within a first signal-to-noise ratio range to obtain a human voice mixed voice signal, the human voice signal can be an independent voice signal corresponding to the human voice obtained by pre-collection or separation, and the first signal-to-noise ratio range is used to limit the signal-to-noise ratio interval of the mixed human voice signal, for example, it can be between 0dB and 5dB. Secondly, the human voice mixed voice signal and the noise signal are mixed within a second signal-to-noise ratio range to obtain a comprehensive voice signal, the noise signal can be an additional signal that interferes with the above voice signal, and the second signal-to-noise ratio range is used to limit the signal-to-noise ratio interval of the mixed two signals, for example, it can be between -6dB and 3dB. Finally, the comprehensive voice signal is processed by a voice signal generation function to obtain mixed voice sample data, and the voice signal generation function can generate a corresponding voice signal based on the corresponding data to achieve the effect of simulating the voice of actual application, specifically, it can be a pyroomacoustics function, and the pyroomacoustics function can quickly construct a simulation scene of a single / multiple sound source and a microphone in a 2D / 3D room, thereby helping to construct simulated voice sample data.

[0028] The above process is described in detail using a specific example. When preparing the mixed speech sample data, first, the time domain signals of the WSJ0 speech signal sample and the WHAM noise sample are resampled at 8kHz, and the two different speaker voices are arbitrarily mixed within a signal-to-noise ratio range of 0dB to 5dB, and then the mixed speech is mixed with the randomly selected noise sample within a signal-to-noise ratio range of -6dB to 3dB. The final mixed speech is based on the room configuration parameters in Table 1 and the room impulse response is obtained based on the pyroomacoustics function, that is, the final mixed speech sample data y containing noise, reverberation and other speaker interference is obtained.

[0029] Table 1

[0030]

[0031] Based on the above implementation, after a certain amount of mixed speech sample data is prepared, the mixed speech sample data can also be divided. Specifically, the mixed speech sample data can be divided into training sample data, verification sample data and test sample data. The training sample data is used to train the model in subsequent steps; the test sample data and verification sample data can be used to test and verify the model after the model training is completed to ensure the effect of the model.

[0032] Using a specific example to illustrate, assuming that the total number of sample data generated based on the above steps is 28,000, 20,000 of them can be divided into training sample data, 3,000 into test sample data, and 5,000 into verification sample data, which are then used for subsequent model training and model verification processes respectively. In actual applications, other proportions can be set according to the total number of samples and application requirements, and are not limited to the above examples.

[0033] S120: constructing an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network.

[0034] The initial speech extraction model is a model constructed for speech extraction for single-channel speech. The initial speech extraction model has a fixed structure, but the various parameters involved in the model are only parameters obtained based on general initialization processing and are not suitable for the requirements of model parameters in actual applications.

[0035] The initial speech extraction model includes a model constructed based on a deep neural network, such as Figure 2 As shown, the initial speech extraction model includes an extractor, a multi-ratio separator and a reconstructor. The extractor is mainly used to extract features from the data, the multi-ratio separator is the core structure of the model, and is used to separate the data corresponding to each speech signal, and the reconstructor is used to reconstruct the speech signal based on the separated data.

[0036] Specifically, the structure of each part in the initial speech extraction model is further introduced. Figure 3 As shown, the extractor mainly includes a layer of convolutional network. When the mixed speech sample data is input into the extractor and calculated based on the convolutional network, the output corresponding to the extractor can be obtained based on layer normalization processing, and the output can be used as the input of the multi-ratio separator.

[0037] The multi-scale separator may include a normalization layer, a convolutional network, five repeated calculation units and a convolutional network in sequence. The repeated calculation unit is mainly used to perform corresponding calculations. The repeated calculation unit includes three multi-scale convolutional units and a graph convolutional network. The multi-scale convolutional unit may specifically include a convolutional unit, a batch normalization network, a depth separation convolutional network, a batch normalization network, a multi-scale connection network and a convolutional network.

[0038] like Figure 4As shown, it is a schematic diagram of the structure of the multi-ratio separator. After the input data is processed based on the layer normalization network and the convolution network, it is calculated based on multiple repeated calculation units in sequence, wherein the results calculated by each multi-ratio convolution unit are combined for convolution operation, and the obtained result is convolved together with the result calculated by the last repeated calculation unit, and then input into the convolution network to obtain the output result corresponding to the multi-ratio separator. Based on the example in the figure, it can be seen that the number of the repeated calculation unit and the multi-ratio convolution unit in each repeated calculation unit can be set based on the needs of the actual application, and there is no restriction on this.

[0039] like Figure 5 As shown in the figure, it is a structural diagram of the multi-scale convolution unit in the repeated calculation unit. After the data is input into the multi-scale convolution unit, the input is sequentially processed using the convolution network, PReLU, global normalization network, deep separation convolution network, PReLU, and global normalization network. The outputs of the two global normalization networks are input into the multi-scale connection network together, and then processed based on the convolution network. The convolution calculation is performed on the original input data to obtain the final output result.

[0040] Among them, PReLU (Parametric Rectified Linear Unit) is a modified linear unit with parameters, which is an activation function set in a neural network. The activation function can be set between network neuron nodes, mainly used to enable the neural network to fit various curves. In this application, the nonlinear relationship between networks is increased by activation functions, and finally a nonlinear mapping between input and output results can be generated. It should be noted that other types of activation functions can also be set in neural networks in actual applications, and are not limited to the above examples, and there is no restriction on this.

[0041] like Figure 6 The figure is a schematic diagram of the structure of the reconstructor. The reconstructor mainly comprises a layer of convolutional network, after receiving the data output by the multi-ratio separator, the data is input into the convolutional network, and the data can be reconstructed to obtain the predicted target speech.

[0042] In some embodiments, after constructing the initial speech extraction model, the model parameters in the initial speech extraction model may be initialized. The model parameters may mainly include weight values ​​and bias values ​​between network neuron nodes, and the model parameters constitute a specific way for the model to process data, specifically in the embodiment of this specification, that is, to extract the target speech from the single-channel speech.

[0043] S130: Inputting the mixed speech sample data into the initial speech extraction model to obtain the predicted target speech.

[0044] After the mixed speech sample data is input into the initial speech extraction model, the data can be processed accordingly based on the structure of each part in the initial speech extraction model and the data flow relationship between different structures to obtain the final predicted target speech.

[0045] Next, the specific process of outputting the predicted target speech is introduced in combination with the structural introduction of the initial speech extraction model in step S120. First, the extractor part of the initial speech extraction model inputs the mixed speech sample data y into the network input end, and extracts higher-level features of the signal through a layer of convolutional network to obtain the primary speech feature H = {h0, ..., h M-1}, where M is the output length corresponding to the last layer of the extractor network.

[0046] For the multi-ratio separator, the primary speech feature H obtained in the previous step is input into the input end of the multi-ratio separator. After normalization and a layer of convolutional neural network calculation, a more advanced first speech feature G1 is obtained. The first speech feature G1 is input into the repeated calculation unit and combined with Figure 4 In the structure of the repeated calculation unit, the first speech feature G1 is first input into the first multi-scale convolution calculation unit, and the second speech feature G2 is calculated through a layer of convolution network and a layer of batch normalization network, and then the third speech feature G3 is calculated using a deep separation convolution network and a layer of batch normalization network. Then, the second speech feature G2 and the third speech feature G3, two high-level features of different levels, are fused as a whole using a multi-scale connection network, and input into the next layer of convolution network for feature calculation to obtain the fourth speech feature G4, and the fourth speech feature G4 is combined with the primary speech feature H to obtain the first unit output G of each multi-scale convolution unit. The first unit output G is input into the next multi-scale convolution unit for feature calculation. As shown in the figure, after calculation by three multi-scale convolution units, the extracted high-level features are input into the last network layer of the repeated calculation unit, namely the graph convolution calculation network. After further global feature extraction through the graph convolution calculation network, the second unit output C of the entire repeated calculation unit is obtained. Based on the number of repeated calculation units, after calculating 5 times according to the above steps, a residual path is established for each result. After obtaining the interim result D, it is input into the subsequent convolutional network to obtain the output of the multi-scale separator, which is the high-dimensional extraction feature map P corresponding to the target speech.

[0047] For the reconstructor part, the high-dimensional extracted feature map P obtained in the previous step is input to the input end of the module. After passing through a layer of convolutional neural network, the predicted target speech is obtained, which is the estimated speech corresponding to the target speaker.

[0048] The above process of obtaining the predicted target speech is a description based on the model structure in the embodiment in step S120. In actual application, the calculation process can be adaptively adjusted according to the specific structure of the model, which will not be repeated here.

[0049] S140: Optimizing the initial speech extraction model based on the predicted target speech and a preset loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in the single-channel speech signal.

[0050] After the predicted target speech is obtained, the model can be optimized using a preset loss function according to the predicted target speech. The preset loss function can be a pre-set loss function corresponding to the initial speech extraction model, which is used to evaluate the loss of the model according to the prediction result, and then modify the model in combination with the calculation result to make it have a more accurate prediction result.

[0051] Specifically, the preset loss function can be In the formula, L is the preset loss function, It is used to represent the effective signal in the speech signal, where s is the ideal target speech. Specifically, it can be reflected by pre-marking in the mixed speech sample data. To predict the target speech, It is used to represent the noise signal in the speech signal, <·,·> represents the dot product between two vectors, and ‖·‖ 2 represents the Euclidean distance, where SNR is the signal-to-noise ratio and MSE is the mean square error.

[0052] In some embodiments, the model can be optimized using a gradient descent method in combination with the preset loss function. The optimization process can be to first calculate a first gradient of the loss function corresponding to the output layer of the initial speech extraction model, then based on the first gradient, sequentially calculate the gradients corresponding to each layer in the initial speech extraction model, and finally update the weights and biases of the initial speech extraction model in combination with the gradients of each layer.

[0053] Specifically, the method of updating the parameters of the multi-scale extraction deep neural network using the gradient descent method can be to fix the parameters of the deep neural network for a certain period of time, and use the above formula to calculate the gradient of the output layer loss function, and then use the initial network level as the Lth layer, and then calculate the gradient corresponding to each layer when the number of network layers is L-1, L-2, ..., 2, where L is the number of layers of the neural network. After all the gradients are calculated, the weights and biases of the entire network are updated according to the calculated gradients, thereby completing the optimization of the model.

[0054] Accordingly, since the differences of the initial speech extraction model are mainly reflected by the model parameters in the model, the optimization process for the initial speech extraction model can mainly optimize the model parameters. The specific optimization process can be adjusted based on the needs of the actual application and will not be repeated here.

[0055] In some implementations, after the speech extraction model is trained, the model may be tested and verified to ensure the training effect of the model. Specifically, based on the implementation in step S110, after the mixed speech sample data is obtained, test sample data and verification sample data may be obtained therefrom.

[0056] The trained speech extraction model is used to extract the test target speech signal in the test sample data, and then the extracted test target speech signal is compared with the verification sample data, and the speech extraction model is optimized according to the comparison result. By analyzing the consistency between the predicted result and the original result, the prediction accuracy of the model can be effectively judged, so as to determine whether the model can be directly applied or trained again, thereby effectively ensuring the training effect of the model.

[0057] After obtaining the speech extraction model, the speech of the target object in the single-channel speech can be accurately and effectively extracted, thereby effectively ensuring the subsequent application process. The specific process of extracting speech can be set based on the needs of actual applications, and will not be repeated here.

[0058] Based on the introduction of the above embodiments, it can be seen that the above method pre-builds a corresponding initial speech extraction model based on a deep neural network architecture, and then optimizes the model by inputting mixed speech sample data into the initial speech extraction model, combining the obtained prediction results, and using a loss function to obtain a final speech extraction model, so that the target speech signal in the single-channel speech signal can be extracted using the speech extraction model. The above method ensures accurate and effective extraction of single-channel speech signals, and can then use the extracted speech for speech recognition and other purposes in subsequent processes, thereby improving the user experience.

[0059] based on Figure 1 The corresponding model training method based on speech extraction introduces a model training device based on speech extraction in the embodiment of this specification. The model training device based on speech extraction is set in a model training device based on speech extraction. Figure 7 As shown, the model training device based on speech extraction includes the following modules.

[0060] The mixed speech sample data acquisition module 710 is used to acquire mixed speech sample data; the mixed speech sample data is a single-channel speech signal; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, a reverberation signal and a target speech signal.

[0061] The initial speech extraction model construction module 720 is used to construct an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network.

[0062] The mixed speech sample data input module 730 is used to input the mixed speech sample data into the initial speech extraction model to obtain the predicted target speech.

[0063] The initial speech extraction model optimization module 740 is used to optimize the initial speech extraction model based on the predicted target speech and the loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in the single-channel speech signal.

[0064] based on Figure 1 Corresponding to the model training method based on speech extraction, the embodiment of this specification provides a model training device based on speech extraction. Figure 8 As shown, the model training device based on speech extraction may include a memory and a processor.

[0065] In this embodiment, the memory may be implemented in any appropriate manner. For example, the memory may be a read-only memory, a mechanical hard disk, a solid-state hard disk, or a USB flash drive, etc. The memory may be used to store computer program instructions.

[0066] In this embodiment, the processor can be implemented in any appropriate manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. The processor can execute the computer program instructions to implement the following steps: obtain mixed speech sample data; the mixed speech sample data is a single-channel speech signal; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, a reverberation signal, and a target speech signal; construct an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; the mixed speech sample data is input into the initial speech extraction model to obtain a predicted target speech; the initial speech extraction model is optimized based on the predicted target speech and the loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in the single-channel speech signal.

[0067] Although the process flows described above include multiple operations that occur in a particular order, it should be clearly understood that the processes may include more or fewer operations, which may be performed sequentially or in parallel (eg, using parallel processors or a multi-threaded environment).

[0068] Although the process flows described above include multiple operations that occur in a particular order, it should be clearly understood that the processes may include more or fewer operations, which may be performed sequentially or in parallel (eg, using parallel processors or a multi-threaded environment).

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0073] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0074] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0075] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0076] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0077] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0078] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A model training method based on speech extraction, characterized in that: include: Acquire mixed speech sample data; the mixed speech sample data is a single-channel speech signal; the mixed speech sample data includes at least one of a noise signal, an interfering speech signal, and a reverberation signal and a target speech signal; Constructing an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; the initial speech extraction model includes an extractor, a multi-scale separator and a reconstructor; wherein the initial speech extraction model includes an extractor, a multi-scale separator and a reconstructor; the extractor includes a layer of convolutional network; the multi-scale separator includes a normalization layer, a convolutional network, five repeated calculation units and a convolutional network in sequence; the reconstructor includes a convolutional network; wherein the repeated calculation unit includes three multi-scale convolutional units and a graph convolutional network; the multi-scale convolutional unit includes a convolutional unit, a batch normalization network, a depth separation convolutional network, a batch normalization network, a multi-scale connection network and a convolutional network in sequence; the repeated calculation unit has a residual path; Inputting the mixed speech sample data into the initial speech extraction model to obtain a predicted target speech; The initial speech extraction model is optimized based on the predicted target speech and the preset loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in the single-channel speech signal; The step of inputting the mixed speech sample data into the initial speech extraction model to obtain the predicted target speech comprises: Using a convolutional network in an extractor to extract features from the mixed speech sample data to obtain primary speech features; Inputting the primary speech feature into a multi-scale separator to obtain a first speech feature by normalizing the feature through a normalization layer and a convolutional network and calculating the feature through a convolutional network; Calculating the first speech feature based on the convolution unit in the multi-scale convolution unit and the batch normalization network to obtain a second speech feature; The second speech feature is calculated using a deep separation convolutional network and a batch normalization network in a multi-scale convolutional unit to obtain a third speech feature; After fusing the second speech feature and the third speech feature through a multi-ratio connection network, the resultant is input into a convolutional network for feature calculation to obtain a fourth speech feature; combining the fourth speech feature with the primary speech feature to obtain a first unit output corresponding to the multi-scale convolutional unit; Based on the output of the first unit, all multi-scale convolution units in the repeated calculation unit are used to calculate and obtain high-level speech features; Inputting high-level speech features into a graph convolutional computing network for global feature extraction to obtain a second unit output corresponding to a repeated computing unit; Based on the output of the second unit, all repeated calculation units in the multi-proportional separator are used to calculate to obtain a stage calculation result; Input the calculation results of all stages into the convolutional network to obtain a high-dimensional extracted feature map; The high-dimensional extracted feature map is calculated using a convolutional neural network in a reconstructor to obtain a predicted target speech.

2. The method according to claim 1, characterized in that The mixed speech sample data is obtained in the following manner: Mixing at least two human voice signals within a first signal-to-noise ratio range to obtain a human voice mixed voice signal; Mixing the human voice mixed speech signal with the noise signal within a second signal-to-noise ratio range to obtain a comprehensive speech signal; The integrated speech signal is processed by using a speech signal generation function to obtain mixed speech sample data.

3. The method according to claim 1, characterized in that The mixed speech sample data includes training sample data, verification sample data and test sample data; the step of inputting the mixed speech sample data into the initial speech extraction model to obtain the predicted target speech includes: Inputting the training sample data into the initial speech extraction model to obtain a predicted target speech; Correspondingly, after optimizing the initial speech extraction model based on the predicted target speech and the loss function to obtain the speech extraction model, the method further includes: Extracting a test target speech signal from the test sample data using the speech extraction model; The speech extraction model is optimized according to the comparison result of the test target speech signal and the verification sample data.

4. The method according to claim 1, characterized in that Before inputting the mixed speech sample data into the initial speech extraction model to obtain the predicted target speech, the method further comprises: Initializing the model parameters in the initial speech extraction model; including: initializing the weights and biases between the neuron nodes in the initial speech extraction model; setting an activation function between the network neuron nodes in the initial speech extraction model; the activation function is used to generate a nonlinear mapping between the input and output corresponding to the network neuron node; Accordingly, the initial speech extraction model is optimized based on the predicted target speech and the loss function to obtain a speech extraction model, including: The model parameters are optimized based on the predicted target speech and the loss function.

5. The method according to claim 1, characterized in that The preset loss function is: In the formula, L is the preset loss function, Where s is the ideal target speech, To predict the target speech, 6. The method according to claim 1, characterized in that The optimizing the initial speech extraction model based on the predicted target speech and the preset loss function includes: Calculating a first gradient of a loss function corresponding to an output layer of the initial speech extraction model; Based on the first gradient, sequentially calculating the gradient corresponding to each layer in the initial speech extraction model; The gradients of each layer are combined to update the weights and biases of the initial speech extraction model.

7. A model training device based on speech extraction, characterized in that: include: A mixed voice sample data acquisition module is used to acquire mixed voice sample data; the mixed voice sample data is a single-channel voice signal; the mixed voice sample data includes at least one of a noise signal, an interfering voice signal, a reverberation signal and a target voice signal; An initial speech extraction model construction module is used to construct an initial speech extraction model; the initial speech extraction model includes a model constructed based on a deep neural network; wherein the initial speech extraction model includes an extractor, a multi-scale separator and a reconstructor; the extractor includes a layer of convolutional network; the multi-scale separator includes a normalization layer, a convolutional network, five repeated calculation units and a convolutional network in sequence; the reconstructor includes a convolutional network; wherein the repeated calculation unit includes three multi-scale convolutional units and a graph convolutional network; the multi-scale convolutional unit includes a convolutional unit, a batch normalization network, a depth separation convolutional network, a batch normalization network, a multi-scale connection network and a convolutional network in sequence; the repeated calculation unit has a residual path; A mixed speech sample data input module, used for inputting the mixed speech sample data into the initial speech extraction model to obtain a predicted target speech; An initial speech extraction model optimization module is used to optimize the initial speech extraction model based on the predicted target speech and the loss function to obtain a speech extraction model; the speech extraction model is used to extract the target speech signal in the single-channel speech signal; The step of inputting the mixed speech sample data into the initial speech extraction model to obtain the predicted target speech comprises: Using a convolutional network in an extractor to extract features from the mixed speech sample data to obtain primary speech features; Inputting the primary speech feature into a multi-scale separator to obtain a first speech feature by normalizing the feature through a normalization layer and a convolutional network and calculating the feature through a convolutional network; Calculating the first speech feature based on the convolution unit in the multi-scale convolution unit and the batch normalization network to obtain a second speech feature; The second speech feature is calculated using a deep separation convolutional network and a batch normalization network in a multi-scale convolutional unit to obtain a third speech feature; After fusing the second speech feature and the third speech feature through a multi-ratio connection network, the resultant is input into a convolutional network for feature calculation to obtain a fourth speech feature; combining the fourth speech feature with the primary speech feature to obtain a first unit output corresponding to the multi-scale convolutional unit; Based on the output of the first unit, all multi-scale convolution units in the repeated calculation unit are used to calculate and obtain high-level speech features; Inputting high-level speech features into a graph convolutional computing network for global feature extraction to obtain a second unit output corresponding to a repeated computing unit; Based on the output of the second unit, all repeated calculation units in the multi-proportional separator are used to calculate to obtain a stage calculation result; Input the calculation results of all stages into the convolutional network to obtain a high-dimensional extracted feature map; The high-dimensional extracted feature map is calculated using a convolutional neural network in a reconstructor to obtain a predicted target speech.

8. A model training device based on speech extraction, comprising a memory for storing processor-executable instructions and a processor, wherein the processor implements the steps of the method described in any one of claims 1 to 6 when executing the instructions.

Images