Speech separation model training method and system based on skip memory network

Abstract

The embodiment of the present invention provides a speech separation model training method and system based on a skip memory network. The method includes: inputting multiple audio features determined by the training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain the historical memory states of the multiple audio features; inputting the historical memory states of the multiple audio features into the long and short-term memory elements for memory processing in the skip memory network to output the predicted future memory states of the multiple audio features; using the actual future memory states of the pre-prepared multiple audio features as training targets, and performing comparative prediction learning training on the speech separation model based on the predicted future memory states and the training targets, so that the trained speech separation model learns causal sequence modeling capabilities. The embodiment of the present invention trains the causal modeling capabilities of the model, thereby improving the speech separation performance without increasing the delay, while reducing the processing delay.

Description

Technical Field

[0001] The present invention relates to the field of intelligent speech, and in particular to a speech separation model training method and system based on a skip memory network. Background Art

[0002] Speech separation technology can be used as the front-end of speech processing systems in complex acoustic environments to isolate the speech of individual speakers from a mixed stream of multiple speakers. However, many conversational scenarios require low latency in speech processing systems, such as remote conferencing, simultaneous interpretation, and hearing aids. The latency of the speech separation front-end determines the lower limit of the overall system latency. Therefore, low-latency speech separation systems are of great importance.

[0003] The SkiM model (Skipping Memory) is typically used for speech separation. The SkiM model is a neural network model used for online speech separation. The actual delay of the speech separation system described above comes from two factors. The first is the ideal delay, or algorithmic delay. During feature encoding, a small window can be used to compress the system's ideal delay to a very small extent. Another factor contributing to the actual delay is processing delay, which is related to the model's computational complexity and hardware speed. However, when a small window is used to compress the system's ideal delay to a very small extent, more feature frames need to be processed, increasing the model's total computational complexity.

[0004] In the process of implementing the present invention, the inventors discovered that there are at least the following problems in the related art:

[0005] To achieve extremely low theoretical latency, the SkiM model uses a relatively fine-grained feature modeling granularity. This fine-grained nature significantly increases the number of features the SkiM model must process per second. This places high demands on computing performance and introduces a certain amount of processing latency. To further improve speech separation, more "future" information is introduced during speech separation (for example, speech from a few seconds after acquisition is used to assist in the separation of the current speech segment). This introduces further latency. Latency and speech separation system performance are often a trade-off. Summary of the Invention

[0006] In order to at least solve the problem of the trade-off between latency and performance in the speech separation system in the prior art, in a first aspect, an embodiment of the present invention provides a speech separation model training method based on a skip memory network, comprising:

[0007] Inputting multiple audio features determined by the training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain historical memory states of the multiple audio features;

[0008] Inputting the historical memory states of the multiple audio features into the long-short-term memory elements for memory processing in the skip memory network, and outputting the predicted future memory states of the multiple audio features;

[0009] The actual future memory state of the pre-prepared multiple audio features is used as a training target, and the speech separation model is subjected to comparative prediction learning training based on the predicted future memory state and the training target, so that the trained speech separation model can learn causal sequence modeling capabilities.

[0010] In a second aspect, an embodiment of the present invention provides a speech separation model training system based on a skip memory network, comprising:

[0011] A historical memory determination program module is used to input multiple audio features determined by training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain the historical memory status of the multiple audio features;

[0012] A prediction program module, configured to input the historical memory states of the plurality of audio features into the long and short-term memory elements for memory processing in the skip memory network, and output the predicted future memory states of the plurality of audio features;

[0013] A training program module is used to take the actual future memory state of the pre-prepared multiple audio features as a training target, and perform comparative predictive learning training on the speech separation model based on the predicted future memory state and the training target, so that the trained speech separation model can learn causal sequence modeling capabilities.

[0014] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the speech separation model training method based on a skip memory network according to any embodiment of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the speech separation model training method based on a skip memory network of any embodiment of the present invention are implemented.

[0016] The beneficial effects of the embodiments of the present invention are as follows: The two long and short-term memory elements of the SkiM model used for speech separation are improved separately. The causal modeling capability of the long and short-term memory element used for memory processing by Mem-LSTM is trained, thereby improving speech separation performance without increasing latency. The structure of the long and short-term memory element used for segment processing by Seg-LSTM is improved to compress local features and reduce the processing latency of the SkiM model. The improved SkiM model can be deployed on devices with lower power consumption, increasing the SkiM model's applicability and processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flowchart of a method for training a speech separation model based on a skip memory network according to an embodiment of the present invention;

[0019] Figure 2 1 is a schematic diagram of a skip memory network of a speech separation model training method based on a skip memory network provided by one embodiment of the present invention;

[0020] Figure 3 1 is a comparative predictive coding diagram of a speech separation model training method based on a skip memory network provided by one embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of local feature visualization of a speech separation model training method based on a skip memory network provided by one embodiment of the present invention;

[0022] Figure 5 2 is a schematic diagram showing comparison of results between models of a speech separation model training method based on a skip memory network provided by one embodiment of the present invention;

[0023] Figure 6 1 is a data comparison diagram of a speech separation model training method based on a skip memory network provided by one embodiment of the present invention;

[0024] Figure 7 1 is a schematic structural diagram of a speech separation model training system based on a skip memory network provided by one embodiment of the present invention;

[0025] Figure 8A schematic structural diagram of an embodiment of an electronic device for training a speech separation model based on a skip memory network provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] like Figure 1 FIG. 1 is a flow chart of a method for training a speech separation model based on a skip memory network according to an embodiment of the present invention, comprising the following steps:

[0028] S11: Inputting multiple audio features determined by the training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain the historical memory states of the multiple audio features;

[0029] S12: Inputting the historical memory states of the multiple audio features into the long-short-term memory element for memory processing in the skip memory network, and outputting the predicted future memory states of the multiple audio features;

[0030] S13: Taking the actual future memory state of the pre-prepared multiple audio features as a training target, the speech separation model is subjected to comparative prediction learning training based on the predicted future memory state and the training target, so that the trained speech separation model can learn causal sequence modeling capabilities.

[0031] In this embodiment, the SkiM model for speech separation consists of two LSTMs: a Mem-LSTM (memory long short-term memory) for long-span modeling and a Seg-LSTM (segmentation long short-term memory) for local modeling. To reduce model latency, these two LSTMs are trained separately.

[0032] This method extends the original SkiM model. CPC (contrastive predictive coding) is employed during Mem-LSTM training. The improved Mem-LSTM is used to estimate future memory from historical memory states, giving the trained SkiM improved causal modeling capabilities. The trained SkiM model no longer uses future information during inference (for example, it no longer waits for subsequent speech input when separating the current speech, reducing ideal latency).

[0033] For step S11, under the low-latency online speech separation goal, it is necessary to process long feature sequences in the speech, such as Figure 2 As shown, the audio features W of the input training speech are identified and the features W are divided into smaller segments Where s = 1, ..., s. s represents the number of segments, K represents the segment length, and N represents the feature size. The overall SkiM model structure for speech separation is as follows: Figure 3 As shown, it is input into Seg-LSTM, and the mapping function of the first block Seg-LSTM is:

[0034]

[0035] in, is the input feature of the sth segment in the lth block, Represents the initial hidden and unit memory states of the LSTM layer in Seg-LSTM. is 0, The memory state of the local information of the output encoding segment s, and then all the memory states of Mem-LSTM are used for large-span modeling:

[0036]

[0037]

[0038] in, is the globally synchronized memory state, which will be used as the initial state of the Seg-LSTM of the next SkiM block. After the last SkiM block, the final output segment are merged to obtain a continuous output memory state of length t, which is determined as the historical memory state of multiple audio features.

[0039] In step S12, the historical memory state of multiple audio features is input into the Mem-LSTM in the SkiM model. By alternating between Seg-LSTMs and Mem-LSTM, the trained SkiM model can model the historical information of a very long sequence in the causal speech separation model. Figure 3 As shown, the input of Mem-LSTM is the historical memory state determined from each Seg-LSTMs Mem-LSTM outputs long-span predicted future memory states:

[0040]

[0041] Among them, long-span prediction of future memory state Can be seen as encoding the context of historical input In the prediction of SkiM, we hope that Mem-LSTM has Predictive features ability, that is Where d is the number of future segments.

[0042] For step S13, the present method uses the actual future memory state of the pre-prepared multiple audio features as the training target. Positively correlated function

[0043]

[0044] By maximizing Can be maximized and The mutual information between Refactoring Easier. Using the logarithmic bilinear model f d :

[0045]

[0046] in, is a parameter of a linear transformation, from estimate In order to make Maximize and use contrastive prediction learning loss to train the speech separation model:

[0047]

[0048] Where W={W1,…,W M} is included and a set of M-1 random negative samples. The final training objective of the SkiM model trained by this method can be written as:

[0049]

[0050] Among them, D is the maximum number of rounds predicted, L PIT It is an invariant source-to-noise ratio (SI-SNR) speech separation loss based on the permutation invariant training (PIT) method. λ is a weighting factor. The trained speech separation model learns causal sequence modeling capabilities and can directly infer future memories from input speech features, thereby ensuring that the speech separation efficiency is improved without increasing the latency of the speech separation model.

[0051] As an embodiment, the structure of the long short-term memory element for segment processing includes: an encoder network, an intermediate network composed of continuous long short-term memory elements, and a decoder network.

[0052] Performing feature compression on the multiple audio features based on an encoder network to obtain multiple audio features at a first frame rate;

[0053] Each of the multiple audio features is input into the continuous long and short-term memory element in sequence, and the continuous long and short-term memory element is used to perform local context encoding and decoding on the audio features of the corresponding segment to obtain the historical memory state of each audio feature segment.

[0054] The historical memory state of multiple audio features of the first frame rate is decoded by the decoder network to obtain the historical memory state of multiple audio features of the second frame rate, wherein the second frame rate is higher than the first frame rate to reduce the computational complexity of the intermediate network.

[0055] In this embodiment, it is considered that the basic layer of the existing speech separation model is divided into SIMO (Single Input and Multiple Outputs) and SISO single input single output (Single Input and Single Outputs) modules. The SIMO module processes the deep features of multi-source mixed speech. It or its subsequent SIMO module separates the deep features into multiple streams, which correspond to the speech of a single speaker. Most traditional blind source separation systems are pure SIMO designs, in which the deep features are separated in the last layer of the separator. The SIMO-SISO model is a pipeline before separation and after enhancement. The deep features are separated in the early stage using the SIMO layer of the encoder network and enhanced in the later stage using the SISO layer of the decoder network. Studies have shown that this encoding-decoding (SIMO-SISO) design can improve separation performance with the same number of parameters. The model of the SIMO-SISO structure of this method can also be called a post-enhancement (PE) model.

[0056] This method trains a SIMO-only SkiM model from the prior art and a PE SkiM model with the encoder network, an intermediate network consisting of continuous long and short-term memory elements, and a decoder network structure described in this method. The mixed speech is then forward propagated through each model. Figure 4 Local segment features are given Visualization results in different intermediate SkiM blocks. It can be observed that the features of the mixed audio SIMO module (in SIMO-only and PE models) do not show a unique pattern. In the PE SkiM model, periodicity can be observed in the features separated from the SISO modules of the two speaker branches. The periodicity may come from the short-term fixation of the small segments of separated speech. The local SISO features have good periodicity, which means that the features have strong redundancy and compressibility. Based on this finding, this method continues to integrate a local context codec (LCC) in the long short-term memory element for segment processing in the trained SkiM model to reduce the computational cost in real-time applications. Before the first SISO, an encoder is inserted to convert Map to in, After the last SISO block, the decoder map length changes from to K. Therefore, the computational cost of the intermediate blocks can be reduced.

[0057] This implementation demonstrates that this method improves both the long and short-term memory elements of the SkiM model used for speech separation. The Mem-LSTM long and short-term memory element, used for memory processing, is trained with causal modeling capabilities, improving speech separation performance without increasing latency. The Seg-LSTM long and short-term memory element used for segment processing is also improved, compressing local features and reducing the processing latency of the SkiM model. This improved SkiM model can be deployed on lower-power devices, increasing its applicability and processing performance.

[0058] This method is experimentally demonstrated on the WSJ0-2mix dataset, a widely used benchmark for speech separation. The baseline model is a causal SkiM model consisting of six SkiM blocks, all of which are SIMO modules. Both the Seg-LSTM and Mem-LSTM are unidirectional LSTMs with 256 hidden units. The convolutional encoder and transposed convolutional decoder have 128 channels.

[0059] We compare the SkiM models trained by our proposed method on four different settings. For ease of illustration, we denote them as pSkiM and existing SkiM models. Their stride sizes are {4, 8, 16, 32}, convolution kernel sizes are {8, 16, 32, 64}, and the corresponding segmentation sizes K in the SkiM blocks are {64, 48, 32, 24}.

[0060] In pSkiM, the estimated maximum number of strides is 10 and λ is 1.0. The batch size is set to 32 during training, and the negative samples for CPC loss are selected from samples in other mini-batches.

[0061] In the local context codec (LCC) experiments, in PE SkiM, the first two SkiM blocks are used as SIMO layers, while the remaining four are used as SISO layers. The number of parameters in PE SkiM is comparable to the SIMO-only baseline. Both the context encoder and decoder are single-layer unidirectional LSTMs with 128 units. The encoder-decoder stride is set to 4. The encoder outputs an encoded feature every 4 steps, while the decoder outputs autoregressively for 4 steps for each input. To avoid increasing the algorithm latency, the current input to the decoder is the feature of the last 4 steps, so there is no need to wait for the encoder 4 steps to decode the current step.

[0062] All models are implemented using the ESPNet-SE toolkit. Adam optimizer is used for training. The initial learning rate (LR) is set to 10 -3 The model was trained for 150 epochs, and LR decreased by 0.98 per cycle.

[0063] First, we examine the effect of CPC training on Mem-LSTM. Figure 5 The SI-SNRi improvements for speech separation under different systems are listed. The number of multiplication-accumulation operations (MACs) per second is also reported. In practical applications, systems with smaller MACs / s will have smaller processing delays. Based on the above experimental results, it is found that the smaller the kernel size of the convolutional encoder, the better the speech separation performance, the smaller the ideal delay, and the larger the MACs / s. Compared with all baseline models, the SkiM model of this method has better separation performance. The improvement is more obvious in the 4ms and 8ms settings, while the performance improvement is the smallest in the 2ms and 16ms settings. The prediction of feature context is only performed in the training phase, so the overall computational cost of the inference phase is the same as the baseline model.

[0064] PE SkiM performance is as follows Figure 6The second row shows that pSkiM trained with CPC is not far behind the baseline model. Significant improvements are observed when our method combines pSkiM training with CPC training. By predicting the memory state of the separated single-speaker streams, we can obtain better contextual dependencies than the mixed stream.

[0065] Three different local context codec (LCC) strategies are compared. The first one applies LCC to the SISO modules of PESkiM. The second one performs LCC on the SIMO modules. The last one applies LCC to the entire network. Figure 6 Results show that applying LCC to the SISO module is superior across all models. It achieves a 1.2dB improvement over the baseline model while requiring less computational cost. Adding LCC to the SIMO module yields a smaller performance improvement. Equipping the entire model with LCC reduces computational cost by 60%, but with a slight performance degradation. This suggests that features in the SISO branch are more suitable for compression using local context coding. The encoding and decoding processes do not damage the information within the deep features while saving computational cost.

[0066] like Figure 7 The figure shows a structural diagram of a speech separation model training system based on a skip memory network provided by an embodiment of the present invention. The system can execute the speech separation model training method based on a skip memory network described in any of the above embodiments and be configured in a terminal.

[0067] The present embodiment provides a speech separation model training system 10 based on a skip memory network, including: a historical memory determination program module 11 , a prediction program module 12 and a training program module 13 .

[0068] Among them, the historical memory determination program module 11 is used to input the multiple audio features determined by the training speech into the long-short-term memory elements used for segment processing in the skip memory network to obtain the historical memory states of the multiple audio features; the prediction program module 12 is used to input the historical memory states of the multiple audio features into the long-short-term memory elements used for memory processing in the skip memory network to output the predicted future memory states of the multiple audio features; the training program module 13 is used to use the actual future memory states of the pre-prepared multiple audio features as training targets, and perform comparative prediction learning training on the speech separation model based on the predicted future memory states and the training targets, so that the trained speech separation model can learn causal sequence modeling capabilities.

[0069] An embodiment of the present invention further provides a non-volatile computer storage medium storing computer-executable instructions, which can execute the speech separation model training method based on the skip memory network in any of the above method embodiments;

[0070] As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0071] Inputting multiple audio features determined by the training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain historical memory states of the multiple audio features;

[0072] Inputting the historical memory states of the multiple audio features into the long-short-term memory elements for memory processing in the skip memory network, and outputting the predicted future memory states of the multiple audio features;

[0073] The actual future memory state of the pre-prepared multiple audio features is used as a training target, and the speech separation model is subjected to comparative prediction learning training based on the predicted future memory state and the training target, so that the trained speech separation model can learn causal sequence modeling capabilities.

[0074] A non-volatile computer-readable storage medium can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods described in the embodiments of the present invention. One or more program instructions stored in the non-volatile computer-readable storage medium, when executed by a processor, perform the skip memory network-based speech separation model training method described in any of the above method embodiments.

[0075] Figure 8 This is a hardware structure diagram of an electronic device for a speech separation model training method based on a skip memory network provided in another embodiment of the present application. Figure 8 As shown, the device includes:

[0076] One or more processors 810 and memory 820, Figure 8 A processor 810 is used as an example. The apparatus for the speech separation model training method based on the skip memory network may further include: an input device 830 and an output device 840.

[0077] The processor 810, the memory 820, the input device 830 and the output device 840 may be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0078] Memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the speech separation model training method based on a skip memory network in the embodiments of the present application. Processor 810 executes the non-volatile software programs, instructions, and modules stored in memory 820 to execute various server functional applications and data processing, thereby implementing the speech separation model training method based on a skip memory network in the above-mentioned method embodiment.

[0079] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data, etc. In addition, the memory 820 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include a memory remotely located relative to the processor 810, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The input device 830 can receive input digital or character information. The output device 840 can include a display device such as a display screen.

[0081] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, the speech separation model training method based on the skip memory network in any of the above method embodiments is executed.

[0082] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0083] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0084] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the speech separation model training method based on the skip memory network of any embodiment of the present invention.

[0085] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0086] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and their primary purpose is to provide voice and data communications. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0087] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers and have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPC devices, such as tablet computers.

[0088] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0089] (4) Other electronic devices with data processing functions.

[0090] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or device. In the absence of further limitations, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0092] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A speech separation model training method based on a skip memory network, comprising: Inputting multiple audio features determined by the training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain historical memory states of the multiple audio features; Inputting the historical memory states of the multiple audio features into the long-short-term memory elements for memory processing in the skip memory network, and outputting the predicted future memory states of the multiple audio features; Using the actual future memory states of the plurality of pre-prepared audio features as training targets, and performing comparative prediction learning training on the speech separation model based on the predicted future memory states and the training targets, so that the trained speech separation model learns causal sequence modeling capabilities; The structure of the long and short-term memory elements for segment processing includes: an encoder network, an intermediate network composed of continuous long and short-term memory elements, and a decoder network; The step of inputting the multiple audio features determined by the training speech into the long-short term memory element for segment processing in the skip memory network to obtain the historical memory state of the multiple audio features comprises: Performing feature compression on the multiple audio features based on an encoder network to obtain multiple audio features at a first frame rate; Inputting the multiple audio features of the first frame rate into the intermediate network composed of the continuous long and short-term memory elements to obtain a historical memory state of the multiple audio features of the first frame rate; The historical memory state of multiple audio features of the first frame rate is decoded by the decoder network to obtain the historical memory state of multiple audio features of the second frame rate, wherein the second frame rate is higher than the first frame rate to reduce the computational complexity of the intermediate network.

2. The method according to claim 1, wherein The step of inputting the multiple audio features of the first frame rate into the intermediate network composed of the continuous long and short-term memory elements includes: Each of the multiple audio features is input into the continuous long and short-term memory element in sequence, and the continuous long and short-term memory element is used to perform local context encoding and decoding on the audio features of the corresponding segment to obtain the historical memory state of each audio feature segment.

3. A speech separation model training system based on a skip memory network, comprising: A historical memory determination program module is used to input multiple audio features determined by training speech into the long and short-term memory elements for segment processing in the skip memory network to obtain the historical memory status of the multiple audio features; A prediction program module, configured to input the historical memory states of the plurality of audio features into the long and short-term memory elements for memory processing in the skip memory network, and output the predicted future memory states of the plurality of audio features; a training program module for using the actual future memory states of the plurality of pre-prepared audio features as training targets, and performing comparative prediction learning training on the speech separation model based on the predicted future memory states and the training targets, so that the trained speech separation model learns causal sequence modeling capabilities; The structure of the long and short-term memory elements for segment processing includes: an encoder network, an intermediate network composed of continuous long and short-term memory elements, and a decoder network; Wherein, the historical memory determination program module is used to: Performing feature compression on the multiple audio features based on an encoder network to obtain multiple audio features at a first frame rate; Inputting the multiple audio features of the first frame rate into the intermediate network composed of the continuous long and short-term memory elements to obtain a historical memory state of the multiple audio features of the first frame rate; The historical memory state of multiple audio features of the first frame rate is decoded by the decoder network to obtain the historical memory state of multiple audio features of the second frame rate, wherein the second frame rate is higher than the first frame rate to reduce the computational complexity of the intermediate network.

4. The system according to claim 3, wherein: The historical memory determination program module is also used to: Each of the multiple audio features is input into the continuous long and short-term memory element in sequence, and the continuous long and short-term memory element is used to perform local context encoding and decoding on the audio features of the corresponding segment to obtain the historical memory state of each audio feature segment.

5. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to claim 1 or 2.

6. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 1 or 2 are implemented.

Images