Voice separation method and model product, and electronic device and computer storage medium

By using voice arrival time difference information in multi-channel voice data for feature extraction and joint attention processing based on gating, direct decoding to achieve speech separation, solving the separation ability of single-channel and multi-channel voice separation in noise and reverberation scenarios Reduce the problem, improve the separation effect and simplify the implementation process.

WO2024248729A1PCT designated stage expired Publication Date: 2024-12-05ALIBABA INNOVATION PRIVATE LIMITED
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/SG2024/050339
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-05-22
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In complex scenarios mixed with noise and reverb, the single-channel voice separation algorithm performs poorly, and the separation performance of the multi-channel voice separation algorithm is also affected by noise and reverb, resulting in a decrease in separation capability.

Method used

By carrying the voice arrival time difference information in the multi-channel voice data for feature extraction, the joint attention processing based on gate is adopted to obtain the speech feature sequence of each speaker, and directly decode it to achieve speech separation, avoiding the encoder's Output dependency.

Benefits of technology

The effect of multi-channel voice separation is improved and the implementation process is simplified, the cost is reduced, and the separation performance in noise and reverberation scenarios is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050339_05122024_PF_FP_ABST
    Figure SG2024050339_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present disclosure are a voice separation method and model product, and an electronic device and a computer storage medium. The voice separation method comprises: acquiring multi-channel voice data to be subjected to voice separation, wherein the multi-channel voice data carries information of the time difference of arrival between voices from a plurality of speakers; performing feature extraction on the multi-channel voice data that carries the information of the time difference of arrival, so as to obtain corresponding multi-channel voice feature sequences; performing gating-based joint attention processing on the multi-channel voice feature sequences, so as to obtain speaker voice feature sequences respectively corresponding to the plurality of speakers; and performing voice separation on the basis of the speaker voice feature sequences, so as to obtain conference voice data corresponding to the plurality of speakers. By means of the embodiments of the present disclosure, a better voice separation effect is achieved regarding multi-channel voice data, the embodiments of the present disclosure are implemented relatively easily, and the solution implementation cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

[0001]Speech Separation Method and Model Product, Electronic Device, and Computer Storage Medium Cross-Reference This disclosure claims priority to Chinese patent application No. 202310650589.9, filed with the Patent Office of the People's Republic of China on June 2, 2023, entitled "Speech Separation Method and Model Product, Electronic Device, and Computer Storage Medium," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of artificial intelligence technology, and more particularly to a speech separation method, a speech separation model product, an electronic device, and a computer storage medium. Background: Speech separation is a technology that separates target speech from interfering background speech. Currently, single-channel speech separation technology is more commonly used. However, in complex scenarios with mixed noise and reverberation, single-channel speech separation algorithms struggle to achieve ideal speech separation performance. The additional tasks of noise reduction and dereverberation severely impact their performance. This results in single-channel speech separation algorithms, which previously performed well in pure speech separation, experiencing a significant decline in separation capabilities when faced with noise and reverberation, rendering them ineffective for speech separation. Consequently, multi-channel speech separation algorithms have emerged. One commonly used multi-channel speech separation algorithm uses sound source localization to isolate the target speech from multi-channel speech. For example, a commonly used sound source localization method based on beamforming is used. In this method, the signals collected by each sensor in a microphone array are filtered, weighted, and summed to form a beam. This beam is then scanned across the entire sound receiving space, visualizing the sound pressure distribution in a plane. The target speech is then identified based on the sound source localization results. However, due to the influence of noise and reverberation in multiple channels, sound source localization algorithms, including beamforming, often produce large localization errors, resulting in poor multi-channel speech separation performance. SUMMARY OF THE INVENTION In view of this, embodiments of the present disclosure provide a speech separation method, a speech separation model product, an electronic device, and a computer storage medium to at least partially address the aforementioned issues. According to a first aspect of embodiments of the present disclosure, a speech separation method is provided, comprising: obtaining multi-channel speech data to be subjected to speech separation, the multi-channel speech data containing information about time difference of arrival of speech from multiple speakers; performing feature extraction on the multi-channel speech data containing the time difference of arrival information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers; and performing speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to each of the multiple speakers.According to a second aspect of an embodiment of the present disclosure, another speech separation method is provided, comprising: generating multi-channel speech data to be speech separated based on conference speech data, wherein the multi-channel speech data carries information about arrival time differences of speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and performing speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively. According to a third aspect of an embodiment of the present disclosure, a speech separation model product is provided, comprising: an encoder, a separator, and a decoder; wherein: the encoder is configured to perform feature extraction on multi-channel speech data carrying information on the time difference of arrival of speech from multiple speakers to obtain a corresponding multi-channel speech feature sequence; the separator is configured to perform gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers; and the decoder is configured to perform speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to each of the multiple speakers. According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is configured to store at least one executable instruction, wherein the executable instruction causes the processor to perform operations corresponding to the method described in the first or second aspect. According to a fourth aspect of an embodiment of the present disclosure, a computer storage medium is provided, storing a computer program, which, when executed by the processor, implements the method described in the first or second aspect. According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, the computer instructions instructing an electronic device to perform operations corresponding to the method described in the first or second aspect. According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method described in the first or second aspect is implemented. According to a seventh aspect of an embodiment of the present disclosure, a computer program is provided, wherein when the computer program is executed by a processor, the method described in the first or second aspect is implemented.According to the solution provided by the embodiments of the present disclosure, for multi-channel speech with noise and reverberation, on the one hand, information about the arrival time differences of the speakers' voices is included in the multi-channel speech data, enabling accurate location of the speakers' speech positions. On the other hand, based on the multi-channel speech data with arrival time difference information, gated joint attention processing is performed on the corresponding multi-channel speech feature sequences. This allows the speaker speech feature sequence for a single speaker to be obtained while fully considering the local feature information of the single speaker and its global feature information within the entire speech. This results in a more accurate feature representation for the single speaker (i.e., the speaker speech feature sequence for the single speaker). Furthermore, after obtaining the speaker speech feature sequences corresponding to multiple speakers, compared to the related art method of combining encoder output with decoding for speech separation, the solution of the embodiments of the present disclosure eliminates the need for encoder output and instead directly decodes based on the speaker speech feature sequences to achieve speech separation. Through the above-described approach, the solution of the embodiments of the present disclosure not only achieves better speech separation results for multi-channel speech data, but also simplifies implementation, reducing implementation costs. To more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below only illustrate some of the embodiments described in the present disclosure. Those skilled in the art can also obtain other drawings based on these drawings. Figure 1 is a schematic diagram of an exemplary system applicable to the solutions of the embodiments of the present disclosure; Figure 2A is a flowchart of the steps of a speech separation method according to the first embodiment of the present disclosure; Figure 2B is a schematic diagram of an exemplary process for speech separation based on a speech separation model in the embodiment shown in Figure 2A; Figure 2C is an example diagram of a scenario in the embodiment shown in Figure 2A; Figure 3A is a block diagram of a speech separation model product according to the second embodiment of the present disclosure; Figure 3B is a schematic diagram of the structure of a speech separation model product according to the embodiment shown in Figure 3A; Figure 3C is a schematic diagram of the structure of a gated joint attention processing module in the speech separation model product shown in Figure 3B; Figure 3D is a schematic diagram of the structure of a convolution module in the gated joint attention processing module shown in Figure 3; Figure 4 is a schematic diagram of the structure of an electronic device according to the third embodiment of the present disclosure; and Figure 5 is a schematic diagram of the structure of a processor according to the embodiment of the present disclosure.DETAILED DESCRIPTION To help those skilled in the art better understand the technical solutions in the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments represent only a portion of the embodiments of the present disclosure, and are not exhaustive. All other embodiments derived by persons skilled in the art based on the embodiments of the present disclosure should fall within the scope of protection of the embodiments of the present disclosure. The specific implementation of the embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings. FIG1 illustrates an exemplary system applicable to the speech separation solution of the embodiments of the present disclosure. As shown in FIG1 , the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106, with FIG1 illustrating multiple user devices. The cloud server 102 may be any suitable device configured to store information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system, a server cluster, a computing cloud service cluster, and the like. In some embodiments, the cloud server 102 may perform any suitable function. For example, in some embodiments, the cloud server 102 can be configured to perform speech separation processing based on the multi-channel speech data sent by the user device 106 to obtain conference speech data corresponding to multiple speaker identities. As an alternative example, in some embodiments, after receiving the original speech data sent by the user device 106, the cloud server 102 processes it into multi-channel speech data to be subjected to speech separation and carries information about the time difference of arrival of the speech of multiple speakers. Next, the cloud server 102 performs feature extraction on the multi-channel speech data to obtain a corresponding multi-channel speech feature sequence. Furthermore, the cloud server 102 performs gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers. Speech separation is then performed based on the speaker speech feature sequences, and the speech separation results are then sent back to the user device 106. As another example, in some embodiments, the cloud server 102 is provided with a speech separation model, and the cloud server 102 can perform the above-mentioned speech separation task using this speech separation model. In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks.For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be connected to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links. The user device 106 can include any one or more user devices suitable for interacting with a user and receiving the user's voice data. In some embodiments, the user device 106 can send the received raw multi-channel voice data to the cloud server 102, which performs the aforementioned voice separation processing. As another example, in some embodiments, the user device 106 can also process the raw multi-channel voice data into the aforementioned multi-channel voice data to be subjected to voice separation, and then send the multi-channel voice data to the cloud server 102, which performs the voice separation processing. However, this is not limiting. If the user device 106 has high-performance software and hardware, voice separation can be performed locally on the user device 106. In this case, user device 106 processes the original speech data to obtain multi-channel speech data for speech separation. Feature extraction is then performed on the multi-channel speech data to obtain a corresponding multi-channel speech feature sequence. Furthermore, gated joint attention processing is performed on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to multiple speakers. Speech separation is then performed based on the speaker speech feature sequences to obtain speech separation results. In some embodiments, user device 106 may comprise any suitable type of device.For example, in some embodiments, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device. Based on the above system, embodiments of the present disclosure provide a speech separation solution, which will be described below through multiple embodiments. Embodiment 1, with reference to FIG2A , shows a flowchart of the steps of a speech separation method according to Embodiment 1 of the present disclosure. The speech separation method of this embodiment includes the following steps: Step S202: Acquire multi-channel speech data to be subjected to speech separation. Multi-channel speech data refers to speech data acquired by collecting speech signals using multiple microphone speech devices. In embodiments of the present disclosure, the multi-channel speech data carries Time Difference of Arrival (TDOA) information of the speech of multiple speakers. TDOA can be used to characterize the time difference between different speech signals arriving at the microphone speech devices. When there are multiple speakers, this time difference can provide speaker position information within a speech frame. It should be noted that, in the embodiments of the present disclosure, unless otherwise specified, "multiple," "multiple," "multi-channel," and other quantities related to "multiple" all mean two or more. In one feasible approach, this step can be implemented as follows: obtaining original voice data of multiple channels; and superimposing the voice data of the multiple channels to obtain multi-channel voice data to be subjected to voice separation. Unlike the traditional approach of forming multi-channel voice data by splicing the voice data of multiple channels in a horizontal direction, in the embodiments of the present disclosure, the voice data of multiple channels are superimposed in a vertical direction to form the multi-channel voice data. For example, assuming there are four channels of voice data 1A, 1B, 1C, and 1D, the multi-channel voice data formed using the traditional approach is in the form of: 1A-1B-1C-1D; while the multi-channel voice data formed using the superposition approach of the embodiments of the present disclosure is in the form of: In addition to characterizing the locations of different speech sounds, it also provides a basis for subsequent processing of speech data based on TDA information. In practical applications, the amount of speech data processed at a time may be limited, making it impossible to process all the original speech data, especially long speech data, all at once. In this case, the original speech data of multiple channels can be aligned and then segmented according to the length of the speech data that can be processed. The speech data of multiple channels corresponding to the same time period can then be superimposed to obtain multi-channel speech data for speech separation. Furthermore, when using a speech separation model to perform a speech separation task, the speech data of multiple channels can be superimposed according to the input dimension of the speech separation model, such as the input dimension of the encoder to be input to the speech separation model, to obtain multi-channel speech data for speech separation. This multi-channel speech data can then be input into the speech separation model, such as the encoder of the speech separation model, for processing. Superimposing the speech data of multiple channels according to the input dimension of the speech separation model ensures that the dimensions of the obtained multi-channel speech data match the input of the speech separation model, facilitating processing by the speech separation model. Step S204: Feature extraction is performed on the multi-channel speech data carrying arrival time difference information to obtain a corresponding multi-channel speech feature sequence. In this step, feature extraction can be performed on the multi-channel speech data using any appropriate method, including but not limited to convolution. Because the multi-channel speech data carries information about the arrival time differences of speech from different speakers, the multi-channel speech feature sequence obtained after feature extraction also carries this information, thereby characterizing the positions of the speech from different speakers within the feature sequence. In one feasible embodiment, the speech separation method of this embodiment can be implemented using a speech separation model comprising an encoder, a separator, and a decoder. The encoder is configured to perform feature extraction on the multi-channel speech data carrying information about the arrival time differences of speech from multiple speakers to obtain a corresponding multi-channel speech feature sequence; the separator is configured to perform gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers; and the decoder is configured to perform speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to each of the multiple speakers. Based on this, an exemplary process of performing speech separation based on the above-mentioned speech separation model is shown in FIG2B . As can be seen from FIG2B , multi-channel speech data is input into the encoder of the speech separation model, and the encoder then implements feature extraction of the multi-channel speech data in this step.This encoder can be used to encode multi-channel speech data into a feature sequence containing speech features of different speakers, which can be used as a digital representation for the speech phasing task. Step S206: Perform gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to multiple speakers. In this disclosed embodiment, joint attention processing means that when performing attention processing on the multi-channel speech feature sequence, attention is paid not only to the feature portion corresponding to a particular speaker, but also to the relationship between the feature portion corresponding to that speaker and the multi-channel speech feature sequence as a whole. The addition of a gating mechanism can better capture the dependencies between features in the sequence. As previously mentioned, the time difference of arrival of speech can be used to represent the position of different speakers in the multi-channel speech feature sequence. Based on this, gated joint attention processing is performed on the multi-channel speech feature sequence. In the feature sequence corresponding to the processing result, different speakers can be associated with different feature sequence portions. Since the goal of speech separation is to isolate speech data corresponding to different speakers, in one feasible approach, this step can be implemented by: performing gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence; and expanding the attention feature sequence according to the number of speakers to obtain speaker speech feature sequences corresponding to each of the multiple speakers. For example, assuming the sequence dimension of the attention feature sequence sS x is "', and the number of speakers is set to C, then the attention feature sequence is expanded according to the number of speakers. Sufficient feature data for speech feature components corresponding to different speakers is provided. In one feasible approach, performing gated joint attention processing on a multi-channel speech feature sequence to obtain an attention feature sequence can be implemented as follows: performing local feature extraction and global feature extraction based on the gated multi-channel speech data on the multi-channel speech feature sequence through attention calculation; and obtaining a mixed attention feature sequence based on the results of the local and global feature extractions. Both local and global feature extractions are implemented through attention calculation. Local feature extraction is based on the speech features of a particular speaker's speech position indicated by the arrival time difference, supplemented by other speech features in the multi-channel speech feature sequence, focusing on these speech features. This is known as local feature extraction. Global feature extraction is based on the overall speech features in the multi-channel speech feature sequence, performing attention calculation on the multi-channel speech feature sequence. This is known as global feature extraction. Furthermore, based on the results of these two feature extractions, a mixed attention feature sequence formed by the results of these two feature extractions can be obtained. This approach not only makes speaker feature extraction more targeted, but also fully considers global information during speaker feature extraction, thereby improving the overall accuracy of the extracted features. After obtaining the attention feature sequence, as mentioned above, it is necessary to expand the attention feature sequence to facilitate subsequent processing. In one feasible approach, after obtaining the attention feature sequence, the attention feature sequence can be expanded according to the number of speakers to obtain multiple corresponding attention feature sequences. Feature extraction is then performed on each of the multiple attention feature sequences, and gating processing is performed on the corresponding feature sequences obtained after feature extraction to obtain gated processing results. Based on the gated processing results, speaker speech feature sequences corresponding to each of the multiple speakers are obtained. By extracting feature from each of the multiple attention feature sequences, more representative speaker speech features can be extracted. Gating processing based on these more representative speaker speech features effectively models the relationships between the speech features of each speaker, facilitating subsequent speech separation. When using the aforementioned speech separation model for speech separation, as shown in FIG2B , this step can be implemented by the speech separation model's separator. Through the aforementioned processing, the separator separates and models the multi-channel speech feature sequences, mapping them to speaker speech feature sequences for different speakers. Step S208: Speech separation is performed based on the speaker speech feature sequences to obtain conference speech data corresponding to each of the multiple speakers.In one feasible approach, speaker speech feature sequences corresponding to different speakers can be decoded separately to obtain decoding results. Speech separation of multi-channel speech data can be performed based on the decoded speaker speech feature sequences. By decoding the speaker speech feature sequences corresponding to different speakers, speech data corresponding to the different speakers can be output, effectively achieving speech separation. When using the aforementioned speech separation model to achieve speech separation, as shown in FIG2B , this step can be implemented by the decoder of the speech separation model. The decoder decodes the individual speaker speech feature sequences corresponding to each speaker output by the separator to achieve speech reconstruction and output speech data for each speaker. During this process, decoder parameters are shared across the decoding process of the speaker speech feature sequences for different speakers, thereby improving decoding speed and efficiency, and thereby improving speech separation speed and efficiency. Furthermore, in the disclosed embodiments, when decoding the speaker speech feature sequences, there is no need to combine them with the feature sequences output by the encoder. Instead, decoding and speech separation operations are performed directly based on the output of the separator. Because the speaker speech feature sequences output by the separator for different speakers are sufficient to fully characterize the current speaker's characteristics, to avoid interference from other features on the current speaker's characteristics and achieve better speech separation results for multi-channel speech data, the embodiments of the present disclosure perform speech separation directly based on the separator's output. The following uses a conference scenario as an example to illustrate the above process. In a conference scenario, there are typically multiple participants, i.e., multiple speakers, and the physical environment of the conference is often noisy and reverberant. In this example, it is assumed that conference audio is captured using multiple microphones to generate raw multi-channel speech data, also referred to as conference speech data. Furthermore, during the collection of conference audio, arrival time difference information for the speech of different speakers is also generated simultaneously. This conference speech data, carrying this arrival time difference information, can then be sent to a backend for speech separation processing. Specifically, this data can be sent to a backend speech separation device, such as a speech separation terminal, a speech separation server, or a cloud-based system, for speech separation processing. For example, as shown in FIG2C , after receiving the conference voice data, the speech separation device processes it into multi-channel voice data for speech separation. For example, the speech data from multiple channels contained in the conference voice data are aligned and then superimposed to obtain the multi-channel voice data for speech separation. Furthermore, this multi-channel voice data also includes information about the time difference of arrival of the voices of multiple speakers.Furthermore, the speech separation device uses the speech separation model shown in Figure 2B to first perform feature extraction on the multi-channel speech data containing arrival time difference information through its encoder, obtaining a corresponding multi-channel speech feature sequence. Its separator then performs gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers. Finally, the decoder performs speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to each of the multiple speakers. The specific implementation of the above operations of the speech separation model can be found in the description of the process in this embodiment and will not be repeated here. After obtaining the speech separation results, conference transcripts corresponding to each of the multiple speakers can be generated based on the conference speech data. For example, speech recognition can be used to convert the speech data into text data. This greatly facilitates and improves the efficiency of conference recording, reducing the cost of conference recording. As can be seen from the above, this embodiment, for multi-channel speech with noise and reverberation, firstly, carries information about the speaker's speech arrival time difference in the multi-channel speech data, enabling accurate location of the speakers' speaking positions. Secondly, based on the multi-channel speech data containing the arrival time difference information, gated joint attention processing is performed on the corresponding multi-channel speech feature sequence. This allows the speaker's speech feature sequence for a single speaker to be fully considered, taking into account both the speaker's local feature information and its global feature information within the entire speech. This results in a more accurate feature representation for the single speaker (i.e., the speaker's speech feature sequence for the single speaker). Furthermore, after obtaining the speaker's speech feature sequences corresponding to multiple speakers, compared to the related art method of combining encoder output with decoding for speech separation, the solution of the present embodiment no longer requires the encoder output, but instead directly decodes based on the speaker's speech feature sequence to achieve speech separation. Through the above approach, the solution of the present embodiment not only achieves better speech separation results for multi-channel speech data, but also simplifies implementation, reducing implementation costs. The following describes the speech separation solution provided by the embodiments of the present disclosure, using a specific implementation of a speech separation model. FIG3A shows a block diagram of a speech separation model product according to the second embodiment of the present disclosure. The speech separation model of this embodiment includes an encoder 302, a separator 304, and a decoder 306.The encoder 302 is configured to perform feature extraction on multi-channel speech data containing information about the time difference of arrival of speech from multiple speakers, obtaining a corresponding multi-channel speech feature sequence. The separator 304 is configured to perform gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers. The decoder 306 is configured to perform speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to each of the multiple speakers. In this embodiment, to enable the speech separation model to effectively process multi-channel speech data, the original speech data is processed into data that can be processed by the speech separation model. This process includes: obtaining original multi-channel speech data; and superimposing the multi-channel speech data according to the encoder input dimension to obtain multi-channel speech data to be subjected to speech separation. The multi-channel speech data contains information about the time difference of arrival of speech from multiple speakers. In this way, speech data from multiple channels can be aligned, and the arrival time difference of speech can be used to represent the speech positions of different speakers, enabling the speech separation model to efficiently perform speech separation processing on multi-channel speech data. The following describes the speech separation model product of this embodiment using the specific structure of a speech separation model as an example, as shown in Figure 3B. In Figure 3B, encoder 302 includes a one-dimensional convolutional layer and a rectified linear unit (ReLU). Encoder 302 is configured to perform feature extraction on multi-channel speech data containing information about the arrival time differences of speech from multiple speakers, obtaining a corresponding multi-channel speech feature sequence. As previously mentioned, in this embodiment, the speech data from multiple channels is first superimposed as the original input features for the speech separation model, i.e., the multi-channel speech data to be input to encoder 302 for speech separation. The number of channels is equal to the dimensionality of the input features, i.e., the dimensionality of the input features that encoder 302 can accept. Encoder 302 not only encodes the input multi-channel speech data into a high-dimensional representation but also preserves the arrival time difference information of the speaker's speech signals received by the multi-channel microphones, providing effective speaker speech frame position information for the subsequent separator 304. The rectified linear unit (RLU) of encoder 302 constrains the output code vector to non-negative values. After processing by the RLU, the code vector output by encoder 302 is a multi-channel speech feature sequence that carries the arrival time difference information of the speech signals of multiple speakers.Separator 304 is configured to process the encoding vectors (i.e., the multi-channel speech feature sequences) output by encoder 302 into speaker-specific speech feature sequences for different speakers, completing the feature-level speech separation task. As shown in FIG3B , after entering separator 304, the multi-channel speech feature sequences are first normalized to produce a normalized multi-channel speech feature sequence. Then, for subsequent attention calculation, positional encoding (position embedding) is added to the normalized multi-channel speech feature sequence, illustrated as "sinusoidal positional encoding" in FIG3B . Next, [the sentence fragment "adds" appears to be a jumble of unrelated fragments and should likely be omitted as it is omitted.] This makes the output more accurate. After triple gating, the multi-channel speech feature sequence is input into the fourth convolutional module for further feature extraction. The result of this feature extraction is summed with the original reshaped multi-channel speech feature sequence input, and the summed attention calculation result is output. As can be seen above, in this example, the gated local & global joint attention module only learns the residual portion of the feature sequence and uses skip connections from the module input to the output, thereby improving the module's feature processing efficiency. As previously mentioned, the gated local & global joint attention module includes N modules. The output of the current gated local & global joint attention module is passed to the next gated local & global joint attention module as input, and processing continues until the last gated local & global joint attention module outputs the final result. In one feasible approach, in the above-mentioned gated local & global joint attention module, an exemplary structure of the first, second, third, and fourth convolutional modules is shown in Figure 3D. As shown in Figure 3D, the convolution module includes, in order: a normalization layer, a linear layer, an activation layer, a first reshaping layer, a one-dimensional depthwise convolution layer, a second reshaping layer, and a dropout layer. The feature sequence input to the first, second, third, or fourth convolution module is first normalized by the normalization layer; then linearly projected by the linear layer; then nonlinearly processed by the activation layer; then reshaped by the first reshaping layer into a feature sequence that conforms to the convolutional form; then convolved by the one-dimensional depthwise convolution layer, and then reshaped by the second reshaping layer into a feature sequence suitable for dropout; finally, the feature sequence reshaped by the second reshaping layer is dropout-processed by the dropout layer and output. Through this gated local and global joint attention module, a more representative multi-channel speech feature sequence can be extracted.Furthermore, the output of the gated local and global joint attention module is reshaped into a feature sequence suitable for subsequent processing and then passed to the next rectified linear unit. Then, a further point-wise convolution is performed to expand the feature sequence to a dimension corresponding to the number of speakers. The result of the point-wise convolution is reshaped again and processed by multiple parallel point-wise convolution modules to obtain 3) a speaker speech feature sequence for each speaker. Furthermore, each speaker's speech feature sequence is gated by a gated linear unit to obtain a gated processing result, which better models the relationship between the speaker speech feature sequences of multiple speakers. The gated speaker speech feature sequence is then processed again by point-wise convolution and a rectified linear unit to obtain more accurate speaker speech feature sequences corresponding to the multiple speakers. These multiple speaker speech feature sequences are then input into the decoder for processing. In this example, the decoder is configured to directly decode the speaker speech feature sequence output by the separator to obtain a decoding result. Based on the decoding result, the decoder performs speech separation on the multi-channel speech data, obtaining conference speech data corresponding to multiple speakers. For example, the decoder may include a one-dimensional transposed convolutional layer, which uses the same kernel size and stride as the encoder to reconstruct the speech data for each speaker based on the decoded speaker speech feature sequence, thereby achieving multi-speaker speech separation. The decoder parameters are shared across all speaker speech feature sequences, meaning that the decoder is a parameter-sharing decoder. This improves the versatility of the entire speech sharing model, including the decoder, and reduces the number of parameters in the decoder and model. Thus, the speech separation model of this embodiment can effectively process multi-channel speech data. Furthermore, the decoder directly decodes the separator output, transforming the separator's design from estimating a mask (MASK) in a correlation approach to estimating speaker speech feature components. This bypasses the sound source localization algorithm and allows direct speech separation in complex scenarios such as noise and reverberation. Furthermore, by leveraging the time difference of arrival (TDA) information carried in multi-channel speech data, combined with gated joint attention processing, the interference of noise and reverberation on separation performance in complex scenarios is reduced, significantly improving the overall separation performance of the speech separation model. Example 3 Referring to FIG4 , a schematic diagram of the structure of an electronic device according to Example 3 of the present disclosure is shown. The present disclosure does not limit the specific implementation of the electronic device.As shown in FIG4 , the electronic device may include a processor 402, a communication interface 404, a memory 406, and a communication bus 408. The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is configured to communicate with other electronic devices or a server. The processor 402 is configured to execute a program 410, which may specifically perform the relevant steps in the above-described embodiment of the speech separation method. Specifically, the program 410 may include program code, which includes computer operating instructions. The processor 402 may be a CPU, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure. The one or more processors included in the smart device may be of the same type, such as one or more CPUs, or of different types, such as one or more CPUs and one or more ASICs. Memory 406 is configured to store program 410. Memory 406 may include high-speed RAM or non-volatile memory, such as at least one disk drive. Program 410 may include multiple computer instructions. Specifically, program 410 may cause processor 402 to perform operations corresponding to the speech separation method described in the aforementioned method embodiment through the multiple computer instructions. The specific implementation of each step in program 410 can be found in the corresponding descriptions of the corresponding steps and units in the aforementioned method embodiment, and corresponding beneficial effects are achieved, so detailed description is omitted here. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes of the devices and modules described above can refer to the corresponding process descriptions in the aforementioned method embodiment, and are not further described here. The present disclosure also provides a processor. Figure 5 shows a schematic structural diagram of a processor used to implement the present disclosure embodiment. As shown in FIG5 , the processor 500 is configured to run a program, wherein when the program is run by the processor, the method in the above embodiment is executed. In the embodiment of the present disclosure, the processor 500 may execute the program of the speech separation method.Optionally, the processor 500 may be configured to perform the following steps: obtaining multi-channel speech data to be subjected to speech separation, the multi-channel speech data carrying information on arrival time differences of speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and performing speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively. Optionally, the processor 500 may be configured to perform the following steps: generating multi-channel speech data to be subjected to speech separation based on conference speech data, wherein the multi-channel speech data carries information about the time difference of arrival of the speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the time difference of arrival information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers; and performing speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to each of the multiple speakers. The present disclosure also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the aforementioned method embodiment. The computer storage medium includes, but is not limited to, a Compact Disc Read-Only Memory (CD-ROM), a Random Access Memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk. A computer storage medium may also be referred to as a computer-readable storage medium. This may include a data signal transmitted in baseband or as part of a carrier wave, carrying readable program code. This transmitted data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable storage medium may transmit, propagate, or transfer a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. The program code contained in the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, radio frequency, or any suitable combination thereof. The present disclosure also provides a computer program product comprising computer instructions that instruct a computing device to perform operations corresponding to the speech separation method described in the above-described method embodiment.The present disclosure also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements operations corresponding to the speech separation method in the above-mentioned method embodiment. The present disclosure also provides a computer program. When executed by a processor, the computer program implements operations corresponding to the speech separation method in the above-mentioned method embodiment. Optionally, the computer program, when executed by a processor, includes program code that implements the following steps: obtaining multi-channel speech data to be subjected to speech separation, the multi-channel speech data containing information about time differences of arrival of speech from multiple speakers; performing feature extraction on the multi-channel speech data containing the time differences of arrival information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to each of the multiple speakers; and performing speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to each of the multiple speakers. Optionally, the computer program implements program code for the following steps when executed by a processor: generating multi-channel speech data to be speech separated based on conference speech data, wherein the multi-channel speech data carries information on arrival time differences of speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and performing speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively. Furthermore, it should be noted that all user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for model training, voice data used for processing, data used for analysis, data stored, and data displayed, etc.) involved in the embodiments of this disclosure are authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or deny. It should be noted that, depending on implementation needs, the various components / steps described in the embodiments of this disclosure may be split into more components / steps, or two or more components / steps or partial operations of components / steps may be combined into new components / steps to achieve the objectives of the embodiments of this disclosure.The above-described methods according to the embodiments of the present disclosure may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and is to be stored in a local recording medium and downloaded via a network, so that the methods described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)). It will be understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, Read-Only Memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code for implementing the methods described herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods described herein. Those skilled in the art will appreciate that the various exemplary units and method steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals may use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments disclosed herein. The above embodiments are merely illustrative of the embodiments of the present disclosure and are not intended to limit the embodiments of the present disclosure. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the embodiments of the present disclosure. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present disclosure. The scope of patent protection of the embodiments of the present disclosure shall be defined by the claims.Industrial Applicability The solution provided by the embodiments of the present disclosure can be applied to the process of speech separation, obtaining multi-channel speech data to be subjected to speech separation, wherein the multi-channel speech data carries information on the arrival time differences of the speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers; and performing speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to the multiple speakers, thereby solving the technical problem of poor speech separation performance of multi-channel speech data.

Claims

Claims 1. A speech separation method, comprising: Acquire multi-channel speech data to be subjected to speech separation, wherein the multi-channel speech data carries information on arrival time differences of speech of multiple speakers; Performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain a corresponding multi-channel speech feature sequence; performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; performing speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively.

2. The method according to claim 1, wherein: The performing speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively includes: decoding the speaker speech feature sequence to obtain a decoding result; performing speech separation on the multi-channel speech data based on the decoding result to obtain conference speech data corresponding to the multiple speakers respectively.

3. The method according to claim 1 or 2, wherein: The obtaining of multi-channel voice data to be subjected to voice separation comprises: obtaining original voice data of multiple channels; and superimposing the voice data of the multiple channels to obtain the multi-channel voice data to be subjected to voice separation.

4. The method according to claim 1 or 2, wherein: The performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively includes: performing gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence; and expanding the attention feature sequence according to the number of the multiple speakers to obtain speaker speech feature sequences corresponding to the multiple speakers respectively.

5. The method according to claim 4, wherein: The step of expanding the attention feature sequence according to the number of the multiple speakers to obtain speaker speech feature sequences corresponding to the multiple speakers includes: expanding the attention feature sequence according to the number of the multiple speakers to obtain multiple corresponding attention feature sequences; performing feature extraction on the multiple attention feature sequences respectively, and performing gating processing on the corresponding multiple feature sequences obtained after the feature extraction to obtain a gating processing result; Based on the gating processing result, speaker speech feature sequences corresponding to the multiple speakers are obtained.

6. The method according to claim 4, wherein: The method of performing gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence includes: performing gated local feature extraction and global feature extraction of the multi-channel speech data on the multi-channel speech feature sequence through attention calculation; and obtaining a mixed attention feature sequence based on the results of the local feature extraction and the results of the global feature extraction.

7. A speech separation method, comprising: According to the conference voice data, multi-channel voice data to be subjected to voice separation is generated, wherein the multi-channel voice data carries information on arrival time differences of voices of multiple speakers; feature extraction is performed on the multi-channel voice data carrying the information on arrival time differences to obtain a corresponding multi-channel voice feature sequence; gated joint attention processing is performed on the multi-channel voice feature sequence to obtain speaker voice feature sequences corresponding to the multiple speakers; based on the speaker voice feature sequence, voice separation is performed to obtain conference voice data corresponding to the multiple speakers respectively.

8. The method according to claim 7, wherein: The performing speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively includes: decoding the speaker speech feature sequence to obtain a decoding result; performing speech separation on the multi-channel speech data based on the decoding result to obtain conference speech data corresponding to the multiple speakers respectively.

9. The method according to claim 7 or 8, wherein: The performing gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively includes: performing gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence; and expanding the attention feature sequence according to the number of the multiple speakers to obtain speaker speech feature sequences corresponding to the multiple speakers respectively.

10. The method according to claim 9, wherein: The step of expanding the attention feature sequence according to the number of the multiple speakers to obtain speaker speech feature sequences corresponding to the multiple speakers includes: expanding the attention feature sequence according to the number of the multiple speakers to obtain multiple corresponding attention feature sequences; performing feature extraction on the multiple attention feature sequences respectively, and performing gating processing on the corresponding multiple feature sequences obtained after the feature extraction to obtain a gating processing result; Based on the gating processing result, speaker speech feature sequences corresponding to the multiple speakers are obtained.

11. The method according to claim 9, wherein: The method of performing gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence includes: performing gated local feature extraction and global feature extraction of the multi-channel speech data on the multi-channel speech feature sequence through attention calculation; and obtaining a mixed attention feature sequence based on the results of the local feature extraction and the results of the global feature extraction.

12. The method according to claim 7, wherein: The method further includes: generating conference record texts corresponding to the multiple speakers respectively according to conference voice data corresponding to the multiple speakers respectively.

13. A speech separation model product, comprising: An encoder, a separator and a decoder; wherein: the encoder is configured to perform feature extraction on multi-channel speech data carrying information on arrival time differences of speech of multiple speakers to obtain a corresponding multi-channel speech feature sequence; the separator is configured to perform gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; the decoder is configured to perform speech separation based on the speaker speech feature sequence to obtain conference speech data corresponding to the multiple speakers respectively.

14. The product according to claim 13, wherein: The decoder is configured to decode the speaker voice feature sequence output by the separator to obtain a decoding result, and perform voice separation of the multi-channel voice data according to the decoding result to obtain conference voice data corresponding to the multiple speakers respectively.

15. The product according to claim 13 or 14, wherein The multi-channel speech data input to the encoder is obtained by: acquiring original speech data of multiple channels; and superimposing the speech data of multiple channels according to the input dimension of the encoder to obtain the multi-channel speech data to be subjected to speech separation.

16. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; The memory is configured to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 12.

17. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

18. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the method as claimed in any one of claims 1 to 12. 18 19. A computer program product, wherein: The invention comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 12 when executed by a processor.

20. A computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented. 19

Citation Information

Patent Citations

  • Feature fusion method, electronic equipment and storage medium

    CN115050374A

  • Speech separation method

    CN116168717A

  • Array geometry agnostic multi-channel personalized speech enhancement

    US20230116052A1