Speech separation method and model product, electronic device, and computer storage medium

Abstract

Embodiments of the present application provide a speech separation method and model product, an electronic device and a computer storage medium, wherein the speech separation method comprises: obtaining multi-channel speech data to be subjected to speech separation, the multi-channel speech data carrying information of time difference of arrival of speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the information of the time difference of arrival to obtain a corresponding multi-channel speech feature sequence; performing gate-based joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and performing speech separation based on the speaker speech feature sequences. Through the embodiments of the present application, the speech separation effect of the multi-channel speech data is better, the implementation is simpler, and the implementation cost of the scheme is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech separation method, a speech separation model product, an electronic device, and a computer storage medium. Background Technology

[0002] Speech separation is a technique that separates target speech from background speech with interference. Currently, single-channel speech separation technology is commonly used. However, in complex scenarios with noise and reverberation, single-channel speech separation algorithms struggle to meet ideal performance requirements. The additional noise reduction and dereverberation tasks severely impact the performance of single-channel speech separation algorithms. As a result, single-channel speech separation algorithms, which initially perform well in separating clean speech, show a significant decline in separation capability when faced with noisy and reverberant speech, rendering them ineffective for speech separation.

[0003] Therefore, multi-channel speech separation algorithms have emerged. In a commonly used multi-channel speech separation algorithm, source localization methods are employed to separate the target speech from the multi-channel audio. For example, a commonly used method is beamforming-based source localization. This method filters and weights the signals collected by each sensor in the microphone array to form a beam, which scans the entire sound reception space, visualizing the sound pressure distribution on a plane. Then, combined with the source localization results, the target speech is determined. However, due to the influence of noise and reverberation in multi-channel audio, source localization algorithms, including beamforming, often produce large localization errors, resulting in poor separation performance in multi-channel speech separation. Summary of the Invention

[0004] In view of this, embodiments of this application provide a speech separation scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, a speech separation method is provided, comprising: acquiring multi-channel speech data to be separated, wherein the multi-channel speech data carries information on the arrival time differences of the speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain corresponding multi-channel speech feature sequences; performing gated joint attention processing on the multi-channel speech feature sequences to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and performing speech separation based on the speaker speech feature sequences.

[0006] According to a second aspect of the embodiments of this application, another speech separation method is provided, comprising: acquiring conference speech data, wherein the conference speech data is multi-channel speech data carrying information on the arrival time differences of the speech of multiple speakers; performing feature extraction on the multi-channel speech data carrying the arrival time difference information to obtain corresponding multi-channel speech feature sequences; performing gated joint attention processing on the multi-channel speech feature sequences to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and performing speech separation based on the speaker speech feature sequences to obtain conference speech data corresponding to the multiple speakers respectively.

[0007] According to a third aspect of the embodiments of this application, a speech separation model product is provided, comprising: an encoder, a separator, and a decoder; wherein: the encoder is used to extract features from multi-channel speech data carrying information on the time difference of arrival of the speech of multiple speakers to obtain corresponding multi-channel speech feature sequences; the separator is used to perform gated joint attention processing on the multi-channel speech feature sequences to obtain speaker speech feature sequences corresponding to the multiple speakers respectively; and the decoder is used to perform speech separation based on the speaker speech feature sequences.

[0008] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0009] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0010] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first or second aspect.

[0011] According to the solution provided in this application embodiment, for multi-channel speech with noise and reverberation, on the one hand, the speech arrival time difference information of the speakers is carried in the multi-channel speech data to accurately locate the speaking positions of different speakers; on the other hand, based on the multi-channel speech data with arrival time difference information, gating-based joint attention processing is performed on the corresponding multi-channel speech feature sequences, so that when obtaining the speaker speech feature sequence of a single speaker, the local feature information of the single speaker and its global feature information in the entire speech can be fully considered, making the obtained feature expression for the single speaker (i.e., the speaker speech feature sequence of the single speaker) more accurate; furthermore, after obtaining the speaker speech feature sequences corresponding to multiple speakers, compared with the related technology that requires decoding based on the encoder output for speech separation, the solution in this application embodiment no longer requires the encoder output, but directly decodes based on the speaker speech feature sequences to achieve speech separation. Through the above methods, the solution in this application embodiment not only has a better speech separation effect for multi-channel speech data, but is also simpler to implement, reducing the implementation cost. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 A schematic diagram of an exemplary system to which the embodiments of this application are applicable;

[0014] Figure 2A This is a flowchart illustrating the steps of a speech separation method according to Embodiment 1 of this application;

[0015] Figure 2B for Figure 2A A schematic diagram illustrating an exemplary process of speech separation based on a speech separation model in the illustrated embodiment;

[0016] Figure 2C for Figure 2A An example scenario diagram of one embodiment is shown;

[0017] Figure 3A This is a structural block diagram of a speech separation model product according to Embodiment 2 of this application;

[0018] Figure 3B for Figure 3A A schematic diagram of the structure of a speech separation model product in the embodiment shown;

[0019] Figure 3C for Figure 3B The diagram shows the structure of the gated joint attention processing module in the speech separation model product.

[0020] Figure 3D Figure 3 shows a schematic diagram of the convolution module in the gate-based joint attention processing module.

[0021] Figure 4 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0023] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0024] Figure 1 An exemplary system for a speech separation scheme applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0025] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to perform speech separation processing based on multi-channel speech data sent by the user equipment 106. As an optional example, in some embodiments, after receiving the raw speech data sent by the user equipment 106, the cloud server 102 processes it into multi-channel speech data to be separated, carrying information on the arrival time differences of the speech of multiple speakers; then, it performs feature extraction on the multi-channel speech data to obtain the corresponding multi-channel speech feature sequence; furthermore, it performs gated joint attention processing on the multi-channel speech feature sequence to obtain speaker speech feature sequences corresponding to multiple speakers respectively; then, it performs speech separation based on the speaker speech feature sequences; and finally, it sends the speech separation result back to the user equipment 106. As another example, in some embodiments, the cloud server 102 is equipped with a speech separation model, which can be used to perform the above-mentioned speech separation task.

[0026] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user equipment 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user equipment 106 and the cloud server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0027] User equipment 106 may include any one or more user equipments suitable for interacting with users and receiving users' voice data. In some embodiments, user equipment 106 may send the received raw multi-channel voice data to cloud server 102, whereby cloud server 102 performs the aforementioned voice separation processing. As another example, in some embodiments, user equipment 106 may also process the raw multi-channel voice data into the aforementioned multi-channel voice data to be separated, and then send the multi-channel voice data to cloud server 102 for voice separation processing. However, it is not limited to this; if user equipment 106 has high software and hardware performance, voice separation can be performed locally on user equipment 106. In this case, user equipment 106 itself processes the raw voice data to obtain multi-channel voice data to be separated; then, it performs feature extraction on the multi-channel voice data to obtain corresponding multi-channel voice feature sequences; furthermore, it performs gated joint attention processing on the multi-channel voice feature sequences to obtain speaker voice feature sequences corresponding to multiple speakers; then, based on the speaker voice feature sequences, it performs voice separation to obtain the voice separation result. In some embodiments, user equipment 106 may include any suitable type of device. For example, in some embodiments, user equipment 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user equipment.

[0028] Based on the above system, this application provides a speech separation scheme, which will be described below through several embodiments.

[0029] Example 1

[0030] Reference Figure 2A The diagram shows a flowchart of the steps of a speech separation method according to Embodiment 1 of this application.

[0031] The speech separation method in this embodiment includes the following steps:

[0032] Step S202: Obtain multi-channel speech data to be separated.

[0033] Multichannel speech data refers to speech data obtained by acquiring speech signals from multiple microphones. In this embodiment, the multichannel speech data carries TDOA (Time Difference of Arrival) information of multiple speakers. TDOA is used to characterize the time difference between the arrival of different speech signals at the microphones. When there are multiple speakers, this time difference can provide the speaker's location information in the speech frame.

[0034] It should be noted that, unless otherwise specified, in the embodiments of this application, "multiple", "various types", "multi-channel", and other quantities related to "multiple" all refer to two or more.

[0035] In one feasible approach, this step can be implemented as follows: acquiring original speech data from multiple channels; and superimposing the speech data from these multiple channels to obtain multi-channel speech data to be separated. Unlike the traditional method of splicing speech data from multiple channels horizontally to form multi-channel speech data, this embodiment superimposes speech data from multiple channels vertically to form multi-channel speech data. For example, assuming there are four channels of speech data A, B, C, and D, the multi-channel speech data formed using the traditional method would look like: ABCD; while the multi-channel speech data formed using the superposition method of this embodiment would look like: By overlaying, TDOA information can be effectively aligned, which can not only effectively represent the position of different speech sounds, but also provide a basis for subsequent processing of speech data based on TDOA information.

[0036] In practical applications, the amount of voice data that can be processed at one time may be limited, and it may be impossible to process the original voice data, especially long voice data, all at once. In this case, the original voice data from multiple channels can be aligned and then segmented according to the length of the voice data that can be processed. Then, the voice data from multiple channels corresponding to the same time period can be superimposed to obtain multi-channel voice data to be separated.

[0037] Alternatively, when using a speech separation model to perform a speech separation task, the speech data from multiple channels can be superimposed according to the input dimensions of the speech separation model, such as the input dimensions of the encoder in the speech separation model, to obtain multi-channel speech data to be separated. This multi-channel speech data can then be input into the speech separation model, such as the encoder of the speech separation model, for processing. Superimposing the speech data from multiple channels according to the input dimensions of the speech separation model ensures that the dimensions of the obtained multi-channel speech data match the input of the speech separation model, making it easier for the speech separation model to process.

[0038] Step S204: Extract features from the multi-channel speech data carrying information about the time difference of arrival to obtain the corresponding multi-channel speech feature sequence.

[0039] In this step, any appropriate method can be used to extract features from the multi-channel speech data, including but not limited to convolution. Since the multi-channel speech data carries information about the arrival time differences of the voices of different speakers, the resulting multi-channel speech feature sequence also carries this information to characterize the position of the voices of different speakers within the feature sequence.

[0040] In one feasible approach, the speech separation method of this embodiment can be implemented by a speech separation model, which includes an encoder, a separator, and a decoder. The encoder extracts features from multi-channel speech data carrying information about the time differences of arrival of multiple speakers, obtaining corresponding multi-channel speech feature sequences. The separator performs gated joint attention processing on the multi-channel speech feature sequences to obtain speaker speech feature sequences for each speaker. The decoder performs speech separation based on the speaker speech feature sequences.

[0041] Based on this, an exemplary process for speech separation based on the above speech separation model is as follows: Figure 2B As shown, by Figure 2B As can be seen, multi-channel speech data is input into the encoder of this speech separation model, and the encoder then performs feature extraction of the multi-channel speech data in this step. Through this encoder, multi-channel speech data can be encoded into a feature sequence containing the speech features of different speakers, which serves as a digital representation for the speech separation task.

[0042] Step S206: Perform gate-based joint attention processing on the multi-channel speech feature sequences to obtain speaker speech feature sequences corresponding to multiple speakers.

[0043] In this embodiment, joint attention processing means that when performing attention processing on a multi-channel speech feature sequence, attention is paid not only to the feature part corresponding to a certain speaker, but also to the relationship between the feature part corresponding to that speaker and the overall multi-channel speech feature sequence. The addition of a gating mechanism can better capture the dependencies between features in the sequence.

[0044] As mentioned earlier, the arrival time difference of speech can be used to characterize the position of different speakers in the multi-channel speech feature sequence. Based on this, gating-based joint attention processing is performed on the multi-channel speech feature sequence. In the feature sequence corresponding to the processing result, different speakers can be assigned to different parts of the feature sequence.

[0045] Since the goal of speech separation is to separate the speech data corresponding to different speakers, in one feasible approach, this step can be implemented as follows: performing gate-based joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence; expanding the attention feature sequence according to the number of speakers to obtain speaker speech feature sequences corresponding to each speaker. For example, assume the sequence dimension of the attention feature sequence is... If the number of speakers is set to C, then after expanding the attention feature sequence according to the number of speakers, the attention feature sequence will be expanded to... This dimension provides sufficient feature data for speech feature components corresponding to different speakers for subsequent speech separation.

[0046] In one feasible approach, gating-based joint attention processing is applied to a multi-channel speech feature sequence to obtain an attention feature sequence. This can be achieved by: performing gating-based local and global feature extraction on the multi-channel speech data through attention computation; and obtaining a hybrid attention feature sequence based on the results of local and global feature extraction. Both local and global feature extraction are implemented through attention computation. Local feature extraction focuses on the speech features at a speaker's speech position indicated by the arrival time difference, using other speech features in the multi-channel speech feature sequence as auxiliary data, and performs attention computation primarily on this part of the speech features (i.e., local feature extraction). Global feature extraction focuses on the overall speech features in the multi-channel speech feature sequence, performing attention computation on the multi-channel speech feature sequence (i.e., global feature extraction). Therefore, based on the results of these two feature extractions, a hybrid attention feature sequence formed by the results of these two feature extractions can be obtained. This approach makes speaker-specific feature extraction more targeted and fully considers global information when extracting features for speakers, thereby improving the overall accuracy of the extracted features.

[0047] After obtaining the attention feature sequence, as mentioned earlier, it needs to be expanded to facilitate subsequent processing. In one feasible approach, after obtaining the attention feature sequence, it can be expanded according to the number of speakers, resulting in multiple corresponding attention feature sequences. Feature extraction is then performed on each of these multiple attention feature sequences, and gating is applied to the resulting sequences. Based on the gating results, speaker speech feature sequences corresponding to each speaker are obtained. Feature extraction from multiple attention feature sequences allows for the extraction of more representative speaker speech features. Gating based on these more representative features effectively models the relationships between speaker speech features, making subsequent speech separation easier.

[0048] When speech separation is achieved using the aforementioned speech separation model, such as Figure 2B As shown, this step can be implemented by the separator of the speech separation model. This separator, through the above processing, performs separation modeling on the multi-channel speech feature sequence, completing the mapping from the multi-channel speech feature sequence to the speaker speech feature sequences of different speakers.

[0049] Step S208: Perform speech separation based on the speaker's speech feature sequence.

[0050] In one feasible approach, speech separation can be achieved by decoding the speech feature sequences corresponding to different speakers separately, and then performing speech separation on the multi-channel speech data based on the decoding results. By decoding the speech feature sequences corresponding to different speakers, speech data corresponding to different speakers can be output, effectively achieving speech separation.

[0051] When speech separation is achieved using the aforementioned speech separation model, such as Figure 2B As shown, this step can be implemented by the decoder of the speech separation model. The decoder decodes the individual speaker speech feature sequences corresponding to each speaker output by the separator to achieve speech reconstruction and output speech data of different speakers. In this process, the decoder parameters are shared during the decoding of speaker speech feature sequences of different speakers to improve decoding speed and efficiency, thereby improving the speed and efficiency of speech separation.

[0052] Furthermore, in this embodiment, when decoding the speaker's speech feature sequence, it is not necessary to combine it with the feature sequence output by the encoder. Instead, decoding and speech separation are performed directly based on the output of the separator. Since the speaker's speech feature sequences from different speakers output by the separator can sufficiently characterize the current speaker's features, to avoid introducing other features that interfere with the current speaker's features and to obtain better speech separation results for multi-channel speech data, the solution in this embodiment directly decodes based on the output of the separator to achieve speech separation.

[0053] The following example, using a meeting scenario, illustrates the above process.

[0054] In a meeting setting, there are typically multiple participants, i.e., multiple speakers, and the physical environment where the meeting takes place often contains noise and reverberation. In this example, we assume that meeting audio is captured using multiple microphones to form raw, multi-channel speech data, referred to as meeting audio data in this example. Furthermore, during the audio capture, information about the arrival time differences (OTDs) of the different speakers' voices is simultaneously generated. This meeting audio data, carrying the OTD information, can then be sent to a backend system for speech separation processing. Exemplarily, this can be sent to a backend speech separation device, such as a speech separation terminal, a speech separation server, or the cloud, for speech separation processing.

[0055] For example, such as Figure 2C As shown, after receiving the conference audio data, the speech separation device processes it into multi-channel audio data for speech separation. For example, it aligns the audio data from multiple channels contained in the conference audio data and then superimposes them to obtain multi-channel audio data for speech separation. Furthermore, this multi-channel audio data also carries information about the arrival time differences of the voices of multiple speakers.

[0056] Furthermore, the voice separation device will, through methods such as Figure 2B The speech separation model shown first extracts features from multi-channel speech data carrying arrival time difference information using its encoder to obtain corresponding multi-channel speech feature sequences. Then, its separator performs gated joint attention processing on the multi-channel speech feature sequences to obtain speaker speech feature sequences for each speaker. Finally, the decoder performs speech separation based on the speaker speech feature sequences to obtain conference speech data for each speaker. The specific implementation of the above operations of the speech separation model can be found in the process description described in this embodiment, and will not be repeated here.

[0057] After obtaining the speech separation results, meeting transcripts for each speaker can be generated from the speech data of multiple speakers. For example, speech recognition can be used to convert the speech data into text data.

[0058] This greatly facilitates meeting minutes, improves meeting minutes efficiency, and saves meeting minutes costs.

[0059] As can be seen from the above, this embodiment addresses multi-channel speech with noise and reverberation by, on the one hand, carrying speaker arrival time difference information into the multi-channel speech data to accurately locate the speaking positions of different speakers; on the other hand, based on the multi-channel speech data with arrival time difference information, gating-based joint attention processing is applied to the corresponding multi-channel speech feature sequences. This ensures that when obtaining the speaker speech feature sequence of a single speaker, the local feature information of the single speaker and its global feature information in the entire speech are fully considered, resulting in a more accurate feature representation for that single speaker (i.e., the speaker speech feature sequence of that single speaker); furthermore, after obtaining the speaker speech feature sequences corresponding to multiple speakers, compared to the related technologies that require decoding based on encoder output for speech separation, the solution in this embodiment eliminates the need for encoder output, directly decoding based on the speaker speech feature sequences to achieve speech separation. Through these methods, the solution in this embodiment not only achieves better speech separation for multi-channel speech data but is also simpler to implement, reducing implementation costs.

[0060] The following describes the speech separation scheme provided in the embodiments of this application, with reference to a specific implementation of a speech separation model.

[0061] Example 2

[0062] Reference Figure 3A The diagram shows a structural block diagram of a speech separation model product according to Embodiment 2 of this application.

[0063] The speech separation model in this embodiment includes: encoder 302, separator 304 and decoder 306.

[0064] in:

[0065] Encoder 302 is used to extract features from multi-channel speech data carrying information about the time difference of arrival of the speech of multiple speakers, and obtain the corresponding multi-channel speech feature sequence.

[0066] The separator 304 is used to perform gated joint attention processing on the multi-channel speech feature sequence to obtain the speaker speech feature sequences corresponding to multiple speakers respectively.

[0067] Decoder 306 is used for speech separation based on the speaker's speech feature sequence.

[0068] In this embodiment, to enable the speech separation model to effectively process multi-channel speech data, the original speech data is processed into data that can be processed by the speech separation model. This includes: acquiring the original speech data from multiple channels; and superimposing the speech data from multiple channels according to the input dimension of the encoder to obtain multi-channel speech data to be separated. This multi-channel speech data carries information about the time difference of arrival of the voices of multiple speakers. In this way, the speech data from multiple channels can be aligned, and the time difference of arrival of the voices can be used to represent the voice positions of different speakers, enabling the speech separation model to efficiently perform speech separation processing on multi-channel speech data.

[0069] The following example uses the specific structure of a speech separation model to illustrate the speech separation model product of this embodiment. Figure 3B As shown.

[0070] Figure 3B In the encoder 302, there are a one-dimensional convolution layer and a rectified linear unit (ReLU). The encoder 302 is used to extract features from multi-channel speech data carrying information on the time difference of arrival of speech from multiple speakers, and obtain the corresponding multi-channel speech feature sequence.

[0071] As mentioned earlier, in this embodiment, the speech data from multiple channels are first superimposed as the original input features of the speech separation model, i.e., the multi-channel speech data to be separated input to encoder 302. The number of channels equals the dimension of the input features, i.e., the dimension of the input features that encoder 302 can receive. Encoder 302 not only encodes the input multi-channel speech data into a high-dimensional representation, but also retains the arrival time difference information of the speaker's speech signals received by the multi-channel microphones, so as to provide effective speaker speech frame position information for the subsequent separator 304. The rectified linear unit of encoder 302 restricts the output encoded vector to non-negative values. After processing by the rectified linear unit, the encoded vector output by encoder 302 is a multi-channel speech feature sequence carrying information on the arrival time differences of the speech of multiple speakers.

[0072] The separator 304 is used to process the encoded vector output by the encoder 302, i.e., the multi-channel speech feature sequence, into speaker speech feature sequences of different speakers, thus completing the speech separation task at the feature level. For example... Figure 3BAs shown, after the multi-channel speech feature sequence enters the separator 304, it is first normalized to obtain a normalized multi-channel speech feature sequence. Then, for subsequent attention calculation, position embedding is added to the normalized multi-channel speech feature sequence. Figure 3B The diagram illustrates "sine positional encoding." Next, the normalized multi-channel speech feature sequence with added positional encoding is subjected to pointwise convolution and reshaping. Pointwise convolution allows for the effective fusion and extraction of feature information; while reshaping transforms the convolutional multi-channel speech feature sequence into a form suitable for attention calculation.

[0073] Based on this, the reshaped multi-channel speech feature sequence is passed to a gated local and global joint attention module for local and global feature extraction through attention computation. In this example, there are N gated local and global joint attention modules, meaning the multi-channel speech feature sequence undergoes N gated local and global joint attention processing steps. Here, N is a hyperparameter that can be appropriately set by those skilled in the art according to actual needs, to ensure that the module achieves good processing results for the multi-channel speech feature sequence. For example, N ≥ 3.

[0074] Among them, the gated local and global joint attention module is a module based on a gated single-head self-attention architecture with convolutional enhancement of joint self-attention, such as... Figure 3C As shown, it includes: four convolutional modules, one scaling, offset, and rotation position encoding module, one local and global joint attention module, and three gating modules (shown in the diagram). ). Here, "&" indicates a "and" relationship.

[0075] The reshaped multi-channel speech feature sequence is first input into the first convolutional module. This module performs linear projection and depthwise convolution on the multi-channel speech feature sequence before passing it to the scaling, offset, and rotation position encoding module. The scaling, offset, and rotation position encoding module performs scaling, offset, and position encoding operations on the multi-channel speech feature sequence processed by the first convolutional module to obtain Q and K, which can be used for subsequent local attention calculations, and Q' and K', which can be used for global attention calculations.

[0076] The second convolutional module also performs linear projection and depthwise convolution on the reshaped multi-channel speech feature sequence, outputting U, which can be used for subsequent local attention calculations. The third convolutional module also performs linear projection and depthwise convolution on the reshaped multi-channel speech feature sequence, outputting V, which can be used for subsequent local attention calculations. Furthermore, based on U and V, which can be used for subsequent local attention calculations, and Q' and K', which have already been obtained for global attention calculations, U' and V' for global attention calculations can be obtained. Further, corresponding local attention calculations and global attention calculations are performed based on U and V, and U' and V', respectively; and based on the attention calculation results of these two parts, the result of joint attention calculation can be obtained. The above attention calculation can be implemented as single-head self-attention calculation.

[0077] Based on this, such as Figure 3C As shown, the results of joint attention computation are processed using triple gating to effectively integrate the local features represented by the local attention computation results and the global features represented by the global attention computation results, thereby achieving long-distance modeling between features. It should be noted that... Figure 3C In the code, an activation function is set after the left-side gating, but it's not limited to this; the activation function can also be set after the right-side gating. This activation function reduces the impact of the result of the gating branch containing the activation function on the overall module output, resulting in more accurate output.

[0078] After triple gating, the multi-channel speech feature sequence is input into the fourth convolutional module for further feature extraction. The result of this feature extraction is then summed with the original reconstructed multi-channel speech feature sequence, and the summed attention calculation result is output.

[0079] As can be seen from the above, in this example, the gated local & global joint attention module only learns the residual part of the feature sequence and uses skip connections to connect the module's input to the output, thereby improving the module's feature processing efficiency.

[0080] As mentioned earlier, the gated local & global joint attention module consists of N modules. The output of the current gated local & global joint attention module will be passed to the next gated local & global joint attention module as input for further processing, until the last gated local & global joint attention module outputs the final result.

[0081] In one feasible approach, in the aforementioned gate-based local & global joint attention module, an exemplary structure of the first, second, third, and fourth convolutional modules is as follows: Figure 3D As shown. By Figure 3DAs can be seen, the convolutional module includes, in sequence: normalization layer, linear layer, activation layer, first reshaping layer, one-dimensional depth convolutional layer, second reshaping layer, and random deactivation (Dropout) layer.

[0082] For the feature sequences input from the first, second, third, or fourth convolutional modules, the following steps are performed: first, normalization is applied through a normalization layer; then, linear projection is applied through a linear layer; next, non-linear numerical processing is applied through an activation layer; then, a first reshaping layer reshapes the feature sequences processed by the activation layer into a convolutional feature sequence; a one-dimensional depthwise convolution layer performs feature convolution on the reshaped feature sequence; then, a second reshaping layer reshapes it into a feature sequence form suitable for random deactivation; finally, a random deactivation layer performs random deactivation on the feature sequence reshaped by the second reshaping layer, and then the result is output.

[0083] Through the processing of the above-mentioned gated local & global joint attention module, more representative multi-channel speech feature sequences can be extracted.

[0084] Furthermore, the output of the gated local & global joint attention module is reshaped into a feature sequence that satisfies the subsequent processing requirements and then passed to the next rectified linear unit (CLU). Next, a pointwise convolution is performed again to expand the dimension of the feature sequence to correspond to the number of speakers. Then, the result of the pointwise convolution is reshaped again, and multiple parallel pointwise convolution modules process the reshaped feature sequences separately to obtain speaker speech feature sequences for each speaker. Then, a gated CLU is used to gate each speaker's speech feature sequence to better model the relationship between the speaker speech feature sequences of multiple speakers. Afterward, the gated speaker speech feature sequence is processed again by pointwise convolution and CLU to obtain more accurate speaker speech feature sequences for each speaker. Finally, the multiple speaker speech feature sequences are input separately into the decoder for processing.

[0085] In this example, the decoder directly decodes the speaker speech feature sequence output by the separator and performs speech separation of multi-channel speech data based on the decoding result. For example, the decoder may include a one-dimensional transposed convolutional layer that uses the same kernel size and stride as the encoder to reconstruct speech data for each speaker based on the decoding result of their speech feature sequence, thus achieving speech separation for multiple speakers. The decoder's parameters are shared across all speaker speech feature sequences; that is, the decoder is a parameter-shared decoder. This improves the versatility of the entire speech sharing model, including the decoder, and reduces the number of parameters in both the decoder and the model.

[0086] Therefore, the speech separation model in this embodiment can effectively process multi-channel speech data. Furthermore, the decoder directly decodes based on the output of the separator, transforming the separator from an estimation masking approach to one that estimates speaker speech feature components. This bypasses the sound source localization algorithm and allows for speech separation in complex scenarios with noise and reverberation. Moreover, by utilizing the time-of-arrival information carried in the multi-channel speech data, combined with gated joint attention processing, the interference of noise and reverberation on separation performance in complex scenarios is reduced, significantly improving the overall separation performance of the speech separation model.

[0087] Example 3

[0088] Reference Figure 4 The diagram shows a structural schematic of an electronic device according to Embodiment 3 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0089] like Figure 4 As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0090] in:

[0091] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0092] Communication interface 404 is used to communicate with other electronic devices or servers.

[0093] The processor 402 is used to execute program 410, which can specifically execute the relevant steps in the above-described speech separation method embodiment.

[0094] Specifically, program 410 may include program code that includes computer operation instructions.

[0095] Processor 402 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0096] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0097] Program 410 may include multiple computer instructions. Specifically, program 410 may use multiple computer instructions to cause processor 402 to perform the operations corresponding to the speech separation method described in the foregoing method embodiments.

[0098] The specific implementation of each step in procedure 410 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0099] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0100] This application also provides a computer program product, including computer instructions that instruct a computing device to perform the operation corresponding to the speech separation method in the above method embodiments.

[0101] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, voice data used for processing, data used for analysis, stored data, and displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0102] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0103] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded over a network. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0104] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0105] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A speech separation method, comprising: Acquire multi-channel speech data to be separated, wherein the multi-channel speech data carries information on the arrival time difference of the speech of multiple speakers. The multi-channel speech data is obtained by aligning the original speech data of multiple channels and then vertically superimposing the speech data of multiple channels in the same time period. Feature extraction is performed on the multi-channel speech data carrying the information of the time difference of arrival to obtain the corresponding multi-channel speech feature sequence; The multi-channel speech feature sequence is subjected to gated joint attention processing to obtain an attention feature sequence; Based on the number of speakers and the attention feature sequence, speaker speech feature sequences are obtained for each of the multiple speakers; Decode the speaker's speech feature sequence; Based on the decoding results of the speaker's speech feature sequence, speech separation is performed on the multi-channel speech data.

2. The method according to claim 1, wherein, The acquisition of multi-channel speech data to be separated includes: Acquire raw voice data from multiple channels; The speech data from the multiple channels are superimposed to obtain the multi-channel speech data to be separated.

3. The method according to claim 1, wherein, The step of obtaining the speaker speech feature sequences corresponding to each of the multiple speakers based on the number of speakers and attention feature sequences includes: The attention feature sequence is expanded according to the number of speakers to obtain the speaker speech feature sequence corresponding to each of the multiple speakers.

4. The method according to claim 3, wherein, The step of expanding the attention feature sequence according to the number of speakers to obtain speaker speech feature sequences corresponding to each of the multiple speakers includes: Based on the number of speakers, the attention feature sequence is expanded to obtain multiple corresponding attention feature sequences; Feature extraction is performed on the multiple attention feature sequences respectively, and gating processing is performed on the corresponding multiple feature sequences obtained after feature extraction; Based on the gating processing results, speaker speech feature sequences corresponding to the multiple speakers are obtained respectively.

5. The method according to claim 1, wherein, The step of performing gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence includes: Attention calculation is used to perform local and global feature extraction of the multi-channel speech data based on gating on the multi-channel speech feature sequence. Based on the results of the local feature extraction and the global feature extraction, a hybrid attention feature sequence is obtained.

6. A speech separation method, comprising: Multi-channel audio data to be separated is generated based on the conference audio data. The multi-channel audio data carries information on the arrival time difference of the audio of multiple speakers. The multi-channel audio data is obtained by aligning the original audio data of multiple channels and then vertically superimposing the audio data of multiple channels in the same time period. Feature extraction is performed on the multi-channel speech data carrying the information of the time difference of arrival to obtain the corresponding multi-channel speech feature sequence; The multi-channel speech feature sequence is subjected to gated joint attention processing to obtain an attention feature sequence; Based on the number of speakers and the attention feature sequence, speaker speech feature sequences are obtained for each of the multiple speakers; Decode the speaker's speech feature sequence; Based on the decoding results of the speaker's speech feature sequence, speech separation is performed on the multi-channel speech data to obtain the conference speech data corresponding to the multiple speakers respectively.

7. The method according to claim 6, wherein, The method further includes: Based on the conference audio data corresponding to the multiple speakers, generate conference transcript text corresponding to each of the multiple speakers.

8. A speech separation model product, comprising: Encoder, splitter, and decoder; in: The encoder is used to extract features from multi-channel speech data carrying information on the time difference of arrival of the speech of multiple speakers, and obtain the corresponding multi-channel speech feature sequence. The multi-channel speech data is obtained by aligning the original speech data of multiple channels and then vertically superimposing the speech data of the multiple channels in the same time period. The separator is used to perform gated joint attention processing on the multi-channel speech feature sequence to obtain an attention feature sequence; and to obtain the speaker speech feature sequences corresponding to the multiple speakers based on the number of speakers and the attention feature sequence. The decoder is used to decode the speaker's speech feature sequence; and based on the decoding result of the speaker's speech feature sequence, to perform speech separation of the multi-channel speech data.

9. The product according to claim 8, wherein, The decoder is used to directly decode the speaker's speech feature sequence output by the separator, and to perform speech separation of the multi-channel speech data based on the decoding result.

10. The product according to claim 8 or 9, wherein, The multi-channel voice data input to the encoder is obtained in the following manner: Acquire raw voice data from multiple channels; According to the input dimension of the encoder, the speech data of the multiple channels are superimposed to obtain multi-channel speech data to be separated.

11. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-7.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-7.

13. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-7.

Images