Neural guidance type speech enhancement method based on personalized brain electrode distribution

Abstract

The invention provides a nerve guidance type speech enhancement method based on personalized brain electrode distribution, and the training method of a speech enhancement model in the speech enhancement method comprises the steps: obtaining a training set which comprises a plurality of training sample combinations corresponding to at least one training user, the training sample combination comprises real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information and an enhanced voice tag of the training user, and the auxiliary information represents the attention degree of the training user to a certain recognition user in the mixed training voice information; performing adversarial training on the plurality of initial enhancement models by using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information and the mixed training voice information to obtain a plurality of pieces of enhanced voice information; generating an initial loss value according to the enhanced voice information and the enhanced voice tag; and iteratively adjusting network parameters of the initial enhancement model according to the initial loss value to obtain a speech enhancement model.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and more specifically, to a training method for a neural-guided speech enhancement model based on personalized brain electrode distribution, a speech enhancement method, a training device, a speech enhancement device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Speech enhancement (SE) aims to extract the target speaker's speech signal from various audio signals mixed with noise interference. It is widely used in hearing aids (HAs) and is the core algorithm of hearing aids. When the noise comes from other speakers, it is also called target speaker extraction (TSE).

[0003] The usual TSE method relies on auxiliary information containing information about the target speaker. The main auxiliary information includes pre-registered audio, visual assistance, spatial information assistance, etc. However, it is difficult to obtain this auxiliary information in actual application scenarios. In recent years, with the development of brain science and the human auditory system, researchers have found that auditory attention information can be extracted from the listener's biological auxiliary information, such as electroencephalogram (EEG), such as the target speech envelope. This has prompted the use of EEG signals to assist TSE. Initially, researchers first extracted and separated the envelope, and then extracted the target speech based on similarity or non-end-to-end solutions. This does not achieve an ideal effect, and there is no need to separate first. To overcome these problems, other researchers have proposed a better end-to-end model that uses a special feature fusion module (FiLM, feature-wise linear modulation).

[0004] In the actual application of brain-assisted speech enhancement, too many electrodes will not significantly improve the performance, but may make the cost too high. This has led to a large number of applications for electrode channel selection in this field. For example, a related technology proposed an end-to-end EEG channel selection method based on a special distribution (Gumbel-Softmax Distribution). However, it has the problem of difficult training and repeated channels. Another related technology adds weighted residual connections to improve the stability of training, but still does not solve the problem of repeated channel selection. Another related technology proposes a new solution based on attention and special constraint functions to achieve a solution that does not contain repeated channels and does not require explicit specification of the number of selected channels. However, the current methods do not pay attention to the problem of personalized selection, and the selection results are unchanged for all users, resulting in a large variance problem. Summary of the invention

[0005] In view of this, the present application provides a training method, a speech enhancement method, a training device, a speech enhancement device, an electronic device, a computer-readable storage medium and a computer program product for a neural-guided speech enhancement model based on personalized brain electrode distribution.

[0006] One aspect of the present application provides a training method for a neural-guided speech enhancement model based on personalized brain electrode distribution, comprising:

[0007] In response to the model training instruction, a training set is obtained, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations include real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information and enhanced voice labels of the training user, any one of the auxiliary information includes EEG information representing the degree of attention of the training user to a certain identified user in the mixed training voice information;

[0008] For each of the above training sample combinations, multiple initial enhancement models are adversarially trained using the above real biological auxiliary information, the above mixed distribution auxiliary information, the above Gaussian distribution auxiliary information and the above mixed training voice information to obtain multiple enhanced voice information;

[0009] For each of the enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced language label;

[0010] According to the above-mentioned multiple initial loss values, the network parameters of the above-mentioned initial enhancement model are iteratively adjusted to obtain a trained speech enhancement model.

[0011] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using the above-mentioned real biological auxiliary information, the above-mentioned mixed distribution auxiliary information, the above-mentioned Gaussian distribution auxiliary information and the above-mentioned mixed training voice information to obtain multiple enhanced voice information, including:

[0012] Using the real biological auxiliary information and the mixed training voice information to train the first initial enhancement model, to obtain the first enhanced voice information;

[0013] Using the mixed distribution auxiliary information and the mixed training speech information to train the second initial enhancement model, to obtain the second enhanced speech information;

[0014] Using the Gaussian distribution auxiliary information and the mixed training voice information to train the third initial enhancement model, to obtain the third enhanced voice information;

[0015] The fourth initial enhanced model is trained using the mixed training voice information to obtain the fourth enhanced voice information.

[0016] According to an embodiment of the present application, generating the enhanced speech information using any of the above-mentioned initial enhancement models includes:

[0017] Processing the target auxiliary information using a target selection module to obtain a biological feature vector, wherein the target auxiliary information includes the real biological auxiliary information, the mixed distribution auxiliary information or the Gaussian distribution auxiliary information;

[0018] The biometric feature vector, the mixed training speech information and the target auxiliary information are processed by a speech enhancement module to generate the enhanced speech information.

[0019] According to an embodiment of the present application, the target selection module is used to process the target auxiliary information to obtain a biometric feature vector, including:

[0020] Using multiple depth separation convolution units to process the target auxiliary information, multiple biological convolution features are obtained;

[0021] Processing multiple biological convolutional features using a linear layer to obtain biological linear features;

[0022] The above biological feature vector is generated according to the above biological linear features and the training parameters of the adaptive neuron.

[0023] According to an embodiment of the present application, multiple depth separation convolution units are used to process the above target auxiliary information to obtain multiple biological convolution features, including:

[0024] For each of the above-mentioned depth separation convolution units, perform depth separation convolution processing on the above-mentioned target auxiliary information to obtain a first convolution feature;

[0025] Perform padding processing on the first convolution feature to obtain a second convolution feature;

[0026] The second convolution feature is pooled to obtain the biological convolution feature.

[0027] According to an embodiment of the present application, the above-mentioned biological feature vector, the above-mentioned mixed training voice information and the above-mentioned target auxiliary information are processed by a voice enhancement module to generate the above-mentioned enhanced voice information, including:

[0028] Generate a biometric feature to be processed according to the biometric feature vector and the target auxiliary information;

[0029] The above-mentioned biological features to be processed and the above-mentioned mixed training voice information are processed using a convolutional time-domain separation network to obtain the above-mentioned enhanced voice information.

[0030] According to an embodiment of the present application, the biometric feature to be processed and the mixed training voice information are processed using a convolutional time-domain separation network to obtain the enhanced voice information, including:

[0031] Encoding the above-mentioned biological features to be processed and the above-mentioned mixed training voice information respectively to obtain biological coding features and language coding features;

[0032] Performing sound source separation processing on the above-mentioned biological coding features and language coding features to obtain speech mask information;

[0033] Generate target coding information according to the speech mask information and the language coding features;

[0034] The target coded information is decoded to obtain the enhanced speech information.

[0035] According to an embodiment of the present application, for each of the enhanced speech information, an initial loss value is generated according to the enhanced speech information and the enhanced language label, including:

[0036] For the first enhanced speech information, generating a signal distortion loss value according to the enhanced speech information and the enhanced language label;

[0037] For the second enhanced speech information, generating a first confusion loss value according to the enhanced speech information and the enhanced language label;

[0038] For the third enhanced speech information, generating a second confusion loss value according to the enhanced speech information and the enhanced language label;

[0039] For the fourth enhanced speech information, a third confusion loss value is generated according to the enhanced speech information and the enhanced language label.

[0040] According to an embodiment of the present application, according to the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained speech enhancement model, including:

[0041] Generate a first loss value according to the signal distortion loss value and the first confusion loss value;

[0042] Generate a second loss value based on the first loss value and the selected mean square error loss;

[0043] Generate a target loss value according to the second loss value, the second confusion loss value and the third confusion loss value;

[0044] The network parameters of the initial enhancement model are iteratively adjusted according to the target loss value to obtain the speech enhancement model.

[0045] Another aspect of the present application provides a neural-guided speech enhancement method based on personalized brain electrode distribution, comprising:

[0046] In response to the model training instruction, obtaining biological auxiliary data and mixed voice data of a target user, wherein the biological auxiliary data includes electroencephalogram information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0047] The above-mentioned biological auxiliary data and mixed voice data are input into the voice enhancement model, and enhanced voice data is output, wherein the voice of the above-mentioned user of interest in the above-mentioned enhanced voice data is enhanced.

[0048] Another aspect of the present application provides a training device for a neural-guided speech enhancement model based on personalized brain electrode distribution, comprising:

[0049] A first acquisition module is configured to acquire a training set in response to a model training instruction, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations include real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice labels of the training user, and any one of the auxiliary information includes EEG information representing the degree of attention of the training user to a certain identified user in the mixed training voice information;

[0050] An adversarial training module is used to perform adversarial training on multiple initial enhancement models for each combination of the training samples using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information and the mixed training voice information to obtain multiple enhanced voice information;

[0051] A generating module, configured to generate an initial loss value for each enhanced speech information according to the enhanced speech information and the enhanced language label;

[0052] The adjustment module is used to iteratively adjust the network parameters of the initial enhancement model according to the multiple initial loss values ​​to obtain a trained speech enhancement model.

[0053] Another aspect of the present application provides a neural-guided speech enhancement device based on personalized brain electrode distribution, comprising:

[0054] A second acquisition module is used to acquire biological auxiliary data and mixed voice data of a target user in response to a model training instruction, wherein the biological auxiliary data includes electroencephalogram information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0055] The enhancement module is used to input the above-mentioned biological auxiliary data and mixed voice data into the voice enhancement model, and output enhanced voice data, wherein the voice of the above-mentioned user of interest in the above-mentioned enhanced voice data is enhanced.

[0056] Another aspect of the present application provides an electronic device, comprising:

[0057] one or more processors;

[0058] a memory for storing one or more programs,

[0059] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0060] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0061] Another aspect of the present application provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the method as described above when executed.

[0062] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information, generate initial loss values ​​based on the enhanced voice information and enhanced language labels, and iteratively adjust the network parameters of the initial enhancement model based on the multiple initial loss values ​​to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0064] Figure 1 An exemplary system architecture to which a training method for a speech enhancement model or a speech enhancement method can be applied according to an embodiment of the present application is shown;

[0065] Figure 2 A flow chart of a method for training a speech enhancement model according to an embodiment of the present application is shown;

[0066] Figure 3 A flow chart of a method for training a speech enhancement model according to another embodiment of the present application is shown;

[0067] Figure 4 A flow chart of a method for generating enhanced voice information according to an embodiment of the present application is shown;

[0068] Figure 5 A processing flow chart of a target selection module according to an embodiment of the present application is shown;

[0069] Figure 6 A schematic diagram showing experimental comparative analysis results of the speech enhancement model of the present application and a model without adversarial training according to an embodiment of the present application is shown;

[0070] Figure 7 A flow chart of a method for training a speech enhancement model according to an embodiment of the present application is shown;

[0071] Figure 8 A block diagram of a training device for a speech enhancement model according to an embodiment of the present application is shown;

[0072] Fig. 9 A block diagram of a speech enhancement device according to an embodiment of the present application is shown; and

[0073] Fig.10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0074] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.

[0075] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0076] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0077] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0078] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0079] In the embodiments of the present application, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0080] Figure 1 An exemplary system architecture 100 to which a training method of a speech enhancement model or a speech enhancement method according to an embodiment of the present application can be applied is shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0081] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0082] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0083] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0084] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0085] It should be noted that the training method of the speech enhancement model and the speech enhancement method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the training device of the speech enhancement model and the speech enhancement device provided in the embodiment of the present application can generally be arranged in the server 105. The training method of the speech enhancement model and the speech enhancement method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the training device of the speech enhancement model and the speech enhancement device provided in the embodiment of the present application can also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the speech enhancement model training method and the speech enhancement method provided in the embodiment of the present application may also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the speech enhancement model training device and the speech enhancement device provided in the embodiment of the present application may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0086] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0087] Figure 2 A flow chart of a method for training a speech enhancement model according to an embodiment of the present application is shown.

[0088] like Figure 2 As shown, the training method of the neural-guided speech enhancement model based on personalized brain electrode distribution includes operations S201~S204.

[0089] In operation S201, in response to a model training instruction, a training set is obtained, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations include real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice tags of the training user, and any one of the auxiliary information includes EEG information representing the degree of attention of the training user to a certain identified user in the mixed training voice information;

[0090] In operation S202, for each training sample combination, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information;

[0091] In operation S203, for each enhanced speech information, an initial loss value is generated according to the enhanced speech information and the enhanced language label;

[0092] In operation S204, network parameters of the initial enhancement model are iteratively adjusted according to the multiple initial loss values ​​to obtain a trained speech enhancement model.

[0093] According to an embodiment of the present application, EEG information refers to electroencephalogram (EEG) auxiliary information, wherein the EEG information can also be replaced by electromyography (EMG) auxiliary information, electrocardiogram (ECG) auxiliary information, electrooculography (EOG) auxiliary information, etc. The present application uses EEG auxiliary information for illustrative purposes. For example, when training users to attend a meeting, the users are trained to pay attention to what speaker A says. At this time, the EEG signal of the training user can be collected as real biological auxiliary information, and at the same time, the voices of other speakers in the meeting place can be collected as mixed training voice information.

[0094] According to the embodiments of the present application, Gaussian distribution auxiliary information refers to auxiliary information obtained by performing Gaussian distribution processing on real biological auxiliary information. Mixed distribution auxiliary information refers to auxiliary information obtained by performing Gaussian distribution and uniform distribution processing on real biological auxiliary information. Enhanced speech label refers to using enhanced speech as a label.

[0095] According to an embodiment of the present application, for each training sample combination, multiple initial enhancement models are simultaneously trained adversarially using the real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training speech information, and enhanced speech labels in the combination, and each initial enhancement model will output an enhanced speech information. Based on the enhanced speech information and enhanced language labels, an initial loss value corresponding to the initial enhancement model can be generated, and the network parameters of the multiple initial enhancement models can be adjusted synchronously according to the initial loss values ​​of the multiple initial enhancement models to obtain a trained speech enhancement model.

[0096] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information, generate initial loss values ​​based on the enhanced voice information and enhanced language labels, and iteratively adjust the network parameters of the initial enhancement model based on the multiple initial loss values ​​to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0097] Figure 3 A flowchart of a method for training a speech enhancement model according to another embodiment of the present application is shown.

[0098] like Figure 3 As shown, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training speech information to obtain multiple enhanced speech information, including:

[0099] Using real biological auxiliary information and mixed training voice messages Training the first initial enhancement model to obtain the first enhanced speech information;

[0100] Utilizing mixed distribution auxiliary information and mixed training voice messages Training the second initial enhanced model to obtain second enhanced speech information;

[0101] Using Gaussian distribution auxiliary information and mixed training voice messages The third initial enhancement model is trained to obtain third enhanced speech information;

[0102] Using mixed training speech information The fourth initial enhanced model is trained to obtain fourth enhanced speech information.

[0103] According to the embodiments of the present application, the present application solves the problem of excessive model memory through multi-process adversarial training to prevent the model from remembering the target speaker. Figure 3 As shown, multiple branch models (i.e., initial enhanced models) share parameters, and three new branches replace the original EEG signals with various invalid signals (such as Figure 2 In , , no auxiliary information ) to simulate the situation where the model may memorize the target speaker, and suppress this with the confusion loss. The confusion loss is the square of SI-SDR.

[0104] Figure 4 A flow chart of a method for generating enhanced voice information according to an embodiment of the present application is shown.

[0105] According to the embodiments of the present application, Figure 4 As shown, any initial enhancement model is used to generate enhanced speech information, including:

[0106] Processing the target auxiliary information using a target selection module to obtain a biological feature vector, wherein the target auxiliary information includes real biological auxiliary information, mixed distribution auxiliary information or Gaussian distribution auxiliary information;

[0107] The speech enhancement module is used to process the biometric feature vector, the mixed training speech information and the target auxiliary information to generate enhanced speech information.

[0108] According to the embodiments of the present application, the biologically assisted speech enhancement model proposed in the present application mainly includes two parts, a personalized target selection module and a subsequent speech enhancement module.

[0109] According to an embodiment of the present application, after the target auxiliary information is input into the initial enhancement model, the target selection module in the initial enhancement model first converts the target auxiliary information into a biometric feature vector, which contains the training user's attention information. Thereafter, the speech enhancement module processes the biometric feature vector, the mixed training speech information and the target auxiliary information to obtain enhanced speech information, i.e. Figure 4 Enhanced speech in .

[0110] Figure 5 A processing flow chart of a target selection module according to an embodiment of the present application is shown.

[0111] According to the embodiments of the present application, Figure 5 As shown, the target selection module is used to process the target auxiliary information to obtain the biometric feature vector, including:

[0112] Multiple deep separation convolution units are used to process target auxiliary information to obtain multiple biological convolution features;

[0113] Use linear layers to process multiple biological convolutional features to obtain biological linear features;

[0114] A biological feature vector is generated according to the biological linear features and the training parameters of the adaptive neurons.

[0115] According to an embodiment of the present application, Figure 5 In That is, the training parameters of the adaptive neuron, is a biological linear feature, That is, the biometric feature vector.

[0116] According to an embodiment of the present application, in order to capture individual differences, the model introduces adaptive neurons. These neurons generate individual difference vectors through training, which are combined with the universal selection vector obtained by the target selection module during pre-training to form the final individual adaptive EEG channel selection vector. The output of the adaptive neuron is limited to the range of [-1, 1] by the activation function to ensure that the captured individual differences are appropriate and controllable. Among them, the universal selection vector is generated by pre-training a universal selection model (i.e., the initial enhancement model) that does not rely on individual differences on a universal data set, and freezing its universal selection layer (i.e., the target selection module). In order to keep the output of the model stable when different signals are input, the input of the universal selection layer is replaced by an all-one vector to ensure that the module is independent of the input. In this way, the pre-trained model can output a stable universal selection vector, called "average fitness", which represents the most informative EEG signal channel among multiple subjects.

[0117] According to an embodiment of the present application, the training process is divided into two stages, the first stage is individual difference learning, and the second stage is selection fine-tuning.

[0118] According to the embodiments of the present application, Figure 5 (a) In the first stage, the model learns the individual differences of each subject through adaptive neurons and combines it with the average fitness to generate an individually adaptive EEG channel selection vector (i.e., biometric feature vector). In this process, the model optimizes a loss function that includes speech enhancement-related losses and regularization terms to ensure that the model can accurately capture individual differences while maintaining a certain channel selection stability. The generated EEG channel selection vector is used to construct an EEG signal subset for each subject and is gradually optimized in multiple iterations until the loss function converges.

[0119] According to the embodiments of the present application, Figure 5 (b) The second stage is the fine-tuning stage, where the model freezes the adaptive neurons and further obtains the final adaptive EEG channel selection scheme through a threshold judge. This threshold judge will decide whether to select the corresponding channel based on the value of the individual difference vector, and finally form an EEG channel selection scheme for each subject. At this stage, the goal of the model is to further optimize the target selection module by combining the convolutional time domain separation network BASEN to improve the performance of the speech enhancement model.

[0120] According to the embodiments of the present application, Figure 5As shown in the figure, multiple deep separation convolution units are used to process the target auxiliary information to obtain multiple biological convolution features, including:

[0121] For each depth separation convolution unit, perform depth separation convolution processing on the target auxiliary information to obtain the first convolution feature;

[0122] Perform padding processing on the first convolution feature to obtain the second convolution feature;

[0123] The second convolutional feature is pooled to obtain the biological convolutional feature.

[0124] According to an embodiment of the present application, the number of depth separation convolution units can be specifically set according to requirements, for example, it can be 8. Figure 5 The sampling vector in is the target auxiliary information.

[0125] According to an embodiment of the present application, a speech enhancement module is used to process a biometric feature vector, mixed training speech information and target auxiliary information to generate enhanced speech information, including:

[0126] Generate a biometric feature to be processed according to the biometric feature vector and target auxiliary information;

[0127] The convolutional time-domain separation network is used to process the biometric features to be processed and the mixed training speech information to obtain enhanced speech information.

[0128] According to an embodiment of the present application, the biometric feature vector The target auxiliary information y(t) is multiplied to generate the biometric feature to be processed, and the biometric feature to be processed and the mixed training speech information x(t) are input into the convolutional time domain separation network BASEN to obtain the enhanced speech information, that is, Figure 5 In Among them, the convolutional time domain separation network BASEN is the Conv-TasNet network.

[0129] According to the embodiments of the present application, referring to Figure 4 , using the convolutional time-domain separation network to process the biometric features to be processed and the mixed training voice information to obtain enhanced voice information, including:

[0130] Encode the biometric features to be processed and the mixed training voice information respectively to obtain the biometric coding features and language encoding features ;

[0131] Encoding characteristics of organisms and language encoding features Perform sound source separation to obtain speech mask information ;

[0132] According to the voice mask information and language encoding features , generate target coding information;

[0133] The target coded information is decoded to obtain enhanced speech information.

[0134] According to an embodiment of the present application, the encoder of the mixed training speech information extracts the embedded features of the audio through multiple layers of one-dimensional convolution, and converts the input audio signal into a representation that can be better processed.

[0135] According to an embodiment of the present application, the encoder of the biometric feature to be processed downsamples the biometric feature to be processed, such as the EEG signal, and extracts multi-level features through a deep convolutional layer to capture auditory attention information in the brain signal.

[0136] According to an embodiment of the present application, the role of the separator used in sound source separation is to combine the embedded features of audio and EEG signals to predict the speech mask of the target speaker. This process relies on deep convolutional layers and cross-layer attention mechanisms to achieve deep fusion of audio and EEG features.

[0137] According to an embodiment of the present application, a decoder for decoding target encoded information uses the mask generated by the separator in combination with the audio features to reconstruct the speech signal of the target speaker. The overall architecture achieves the separation and reconstruction tasks of the speech signal by effectively fusing multimodal information.

[0138] According to an embodiment of the present application, for each enhanced speech information, an initial loss value is generated according to the enhanced speech information and the enhanced language label, including:

[0139] For the first enhanced voice information, according to the enhanced voice information and enhanced language tags to generate signal distortion loss values ​​SI-SDR;

[0140] For the second enhanced voice information, according to the enhanced voice information and enhanced language labels, generating a first confusion loss value;

[0141] For the third enhanced voice information, according to the enhanced voice information and enhanced language labels to generate a second confusion loss value;

[0142] For the fourth enhanced voice information, according to the enhanced voice information and enhanced language labels to generate a third confusion loss value.

[0143] According to an embodiment of the present application, according to a plurality of initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained speech enhancement model, including:

[0144] Generate a first loss value according to the signal distortion loss value and the first confusion loss value;

[0145] Generate a second loss value based on the first loss value and the selected mean square error loss;

[0146] Generate a target loss value according to the second loss value, the second confusion loss value, and the third confusion loss value;

[0147] The network parameters of the initial enhancement model are iteratively adjusted according to the target loss value to obtain a speech enhancement model.

[0148] According to an embodiment of the present application, a special constraint function is used to train the target selection layer:

[0149]

[0150]

[0151] in The loss function is SI-SDR, The loss function is the selection vector The square of the loss, that is, the mean square error loss, The loss function is the square of the SI-SDR output by the three adversarial branches. This loss function can effectively suppress the phenomenon of the model remembering the target speaker.

[0152] In a specific embodiment, the present application takes into account the problem of electrode channel redundancy in the field of brain-assisted speech enhancement in related technologies. More electrodes do not have a significant impact on performance, but will cause excessive overhead and loss, and the existing electrode channel selection method cannot effectively achieve personalized selection, resulting in a large final enhancement variance. Therefore, the present application designs a new speech enhancement method and a personalized speech enhancement model to achieve biologically assisted speech enhancement with personalized electrode distribution, and uses a new adversarial training method to solve the problem of abnormal model memory. The details are as follows:

[0153] (1) Experimental setup

[0154] In the performance evaluation, three objective indicators are used to measure the overall quality of the enhanced speech signal, including SI-SDR (dB) for evaluating the signal-to-distortion ratio, perceptual evaluation of speech quality (PESQ), and short-term objective intelligibility (STOI). These indicators can comprehensively evaluate the quality, intelligibility, and clarity of the enhanced speech. This application also designs an indicator to measure the abnormal memory of the model, namely excess memory (OM).

[0155] A general dataset of 26 subjects was used for pre-training of the general selection model. The Adam optimizer was used for training, and the learning rate was set to 0.0001. At the same time, in order to optimize the deep neural network (DNN) weights, the exponential moving average method was used, and the decay rate was set to 0.999. In order to obtain the corresponding EEG channel selection, the regularization term weight γ was set to 0.05. The total training cycle of pre-training was 60 rounds, and the batch size was set to 8.

[0156] In the individual adaptive training, 1000 rounds of individual variation learning were first performed, followed by 1000 rounds of selection fine-tuning. During the selection fine-tuning, the model training was stopped at the appropriate time point by monitoring the loss function of the validation set and setting an early stopping mechanism of 10 rounds. All models converged before reaching the maximum number of training rounds. Other settings of the individual adaptive training were consistent with the pre-training stage.

[0157] The training set used in this application evaluation contains 33 subjects (28 males and 5 females), with an average age of 27.3±3.2 years. All subjects are native English speakers with normal hearing and no history of neurological diseases. Due to the poor data quality of the sixth subject, his recorded data was excluded.

[0158] Each subject participated in 30 trials, each trial lasting 60 seconds. The audio stimulus material was a different story read by two men. In each trial, one story was played through the left ear and the other story was played through the right ear. Half of the subjects were asked to focus on the story in the left ear (left-side attention), and the rest of the subjects (including the excluded subjects) focused on the story in the right ear (right-side attention). After each trial, the subjects were asked to answer multiple-choice questions to confirm that their attention to the instructed story was effective. To ensure the continuity of the story, the story in each trial continued the playback progress of the previous trial. To reduce other interfering EEG signals, the subjects were asked to focus their vision on a crosshair in the center of the screen.

[0159] During the experiment, the subjects wore an EEG cap with 128 channels (plus two mastoid electrodes) and recorded EEG signals at a sampling rate of 512 Hz using the BioSemi ActiveTwo system. To be consistent with previous studies, the EEG data were downsampled to 128 Hz. The audio stimuli were played through Sennheiser HD650 headphones with a sampling rate of 44.1kHz. To avoid the influence of audio intensity on attention, the RMS amplitude of all audio stimuli was normalized.

[0160] Both audio and EEG data are processed to 14.7KHZ, and mixed noise audio is generated by mixing the focused audio and the unfocused audio with equal intensity. This application divides the training set into three groups: 5 experiments are randomly selected from all subjects for testing, 2 experiments are selected for verification, and the remaining experiments are used for training. For the training set and the verification set, each experiment is cut into 2-second segments; for the test set, each 60-second experiment is cut into 20-second segments. Since individual adaptive training is required, we further divide the data set according to the subjects. Three subjects were randomly selected from the left attention group and the right attention group for individual adaptive training. The remaining 26 subjects were used for general training of the model. In the individual adaptive training of these 6 subjects, the data of 26 subjects were inaccessible to ensure that the individual adaptive training was independent of the general training data.

[0161] (2) Experimental results

[0162] Figure 6 In order to compare and analyze the experimental results of the speech enhancement model of the present application (the "adversarial training option" in the figure) and the model without adversarial training, the experimental results are displayed in the form of a violin plot, which can well reflect the distribution and range of the experimental results. It can be seen from the figure that the speech enhancement model of the present application has well realized the brain-assisted speech enhancement of personalized electrode distribution, and its performance is comparable to that of the full-channel brain-assisted speech enhancement model and is due to the mainstream model of the full channel UBESD. After the introduction of adversarial training, the model of the present application can still maintain its performance advantage, and the present application has greatly reduced the variance.

[0163] According to the embodiments of the present application, in Table 1, the present application evaluated the branches of adversarial training and fully verified the performance of the model of the present application. The combination of its three adversarial branches effectively reduced the abnormal memory phenomenon and remembered the target speaker.

[0164] Table 1. Evaluation of three adversarial training branches

[0165]

[0166] Figure 7 A flow chart of a method for training a speech enhancement model according to an embodiment of the present application is shown.

[0167] like Figure 7 As shown, the neural-guided speech enhancement method based on personalized brain electrode distribution includes S701~S702.

[0168] In operation S701, in response to a model training instruction, biological auxiliary data and mixed voice data of a target user are acquired, wherein the biological auxiliary data includes electroencephalogram information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0169] In operation S702, the biological auxiliary data and the mixed voice data are input to a voice enhancement model, and enhanced voice data is output, wherein the voice of the user of interest in the enhanced voice data is enhanced.

[0170] According to an embodiment of the present application, in a scenario where the target user is speaking with multiple people, bio-auxiliary data, such as EEG, can be detected in real time through electrodes and other devices, and the speech enhancement model can process the bio-auxiliary data and mixed speech data in real time. When paying attention to a certain speaker in the bio-auxiliary data of the target user, the speech enhancement model can enhance the speech of the speaker in the mixed speech data, thereby outputting the enhanced speech data of the focused user to the target user.

[0171] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information, generate initial loss values ​​based on the enhanced voice information and enhanced language labels, and iteratively adjust the network parameters of the initial enhancement model based on the multiple initial loss values ​​to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0172] Figure 8 A block diagram of a training device for a speech enhancement model according to an embodiment of the present application is shown.

[0173] like Figure 8 As shown, the training device 800 of the neural-guided speech enhancement model based on personalized brain electrode distribution includes a first acquisition module 810, an adversarial training module 820, a generation module 830, and an adjustment module 840.

[0174] A first acquisition module 810 is used to acquire a training set in response to a model training instruction, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations include real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information and enhanced voice tags of the training user, and any one of the auxiliary information includes EEG information representing the degree of attention of the training user to a certain identified user in the mixed training voice information;

[0175] An adversarial training module 820 is used to perform adversarial training on multiple initial enhancement models for each training sample combination using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information and mixed training voice information to obtain multiple enhanced voice information;

[0176] A generating module 830, for generating an initial loss value for each enhanced speech information according to the enhanced speech information and the enhanced language label;

[0177] The adjustment module 840 is used to iteratively adjust the network parameters of the initial enhancement model according to multiple initial loss values ​​to obtain a trained speech enhancement model.

[0178] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information, generate initial loss values ​​based on the enhanced voice information and enhanced language labels, and iteratively adjust the network parameters of the initial enhancement model based on the multiple initial loss values ​​to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0179] Fig. 9 A block diagram of a speech enhancement device according to an embodiment of the present application is shown.

[0180] like Fig. 9 As shown, the neural-guided speech enhancement device 900 based on personalized brain electrode distribution includes a second acquisition module 910 and an enhancement module 920.

[0181] A second acquisition module 910 is used to acquire biological auxiliary data and mixed voice data of a target user in response to a model training instruction, wherein the biological auxiliary data includes EEG information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0182] The enhancement module 920 is used to input the biological auxiliary data and the mixed speech data into the speech enhancement model, and output the enhanced speech data, wherein the speech of the user of interest in the enhanced speech data is enhanced.

[0183] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information, generate initial loss values ​​based on the enhanced voice information and enhanced language labels, and iteratively adjust the network parameters of the initial enhancement model based on the multiple initial loss values ​​to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0184] According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0185] For example, any of the first acquisition module 810, the adversarial training module 820, the generation module 830, the adjustment module 840, or the second acquisition module 910, the enhancement module 920 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present application, at least one of the first acquisition module 810, the adversarial training module 820, the generation module 830, the adjustment module 840, or the second acquisition module 910, the enhancement module 920 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 810, the adversarial training module 820, the generation module 830, the adjustment module 840, or the second acquisition module 910, the enhancement module 920 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0186] It should be noted that the training device and speech enhancement device parts of the speech enhancement model in the embodiments of the present application correspond to the training method and speech enhancement method parts of the speech enhancement model in the embodiments of the present application. The description of the training device and speech enhancement device parts of the speech enhancement model specifically refers to the training method and speech enhancement method parts of the speech enhancement model, which will not be repeated here.

[0187] Fig.10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Fig.10As shown, the electronic device 1000 according to an embodiment of the present application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0189] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present application by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present application by executing the program stored in the one or more memories.

[0190] According to an embodiment of the present application, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that the computer program read therefrom is installed into the storage portion 1008 as needed.

[0191] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0192] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0193] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0194] For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003 .

[0195] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0196] When the computer program is executed by the processor 1001, the above functions defined in the system / device of the embodiment of the present application are executed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0197] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0198] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages, and specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It will be appreciated by those skilled in the art that the features recorded in the various embodiments of the present application can be combined and / or combined in a variety of ways, even if such a combination or combination is not clearly recorded in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application.

[0200] The embodiments of the present application are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present application. Although each embodiment is described above, this does not mean that the measures in each embodiment cannot be used in combination advantageously. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A training method for a neural-guided speech enhancement model based on personalized brain electrode distribution, characterized in that: include: In response to the model training instruction, a training set is obtained, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations include real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information and enhanced voice labels of the training user, any one of the auxiliary information includes EEG information representing the degree of attention of the training user to a certain identified user in the mixed training voice information; For each of the training sample combinations, multiple initial enhancement models are adversarially trained using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information and the mixed training voice information to obtain multiple enhanced voice information; For each enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced language label; According to the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained speech enhancement model.

2. The method according to claim 1, characterized in that Using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information and the mixed training voice information to perform adversarial training on multiple initial enhancement models to obtain multiple enhanced voice information, including: Using the real biological auxiliary information and the mixed training voice information to train the first initial enhancement model to obtain the first enhanced voice information; Using the mixed distribution auxiliary information and the mixed training speech information to train the second initial enhancement model to obtain the second enhanced speech information; Using the Gaussian distribution auxiliary information and the mixed training speech information to train the third initial enhancement model to obtain the third enhanced speech information; The fourth initial enhanced model is trained using the mixed training voice information to obtain the fourth enhanced voice information.

3. The method according to claim 2, characterized in that Generating the enhanced speech information using any of the initial enhancement models comprises: Processing the target auxiliary information using a target selection module to obtain a biometric feature vector, wherein the target auxiliary information includes the real biometric auxiliary information, the mixed distribution auxiliary information or the Gaussian distribution auxiliary information; The biometric feature vector, the mixed training speech information and the target auxiliary information are processed by a speech enhancement module to generate the enhanced speech information.

4. The method according to claim 3, characterized in that The target auxiliary information is processed by a target selection module to obtain a biometric feature vector, including: Processing the target auxiliary information using a plurality of depth separation convolution units to obtain a plurality of biological convolution features; Processing a plurality of the biological convolutional features using a linear layer to obtain a biological linear feature; The biological feature vector is generated according to the biological linear feature and the training parameters of the adaptive neuron.

5. The method according to claim 4, characterized in that The target auxiliary information is processed by using multiple depth separation convolution units to obtain multiple biological convolution features, including: For each of the depth separation convolution units, performing depth separation convolution processing on the target auxiliary information to obtain a first convolution feature; Performing padding processing on the first convolution feature to obtain a second convolution feature; The second convolution feature is pooled to obtain the biological convolution feature.

6. The method according to claim 3, characterized in that Processing the biometric feature vector, the mixed training voice information and the target auxiliary information using a voice enhancement module to generate the enhanced voice information includes: Generate a biometric feature to be processed according to the biometric feature vector and the target auxiliary information; The biometric feature to be processed and the mixed training voice information are processed using a convolutional time-domain separation network to obtain the enhanced voice information.

7. The method according to claim 6, characterized in that Processing the biometric feature to be processed and the mixed training voice information using a convolutional time-domain separation network to obtain the enhanced voice information includes: Encoding the to-be-processed biometric feature and the mixed training voice information respectively to obtain a biometric coding feature and a language coding feature; Performing sound source separation processing on the biological coding features and the language coding features to obtain speech mask information; Generating target coding information according to the speech mask information and the language coding features; The target coded information is decoded to obtain the enhanced speech information.

8. The method according to claim 2, characterized in that: For each enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced language label, including: For the first enhanced speech information, generating a signal distortion loss value according to the enhanced speech information and the enhanced language label; For the second enhanced speech information, generating a first confusion loss value according to the enhanced speech information and the enhanced language label; For the third enhanced speech information, generating a second confusion loss value according to the enhanced speech information and the enhanced language label; For the fourth enhanced speech information, a third confusion loss value is generated according to the enhanced speech information and the enhanced language label.

9. The method according to claim 8, characterized in that Iteratively adjusting the network parameters of the initial enhancement model according to the multiple initial loss values ​​to obtain a trained speech enhancement model, including: Generate a first loss value according to the signal distortion loss value and the first confusion loss value; Generate a second loss value according to the first loss value and a selected mean square error loss; Generate a target loss value according to the second loss value, the second confusion loss value, and the third confusion loss value; The network parameters of the initial enhancement model are iteratively adjusted according to the target loss value to obtain the speech enhancement model.

10. A neural-guided speech enhancement method based on personalized brain electrode distribution, characterized in that: include: In response to the model training instruction, acquiring biological auxiliary data and mixed voice data of a target user, wherein the biological auxiliary data includes electroencephalogram information representing the degree of attention of the target user to a certain attention user in the mixed voice data; Inputting the biological auxiliary data and the mixed voice data into a voice enhancement model, and outputting enhanced voice data, wherein the voice of the user of interest in the enhanced voice data is enhanced; Wherein, the speech enhancement model is trained by the method described in any one of claims 1 to 9.

Images