Neurally guided speech enhancement method based on personalized brain electrode distribution

Abstract

The present application provides a neural-guided speech enhancement method based on personalized brain electrode distribution. In the speech enhancement method, a training method of a speech enhancement model includes obtaining a training set, the training set including multiple training sample combinations corresponding to at least one training user, the training sample combinations including the training user's real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training speech information and enhanced speech labels, the auxiliary information representing the degree of attention of the training user to a certain identified user in the mixed training speech information; using the real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information and mixed training speech information to perform adversarial training on multiple initial enhancement models to obtain multiple enhanced speech information; generating an initial loss value based on the enhanced speech information and the enhanced language label; iteratively adjusting the network parameters of the initial enhancement model based on the initial loss value to obtain a speech enhancement model.

Description

Technical Field

[0001] The present application relates to the field of speech signal processing technology, and more specifically, to a training method for a neural-guided speech enhancement model based on personalized brain electrode distribution, a speech enhancement method, a training device, a speech enhancement device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Speech enhancement (SE) aims to extract the target speaker's speech signal from various audio signals mixed with noise. It is widely used in hearing aids (HAs) and is the core algorithm for these devices. When the noise originates from other speakers, it is also called target speaker extraction (TSE).

[0003] Conventional TSE methods rely on auxiliary information containing information about the target speaker. These auxiliary information primarily includes pre-registered audio, visual aids, and spatial information. However, obtaining this auxiliary information is challenging in practical applications. Recent advances in brain science and the human auditory system have led researchers to discover that auditory attention information, such as the target speech envelope, can be extracted from listener bio-analogue information, such as electroencephalogram (EEG). This has led to the development of TSE approaches using EEG signals. Initially, researchers performed envelope extraction and separation, then extracted the target speech based on similarity, or employed non-end-to-end approaches. This approach did not achieve ideal results, and the initial separation was unnecessary. To overcome these issues, other researchers have proposed more advanced end-to-end models that utilize a specialized feature fusion module (FiLM).

[0004] In the actual application of brain-assisted speech enhancement, however, excessive electrodes do not significantly improve performance and may even lead to excessive costs. This has led to widespread application of electrode channel selection in this field. For example, a related technique proposed an end-to-end EEG channel selection method based on a special distribution (Gumbel-Softmax Distribution). However, this method faces difficulties in training and duplicate channels. Another related technique added weighted residual connections to improve training stability, but still did not solve the duplicate channel selection problem. Another related technique proposed a new solution based on attention and special constraint functions to achieve a solution without duplicate channels and without explicitly specifying the number of selected channels. However, current methods do not address the issue of personalized selection, and the selection results are constant for all users, resulting in high variance. Summary of the Invention

[0005] In view of this, the present application provides a training method, speech enhancement method, training device, speech enhancement device, electronic device, computer-readable storage medium and computer program product for a neural-guided speech enhancement model based on personalized brain electrode distribution.

[0006] One aspect of the present application provides a training method for a neural-guided speech enhancement model based on personalized brain electrode distribution, comprising:

[0007] In response to a model training instruction, a training set is obtained, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations including real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice labels of the training user, any of which includes EEG information representing the degree of attention of the training user to a recognized user in the mixed training voice information;

[0008] For each of the above training sample combinations, multiple initial enhancement models are adversarially trained using the above real biological auxiliary information, the above mixed distribution auxiliary information, the above Gaussian distribution auxiliary information, and the above mixed training speech information to obtain multiple enhanced speech information;

[0009] For each enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced language label;

[0010] According to the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained speech enhancement model.

[0011] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using the above-mentioned real biological auxiliary information, the above-mentioned mixed distribution auxiliary information, the above-mentioned Gaussian distribution auxiliary information, and the above-mentioned mixed training voice information to obtain multiple enhanced voice information, including:

[0012] Using the real biological auxiliary information and the mixed training voice information to train the first initial enhancement model, to obtain the first enhanced voice information;

[0013] Using the mixed distribution auxiliary information and the mixed training speech information, the second initial enhancement model is trained to obtain the second enhanced speech information.

[0014] Using the Gaussian distribution auxiliary information and the mixed training speech information to train the third initial enhancement model, to obtain the third enhanced speech information;

[0015] The fourth initial enhanced model is trained using the mixed training voice information to obtain the fourth enhanced voice information.

[0016] According to an embodiment of the present application, generating the enhanced speech information using any of the above-mentioned initial enhancement models includes:

[0017] Processing the target auxiliary information using a target selection module to obtain a biometric feature vector, wherein the target auxiliary information includes the real biometric auxiliary information, the mixed distribution auxiliary information, or the Gaussian distribution auxiliary information;

[0018] The biometric feature vector, the mixed training speech information and the target auxiliary information are processed by a speech enhancement module to generate the enhanced speech information.

[0019] According to an embodiment of the present application, the target selection module is used to process the target auxiliary information to obtain a biometric feature vector, including:

[0020] Utilizing multiple depth-separated convolution units to process the target auxiliary information, multiple biological convolution features are obtained;

[0021] Use linear layers to process multiple biological convolutional features to obtain biological linear features;

[0022] The above-mentioned biological feature vector is generated according to the above-mentioned biological linear features and the training parameters of the adaptive neurons.

[0023] According to an embodiment of the present application, multiple depth-separated convolution units are used to process the target auxiliary information to obtain multiple biological convolution features, including:

[0024] For each of the above-mentioned depth separation convolution units, perform depth separation convolution processing on the above-mentioned target auxiliary information to obtain a first convolution feature;

[0025] Perform padding processing on the first convolution feature to obtain the second convolution feature;

[0026] The second convolution feature is pooled to obtain the biological convolution feature.

[0027] According to an embodiment of the present application, a speech enhancement module is used to process the biometric feature vector, the mixed training speech information, and the target auxiliary information to generate the enhanced speech information, including:

[0028] Generate a biometric feature to be processed according to the biometric feature vector and the target auxiliary information;

[0029] The biometric features to be processed and the mixed training voice information are processed using a convolutional time-domain separation network to obtain the enhanced voice information.

[0030] According to an embodiment of the present application, a convolutional time-domain separation network is used to process the above-mentioned biometric features to be processed and the above-mentioned mixed training voice information to obtain the above-mentioned enhanced voice information, including:

[0031] Encoding the aforementioned biometric features to be processed and the aforementioned mixed training voice information respectively to obtain biometric coding features and language coding features;

[0032] Performing sound source separation processing on the above-mentioned biological coding features and language coding features to obtain speech mask information;

[0033] Generating target coding information according to the speech mask information and the language coding features;

[0034] The target coded information is decoded to obtain the enhanced speech information.

[0035] According to an embodiment of the present application, for each enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced language label includes:

[0036] For the first enhanced speech information, generating a signal distortion loss value according to the enhanced speech information and the enhanced language label;

[0037] For the second enhanced speech information, generating a first confusion loss value according to the enhanced speech information and the enhanced language label;

[0038] For the third enhanced speech information, generating a second confusion loss value according to the enhanced speech information and the enhanced language label;

[0039] For the fourth enhanced speech information, a third confusion loss value is generated according to the enhanced speech information and the enhanced language label.

[0040] According to an embodiment of the present application, the network parameters of the initial enhancement model are iteratively adjusted according to the multiple initial loss values ​​to obtain a trained speech enhancement model, including:

[0041] Generate a first loss value according to the signal distortion loss value and the first confusion loss value;

[0042] Generate a second loss value based on the first loss value and the selected mean square error loss;

[0043] Generate a target loss value according to the second loss value, the second confusion loss value, and the third confusion loss value;

[0044] The network parameters of the initial enhancement model are iteratively adjusted according to the target loss value to obtain the speech enhancement model.

[0045] Another aspect of the present application provides a neural-guided speech enhancement method based on personalized brain electrode distribution, comprising:

[0046] In response to the model training instruction, obtaining biological auxiliary data and mixed voice data of a target user, wherein the biological auxiliary data includes electroencephalogram (EEG) information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0047] The above-mentioned biological auxiliary data and mixed voice data are input into the voice enhancement model, and enhanced voice data is output, wherein the voice of the above-mentioned user of interest in the above-mentioned enhanced voice data is enhanced.

[0048] Another aspect of the present application provides a training device for a neural-guided speech enhancement model based on personalized brain electrode distribution, comprising:

[0049] A first acquisition module is configured to acquire a training set in response to a model training instruction, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations including real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice labels of the training user, any of which includes EEG information representing the degree of attention of the training user to a recognized user in the mixed training voice information;

[0050] An adversarial training module is configured to perform adversarial training on multiple initial enhancement models for each combination of the training samples using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information, and the mixed training speech information to obtain multiple enhanced speech information;

[0051] A generating module, configured to generate an initial loss value for each enhanced speech information according to the enhanced speech information and the enhanced language label;

[0052] An adjustment module is used to iteratively adjust the network parameters of the initial enhancement model according to the multiple initial loss values ​​to obtain a trained speech enhancement model.

[0053] Another aspect of the present application provides a neural-guided speech enhancement device based on personalized brain electrode distribution, comprising:

[0054] A second acquisition module is configured to acquire, in response to the model training instruction, biological auxiliary data and mixed voice data of a target user, wherein the biological auxiliary data includes electroencephalogram (EEG) information representing the degree of attention paid by the target user to a certain attention user in the mixed voice data;

[0055] The enhancement module is used to input the above-mentioned biological auxiliary data and mixed voice data into the voice enhancement model and output enhanced voice data, wherein the voice of the above-mentioned user of interest in the above-mentioned enhanced voice data is enhanced.

[0056] Another aspect of the present application provides an electronic device, comprising:

[0057] one or more processors;

[0058] a memory for storing one or more programs,

[0059] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0060] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0061] Another aspect of the present application provides a computer program product, which includes computer-executable instructions. When the instructions are executed, the instructions are used to implement the method described above.

[0062] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information. An initial loss value is generated based on the enhanced voice information and the enhanced language label. Based on the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0064] Figure 1 An exemplary system architecture to which a training method for a speech enhancement model or a speech enhancement method according to an embodiment of the present application can be applied is shown;

[0065] Figure 2 A flowchart of a method for training a speech enhancement model according to an embodiment of the present application is shown;

[0066] Figure 3 A flowchart of a method for training a speech enhancement model according to another embodiment of the present application is shown;

[0067] Figure 4 A flow chart of a method for generating enhanced voice information according to an embodiment of the present application is shown;

[0068] Figure 5 shows a processing flow chart of a target selection module according to an embodiment of the present application;

[0069] Figure 6 A schematic diagram showing experimental comparative analysis results of the speech enhancement model of the present application and a model without adversarial training according to an embodiment of the present application is shown;

[0070] Figure 7 A flowchart of a method for training a speech enhancement model according to an embodiment of the present application is shown;

[0071] Figure 8 A block diagram of a training device for a speech enhancement model according to an embodiment of the present application is shown;

[0072] Figure 9 A block diagram showing a speech enhancement device according to an embodiment of the present application is shown; and

[0073] Figure 10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0074] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0075] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0076] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0077] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0078] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0079] In the embodiments of the present application, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0080] Figure 1 FIG1 shows an exemplary system architecture 100 to which a training method of a speech enhancement model or a speech enhancement method according to an embodiment of the present application can be applied. It should be noted that, Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0081] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0082] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0083] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0084] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0085] It should be noted that the training method of the speech enhancement model and the speech enhancement method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the training device of the speech enhancement model and the speech enhancement device provided in the embodiment of the present application can generally be set in the server 105. The training method of the speech enhancement model and the speech enhancement method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the training device of the speech enhancement model and the speech enhancement device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the speech enhancement model training method and the speech enhancement method provided in the embodiments of the present application may also be performed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the speech enhancement model training device and the speech enhancement device provided in the embodiments of the present application may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be provided in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0086] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0087] Figure 2 A flowchart of a method for training a speech enhancement model according to an embodiment of the present application is shown.

[0088] like Figure 2 As shown, the training method of the neural-guided speech enhancement model based on personalized brain electrode distribution includes operations S201 to S204.

[0089] In operation S201, in response to a model training instruction, a training set is obtained, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations including real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice labels of the training user, wherein any of the auxiliary information includes EEG information representing the degree of attention of the training user to a recognized user in the mixed training voice information;

[0090] In operation S202, for each training sample combination, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training speech information to obtain multiple enhanced speech information;

[0091] In operation S203, for each enhanced speech information, an initial loss value is generated according to the enhanced speech information and the enhanced language label;

[0092] In operation S204, network parameters of the initial enhancement model are iteratively adjusted according to the multiple initial loss values ​​to obtain a trained speech enhancement model.

[0093] According to an embodiment of the present application, EEG information refers to electroencephalogram (EEG) auxiliary information, wherein the EEG information can also be replaced by electromyography (EMG) auxiliary information, electrocardiogram (ECG) auxiliary information, electrooculography (EOG) auxiliary information, etc. The present application uses EEG auxiliary information for illustrative purposes. For example, when training a user to attend a meeting, the user is trained to pay attention to what speaker A says. At this time, the EEG signal of the training user can be collected as real biological auxiliary information, and the voice of other speakers in the meeting place can be collected as mixed training voice information.

[0094] According to the embodiments of the present application, Gaussian distribution auxiliary information refers to auxiliary information obtained by applying Gaussian distribution processing to real biological auxiliary information. Mixed distribution auxiliary information refers to auxiliary information obtained by applying Gaussian distribution and uniform distribution processing to real biological auxiliary information. Enhanced speech labeling refers to using enhanced speech as a label.

[0095] According to an embodiment of the present application, for each training sample combination, multiple initial enhancement models are simultaneously trained adversarially using the real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training speech information, and enhanced speech labels within the combination. At this time, each initial enhancement model outputs an enhanced speech information. Based on the enhanced speech information and enhanced language labels, an initial loss value corresponding to the initial enhancement model can be generated. The network parameters of the multiple initial enhancement models are synchronously adjusted based on the initial loss values ​​of the multiple initial enhancement models to obtain a trained speech enhancement model.

[0096] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information. An initial loss value is generated based on the enhanced voice information and the enhanced language label. Based on the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0097] Figure 3 A flowchart of a method for training a speech enhancement model according to another embodiment of the present application is shown.

[0098] like Figure 3 As shown in FIG, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training speech information to obtain multiple enhanced speech information, including:

[0099] Using real biological auxiliary information and mixed training voice messages Training the first initial enhancement model to obtain the first enhanced speech information;

[0100] Utilizing mixed distribution auxiliary information and mixed training voice messages Training the second initial enhancement model to obtain second enhanced speech information;

[0101] Using Gaussian distribution auxiliary information and mixed training voice messages Training the third initial enhancement model to obtain third enhanced speech information;

[0102] Using mixed training speech information The fourth initial enhanced model is trained to obtain fourth enhanced speech information.

[0103] According to the embodiment of the present application, the present application solves the problem of excessive model memory through multi-process adversarial training to prevent the model from memorizing the target speaker. Figure 3 As shown, multiple branch models (i.e., initial enhanced models) share parameters, and three new branches replace the original EEG signals with various invalid signals (such as Figure 2 in 、 , no auxiliary information ) to simulate the possibility that the model may memorize the target speaker and suppress this with the confusion loss, which is the square of SI-SDR.

[0104] Figure 4 A flow chart of a method for generating enhanced voice information according to an embodiment of the present application is shown.

[0105] According to the embodiments of the present application, Figure 4 As shown, any initial enhancement model is used to generate enhanced speech information, including:

[0106] Processing target auxiliary information using a target selection module to obtain a biometric feature vector, wherein the target auxiliary information includes real biometric auxiliary information, mixed distribution auxiliary information, or Gaussian distribution auxiliary information;

[0107] The speech enhancement module is used to process the biometric feature vector, the mixed training speech information and the target auxiliary information to generate enhanced speech information.

[0108] According to the embodiments of the present application, the bio-assisted speech enhancement model proposed in the present application mainly includes two parts, a personalized target selection module and a subsequent speech enhancement module.

[0109] According to an embodiment of the present application, after the target auxiliary information is input into the initial enhancement model, the target selection module in the initial enhancement model first converts the target auxiliary information into a biometric feature vector, which implicitly contains the training user's attention information. Thereafter, the speech enhancement module processes the biometric feature vector, the mixed training speech information and the target auxiliary information to obtain the enhanced speech information, i.e. Figure 4 Enhanced speech in .

[0110] Figure 5 The figure shows a processing flow chart of the target selection module according to an embodiment of the present application.

[0111] According to the embodiments of the present application, Figure 5 As shown, the target selection module is used to process the target auxiliary information to obtain the biometric feature vector, including:

[0112] Multiple depth-separated convolutional units are used to process target auxiliary information to obtain multiple biological convolutional features;

[0113] Use linear layers to process multiple biological convolutional features to obtain biological linear features;

[0114] A biological feature vector is generated according to the biological linear features and the training parameters of the adaptive neurons.

[0115] According to the embodiments of the present application, Figure 5 in That is, the training parameters of the adaptive neurons, is a biological linear feature, That is, the biological feature vector.

[0116] According to an embodiment of the present application, to capture individual differences, the model introduces adaptive neurons. These neurons are trained to generate individual difference vectors, which are combined with the universal selection vector obtained during pre-training of the target selection module to form the final individual adaptive EEG channel selection vector. The output of the adaptive neurons is constrained to the range [-1, 1] by an activation function to ensure that the captured individual differences are appropriate and controllable. The universal selection vector is generated by pre-training a universal selection model that does not rely on individual differences (i.e., the initial enhancement model) on a common dataset and freezing its universal selection layer (i.e., the target selection module). To ensure that the model's output remains stable despite different input signals, the input of the universal selection layer is replaced with an all-one vector to ensure that the module is independent of the input. In this way, the pre-trained model can output a stable universal selection vector, called "average fitness," which represents the most informative EEG signal channel across multiple subjects.

[0117] According to an embodiment of the present application, the training process is divided into two stages: the first stage is individual difference learning, and the second stage is selection fine-tuning.

[0118] According to the embodiments of the present application, Figure 5 (a) In the first stage, the model learns individual differences for each subject through adaptive neurons and combines this with the average fitness to generate an individually adaptive EEG channel selection vector (i.e., a biometric feature vector). During this process, the model optimizes a loss function that includes a speech enhancement-related loss and a regularization term to ensure that the model accurately captures individual differences while maintaining a certain degree of channel selection stability. The generated EEG channel selection vector is used to construct a subset of the EEG signal for each subject, which is gradually optimized over multiple iterations until the loss function converges.

[0119] According to the embodiments of the present application, Figure 5 (b) The second stage is fine-tuning. The model freezes the adaptive neurons and uses a threshold decision maker to determine the final adaptive EEG channel selection scheme. This threshold decision maker determines whether to select the corresponding channel based on the value of the individual difference vector, ultimately forming a customized EEG channel selection scheme for each subject. In this stage, the model aims to further optimize the target selection module by combining it with a convolutional time-domain separation network (BASEN) to improve the performance of the speech enhancement model.

[0120] According to the embodiments of the present application, Figure 5As shown in the figure, multiple depth-separated convolution units are used to process the target auxiliary information to obtain multiple biological convolution features, including:

[0121] For each depth-separable convolution unit, perform depth-separable convolution processing on the target auxiliary information to obtain the first convolution feature;

[0122] Perform padding on the first convolution feature to obtain the second convolution feature;

[0123] The second convolution feature is pooled to obtain the biological convolution feature.

[0124] According to an embodiment of the present application, the number of depth separation convolution units can be set according to specific needs, for example, it can be 8. Figure 5 The sampling vector in is the target auxiliary information.

[0125] According to an embodiment of the present application, a speech enhancement module is used to process a biometric feature vector, mixed training speech information, and target auxiliary information to generate enhanced speech information, including:

[0126] Generate a biometric feature to be processed according to the biometric feature vector and target auxiliary information;

[0127] The convolutional time domain separation network is used to process the biometric features to be processed and the mixed training speech information to obtain enhanced speech information.

[0128] According to an embodiment of the present application, the biometric feature vector The product of the target auxiliary information y(t) is processed to generate the biometric feature to be processed, and the biometric feature to be processed and the mixed training speech information x(t) are input into the convolutional time domain separation network BASEN to obtain the enhanced speech information, that is, Figure 5 in Among them, the convolutional time domain separation network BASEN is the Conv-TasNet network.

[0129] According to the embodiments of this application, referring to Figure 4 , using the convolutional time domain separation network to process the biometric features to be processed and the mixed training speech information to obtain enhanced speech information, including:

[0130] Encode the biometric features to be processed and the mixed training voice information respectively to obtain the biometric coding features and language encoding features ;

[0131] Encoding characteristics of organisms and language encoding features Perform sound source separation processing to obtain speech mask information ;

[0132] According to the voice mask information and language encoding features , generate target coding information;

[0133] The target coded information is decoded to obtain enhanced speech information.

[0134] According to an embodiment of the present application, the encoder of the mixed training speech information extracts the embedded features of the audio through multiple layers of one-dimensional convolution, and converts the input audio signal into a representation that can be better processed.

[0135] According to an embodiment of the present application, the encoder of the biometric feature to be processed downsamples the biometric feature to be processed, such as the EEG signal, and extracts multi-level features through a deep convolutional layer to capture auditory attention information in the brain signal.

[0136] According to embodiments of the present application, the separator used for sound source separation combines the embedded features of the audio and EEG signals to predict the target speaker's speech mask. This process relies on deep convolutional layers and a cross-layer attention mechanism to achieve a deep fusion of audio and EEG features.

[0137] According to an embodiment of the present application, the decoder used to decode the target encoded information combines the mask generated by the separator with the audio features to reconstruct the speech signal of the target speaker. The overall architecture achieves the task of separating and reconstructing speech signals by effectively fusing multimodal information.

[0138] According to an embodiment of the present application, for each enhanced speech information, an initial loss value is generated according to the enhanced speech information and the enhanced language label, including:

[0139] For the first enhanced voice information, according to the enhanced voice information and enhanced language tags to generate signal distortion loss values ​​SI-SDR;

[0140] For the second enhanced voice information, according to the enhanced voice information and enhanced language labels to generate the first confusion loss value;

[0141] For the third enhanced voice information, according to the enhanced voice information and enhanced language labels to generate a second confusion loss value;

[0142] For the fourth enhanced voice information, according to the enhanced voice information and enhanced language labels to generate a third confusion loss value.

[0143] According to an embodiment of the present application, the network parameters of the initial enhancement model are iteratively adjusted according to multiple initial loss values ​​to obtain a trained speech enhancement model, including:

[0144] generating a first loss value according to the signal distortion loss value and the first confusion loss value;

[0145] Generate a second loss value based on the first loss value and the selected mean square error loss;

[0146] Generate a target loss value according to the second loss value, the second confusion loss value, and the third confusion loss value;

[0147] The network parameters of the initial enhancement model are iteratively adjusted according to the target loss value to obtain a speech enhancement model.

[0148] According to an embodiment of the present application, a special constraint function is used to train the target selection layer:

[0149]

[0150]

[0151] in The loss function is SI-SDR, The loss function is the selection vector The square of the loss, that is, the mean square error loss, The loss function is the square of the SI-SDR output of the three adversarial branches. This loss function can effectively suppress the phenomenon of the model memorizing the target speaker.

[0152] In a specific embodiment, the present application takes into account the problem of electrode channel redundancy in the field of brain-assisted speech enhancement in related technologies. The extra electrodes do not have a significant impact on performance, but will result in excessive overhead and loss, and the existing electrode channel selection method cannot effectively achieve personalized selection, resulting in excessively large variance in the final enhancement. Therefore, the present application designs a new speech enhancement method and a personalized speech enhancement model to achieve biologically assisted speech enhancement with personalized electrode distribution, and utilizes a new adversarial training method to solve the problem of abnormal model memory. The details are as follows:

[0153] (1) Experimental setup

[0154] In the performance evaluation, three objective metrics were used to measure the overall quality of the enhanced speech signal: SI-SDR (dB) for evaluating the signal-to-distortion ratio, perceptual evaluation of speech quality (PESQ), and short-term objective intelligibility (STOI). These metrics comprehensively assess the quality, intelligibility, and clarity of the enhanced speech. This application also designed a metric to measure abnormal model memory, namely excess memory (OM).

[0155] A general dataset containing 26 subjects was used for pre-training the general selection model. The Adam optimizer was used for training, with a learning rate of 0.0001. Furthermore, an exponential moving average method with a decay rate of 0.999 was used to optimize the deep neural network (DNN) weights. To achieve consistent EEG channel selection, the regularization term weight γ was set to 0.05. Pre-training consisted of 60 epochs with a batch size of 8.

[0156] During individual adaptive training, 1000 epochs of individual variation learning were performed, followed by another 1000 epochs of selective fine-tuning. During selective fine-tuning, the model training was stopped at the appropriate point by monitoring the validation set loss function and implementing an early stopping mechanism of 10 epochs. All models converged before reaching the maximum number of training epochs. Other settings for individual adaptive training remained the same as those for pre-training.

[0157] The training set used for this application evaluation consisted of 33 subjects (28 males, 5 females) with an average age of 27.3 ± 3.2 years. All subjects were native English speakers with normal hearing and no history of neurological disease. Due to poor data quality, the recordings of the sixth subject were excluded.

[0158] Each participant participated in 30 trials, each lasting 60 seconds. The audio stimuli consisted of two different stories read aloud by two men. In each trial, one story was played in the left ear and the other in the right ear. Half of the participants were instructed to focus their attention on the story in their left ear (left-sided attention), while the remaining participants (including those who were excluded) focused their attention on the story in their right ear (right-sided attention). After each trial, participants answered a multiple-choice question to confirm that their attention to the instructed story was effective. To ensure story continuity, each trial continued from the previous trial. To reduce other EEG interference signals, participants were instructed to focus their vision on a crosshair in the center of the screen.

[0159] During the experiment, participants wore a 128-channel EEG cap (plus two mastoid electrodes) and recorded EEG signals at a 512 Hz sampling rate using a BioSemi ActiveTwo system. To maintain consistency with previous research, EEG data were downsampled to 128 Hz. Audio stimuli were played through Sennheiser HD650 headphones at a sampling rate of 44.1 kHz. To minimize the influence of audio intensity on attention, the RMS amplitude of all audio stimuli was normalized.

[0160] Both audio and EEG data are processed to 14.7KHZ, and mixed noise audio is generated by mixing the focused audio and the unfocused audio with equal intensity. This application divides the training set into three groups: 5 experiments are randomly selected from all subjects for testing, 2 experiments are selected for validation, and the remaining experiments are used for training. For the training set and validation set, each experiment is cut into 2-second segments; for the test set, each 60-second experiment is cut into 20-second segments. Since individual adaptive training is required, we further divide the data set according to the subjects. 3 subjects were randomly selected from the left attention group and the right attention group for individual adaptive training. The remaining 26 subjects are used for general training of the model. In the individual adaptive training of these 6 subjects, the data of 26 subjects are inaccessible to ensure that the individual adaptive training is independent of the general training data.

[0161] (2) Experimental results

[0162] Figure 6 In order to compare and analyze the experimental results of the speech enhancement model of this application (the "adversarial training option" in the figure) and the model without adversarial training, the experimental results are displayed in the form of a violin plot, which can well reflect the distribution and range of the experimental results. It can be seen from the figure that the speech enhancement model of this application has well realized the brain-assisted speech enhancement of personalized electrode distribution, and its performance is comparable to that of the full-channel brain-assisted speech enhancement model and is due to the mainstream model UBESD of the full channel. After the introduction of adversarial training, the model of this application can still maintain its performance advantage, and this application has greatly reduced the variance.

[0163] According to the embodiments of the present application, in Table 1, the present application evaluated the branches of adversarial training and fully verified the performance of the model of the present application. The combination of its three adversarial branches effectively reduced the abnormal memory phenomenon and remembered the target speaker.

[0164] Table 1. Evaluation of three adversarial training branches

[0165]

[0166] Figure 7 A flowchart of a method for training a speech enhancement model according to an embodiment of the present application is shown.

[0167] like Figure 7 As shown, the neural-guided speech enhancement method based on personalized brain electrode distribution includes S701~S702.

[0168] In operation S701, in response to a model training instruction, biological auxiliary data and mixed voice data of a target user are acquired, wherein the biological auxiliary data includes electroencephalogram (EEG) information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0169] In operation S702 , the biometric auxiliary data and the mixed voice data are input to a voice enhancement model, and enhanced voice data is output, wherein the voice of the user of interest in the enhanced voice data is enhanced.

[0170] According to an embodiment of the present application, in a scenario where the target user is speaking with multiple people, bio-auxiliary data, such as EEG, can be detected in real time through electrodes and other devices, and the speech enhancement model processes the bio-auxiliary data and mixed speech data in real time. When focusing on a certain speaker in the bio-auxiliary data of the target user, the speech enhancement model can enhance the speech of the speaker in the mixed speech data, thereby outputting the enhanced speech data of the focused user to the target user.

[0171] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information. An initial loss value is generated based on the enhanced voice information and the enhanced language label. Based on the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0172] Figure 8 A block diagram of a training device for a speech enhancement model according to an embodiment of the present application is shown.

[0173] like Figure 8 As shown, the training device 800 of the neural-guided speech enhancement model based on personalized brain electrode distribution includes a first acquisition module 810, an adversarial training module 820, a generation module 830, and an adjustment module 840.

[0174] A first acquisition module 810 is configured to acquire a training set in response to a model training instruction, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations including the training user's real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice labels, wherein any of the auxiliary information includes EEG information representing the degree of attention of the training user to a recognized user in the mixed training voice information;

[0175] An adversarial training module 820 is configured to perform adversarial training on multiple initial enhancement models for each training sample combination using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training speech information to obtain multiple enhanced speech information;

[0176] A generating module 830 is configured to generate an initial loss value for each enhanced speech information according to the enhanced speech information and the enhanced language label;

[0177] The adjustment module 840 is used to iteratively adjust the network parameters of the initial enhancement model according to multiple initial loss values ​​to obtain a trained speech enhancement model.

[0178] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information. An initial loss value is generated based on the enhanced voice information and the enhanced language label. Based on the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0179] Figure 9 A block diagram of a speech enhancement device according to an embodiment of the present application is shown.

[0180] like Figure 9 As shown, the neural-guided speech enhancement device 900 based on personalized brain electrode distribution includes a second acquisition module 910 and an enhancement module 920.

[0181] A second acquisition module 910 is configured to acquire, in response to a model training instruction, biological auxiliary data and mixed voice data of a target user, wherein the biological auxiliary data includes EEG information representing the degree of attention of the target user to a certain attention user in the mixed voice data;

[0182] The enhancement module 920 is used to input the biological auxiliary data and the mixed speech data into the speech enhancement model and output enhanced speech data, wherein the speech of the user of interest in the enhanced speech data is enhanced.

[0183] According to an embodiment of the present application, multiple initial enhancement models are adversarially trained using real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, and mixed training voice information to obtain multiple enhanced voice information. An initial loss value is generated based on the enhanced voice information and the enhanced language label. Based on the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained voice enhancement model. The voice enhancement model of the present application only requires a single type of biological auxiliary information, which reduces the number of electrodes used. At the same time, the voice enhancement model trained using biological auxiliary information can respond to the user's personalized selection, thereby enhancing the voice of the user of interest, effectively avoiding the abnormal memory problem of the voice enhancement model in the related art.

[0184] According to the embodiments of the present application, any number of modules, submodules, units, and subunits, or at least part of the functions of any number of them, can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, which can perform the corresponding functions when the computer program module is executed.

[0185] For example, any of the first acquisition module 810, the adversarial training module 820, the generation module 830, the adjustment module 840, or the second acquisition module 910 and the enhancement module 920 can be combined into one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present application, at least one of the first acquisition module 810, the adversarial training module 820, the generation module 830, the adjustment module 840, or the second acquisition module 910, the enhancement module 920 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable manner of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 810, the adversarial training module 820, the generation module 830, the adjustment module 840, or the second acquisition module 910, the enhancement module 920 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0186] It should be noted that the training device and speech enhancement device parts of the speech enhancement model in the embodiments of the present application correspond to the training method and speech enhancement method parts of the speech enhancement model in the embodiments of the present application. The description of the training device and speech enhancement device parts of the speech enhancement model specifically refers to the training method and speech enhancement method parts of the speech enhancement model, and will not be repeated here.

[0187] Figure 10 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 10 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0188] like Figure 10As shown, the electronic device 1000 according to an embodiment of the present application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0189] Various programs and data required for the operation of the electronic device 1000 are stored in the RAM 1003. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 1002 and / or the RAM 1003. It should be noted that the programs may also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0190] According to an embodiment of the present application, electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to bus 1004. Electronic device 1000 may also include one or more of the following components connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or modem. Communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1010 as needed, so that computer programs read from the removable media can be installed into storage section 1008 as needed.

[0191] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0192] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0193] According to embodiments of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003 .

[0195] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0196] When the computer program is executed by the processor 1001, the above functions defined in the system / device of the embodiment of the present application are performed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0197] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0198] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments of the present application may be combined and / or coupled in various ways, and all of these combinations and / or couplings fall within the scope of the present application.

[0200] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A training method for a neural-guided speech enhancement model based on personalized brain electrode distribution, characterized in that: include: In response to a model training instruction, a training set is obtained, wherein the training set includes a plurality of training sample combinations corresponding to at least one training user, the training sample combinations including real biological auxiliary information, mixed distribution auxiliary information, Gaussian distribution auxiliary information, mixed training voice information, and enhanced voice labels of the training user, wherein any type of auxiliary information includes EEG information representing the degree of attention of the training user to a recognized user in the mixed training voice information; For each of the training sample combinations, adversarial training is performed on multiple initial enhancement models using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information, and the mixed training speech information to obtain multiple enhanced speech information; For each enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced speech label; According to the multiple initial loss values, the network parameters of the initial enhancement model are iteratively adjusted to obtain a trained speech enhancement model.

2. The method according to claim 1, characterized in that The multiple initial enhancement models are subjected to adversarial training using the real biological auxiliary information, the mixed distribution auxiliary information, the Gaussian distribution auxiliary information, and the mixed training speech information to obtain multiple enhanced speech information, including: Using the real biological auxiliary information and the mixed training voice information to train the first initial enhancement model to obtain the first enhanced voice information; Training the second initial enhancement model using the mixed distribution auxiliary information and the mixed training speech information to obtain the second enhanced speech information; Training the third initial enhancement model using the Gaussian distribution auxiliary information and the mixed training speech information to obtain the third enhanced speech information; The fourth initial enhancement model is trained using the mixed training voice information to obtain the fourth enhanced voice information.

3. The method according to claim 2, characterized in that Generating the enhanced speech information using any of the initial enhancement models comprises: Processing target auxiliary information using a target selection module to obtain a biometric feature vector, wherein the target auxiliary information includes the real biometric auxiliary information, the mixed distribution auxiliary information, or the Gaussian distribution auxiliary information; The biometric feature vector, the mixed training speech information and the target auxiliary information are processed by a speech enhancement module to generate the enhanced speech information.

4. The method according to claim 3, characterized in that The target auxiliary information is processed by a target selection module to obtain a biometric feature vector, including: Processing the target auxiliary information using multiple depth-separated convolution units to obtain multiple biological convolution features; Processing a plurality of the biological convolutional features using a linear layer to obtain a biological linear feature; The biological feature vector is generated according to the biological linear feature and the training parameters of the adaptive neuron.

5. The method according to claim 4, characterized in that The target auxiliary information is processed using multiple depth-separated convolution units to obtain multiple biological convolution features, including: For each of the depthwise separation convolution units, performing depthwise separation convolution processing on the target auxiliary information to obtain a first convolution feature; Performing padding processing on the first convolution feature to obtain a second convolution feature; Pooling is performed on the second convolutional features to obtain the biological convolutional features.

6. The method according to claim 3, characterized in that Processing the biometric feature vector, the mixed training speech information, and the target auxiliary information using a speech enhancement module to generate the enhanced speech information includes: generating a biometric feature to be processed according to the biometric feature vector and the target auxiliary information; The biometric feature to be processed and the mixed training voice information are processed using a convolutional time-domain separation network to obtain the enhanced voice information.

7. The method according to claim 6, characterized in that Processing the biometric feature to be processed and the mixed training voice information using a convolutional time-domain separation network to obtain the enhanced voice information includes: Encoding the to-be-processed biometric feature and the mixed training voice information respectively to obtain a biometric coding feature and a language coding feature; Performing sound source separation processing on the biological coding features and the language coding features to obtain speech mask information; generating target coding information according to the speech mask information and the language coding features; The target coded information is decoded to obtain the enhanced speech information.

8. The method according to claim 2, characterized in that For each enhanced speech information, generating an initial loss value according to the enhanced speech information and the enhanced speech label, including: For the first enhanced voice information, generating a signal distortion loss value according to the enhanced voice information and the enhanced voice label; For the second enhanced speech information, generating a first confusion loss value according to the enhanced speech information and the enhanced speech label; For the third enhanced speech information, generating a second confusion loss value according to the enhanced speech information and the enhanced speech label; For the fourth enhanced speech information, a third confusion loss value is generated according to the enhanced speech information and the enhanced speech label.

9. The method according to claim 8, characterized in that Iteratively adjusting the network parameters of the initial enhancement model according to the multiple initial loss values ​​to obtain a trained speech enhancement model, including: generating a first loss value according to the signal distortion loss value and the first confusion loss value; Generating a second loss value according to the first loss value and selecting a mean square error loss; generating a target loss value according to the second loss value, the second confusion loss value, and the third confusion loss value; The network parameters of the initial enhancement model are iteratively adjusted according to the target loss value to obtain the speech enhancement model.

10. A neural-guided speech enhancement method based on personalized brain electrode distribution, characterized in that: include: In response to the model training instruction, obtaining biological auxiliary data and mixed voice data of a target user, wherein the biological auxiliary data includes electroencephalogram (EEG) information representing the degree of attention of the target user to a certain attention user in the mixed voice data; Inputting the biological auxiliary data and the mixed voice data into a voice enhancement model, and outputting enhanced voice data, wherein the voice of the user of interest in the enhanced voice data is enhanced; Wherein, the speech enhancement model is trained by the method according to any one of claims 1 to 9.

Images