Speech enhancement method, speech enhancement model training method, speech enhancement system, glasses, computer storage medium and program product

Abstract

The embodiment of the invention provides a speech enhancement method, a speech enhancement model training method, a speech enhancement system, glasses, a computer storage medium and a program product, and the speech enhancement method comprises the steps: obtaining sound information containing the speech of a target speaker through a microphone array, and carrying out the recognition of the sound information through the microphone array; obtaining the spatial orientation information of the voice of the target speaker based on the microphone array; performing sound feature extraction on the sound information to obtain corresponding sound features, and performing spatial orientation feature extraction on the sound information based on the spatial orientation information to obtain corresponding spatial orientation features; and based on the sound features and the spatial orientation features, performing speech enhancement for the speech of the target speaker, and obtaining the speech of the target speaker after speech enhancement. According to the embodiment of the invention, accurate and efficient speech enhancement for the target speaker is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech processing technology, and in particular to a speech enhancement method, a speech enhancement model training method, a speech enhancement system, a pair of glasses, a computer storage medium, and a computer program product. Background Art

[0002] Speech Enhancement (SE) is a technology for processing and improving the quality of speech signals. It can separate clean speech signals from disturbed speech signals, reduce the impact of noise and reverberation on speech, and improve speech clarity and intelligibility. With the growing attention to personal information, directional speech enhancement has been proposed and is increasingly used.

[0003] Directional Speech Enhancement (DSE) is a signal processing method that uses a multi-channel microphone device to extract speech signals from only a certain direction (including but not limited to the direction of the target speaker) while suppressing signals from other directions, thereby enhancing the speech component. However, current directional speech enhancement still has poor accuracy when performing targeted speech extraction for the target speaker, and the sound of non-target speakers is leaked. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a speech enhancement and model training solution to at least partially solve the above problems.

[0005] According to a first aspect of an embodiment of the present application, a speech enhancement method is provided, comprising: acquiring sound information containing a target speaker's speech through a microphone array, and spatial orientation information of the target speaker's speech obtained based on the microphone array; performing sound feature extraction on the sound information to obtain corresponding sound features, and performing spatial orientation feature extraction on the sound information based on the spatial orientation information to obtain corresponding spatial orientation features; performing speech enhancement on the target speaker's speech based on the sound features and the spatial orientation features to obtain the target speaker's speech after speech enhancement.

[0006] According to a second aspect of an embodiment of the present application, a speech enhancement model training method is provided, comprising: obtaining a training sample, the training sample comprising: sound sample information containing a target speaker's speech, speech annotations of the target speaker's speech, speech annotations of a non-target speaker's speech, and spatial orientation sample information of the target speaker's speech; performing sound feature extraction on the sound sample information respectively to obtain corresponding sound sample features, and performing spatial orientation feature extraction on the sound sample information based on the spatial orientation sample information to obtain corresponding spatial orientation sample features; performing speech enhancement on the target speaker's speech based on the sound sample features and the spatial orientation sample features to obtain a predicted speech of the target speaker after speech enhancement; and training a speech enhancement model based on the predicted speech and the speech annotated by the speech annotations.

[0007] According to a third aspect of an embodiment of the present application, a speech enhancement system is provided, comprising: a microphone array and a microphone chip; wherein: the microphone array is used to collect sound information from a physical environment; the microphone chip is used to call a speech enhancement instruction from a memory when the sound information collected by the microphone array is sound information containing the voice of a target speaker, so as to perform speech enhancement processing on the sound information; wherein the memory is used to store at least one executable instruction, the executable instruction includes the speech enhancement instruction, and the speech enhancement instruction is used to enable the processor to perform an operation corresponding to the method described in the first aspect.

[0008] According to a fourth aspect of an embodiment of the present application, a pair of glasses is provided, comprising: a microphone array, the microphone array comprising a plurality of microphones arranged at different positions of the glasses; the microphone array is used to collect sound information from a physical environment; a processor, the processor is communicatively connected to the microphone array, the processor is used to call a speech enhancement instruction from a memory when the sound information collected by the microphone array is sound information containing a target speaker's voice, so as to perform speech enhancement processing on the sound information; wherein the processor is communicatively connected to the memory, the memory is used to store at least one executable instruction, the executable instruction includes the speech enhancement instruction, and the speech enhancement instruction is used to enable the processor to perform an operation corresponding to the method described in the first aspect.

[0009] According to a fifth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0010] According to a sixth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the method described in the first aspect or the second aspect.

[0011] According to the scheme provided by the embodiment of the present application, when performing speech enhancement for the target speaker, in addition to the sound information, the spatial orientation information of the target speaker's speech is also combined, and the spatial orientation information can effectively indicate the source direction of the target speaker's sound signal. On this basis, the sound features and spatial orientation features of the sound information containing the target speaker's speech are extracted respectively, and targeted speech enhancement is performed based on the extracted sound features and spatial orientation features. Because the spatial orientation information effectively indicates the source direction of the target speaker's sound signal, the feature extraction of the sound information in this direction, and the subsequent processing based on this, can mainly process the speech signal in this direction, that is, the speech signal of the target speaker, and suppress the speech signals in other directions, thereby achieving accurate and efficient speech enhancement for the target speaker, and avoiding the phenomenon of sound leakage when performing speech enhancement for the target speaker's speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 A schematic diagram of an exemplary system applicable to the embodiment of the present application;

[0014] Figure 2A A flowchart of a method for training a speech enhancement model according to an embodiment of the present application;

[0015] Figure 2B for Figure 2A A structural block diagram of a speech enhancement model in the illustrated embodiment;

[0016] Figure 2C for Figure 2A An exemplary structural diagram of a speech enhancement model in the illustrated embodiment;

[0017] Figure 2D for Figure 2A A schematic diagram of the structure of a microphone array and its partitions in the illustrated embodiment;

[0018] Figure 2E for Figure 2AA schematic diagram of the structure of a dense encoder in the illustrated embodiment;

[0019] Figure 2F for Figure 2A A schematic diagram of the structure of a cross-band neural network module in the illustrated embodiment;

[0020] Figure 2G for Figure 2A A schematic diagram of the structure of a spatial narrowband neural network module in the illustrated embodiment;

[0021] Figure 3 is a flowchart of a method for speech enhancement according to an embodiment of the present application;

[0022] Figure 4 Schematic diagram of the structure of a pair of glasses according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0024] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0025] Figure 1 An exemplary system applicable to the embodiment of the present application is shown. Figure 1 As shown, the system 100 may include a cloud service end 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in FIG. 1 is a plurality of user devices 106. Exemplarily, the plurality of user devices 106 may include but are not limited to wearable smart devices.

[0026] The cloud server 102 may be any suitable device for storing information, data, programs and / or any other suitable type of content, including but not limited to a distributed storage system device, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 may perform any suitable function. For example, in some embodiments, the cloud server 102 may be used to train a speech enhancement model and deploy the trained speech enhancement model to the user device 106.

[0027] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN) and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0028] The user device 106 may include any one or more user devices suitable for interacting with a user and capable of sound collection. In some embodiments, the user device 106 may include any suitable type of device. For example, in some embodiments, the user device 106 may include a wearable smart device, a mobile device, a tablet computer, a laptop computer, a desktop computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device.

[0029] Based on the above system, an embodiment of the present application provides a speech enhancement and model training scheme. The following first describes the training of the speech enhancement model, and then, on this basis, describes the process of performing speech enhancement using the trained speech enhancement model.

[0030] The following combination Figure 2A , Figure 2B and Figure 2C , a step flow chart of a speech enhancement model training method provided in an embodiment of the present application is described.

[0031] First, refer to Figure 2B The structure of a speech enhancement model in an embodiment of the present application includes: a first feature extraction part, a second feature extraction part, a feature fusion layer, a speech separation part, and a prediction training part.

[0032] The first feature extraction part is used to extract the sound features of the sound information to obtain the corresponding sound features. Figure 2C As shown, the multi-channel mixed speech data including the target speaker's speech collected by the microphone array (that is, the sound information including the target speaker's speech, Figure 2C Indicated as y1...y c ) performs Fourier transform, such as short-time Fourier transform STFT, to obtain corresponding spectrum data, and then extracts features from the spectrum data through an encoder to obtain corresponding sound features. Figure 2C In the example, the encoder is an encoder block (Dense Encoder Block) of a dense encoder, hereinafter referred to as the first dense encoder. Compared with other types of encoders, dense encoders can effectively process sequence data such as sound information, capture the timing dependency and context information in the sound information, so that sound feature extraction can be performed more accurately and efficiently. However, it is not limited to this, and other structures or encoders (such as convolution structures, Transformer structures, etc.) that have feature extraction functions and can perform sound feature extraction can also be applied to the solutions of the embodiments of the present application.

[0033] The second feature extraction part is used to extract the spatial orientation features of the sound information based on the spatial orientation information of the target speaker's voice to obtain the corresponding spatial orientation features. Figure 2C As shown, based on the structure (geometry) of the microphone array, the sound information in the direction indicated by the spatial orientation information (usually the direction of the target speaker's voice source) can be obtained from the collected sound information, and then, through an encoder, the sound information in this direction is extracted to obtain the corresponding spatial orientation features. Figure 2C In the embodiment, the encoder is also exemplified as an encoder block of a dense encoder to perform more accurate and efficient spatial orientation feature extraction, hereinafter referred to as the second dense encoder. Similarly, other structures or encoders having feature extraction functions and capable of performing sound feature extraction can also be applied to the solutions of the embodiments of the present application.

[0034] Among them, the spatial orientation information can effectively indicate the source direction of the target speaker's voice signal, which is related to the microphone array structure and the partition of the microphone array structure. Figure 2D As shown in FIG. 1 , a circular microphone array with a radius of 5 cm is shown, and the microphone array includes six microphones evenly distributed around the circular circumference. If one of the microphones is set to an azimuth angle of 0 degrees, then the azimuth angle of the other microphone opposite to it in the horizontal direction is 180 degrees. Furthermore, the azimuth angles of other microphones can be determined based on the positional relationship between the other microphones and the two microphones. Figure 2D , the azimuth angle of mic0 (microphone No. 0) is 0 degrees, the azimuth angle of mic1 (microphone No. 1) is 60 degrees, the azimuth angle of mic2 (microphone No. 2) is 120 degrees, the azimuth angle of mic3 (microphone No. 3) is 180 degrees, the azimuth angle of mic4 (microphone No. 4) is 240 degrees, and the azimuth angle of mic5 (microphone No. 5) is 300 degrees. A partition of the microphone array is formed between adjacent azimuth angles. However, it is not limited to this. In order to be able to determine the direction of the source signal more accurately, in a feasible method, the microphone array can also be partitioned with a finer granularity. The partition granularity can be appropriately set by those skilled in the art according to actual needs, and the embodiments of the present application are not limited to this. For example, if Figure 2D As shown in , the microphone array is partitioned into 30-degree units. Figure 2D The partitions shown in the figure may include 12, for example, the first partition is the positive 30 degree range from the horizontal line where mic0 is located, the second partition is the positive 30 degree to the 60 degree range where mic1 is located, and so on, the twelfth partition is the range from 330 degrees to the horizontal line where mic0 is located. Therefore, the microphone array has 12 azimuth angles, from θ0-θ 11 ,like Figure 2D as shown in .

[0035] For the sound from the source direction, i.e. the speech of the target speaker, the spatial orientation information of the target speaker's speech can be determined by the signal strength in each azimuth direction of the microphone array. The spatial orientation features extracted based on it are closely related to the directional speech of the target speaker (such as the speech in the 0-degree azimuth direction and the 180-degree azimuth direction), but are unrelated to the speech in other interference directions.

[0036] After obtaining the sound features and the spatial orientation features, the two features can be fused through the feature fusion layer to obtain fused features. The fusion can be achieved by appropriate methods, such as feature splicing.

[0037] Then, based on the fusion features obtained after fusion, the speech separation part performs speech separation and enhancement for the target speaker. Figure 2C As shown in FIG. 1 , this part includes cross-band feature extraction by a cross-band neural network, directional speech feature extraction by a spatial narrow-band neural network, and a linearization layer for the directional speech features output by the spatial narrow-band neural network. Thus, through the speech separation part ( Figure 2C The “SpatialNet Block” in the image and the subsequent linearization layer can predict the output target speaker speech and non-target speaker speech (other people's speech).

[0038] At this point, in the inference phase of the speech enhancement model, the speech enhancement model can output the speech of the target speaker. In the training phase of the speech enhancement model, the output of the speech separation part will also go through the prediction training part, using the loss function to calculate the difference between the predicted target speaker's speech and the true value of the target speaker's speech marked in the training sample, and the model training is performed accordingly.

[0039] Below, based on the above model structure, the training of the speech enhancement model provided in the embodiment of the present application is described. It should be noted that in the training stage, the above data is in the form of sample data. It should be understood by those skilled in the art that the other parts of the speech enhancement model (including the first and second feature extraction parts, the feature fusion layer, and the speech separation part) except the prediction training part are the same whether they are processing sample data or actual data in the reasoning stage.

[0040] Reference Figure 2A According to an embodiment of the present application, a speech enhancement model training method includes the following steps:

[0041] Step S202: Obtain training samples.

[0042] The training samples are used to train the speech enhancement model, and at least include positive samples, which at least include: sound sample information containing the target speaker's speech, speech annotations of the target speaker's speech, speech annotations of non-target speaker's speech, and spatial orientation sample information of the target speaker's speech. In order to make the model more robust and avoid overfitting, the training samples also include negative samples, in which the sound sample information may not contain the target speaker's speech.

[0043] In the training phase, the spatial orientation sample information may be pre-set information, and does not need to be determined based on the sound information temporarily collected by the microphone array. The sound sample information may be multi-channel sound sample information.

[0044] Step S204: extracting sound features from the sound sample information respectively to obtain corresponding sound sample features, and extracting spatial orientation features from the sound sample information based on the spatial orientation sample information to obtain corresponding spatial orientation sample features.

[0045] The sound feature extraction and the spatial orientation feature extraction may be performed in any order or in parallel.

[0046] Since the sound sample information contains not only the target speaker's voice, but also other sounds (such as noise, echo or reverberation, etc.) and / or the voices of other speakers, the extracted sound sample features also contain the features of these sounds.

[0047] In one feasible way, sound feature extraction can be implemented as spectrum feature extraction, that is, extracting the spectrum features of sound sample information to obtain the corresponding spectrum sample features. Although sound features can also be implemented in other forms, such as voiceprint features, spectrum features have a wider range of applicability and have better performance in speech enhancement. Therefore, in the embodiment of the present application, sound feature extraction is implemented in the form of spectrum feature extraction. In the specific implementation, such as Figure 2C In the form shown in , the sound sample information can be first subjected to STFT transformation to obtain corresponding spectrum data, and then the spectrum data can be subjected to feature extraction by a first dense encoder to obtain spectrum sample features.

[0048] As for the extraction of spatial orientation features, it can be achieved through spatial filtering. In one feasible method, the sound sample information can be spatially filtered in the spatial direction indicated by the spatial orientation sample information to obtain the corresponding spatial orientation sample features. Spatial filtering can filter the sound signal in the spatial domain, remove or weaken unnecessary sound components, and enhance or retain the required sound components. Specifically in the embodiment of the present application, the sound sample information can be spatially filtered in multiple directions (azimuth indication) of the microphone array, and the spatial filtering results are used as a rough estimate of the source orientation of the target speaker's voice, and then, based on this, the part of the sound sample information that is most likely the target speaker's voice is determined, and then feature extraction is performed on it to obtain the spatial orientation sample features.

[0049] In a specific feasible manner, spatial filtering is performed on the sound sample information in the spatial direction indicated by the spatial orientation sample information to obtain the corresponding spatial orientation sample features, which can be realized by: determining the IPD (Interaural Phase Difference) and TPD (Target-dependent Phase Difference) of the sound sample information in the spatial direction indicated by the spatial orientation information; based on the similarity between the IPD and the TPD, spatial filtering is performed on the sound sample information in the spatial direction indicated by the spatial orientation information to obtain the corresponding spatial orientation sample features. In this way, the spatial orientation sample features of the target speaker's voice can be effectively extracted.

[0050] For example, the similarity between IPD and TPD can be calculated using the following formula (1) to obtain the spatial orientation sample features:

[0051]

[0052] Where M represents the number of azimuth angles of the microphone array; DF(θ i, t, f) represents the target speaker's voice in frequency band f, at time t relative to the i-th azimuth angle θ i The spatial orientation sample characteristics; represents a two-dimensional vector composed of the cosine value and sine value of the phase difference; p represents a microphone pair, which can be determined by a combination algorithm to determine the combination of multiple microphones, or it can be based on one of the microphones and combine the microphone with other microphones to form a microphone pair. DF(θ i , t, f) is larger, indicating that θ i The higher the direction, the higher the probability that the sound signal is the target speaker's voice.

[0053] In order to obtain richer information from the spectral data corresponding to multi-channel speech and the extracted spatial orientation sample features, such as Figure 2C As shown in , the first and second dense encoders can be used to comprehensively model the spectrum features, spatial orientation sample features (representing spatial information) and their mutual relationship. Figure 2E As shown, the first and second dense encoder blocks have the same structure, which includes two convolution blocks, and a dilated Dense Block is arranged between the two convolution blocks. The DenseNet includes four layers of two-dimensional dilated convolutions, and the dilation rates corresponding to each layer can be 1, 2, 4 and 8. Each convolution block includes a 2D convolution layer, a normalization layer and a ReLU layer (activation function layer), and the functional implementation of each layer of this part is the same as the conventional one.

[0054] Step S206: Based on the sound sample features and the spatial orientation sample features, speech enhancement is performed on the target speaker's speech to obtain the predicted speech of the target speaker after speech enhancement.

[0055] After obtaining the sound sample features and the spatial orientation sample features, these two parts of features can be fused, and then speech enhancement processing can be performed on the target speaker's voice based on the sample fusion features obtained after the fusion.

[0056] Exemplarily, the sound sample features and the spatial orientation sample features may be fused through a feature fusion layer, and sample fusion features may be output.

[0057] On this basis, the speech separation part can be used to separate and enhance the speech of the target speaker. In one feasible method, the speech separation part can be implemented as follows Figure 2CThe SpatialNet shown in and the linearization layer for linearizing its output. SpatialNet integrates a cross-band neural network (Cross-band Block) and a spatial narrow-band neural network (Narrow-band Block). The cross-band neural network can perform cross-band feature extraction based on sample fusion features, so that the features of each frequency band in the extracted multiple frequency bands fully consider the features of other frequency bands. The spatial narrow-band neural network can perform directional speech feature extraction. Specifically, in the embodiment of the present application, the speech of the target speaker can be extracted in a directional manner. Therefore, the speech separation of the target speaker can be effectively achieved through SpatialNet. In addition, through SpatialNet, the effects of noise reduction and dereverberation can also be achieved.

[0058] An exemplary cross-band neural network is as follows: Figure 2F As shown, it includes two frequency domain convolution blocks and one full-band linear block. Among them, the two frequency domain convolution blocks each include: a normalization layer, a group convolution layer along the frequency axis (F-GConv1d) and a ReLU layer. The full-band linear block includes: a normalization layer, two SiLU linearization layers, and an F-linear network group layer.

[0059] An exemplary spatial narrowband neural network is as follows Figure 2G As shown in FIG. 1 , it includes: a one-dimensional convolution layer, a normalization layer, a SiLU layer, an SSM (Selective State Space Model) layer and a linearization layer. The relationship between the layers is as follows: Figure 2G As shown, no further details are given here.

[0060] The input of SpatialNet is processed by the linearization layer to generate predicted speech, including the predicted speech of the target speaker ( Figure 2C Indicated in ) and the predicted speech of non-target speakers ( Figure 2C Indicated in ).

[0061] Step S208: training a speech enhancement model based on the predicted speech and the speech annotated by the speech annotation.

[0062] In the embodiment of the present application, the speech enhancement model is trained by supervised training. Therefore, after the predicted speech (including the predicted speech of the target speaker and the predicted speech of the non-target speaker) is obtained, the speech enhancement model can be trained based on the difference between the predicted speech and the target speaker's speech marked in the training sample.

[0063] Although in the embodiment of the present application, conventional loss functions, such as loss functions that characterize signal-to-noise ratio differences and loss functions that characterize spectrum differences, can also be used to determine the difference between the predicted speech and the speech annotated by the speech annotation, and then train the speech enhancement model based on the difference, this method, however, easily leads to speech loss and distortion in the trained model.

[0064] To this end, the loss function used in the solution of the embodiment of the present application includes three parts, namely: a second sub-loss function for comparing the difference in signal-to-noise ratio between the predicted speech and the speech marked by the speech annotation, and a third sub-loss function for comparing the spectral difference between the predicted speech and the speech marked by the speech annotation. In addition, a first sub-loss function for comparing the phoneme difference between the predicted speech and the speech marked by the speech annotation is also included. Through the first sub-loss function for measuring the phoneme loss, the speech enhancement model can introduce speech distortion to a minimum, thereby better protecting the speech components, reducing speech damage and distortion, and thus more effectively improving the effects of real-time voice communication and voice interaction.

[0065] In order to obtain the phonemes of speech, an encoder is used in the embodiment of the present application to obtain the phonemes of speech. Exemplarily, the encoder can be implemented as an encoder in a pre-trained ASR (Automatic Speech Recognition) model. Through the encoder, the predicted speech of the target speaker and the predicted speech of the non-target speaker output by the linearization layer are both subjected to phoneme feature extraction to obtain the phoneme features corresponding to the target speaker and the phoneme features corresponding to the non-target speaker.

[0066] On this basis, illustratively, the first sub-loss function It can be expressed as:

[0067]

[0068] Wherein, MSE represents the mean square error function; represents the output of the speech separation part, i.e., the predicted speech of the target speaker and the predicted speech of the non-target speaker; X represents the speech annotated by the speech annotation in the training sample; Enc represents the output of the encoder, i.e., the phoneme feature.

[0069] For example, the second sub-loss function It can be expressed as:

[0070]

[0071] in,

[0072] and X have the same meaning as above, ||X|| 2 =<X,X> , represents the power of the speech signal; ‖ ‖ 2 represents the L2 norm; X target represents the pure sound signal corresponding to X, e noise Indicates noise.

[0073] For example, the third sub-loss function is It can be expressed as:

[0074]

[0075] in:

[0076]

[0077] Where STFT stands for short-time Fourier transform, ||·|| F represents the Frobenius norm, and ||·||1 represents the L1 norm.

[0078] Based on this, in the embodiment of the present application, the total loss function used in the training of the speech enhancement model is expressed as:

[0079]

[0080] Among them, α1 and α2 are weight coefficients, and through α1 and α2, the first, second and third sub-loss functions can contribute equally to the total loss function. For example, α1 can be set to 0.1, and α2 can be set to 0.05.

[0081] Based on the loss function, the difference between the predicted speech of the model and the speech annotated by the speech annotation in the training sample (including the phoneme difference) is obtained, and then the model is trained based on the difference. The training cycle is iterated until the training termination condition is met, such as reaching a preset number of training times, or the loss value calculated by the above formula (5) meets a preset threshold, etc.

[0082] It should be noted that in Figure 2C In the example shown, the predicted speech (including the predicted speech of the target speaker) and predicted speech of non-target speakers ) will first be transformed into a speech enhanced waveform through inverse Fourier transform (ISTFT), and then the corresponding loss function processing will be performed based on the waveform. However, this method is only an optional method. In practical applications, subsequent loss function processing can also be performed directly based on the predicted speech. The ISTFT method makes it easier for the encoder used to extract phonemes to extract phonemes.

[0083] In addition, if Figure 2CAs shown in FIG, if the ISTFT method is used, the training samples can also include waveform annotations for the target speaker and the non-target speaker, that is, Figure 2C The “net waveform” shown in is used as the true value corresponding to the “enhanced waveform”.

[0084] Through the training method of this embodiment, a spatial narrowband neural network is used to realize directional speech extraction, and noise and reverberation operations are performed to achieve the directional speech extraction function; at the same time, more explicit spatial orientation information and the introduction of a phoneme loss function are used to guide the model to better protect speech components and reduce speech damage and distortion, so that a speech enhancement model that can more accurately identify and extract the target speaker's speech can be trained, and the speech enhancement model can also more effectively improve the effects of real-time voice communication and voice interaction. In the subsequent use of the model, speech enhancement can be mainly performed on the speech signal in the direction of the target speaker's speech, while suppressing speech signals in other directions, thereby achieving accurate and efficient speech enhancement for the target speaker and avoiding the phenomenon of sound leakage when performing speech enhancement on the target speaker's speech.

[0085] Based on the speech enhancement model obtained through the above training, the process of performing speech enhancement using the speech enhancement model is described below.

[0086] Reference Figure 3 , shows a step flow chart of a speech enhancement method according to an embodiment of the present application.

[0087] The speech enhancement method of this embodiment comprises the following steps:

[0088] Step S302: collecting sound information including the target speaker's voice through a microphone array, and spatial orientation information of the target speaker's voice obtained based on the microphone array.

[0089] In practical applications, a microphone array is needed to collect sounds in a physical environment in real time, especially during a conversation between a target speaker and other people. A microphone array generally includes multiple microphones. An exemplary microphone array is Figure 2D However, it should be noted that Figure 2D Only a microphone array with an annular structure is shown, but microphone arrays of other shapes, such as linear and spherical microphone arrays, are also applicable to the solution of the embodiment of the present application. Moreover, for microphone arrays with non-annular structures, the determination of the azimuth angle can also be achieved by those skilled in the art according to actual needs and with reference to the relevant azimuth angle setting method, and the embodiment of the present application will not be described in detail here.

[0090] In the scenario where the target speaker speaks himself or talks with other speakers, the microphone array can collect sound information that at least includes the target speaker's voice. In addition, in the embodiment of the present application, the spatial orientation information of the target speaker's voice relative to the microphone array is also obtained, and the spatial orientation information can effectively characterize the source direction of the target speaker's voice signal, which is convenient for subsequent directional extraction of the target speaker's voice.

[0091] In one feasible manner, the spatial orientation information of the target speaker's voice obtained based on the microphone array can be implemented as follows: based on multiple azimuths in the microphone array and the sound source direction of the target speaker's voice, the spatial orientation information of the target speaker's voice is determined. In this way, not only can the spatial orientation information be accurately determined, but it is also simple and efficient, and the implementation cost is low.

[0092] The azimuth angle of the microphone array can be optionally determined in the following manner: based on the structure of the microphone array and the positions of the microphones in the microphone array, the space where the microphone array is located is divided into multiple regions; and multiple azimuth angles are determined for the microphone array according to the multiple divided regions. The specific determination method can refer to the relevant description in the above embodiment and will not be described in detail here.

[0093] For example, Figure 2D The microphone array and its azimuth shown in are examples. When determining the spatial orientation information, such as the azimuth, for the target speaker's voice, the time difference of the target speaker's voice reaching each microphone can be calculated; then the time difference is multiplied by the speed of sound to obtain the distance from the target speaker's voice to each microphone; then, based on the sign of the distance difference between each distance and the comparison relationship between their absolute values, the partition of the microphone array into which the target speaker's voice falls can be preliminarily determined; on this basis, an algorithm such as any appropriate optimization algorithm (such as the 0.618 method, etc.) is used to approximate and optimize the distance between the target speaker's voice and the center of the circle and the azimuth to obtain the final azimuth. However, it is not limited to this, and other methods of determining the azimuth of the target speaker's voice are also applicable to the scheme of the embodiments of the present application.

[0094] Step S304: extracting sound features from the sound information to obtain corresponding sound features, and extracting spatial orientation features from the sound information based on the spatial orientation information to obtain corresponding spatial orientation features.

[0095] The sound feature extraction and spatial orientation feature extraction may be performed in any order or in parallel.

[0096] In one feasible manner, extracting sound features from sound information to obtain corresponding sound features can be implemented as follows: extracting spectral features from sound information to obtain corresponding spectral features. The spectral features of sound have a wide range of applicability and have better performance in speech enhancement. Therefore, the embodiment of the present application adopts the method of extracting spectral features.

[0097] As for the extraction of spatial orientation features, the sound information can be spatially filtered in the spatial direction indicated by the spatial orientation information to obtain the corresponding spatial orientation features. Thus, the sound information can be filtered in the spatial domain according to the sound source direction of the target speaker, and unnecessary sound components can be removed or weakened, and necessary sound components can be enhanced or retained, providing a guarantee for directional speech extraction.

[0098] In a feasible manner, performing spatial filtering on the sound information in the spatial direction indicated by the spatial orientation information to obtain the corresponding spatial orientation feature may include: determining the IPD and TPD of the sound information in the spatial direction indicated by the spatial orientation information; performing spatial filtering on the sound information in the spatial direction indicated by the spatial orientation information according to the similarity between the IPD and the TPD to obtain the corresponding spatial orientation feature. Exemplarily, the spatial orientation feature may be calculated based on the aforementioned formula (1). In this way, the spatial orientation feature may be obtained effectively and accurately.

[0099] Step S306: Based on the sound features and the spatial orientation features, speech enhancement is performed on the target speaker's speech to obtain the speech of the target speaker after speech enhancement.

[0100] After obtaining the sound features and spatial orientation features, the two can be fused to obtain fused features; then, based on the fused features, speech separation and enhancement can be performed, including cross-band feature extraction through a cross-band neural network and directional speech feature extraction through a spatial narrow-band neural network; based on the results of speech separation and enhancement, processing (such as linearization) can be performed to obtain the speech of the target speaker and the speech of the non-target speaker.

[0101] It can be seen that through this embodiment, when performing speech enhancement for the target speaker, in addition to the sound information, the spatial orientation information of the target speaker's speech is also combined, and the spatial orientation information can effectively indicate the source direction of the target speaker's sound signal. On this basis, the sound features and spatial orientation features of the sound information containing the target speaker's speech are extracted respectively, and targeted speech enhancement is performed based on the extracted sound features and spatial orientation features. Because the spatial orientation information effectively indicates the source direction of the target speaker's sound signal, the feature extraction of the sound information in this direction, and the subsequent processing based on this, can mainly process the speech signal in this direction, that is, the speech signal of the target speaker, and suppress the speech signals in other directions, thereby achieving accurate and efficient speech enhancement for the target speaker, and avoiding the sound leakage phenomenon when performing speech enhancement for the target speaker's speech.

[0102] The embodiment of the present application also provides a pair of glasses, which may be AI (Artificial Intelligence) glasses or AR (Augmented Reality, AR) glasses, but are not limited thereto. The glasses may also be used in other scenarios and need to have sound collection and voice enhancement functions.

[0103] Figure 4 A schematic diagram of the structure of glasses according to an embodiment of the present application is shown. Figure 4 As shown in the figure, a microphone array is arranged on the glasses, and the microphone array includes 5 microphones, which are respectively indicated as mic0-mic4. Among them, mic0 is arranged in the middle position of the upper frame of the glasses, mic1 and mic2 are respectively arranged at the outer side of the lower edge of the lens frame, and mic3 and mic4 are respectively arranged at the two ends of the upper edge of the lens frame close to the outer side. It should be noted that this arrangement is only an exemplary description. In actual applications, those skilled in the art can arrange microphones at any appropriate position of the glasses according to the actual structure of the glasses and usage requirements. For example, two microphones can also be arranged behind the two temples, such as corresponding to the positions of the ears, to form a microphone array including 7 microphones.

[0104] In addition, a processor such as an MCU (Microcontroller Unit) can be placed at any appropriate position in the glasses, and a micro memory can also be provided. The microphone array and the processor, and the processor and the memory can be communicatively connected.

[0105] Among them, at least one executable instruction can be stored in the memory, and the executable instruction includes a speech enhancement instruction, and the speech enhancement instruction can enable the processor to perform operations corresponding to the aforementioned speech enhancement method.

[0106] Then, in actual application, the microphone array can collect sound information from the physical environment where the glasses are located; then, when the sound information collected by the microphone array contains the voice of the target speaker, the processor calls the voice enhancement instruction from the memory to perform the aforementioned voice enhancement processing on the sound information. The voice enhancement processing can refer to the aforementioned Figure 3 The method implementation described in the illustrated embodiment will not be described in detail here.

[0107] It should be noted that in this embodiment, glasses are used as an example to illustrate the implementation of voice enhancement on wearable smart devices. However, it is not limited to this, and other forms of wearable smart devices, such as smart watches, smart bracelets, smart headphones, etc., can also refer to this embodiment to implement corresponding voice enhancement solutions.

[0108] As a result, wearable smart devices are more intelligent in voice processing. Moreover, since wearable smart devices are usually associated with more other functions and applications, the voice enhancement processing results of wearable smart devices also provide a solid foundation for the subsequent function and application realization.

[0109] According to another embodiment of the present application, a speech processing system is also provided. The speech processing system may include a microphone array and a microphone chip.

[0110] The microphone array may be any suitable shape or structure, including but not limited to: Figure 2D The embodiment of the present application does not limit the specific shape and structure of the microphone array, such as the circular array shown in FIG.

[0111] The microphone array is communicatively connected to the microphone chip, and the microphone chip is used to call a speech enhancement instruction from a memory when the sound information collected by the microphone array contains the speech of a target speaker, so as to perform speech enhancement processing on the sound information.

[0112] The memory may be a memory outside the speech processing system or a memory inside the speech processing system. If it is an internal memory, the speech processing system of the embodiment of the present application also includes the internal memory. The memory is used to store at least one executable instruction, which may include a speech enhancement instruction, and the speech enhancement instruction is used to enable the microphone chip to perform the operation corresponding to the aforementioned speech enhancement method.

[0113] In addition, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-mentioned multiple method embodiments is implemented. The computer storage medium includes, but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk or a magneto-optical disk, etc.

[0114] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0115] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, collected sound information, data used for analysis, stored data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0116] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0117] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0118] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0119] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A speech enhancement method, comprising: Acquiring sound information including a target speaker's voice through a microphone array, and spatial orientation information of the target speaker's voice obtained based on the microphone array; Extracting sound features from the sound information respectively to obtain corresponding sound features, and extracting spatial orientation features from the sound information based on the spatial orientation information to obtain corresponding spatial orientation features; Based on the sound features and the spatial orientation features, speech enhancement is performed on the speech of the target speaker to obtain the speech of the target speaker after speech enhancement.

2. The method according to claim 1, wherein: The spatial orientation information of the target speaker's voice obtained based on the microphone array includes: Based on multiple azimuth angles in the microphone array and the sound source direction of the target speaker's voice, the spatial orientation information of the target speaker's voice is determined.

3. The method according to claim 2, wherein: Before determining the spatial orientation information of the target speaker's voice based on the multiple azimuths in the microphone array and the sound source direction of the target speaker's voice, the method further includes: Based on the structure of the microphone array and the positions of the microphones in the microphone array, dividing the space where the microphone array is located into a plurality of areas; According to the divided multiple areas, multiple azimuth angles are determined for the microphone array.

4. The method according to any one of claims 1 to 3, wherein: The extracting the spatial orientation feature of the sound information based on the spatial orientation information to obtain the corresponding spatial orientation feature includes: The sound information is spatially filtered in the spatial direction indicated by the spatial orientation information to obtain corresponding spatial orientation features.

5. The method according to claim 4, wherein: The performing spatial filtering on the sound information in the spatial direction indicated by the spatial orientation information to obtain corresponding spatial orientation features includes: Determine an interaural phase difference IPD and a target correlation phase difference TPD of the sound information in the spatial direction indicated by the spatial orientation information; According to the similarity between the IPD and the TPD, spatial filtering is performed on the sound information in the spatial direction indicated by the spatial orientation information to obtain corresponding spatial orientation features.

6. The method according to any one of claims 1 to 3, wherein: The extracting sound features from the sound information to obtain corresponding sound features includes: Spectrum features are extracted from the sound information to obtain corresponding spectrum features.

7. A speech enhancement model training method, comprising: Acquire a training sample, wherein the training sample includes: sound sample information including the speech of a target speaker, speech annotations of the speech of the target speaker, speech annotations of the speech of a non-target speaker, and spatial orientation sample information of the speech of the target speaker; Extracting sound features from the sound sample information respectively to obtain corresponding sound sample features, and extracting spatial orientation features from the sound sample information based on the spatial orientation sample information to obtain corresponding spatial orientation sample features; Based on the sound sample features and the spatial orientation sample features, performing speech enhancement on the speech of the target speaker to obtain a predicted speech of the target speaker after speech enhancement; A speech enhancement model is trained based on the predicted speech and the speech annotated by the speech annotation.

8. The method according to claim 7, wherein: The extracting the spatial orientation features of the sound sample information based on the spatial orientation sample information to obtain the corresponding spatial orientation sample features includes: The sound sample information is spatially filtered in the spatial direction indicated by the spatial orientation sample information to obtain corresponding spatial orientation sample features.

9. The method according to claim 8, wherein: The performing spatial filtering on the sound sample information in the spatial direction indicated by the spatial orientation sample information to obtain corresponding spatial orientation sample features includes: Determine an interaural phase difference IPD and a target correlation phase difference TPD of the sound sample information in the spatial direction indicated by the spatial orientation information; According to the similarity between the IPD and the TPD, spatial filtering is performed on the sound sample information in the spatial direction indicated by the spatial orientation information to obtain corresponding spatial orientation sample features.

10. The method according to any one of claims 7 to 9, wherein: The extracting sound features from the sound sample information to obtain corresponding sound sample features includes: Spectrum feature extraction is performed on the sound sample information to obtain corresponding spectrum sample features.

11. The method according to any one of claims 7 to 9, wherein: The training of the speech enhancement model based on the predicted speech and the speech annotated by the speech annotation comprises: Training a speech enhancement model based on the predicted speech, the speech annotated by the speech annotation, and the predicted loss function; The loss function includes: a first sub-loss function for comparing the phoneme difference between the predicted speech and the speech annotated by the speech annotation.

12. The method according to claim 11, wherein: The loss function also includes: a second sub-loss function for comparing the scale-invariant signal-to-noise ratio difference between the predicted speech and the speech annotated by the speech annotation, and a third sub-loss function for comparing the spectrum difference between the predicted speech and the speech annotated by the speech annotation.

13. A speech enhancement system, comprising: Microphone arrays and microphone chips; in: The microphone array is used to collect sound information from the physical environment; The microphone chip is used to call a speech enhancement instruction from a memory to perform speech enhancement processing on the sound information when the sound information collected by the microphone array is sound information containing the speech of a target speaker; Wherein, the memory is used to store at least one executable instruction, and the executable instruction includes the voice enhancement instruction, and the voice enhancement instruction is used to enable the microphone chip to perform an operation corresponding to the method as described in any one of claims 1-6.

14. A pair of glasses, comprising: A microphone array, the microphone array comprising a plurality of microphones disposed at different positions of the glasses; The microphone array is used to collect sound information from the physical environment; A processor, the processor being communicatively connected to the microphone array, the processor being configured to call a speech enhancement instruction from a memory to perform speech enhancement processing on the sound information when the sound information collected by the microphone array is sound information containing the speech of a target speaker; The processor and the memory are communicatively connected, the memory is used to store at least one executable instruction, the executable instruction includes the speech enhancement instruction, and the speech enhancement instruction is used to enable the processor to perform an operation corresponding to the method as described in any one of claims 1-6.

15. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the method according to any one of claims 1 to 6, or the method according to any one of claims 7 to 12.

16. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the method as claimed in any one of claims 1 to 6, or to execute operations corresponding to the method as claimed in any one of claims 7 to 12.

Images