Speech recognition method and system for target speaker and related equipment

Abstract

The invention provides a speech recognition method and system for a target speaker and related equipment. The method comprises the following steps: acquiring a speaker reference audio and a to-be-recognized audio; inputting the speaker reference audio and the audio to be recognized into a target speaker voice recognition model, and extracting tone features of a target speaker from the speaker reference audio through an ECAPA-TDNN sub-model in the model; through a Wav2Vec2.0 sub-model in the model, extracting audio universal acoustic features from the to-be-identified audio; inputting the timbre features and the audio general acoustic features into a TS-Conformer encoder in a model for feature screening to obtain acoustic features of a target speaker; and inputting the acoustic features of the target speaker into a Transform decoder in the model to generate a prediction text of the target speaker. Through the technical scheme of the invention, the voice of the target speaker can be effectively separated from the audio to be recognized, and the corresponding text can be output.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method, system and related equipment for a target speaker. Background Art

[0002] Automatic Speech Recognition (ASR) is a cutting-edge computer technology that rapidly transcribes audio into text. With the rapid development of artificial intelligence, speech recognition technology has undergone numerous evolutions and innovations, becoming a key highlight of AI applications. The surge in online voice data and improvements in hardware have further advanced this technology. Currently, speech recognition is widely used in internet products. For example, search engines' voice search functions improve search efficiency, speech-to-text functions speed up input, and simultaneous interpretation technology solves many problems in real-time translation scenarios, greatly improving the convenience of life. However, most existing speech recognition models only support single-speaker scenarios and lack the ability to effectively handle overlapping speech interference when multiple speakers are speaking simultaneously in noisy environments.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0004] The present invention provides a target speaker's speech recognition method, system and related equipment. The main purpose of the present invention is to solve the technical problems mentioned in the background art of the prior art.

[0005] A first aspect of the present invention provides a method for speech recognition of a target speaker, comprising: Obtain speaker reference audio and audio to be recognized; Inputting the speaker reference audio and the audio to be recognized into a target speaker speech recognition model, wherein the target speaker speech recognition model includes an ECAPA-TDNN sub-model, a Wav2Vec2.0 sub-model, a TS-Conformer encoder, and a Transformer decoder; Extracting the timbre features of the target speaker from the speaker reference audio using the ECAPA-TDNN sub-model; Extracting universal audio acoustic features from the audio to be recognized using the Wav2Vec2.0 sub-model; Inputting the timbre features and the audio general acoustic features into a TS-Conformer encoder for feature screening to obtain the acoustic features of the target speaker; The target speaker's acoustic features are input into the Transformer decoder to generate a predicted text of the target speaker.

[0006] In an optional implementation of the first aspect of the present invention, extracting the timbre features of the target speaker from the speaker reference audio using the ECAPA-TDNN sub-model includes: Inputting the speaker reference audio into the ECAPA-TDNN sub-model; Preprocessing the speaker reference audio using the ECAPA-TDNN sub-model, wherein the preprocessing includes frame windowing, pre-emphasis filtering, FBank feature extraction, and normalization; Use 1D convolution to map multi-dimensional FBank features to multi-dimensional channels to capture short-term context information; The feature channels are grouped through the Res2Net module, and multi-scale acoustic features are captured through layered convolution; Dynamically weighting the multi-scale acoustic features through the SE attention mechanism to enhance the discriminative timbre information; Aggregating and statistically pooling the weighted multi-scale acoustic features to form a multi-dimensional vector; A fully connected layer is used to compress the multidimensional vector into a multidimensional timbre embedding vector to serve as the timbre feature of the target speaker.

[0007] In an optional implementation of the first aspect of the present invention, extracting the universal audio acoustic features from the audio to be recognized by using the Wav2Vec2.0 sub-model includes: Input the audio to be recognized into the Wav2Vec2.0 sub-model; Performing mono conversion and resampling on the audio to be recognized to obtain preprocessed audio data; Performing waveform normalization on the pre-processed audio data by a feature extractor; Performing forward inference on the preprocessed audio data after waveform normalization to obtain all hidden layer features; Perform frame-level feature extraction in the time dimension on all the hidden layer features to obtain universal acoustic features of the audio.

[0008] In an optional implementation of the first aspect of the present invention, inputting the timbre feature and the audio universal acoustic feature into a TS-Conformer encoder for feature screening to obtain the target speaker acoustic feature includes: Inputting the timbre feature and the audio universal acoustic feature into a TS-Conformer encoder; Position encoding of the audio universal acoustic features is performed by a position encoding module of the TS-Conformer encoder; Repeating the timbre feature according to the number of frame blocks of the audio universal acoustic feature to achieve alignment between the timbre feature and the audio universal acoustic feature; Extracting speaker high-dimensional features that affect text generation from the aligned timbre features through the PConv module of the TS-Conformer encoder; The position-encoded universal audio acoustic features and the speaker high-dimensional features are combined using a multiplication operation and then input into a stacked Conformer module, wherein the Conformer module includes a multi-head self-attention submodule, a convolution submodule, and two groups of feedforward submodules; The stacked Conformer modules capture the global context information and local time domain structure in the input features and perform feature screening to obtain the acoustic features of the target speaker.

[0009] In an optional embodiment of the first aspect of the present invention, the stacking of the Conformer modules to capture global context information and local time domain structure in the input features and perform feature screening to obtain the acoustic features of the target speaker includes: extracting and encoding semantic information in the input features layer by layer in a stacking manner through the Conformer modules, and establishing global context information based on the obtained multiple layers of semantic information; Each level of extraction is performed by calculating the attention weight of the input feature based on linear transformation through the multi-head self-attention submodule in the Conformer module and applying it to the value sequence to obtain the attention feature representation, and performing nonlinear transformation and feature extraction on the attention feature representation through the convolution submodule in the Conformer module to capture the local time domain structure in the input feature; The input features are screened based on the obtained global context information and the local time domain structure to obtain acoustic features of the target speaker.

[0010] In an optional implementation of the first aspect of the present invention, the speech recognition method of the target speaker also includes: pre-stage training of the target speaker speech recognition model; the stage training includes a first stage, a second stage and a third stage, in the first stage, the speaker reference audio does not exist, the audio to be recognized is a single-person speaking audio, and only the text in the audio to be recognized needs to be output; in the second stage, the speaker reference audio exists, the audio to be recognized is a single-person speaking audio, and only the text in the audio to be recognized needs to be output; in the third stage, the speaker reference audio exists, the audio to be recognized is a multi-person speaking audio, and the text of the target speaker needs to be predicted from the audio to be recognized; in each stage, the model is first trained until convergence, and then training for the next stage is performed.

[0011] In an optional implementation of the first aspect of the present invention, in the training of the target speaker speech recognition model at each stage, for all training sample sets at each stage, the spectral entropy method is used to sort the complexity of the audio to be recognized in all training sample sets, and the audio to be recognized is input into the target speaker speech recognition model in order from low to high complexity for training.

[0012] A second aspect of the present invention provides a target speaker speech recognition system, the target speaker speech recognition system comprising: An audio acquisition module is used to obtain speaker reference audio and audio to be recognized; An audio input module is used to input the speaker reference audio and the audio to be recognized into a target speaker speech recognition model, wherein the target speaker speech recognition model includes an ECAPA-TDNN sub-model, a Wav2Vec2.0 sub-model, a TS-Conformer encoder, and a Transformer decoder; A speaker timbre feature processing module, configured to extract the timbre features of the target speaker from the speaker reference audio using the ECAPA-TDNN sub-model; An audio acoustic feature processing module for identifying audio, configured to extract universal audio acoustic features from the audio to be identified using the Wav2Vec2.0 sub-model; The module for filtering the acoustic features of the audio to be identified is used to input the timbre features and the general acoustic features of the audio into the TS-Conformer encoder for feature filtering to obtain the acoustic features of the target speaker; The speaker text generation module is used to input the acoustic features of the target speaker into the Transformer decoder to generate the predicted text of the target speaker.

[0013] A third aspect of the present invention provides a target speaker speech recognition device, the target speaker speech recognition device comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via a circuit; The at least one processor calls the instructions in the memory to enable the speech recognition device of the target speaker to perform the speech recognition method of the target speaker as described in any one of the first aspects of the present invention.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for speech recognition of a target speaker as described in any one of the first aspects of the present invention is implemented.

[0015] Beneficial Effects: The present invention provides a method, system, and related equipment for speech recognition of a target speaker. The method comprises obtaining speaker reference audio and audio to be recognized; inputting the speaker reference audio and the audio to be recognized into a target speaker speech recognition model, extracting the target speaker's timbre features from the speaker reference audio using the ECAPA-TDNN sub-model in the model; extracting general audio acoustic features from the audio to be recognized using the Wav2Vec2.0 sub-model in the model; inputting the timbre features and the general audio acoustic features into a TS-Conformer encoder in the model for feature screening to obtain the target speaker's acoustic features; and inputting the target speaker's acoustic features into a Transformer decoder in the model to generate predicted text for the target speaker. The technical solution of the present invention can effectively separate the target speaker's speech from the audio to be recognized and output the corresponding text. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an embodiment of a method for speech recognition of a target speaker according to the present invention; Figure 2 A schematic diagram of an embodiment of a target speaker speech recognition model of the present invention; Figure 3 A schematic diagram of an embodiment of a course learning phase idea of ​​the present invention; Figure 4 A schematic diagram of an embodiment of a method for calculating audio spectrum entropy according to the present invention; Figure 5 A schematic diagram of an embodiment of a TS-Conformer encoder according to the present invention; Figure 6 A schematic diagram of an embodiment of a comparison of verification experimental results of the present invention; Figure 7 A schematic diagram of an embodiment of a target speaker speech recognition system of the present invention; Figure 8 The figure is a schematic diagram of an embodiment of a speech recognition device for a target speaker according to the present invention. DETAILED DESCRIPTION

[0017] The terms "first," "second," "third," "fourth," and so on (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0018] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A first aspect of the present invention provides a method for speech recognition of a target speaker, comprising: S100. Acquire speaker reference audio and audio to be recognized. In the present invention, the audio to be recognized refers to speech audio with overlapping speech of one or more people, wherein the one or more people include a target speaker. The speaker reference audio refers to audio in which only the target speaker speaks. The inventive point of the technical solution of the present invention is to find audio of the same or similar speaker in the audio to be recognized based on the speaker timbre of the speaker reference audio, and extract the text of the target speaker audio in the speech recognition model.

[0019] S200: Input the speaker reference audio and the audio to be recognized into the target speaker speech recognition model. Figure 2In the present invention, the target speaker speech recognition model includes an ECAPA-TDNN sub-model, a Wav2Vec2.0 sub-model, a TS-Conformer encoder, and a Transformer decoder. In general, the target speaker speech recognition model of the present invention mainly consists of three parts, namely, an audio feature extraction model (i.e., an ECAPA-TDNN sub-model and a Wav2Vec2.0 sub-model), an encoder model (i.e., a TS-Conformer encoder), and a decoder model (i.e., a Transformer decoder). The feature extraction model is mainly used to obtain the acoustic features of the audio. The quality of the acoustic features can greatly affect the final speech recognition performance. The encoder model is mainly used to further analyze the acoustic features and extract speech features. The decoder model is used to predict text based on the speech features. Most current speech recognition models are unable to provide excellent speech recognition performance for overlapping human voice scenes. In the network model design, the present invention adds speaker feature registration and uses speaker features to control acoustic features to achieve speech recognition effects for the target speaker.

[0020] To make the model achieve better audio recognition results, see Figure 3 The present invention also utilizes the concept of classroom learning, sorting training data from simple to difficult and conducting model training in stages. Specifically, before using the target speaker speech recognition model of the present invention, the target speaker speech recognition model is pre-trained in stages. The stage training includes a first stage, a second stage, and a third stage. In the first stage, the speaker reference audio does not exist, the audio to be recognized is a single-person audio, and only the text in the audio to be recognized needs to be output. In the second stage, the speaker reference audio exists, the audio to be recognized is a single-person audio, and only the text in the audio to be recognized needs to be output. In the third stage, the speaker reference audio exists, the audio to be recognized is multi-person audio, and the target speaker's text needs to be predicted from the audio to be recognized. In each stage, the model is trained until convergence before proceeding to the next stage of training. Furthermore, in each stage of training the target speaker speech recognition model, the audio to be recognized in all training sample sets in each stage is sorted by complexity using the spectral entropy method, and the audio to be recognized is input into the target speaker speech recognition model in ascending order of complexity for training. Simply put, the overall idea of ​​the model training of the present invention is to make the model training transition from simple to complex.

[0021] Specifically, the spectral entropy method (SEM) is a method used to quantify the spectral complexity of a signal and is widely used in fields such as signal processing, feature extraction, and classification. Its core concept is to describe the spectral complexity of a signal by calculating the entropy of the signal spectrum, thereby enabling signal analysis and classification. In the audio field, spectral entropy is often used for endpoint detection, accurately identifying the boundaries between speech and non-speech segments by analyzing the spectral characteristics of speech signals. Because spectral entropy quantifies the spectral complexity of speech signals and reflects the signal's information content and randomness, it can effectively distinguish between different speech signals, providing important feature support for subsequent speech recognition tasks.

[0022] The calculation process of audio spectrum entropy is similar to that of audio feature extraction, such as Figure 4 First, the audio signal is preprocessed, including pre-emphasis, framing, and windowing operations; second, each frame of the signal is Fourier transformed to obtain its spectrum information; then, the spectrum data is squared to obtain the energy spectrum of each frequency component. E i Then, the energy spectrum is normalized and divided by the window length to obtain the power spectrum probability density of each frequency component, as shown in formula (1). This process lays a solid foundation for subsequent spectrum complexity calculation and endpoint detection.

[0023] (1) In formula (1), P i is the probability density of the i-th power spectrum, N is the window length. Finally, the entropy value is calculated using the power spectrum probability density of the spectrum data, as shown in formula (2).

[0024] (2) In formula (2), P i Represents the probability density of the i-th power spectrum. The calculated entropy value H As the spectral entropy value of the signal.

[0025] Spectral entropy is a quantitative measure of a signal's spectral complexity. Higher entropy values ​​indicate a more complex signal spectrum and contain more information. Conversely, lower entropy values ​​indicate a simpler spectrum and contain less information. This characteristic makes spectral entropy methods valuable in endpoint detection and speech signal analysis.

[0026] S300. The timbre features of the target speaker are extracted from the speaker reference audio through the ECAPA-TDNN sub-model. The speaker reference audio of the present invention is extracted using the ECAPA-TDNN model. The ECAPA-TDNN model is a mainstream speaker feature extraction model. Based on the traditional TDNN, this model introduces a channel attention mechanism (SE module) to enhance the expressive power of important feature channels. It uses a 1D Res2Net module to expand the receptive field and improve multi-scale feature extraction capabilities. It also generates a fixed-length speaker embedding vector (i.e., timbre feature) through multi-layer feature aggregation and statistical pooling (such as mean and standard deviation). This feature is used to extract the timbre features of the target speaker and mainly includes the timbre of the target speaker. This feature is used to filter out the semantic features of other speakers in the audio to be recognized in the subsequent speech recognition step, retaining only the semantic features of the target speaker.

[0027] More specifically, in an optional embodiment of step S300 of the present invention, the extracting the timbre features of the target speaker from the speaker reference audio through the ECAPA-TDNN sub-model includes: inputting the speaker reference audio into the ECAPA-TDNN sub-model; preprocessing the speaker reference audio through the ECAPA-TDNN sub-model, and the preprocessing includes frame windowing, pre-emphasis filtering, FBank feature extraction and normalization; using 1D convolution to map multidimensional FBank features to multidimensional channels to capture short-term context information; grouping feature channels through the Res2Net module, and capturing multi-scale acoustic features through layered convolution; dynamically weighting the multi-scale acoustic features through the SE attention mechanism to enhance discriminative timbre information; aggregating and statistically pooling the weighted multi-scale acoustic features and splicing them into a multi-dimensional vector; using a fully connected layer to compress the multi-dimensional vector into a multi-dimensional timbre embedding vector to serve as the timbre features of the target speaker.

[0028] S400. The general acoustic features of the audio are extracted from the audio to be identified through the Wav2Vec2.0 sub-model; in the present invention, the acoustic features of the audio to be identified are extracted through the Wav2Vec2.0 pre-trained model. The Wav2Vec2.0 pre-trained model performs downsampling through a multi-layer CNN and uses a Transformer encoder to output relevant features. This pre-trained model requires tens of thousands of hours of unlabeled data for self-supervised learning, so it can more effectively extract the acoustic features of the input audio than traditional feature extraction techniques. Extracting the acoustic features of audio can compress the audio signal into a feature vector, reducing computational complexity, while retaining the general acoustic information of each frame of audio, such as human voice, background sound, noise, reverberation and all other audio acoustic information. Moreover, the features extracted using the pre-trained model can significantly improve the performance of audio-related tasks, reduce computational costs, and enhance the adaptability and generalization capabilities of the model in different scenarios.

[0029] Specifically, in an optional implementation of step S400 of the present invention, the extracting of universal audio acoustic features from the audio to be identified through the Wav2Vec2.0 sub-model includes: inputting the audio to be identified into the Wav2Vec2.0 sub-model; performing mono conversion and resampling (for example, audio with a sampling rate of 16kHz) on the audio to be identified to obtain preprocessed audio data; performing waveform normalization on the preprocessed audio data through a feature extractor; performing forward inference on the preprocessed audio data after waveform normalization to obtain all hidden layer features; performing frame-level feature extraction in the time series dimension on all the hidden layer features to obtain universal audio acoustic features. In this embodiment, after the model receives the preprocessed audio data, the audio data is automatically cut into overlapping short-time frames (for example, through the sliding window of the convolutional layer) to capture local acoustic information. It is composed of a multi-layer one-dimensional convolutional neural network (CNN) that extracts local acoustic features (such as phonemes, tones, etc.) layer by layer and converts the waveform into a high-dimensional latent representation. After the input waveform passes through the CNN, a 512-dimensional feature vector may be generated for each frame. The global acoustic information of the feature sequence is then analyzed through the context editor (Transformer architecture) to obtain a universal acoustic feature containing all the acoustic information of the audio.

[0030] S500: Input the timbre features and the universal acoustic features of the audio into the TS-Conformer encoder for feature screening to obtain the acoustic features of the target speaker. In the present invention, after obtaining the timbre features of the target speaker and the universal acoustic features of the audio to be recognized, these two features are fused and filtered in the acoustic encoder through the TS-Conformer encoder. The universal acoustic features of the audio to be recognized and the timbre features of the speaker reference audio are fused and filtered multiple times, allowing the Conformer encoder network to retain semantic information related to the target speaker and filter out other acoustic information, such as other people's speech, background noise, etc.

[0031] In an optional embodiment of the first aspect of the present invention, the inputting the timbre features and the universal audio acoustic features into the TS-Conformer encoder for feature screening to obtain the target speaker acoustic features includes: inputting the timbre features and the universal audio acoustic features into the TS-Conformer encoder; position encoding the universal audio acoustic features through the position encoding module of the TS-Conformer encoder; repeating the timbre features according to the number of frame blocks of the universal audio acoustic features to achieve alignment of the timbre features and the universal audio acoustic features; extracting the speaker high-dimensional features that affect text generation from the aligned timbre features through the PConv module of the TS-Conformer encoder; using a multiplication operation to combine the position-encoded universal audio acoustic features and the speaker high-dimensional features and then inputting them into a stacked Conformer module, the Conformer module including a multi-head self-attention submodule, a convolution submodule and two groups of feedforward submodules; capturing the global context information and local time domain structure in the input features through the stacked Conformer modules and performing feature screening to obtain the target speaker acoustic features.

[0032] In an optional embodiment of the first aspect of the present invention, the stacked Conformer modules capture global context information and local time domain structures in the input features and perform feature screening to obtain the acoustic features of the target speaker, including: extracting and encoding semantic information in the input features layer by layer in a stacked manner through the Conformer modules, and establishing global context information based on the obtained multi-layer semantic information; for each level of extraction, the multi-head self-attention submodule in the Conformer module calculates the attention weight of the input features based on linear transformation and applies it to a value sequence to obtain an attention feature representation, and performs nonlinear transformation and feature extraction on the attention feature representation through the convolution submodule in the Conformer module to capture the local time domain structure in the input features; and screening the input features based on the obtained global context information and the local time domain structure to obtain the acoustic features of the target speaker.

[0033] Specifically, the TS-Conformer model mainly filters the required acoustic features multiple times based on the target speaker characteristics and generates high-dimensional speech features. Its structure can be shown as follows Figure 5 TS-Conformer is a speech recognition model that combines the advantages of Transformer and convolutional neural networks (CNNs). Through the combination of pre-convolutional layers, stacked Conformer modules, and deep convolutional networks, it can simultaneously capture global contextual information and local temporal structure, thereby significantly improving speech recognition performance. The core of TS-Conformer lies in its modular design. Each Conformer module consists of four key components: a multi-head self-attention module, a convolution module, and two sets of feedforward neural network modules. These modules are stacked layer by layer to extract and encode semantic information from input features and establish global contextual relationships.

[0034] The multi-head self-attention submodule, similar to the multi-head attention mechanism in the Transformer, allows the model to concurrently focus on different positions of input features at the same level, thereby capturing global context. This mechanism computes attention weights through linear transformations of the query, key, and value, and applies these weights to the value sequence to generate the final attention feature representation. Furthermore, the convolution module in the Conformer module further enhances the modeling of local context. Through a series of convolutional layers and activation functions, it performs nonlinear transformations and feature extraction on the attention feature representation, effectively capturing the local temporal structure of the input features. This combination of global and local features enables the Conformer to achieve outstanding performance in speech recognition tasks, making it a key technology in the field. Since the target speaker features lack temporal sequence characteristics, to ensure alignment with the acoustic features, they are repeatedly generated based on the number of acoustic feature frames. The PConv module is then used to extract high-dimensional speaker features that influence text generation. After the acoustic features are positionally encoded, they are combined with the target speaker features using multiplication operations before passing through each TS-Conformer block. The TS-Conformer block then extracts the global contextual relationships and local time domain structures. Finally, the target speaker acoustic features are obtained based on the extracted global contextual relationships and local time domain structures.

[0035] S600: Input the target speaker's acoustic features into the Transformer decoder to generate the target speaker's predicted text. After extracting the target speaker's acoustic features from the general acoustic features of the audio to be recognized in step S500, the target speaker's acoustic features are input into the Transformer decoder for decoding to obtain the target speaker's predicted text.

[0036] In order to better illustrate the effect of speech recognition of the technical solution of the present invention, the present application constructed the following experiment for verification.

[0037] Dataset Selection: We used Chinese as the training language for this model. To better simulate the distortion and noise introduced by real-world scenarios, we used the open-source Common Voice and Emilia datasets, as well as the publicly available noisy ESC50 dataset. We added various noises to the speech datasets to improve the robustness of the model.

[0038] Comparison of experimental results: The experiment uses the Chinese test set of Common Voice and the randomly selected data of Emilia to superimpose each other for the effect test of the speech recognition model. The test models are the whisper model and the target speaker speech recognition model of the present invention. The test results are as follows: Figure 6 As shown. Since the whisper model does not have a relevant module for identifying speaker features, it is not possible to perform speech recognition on the target speaker very well. Most of the effects are text output based on the logic of the text at this stage. The target speaker speech recognition of the present invention can select the speaker, can better perform speech recognition on the speaker's audio, and predict a more accurate text. In general, the target speaker speech recognition method based on audio registration of the present invention, under the control of speaker features, will produce better results in two-person overlapping scenarios and is more robust.

[0039] See also Figure 7 A second aspect of the present invention provides a speech recognition system for a target speaker, the speech recognition system for the target speaker comprising: An audio acquisition module 10 is used to acquire a speaker reference audio and an audio to be recognized; An audio input module 20 is configured to input the speaker reference audio and the audio to be recognized into a target speaker speech recognition model, wherein the target speaker speech recognition model includes an ECAPA-TDNN sub-model, a Wav2Vec2.0 sub-model, a TS-Conformer encoder, and a Transformer decoder; A speaker timbre feature processing module 30 is configured to extract the timbre features of the target speaker from the speaker reference audio using the ECAPA-TDNN sub-model; The audio acoustic feature processing module 40 is used to extract the universal acoustic features of the audio from the audio to be recognized by using the Wav2Vec2.0 sub-model; The audio acoustic feature screening module 50 is used to input the timbre feature and the general audio acoustic feature into the TS-Conformer encoder for feature screening to obtain the acoustic features of the target speaker; The speaker text generation module 60 is configured to input the target speaker's acoustic features into a Transformer decoder to generate a predicted text of the target speaker.

[0040] In an optional implementation of the second aspect of the present invention, the speaker timbre feature processing module includes: A reference audio input unit, configured to input the speaker reference audio into the ECAPA-TDNN sub-model; An audio preprocessing unit, configured to preprocess the speaker reference audio using the ECAPA-TDNN sub-model, wherein the preprocessing includes frame windowing, pre-emphasis filtering, FBank feature extraction, and normalization; Feature mapping unit, which uses 1D convolution to map multi-dimensional FBank features to multi-dimensional channels to capture short-term context information; Channel grouping unit, used to group feature channels through the Res2Net module and capture multi-scale acoustic features through layered convolution; A feature weighting unit, configured to dynamically weight the multi-scale acoustic features through an SE attention mechanism to enhance discriminative timbre information; An aggregation and pooling unit, configured to aggregate and statistically pool the weighted multi-scale acoustic features and concatenate them into a multi-dimensional vector; A fully connected processing unit is used to compress the multidimensional vector into a multidimensional timbre embedding vector using a fully connected layer to serve as the timbre feature of the target speaker.

[0041] In an optional implementation of the second aspect of the present invention, the module for processing the audio acoustic features to be identified includes: An audio recognition input unit, configured to input the audio to be recognized into a Wav2Vec2.0 sub-model; An audio conversion unit, configured to perform mono conversion and resample the audio to be recognized to obtain preprocessed audio data; a waveform normalization unit, configured to perform waveform normalization on the preprocessed audio data using a feature extractor; A feature inference unit, configured to perform forward inference on the preprocessed audio data after waveform normalization to obtain all hidden layer features; The frame-level feature extraction unit is used to perform frame-level feature extraction in the time series dimension on all the hidden layer features to obtain universal acoustic features of the audio.

[0042] In an optional implementation of the second aspect of the present invention, the module for screening audio acoustic features to be identified includes: A feature input unit, configured to input the timbre feature and the audio universal acoustic feature into a TS-Conformer encoder; a position encoding unit, configured to perform position encoding on the audio universal acoustic features through a position encoding module of the TS-Conformer encoder; a feature alignment unit, configured to repeat the timbre feature according to the number of frame blocks of the audio universal acoustic feature to achieve alignment between the timbre feature and the audio universal acoustic feature; A high-dimensional feature extraction unit, configured to extract high-dimensional features of the speaker that affect text generation from the aligned timbre features through a PConv module of the TS-Conformer encoder; A feature combination input unit is used to combine the position-encoded universal audio acoustic features and the speaker high-dimensional features using a multiplication operation and then input the combined features into a stacked Conformer module, wherein the Conformer module includes a multi-head self-attention submodule, a convolution submodule, and two groups of feedforward submodules; The feature screening unit is used to capture the global context information and local time domain structure in the input features through the stacked Conformer modules and perform feature screening to obtain the acoustic features of the target speaker.

[0043] In an optional embodiment of the second aspect of the present invention, the feature screening unit includes: a global context information establishment subunit, configured to extract and encode semantic information in the input features layer by layer based on a stacking manner through the Conformer module, and establish global context information based on the obtained multiple layers of semantic information; A local time domain structure extraction subunit is configured to calculate the attention weight of the input feature based on linear transformation through the multi-head self-attention submodule in the Conformer module and apply it to the value sequence at each level of extraction to obtain an attention feature representation, and perform nonlinear transformation and feature extraction on the attention feature representation through the convolution submodule in the Conformer module to capture the local time domain structure in the input feature; A screening subunit is configured to screen the input features based on the obtained global context information and the local time domain structure to obtain acoustic features of a target speaker.

[0044] In an optional embodiment of the second aspect of the present invention, the speech recognition system of the target speaker also includes: a model pre-training module, which is used to pre-perform stage training on the target speaker speech recognition model; the stage training includes a first stage, a second stage and a third stage, in the first stage, the speaker reference audio does not exist, the audio to be recognized is a single-person speaking audio, and only the text in the audio to be recognized needs to be output; in the second stage, the speaker reference audio exists, the audio to be recognized is a single-person speaking audio, and only the text in the audio to be recognized needs to be output; in the third stage, the speaker reference audio exists, the audio to be recognized is a multi-person speaking audio, and the text of the target speaker needs to be predicted from the audio to be recognized; in each stage, the model is first trained until convergence, and then training for the next stage is carried out.

[0045] In an optional implementation of the second aspect of the present invention, in the training of the target speaker speech recognition model at each stage, for all training sample sets at each stage, the spectral entropy method is used to sort the complexity of the audio to be recognized in all training sample sets, and the audio to be recognized is input into the target speaker speech recognition model in order from low to high complexity for training.

[0046] Figure 8 This is a schematic diagram of the structure of a target speaker speech recognition device provided by an embodiment of the present invention. This target speaker speech recognition device may vary significantly due to different configurations or performance. It may include one or more processors (central processing units, CPUs) 80 (e.g., one or more processors), memory 90, and one or more storage media 100 (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media may be either transient or persistent. The program stored on the storage medium may include one or more modules (not shown), each of which may include a series of instructions for operating on the target speaker speech recognition device. Furthermore, the processor may be configured to communicate with the storage medium to execute the series of instructions stored in the storage medium on the target speaker speech recognition device.

[0047] The target speaker speech recognition device of the present invention may further include one or more power supplies 110, one or more wired or wireless network interfaces 120, one or more input and output interfaces 140, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 8 The structure of the target speaker's speech recognition device shown does not limit the target speaker's speech recognition device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0048] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of the target speaker's speech recognition system.

[0049] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system or system or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0050] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a speech recognition device for a target speaker, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0051] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for speech recognition of a target speaker, characterized in that: include: Obtain speaker reference audio and audio to be recognized; Inputting the speaker reference audio and the audio to be recognized into a target speaker speech recognition model, wherein the target speaker speech recognition model includes an ECAPA-TDNN sub-model, a Wav2Vec2.0 sub-model, a TS-Conformer encoder, and a Transformer decoder; Extracting the timbre features of the target speaker from the speaker reference audio using the ECAPA-TDNN sub-model; Extracting universal audio acoustic features from the audio to be recognized using the Wav2Vec2.0 sub-model; Inputting the timbre features and the audio general acoustic features into a TS-Conformer encoder for feature screening to obtain the acoustic features of the target speaker; The target speaker's acoustic features are input into the Transformer decoder to generate a predicted text of the target speaker.

2. The target speaker speech recognition method according to claim 1, wherein: The extracting of the timbre features of the target speaker from the speaker reference audio by the ECAPA-TDNN sub-model includes: Inputting the speaker reference audio into the ECAPA-TDNN sub-model; Preprocessing the speaker reference audio using the ECAPA-TDNN sub-model, wherein the preprocessing includes frame windowing, pre-emphasis filtering, FBank feature extraction, and normalization; Use 1D convolution to map multi-dimensional FBank features to multi-dimensional channels to capture short-term context information; The feature channels are grouped through the Res2Net module, and multi-scale acoustic features are captured through layered convolution; Dynamically weighting the multi-scale acoustic features through the SE attention mechanism to enhance the discriminative timbre information; Aggregating and statistically pooling the weighted multi-scale acoustic features to form a multi-dimensional vector; A fully connected layer is used to compress the multidimensional vector into a multidimensional timbre embedding vector to serve as the timbre feature of the target speaker.

3. The target speaker speech recognition method according to claim 1, wherein: The audio general acoustic features extracted from the audio to be recognized by the Wav2Vec2.0 sub-model include: Input the audio to be recognized into the Wav2Vec2.0 sub-model; Performing mono conversion and resampling on the audio to be recognized to obtain preprocessed audio data; Performing waveform normalization on the pre-processed audio data by a feature extractor; Performing forward inference on the preprocessed audio data after waveform normalization to obtain all hidden layer features; Perform frame-level feature extraction in the time dimension on all the hidden layer features to obtain universal acoustic features of the audio.

4. The method for speech recognition of a target speaker according to claim 1, wherein: Inputting the timbre feature and the audio universal acoustic feature into the TS-Conformer encoder for feature screening to obtain the acoustic feature of the target speaker includes: Inputting the timbre feature and the audio universal acoustic feature into a TS-Conformer encoder; Position encoding of the audio universal acoustic features is performed by a position encoding module of the TS-Conformer encoder; Repeating the timbre feature according to the number of frame blocks of the audio universal acoustic feature to achieve alignment between the timbre feature and the audio universal acoustic feature; Extracting speaker high-dimensional features that affect text generation from the aligned timbre features through the PConv module of the TS-Conformer encoder; The position-encoded universal audio acoustic features and the speaker high-dimensional features are combined using a multiplication operation and then input into a stacked Conformer module, wherein the Conformer module includes a multi-head self-attention submodule, a convolution submodule, and two groups of feedforward submodules; The stacked Conformer modules capture the global context information and local time domain structure in the input features and perform feature screening to obtain the acoustic features of the target speaker.

5. The target speaker speech recognition method according to claim 1, wherein: The stacked Conformer modules capture global context information and local time domain structures in the input features and perform feature screening to obtain the acoustic features of the target speaker, including: extracting and encoding semantic information in the input features layer by layer in a stacked manner through the Conformer modules, and establishing global context information based on the obtained multiple layers of semantic information; Each level of extraction is performed by calculating the attention weight of the input feature based on linear transformation through the multi-head self-attention submodule in the Conformer module and applying it to the value sequence to obtain the attention feature representation, and performing nonlinear transformation and feature extraction on the attention feature representation through the convolution submodule in the Conformer module to capture the local time domain structure in the input feature; The input features are screened based on the obtained global context information and the local time domain structure to obtain acoustic features of the target speaker.

6. The target speaker speech recognition method according to claim 1, wherein: Also includes: The target speaker speech recognition model is pre-trained in stages; the stage training includes a first stage, a second stage, and a third stage. In the first stage, the speaker reference audio does not exist, the audio to be recognized is a single-person speech audio, and only the text in the audio to be recognized needs to be output; in the second stage, the speaker reference audio exists, the audio to be recognized is a single-person speech audio, and only the text in the audio to be recognized needs to be output; in the third stage, the speaker reference audio exists, the audio to be recognized is a multi-person speech audio, and the text of the target speaker needs to be predicted from the audio to be recognized; in each stage, the model is first trained until convergence, and then training for the next stage is carried out.

7. The target speaker speech recognition method according to claim 6, characterized in that: In the training of the target speaker speech recognition model at each stage, for all training sample sets at each stage, the spectral entropy method is used to sort the complexity of the audio to be recognized in all training sample sets, and the audio to be recognized is input into the target speaker speech recognition model in order from low complexity to high complexity for training.

8. A target speaker speech recognition system, characterized in that: The target speaker's speech recognition system comprises: An audio acquisition module is used to obtain speaker reference audio and audio to be recognized; An audio input module is used to input the speaker reference audio and the audio to be recognized into a target speaker speech recognition model, wherein the target speaker speech recognition model includes an ECAPA-TDNN sub-model, a Wav2Vec2.0 sub-model, a TS-Conformer encoder, and a Transformer decoder; A speaker timbre feature processing module, configured to extract the timbre features of the target speaker from the speaker reference audio using the ECAPA-TDNN sub-model; An audio acoustic feature processing module for identifying audio, configured to extract universal audio acoustic features from the audio to be identified using the Wav2Vec2.0 sub-model; The module for filtering the acoustic features of the audio to be identified is used to input the timbre features and the general acoustic features of the audio into the TS-Conformer encoder for feature filtering to obtain the acoustic features of the target speaker; The speaker text generation module is used to input the acoustic features of the target speaker into the Transformer decoder to generate the predicted text of the target speaker.

9. A target speaker speech recognition device, characterized in that The speech recognition device for the target speaker includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; The at least one processor calls the instructions in the memory to enable the speech recognition device of the target speaker to perform the speech recognition method of the target speaker according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target speaker speech recognition method according to any one of claims 1 to 7 is implemented.

Images