Speech variability coding method for pathological speech recognition

Abstract

The invention provides a speech variability coding method for pathological speech recognition, which comprises the following steps: constructing a speech variability variation auto-encoder model to capture and characterize acoustic variability of pathological speech and reconstruct acoustic features; an acoustic variability coding optimization training technology is adopted to enable the voice variability variation auto-encoder model to avoid phoneme information so as to encode phoneme-independent acoustic variability; and based on the reconstructed acoustic features, applying the speech variability variation auto-encoder model to a pathological speech recognition system and constructing a normal speech synthesis system. According to the method, the performance and robustness of the acoustic model in the speech recognition system can be improved, so that the acoustic model can adapt to a pathological speech recognition scene, and therefore, special language groups can be benefited from speech-driven auxiliary science and technology and intelligent equipment.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a speech variability coding method for pathological speech recognition. Background Art

[0002] Pathological speech production may be caused by a variety of diseases and physiological conditions, such as neurological diseases like stroke, Parkinson's disease, Alzheimer's disease, abnormal or damaged vocal organs like laryngeal cancer, cleft palate, and dysarthria caused by different conditions. Pathological speech recognition technology and pathological speech reconstruction technology based on speech recognition results to synthesize normal speech are of great significance to assist patients in their daily lives, such as assisting patients in verbal communication, controlling smart devices, and conducting speech rehabilitation training.

[0003] Pathological speech is characterized by slurred speech, monotonous intonation, slow or labored pronunciation, heavy breath, low volume and other variations, which usually reduce the intelligibility of speech. The acoustic performance of speech variability caused by these various reasons varies greatly, making this variation difficult to accurately model. Coupled with the limited amount of pathological speech data, pathological speech recognition is a challenging task. At the same time, due to the huge difference between training and testing, commercial automatic speech recognition systems trained with normal speech cannot be directly used for dysarthric speech.

[0004] For pathological speech recognition, the main processing methods are: (1) Optimizing features: reducing the difference between pathological speech and normal speech by modifying the speech rhythm in the signal domain and the feature domain; supplementing pronunciation knowledge can also reduce the variability between speakers; using deep neural networks or convolutional bottleneck networks trained with normal speech to extract bottleneck features to improve the performance of pathological speech recognition models. (2) Adding visual modality: using visual information to reduce the variability of deep neural networks. (3) Adaptive technology: speaker adaptation training is used to eliminate differences between speakers, and pronunciation dictionary adaptation helps to recognize pathological speech in large vocabulary tasks. (4) Adopting new models: new sequence-to-sequence models such as RNN-transducer and transformer have been used to model dysarthric speech.

[0005] Currently, the existing technologies use common models such as autoencoders, LSTM, and Transformer for speech recognition of dialects, accents, and articulation disorders. For example, the irregularity of pathological speech is captured by speech signal processing technology, the LSTM-Attention model is used for Parkinson's speech recognition, the transformer structure is changed by dual encoders for dialect recognition, and the activation function of dual LSTM is improved to improve speech recognition in high noise environments. However, these existing technologies have the following disadvantages:

[0006] (1) The existing technology mainly attributes the characteristics of pathological speech to speaker differences and processes them by using speaker-related information;

[0007] (2) Without optimization for pathological speech, it is difficult to obtain stable and robust performance in pathological speech recognition with insufficient data resources by adopting new speech recognition models or improving model structures;

[0008] (3) The encoder is used to model the speech recognition system instead of encoding speech variability. Therefore, it does not accurately represent the difference between pathological speech and normal speech, resulting in the lack of universality of the pathological speech recognition system. Summary of the invention

[0009] In view of this, the present invention provides a speech variability coding method for pathological speech recognition to at least solve the above-mentioned problem.

[0010] The present invention provides a speech variability coding method for pathological speech recognition, comprising: constructing a speech variability variational autoencoder model to capture and characterize the acoustic variability of pathological speech signals and reconstruct acoustic features; using acoustic variability coding optimization training technology to enable the speech variability variational autoencoder model to avoid phoneme information and encode phoneme-independent acoustic variability; based on the reconstructed acoustic features, applying the speech variability variational autoencoder model to a pathological speech recognition system and constructing a normal speech synthesis system.

[0011] In one implementation, the speech variability variational autoencoder model is constructed to capture and characterize the acoustic variability of pathological speech signals and reconstruct acoustic features, including: the speech variability variational autoencoder in the speech variability variational autoencoder model adopts an encoder-decoder structure, and deploys a time series neural network to extract time series information in the pathological speech signal, wherein the time series neural network is a BiLSTM network or a Transformer layer based on multi-head self-attention; based on the time series information, the speech variability variational autoencoder reconstructs the acoustic input features into acoustic features containing speech variability through phoneme information and acoustic information.

[0012] In another implementation, the speech variability variational autoencoder based on time series information reconstructs the acoustic input features into acoustic features containing speech variability through phoneme information and acoustic information, including: converting the acoustic input features into low-dimensional latent space variables through the encoder of the speech variability variational autoencoder, constraining the encoder during the conversion process so that the encoder encodes the acoustic variability that is independent of the phoneme information, and the decoder reconstructs the low-dimensional latent space variables and the phoneme information into acoustic features containing speech variability.

[0013] In another implementation, in the speech variability variational autoencoder model, let o = {ot＝1：T} is the acoustic feature sequence, c={c t＝1：T} is a phoneme sequence, z={z t＝1：T} is the acoustic variation, and the generated speech variability variational autoencoder model can be expressed as o~p(o|c,z). In order to show the encoded acoustic variation, z is regarded as a latent variable generated by the encoder. The posterior distribution of z can be approximated by the probability encoder output distribution q(z|o,φ) using the stochastic gradient variational Bayes algorithm. The conditional probability of o is calculated as follows:

[0014]

[0015] where θ and φ represent the model parameters of the encoder and decoder, respectively, and KL(q||p) is the KL divergence between q(z|o, φ) and the prior probability p(z). By fixing θ, we maximize the lower bound It is equivalent to minimizing the KL divergence between the true posterior probability distribution p(z|o, c) and q(z|o, φ), so q(z|o, φ) represents an approximation of the true posterior probability distribution.

[0016] In another implementation, the encoder output probability distribution q(z|o, φ) can be calculated as a Gaussian distribution with diagonal covariance, and for time t in the sentence, we have The diagonal vectors of mean and standard deviation are implemented by a temporal neural network, using the entire sentence as input in order to take into account temporal information, which is represented as follows:

[0017]

[0018] Then, the latent variable z t Sampled as in In time, the latent variable z t The dimension is lower than

[0019] In another implementation, in the decoder, p(o|c, z, θ) is modeled as an isotropic Gaussian distribution, and for the tth time, there is When the time series neural network is a BiLSTM network, the mean vector can be implemented as:

[0020]

[0021] Among them, c t represents the one-hot vector of the t-th phoneme label. Variance σ 2 is assumed to be a tunable scalar constant.

[0022] In another implementation, the method further includes training and fine-tuning the speech variability variational autoencoder, and the specific steps include: first pre-training the decoder part of the speech variability variational autoencoder layer by layer until it converges, and then fine-tuning all parameters of the speech variability variational autoencoder together. During the training process of the speech variability variational autoencoder, the prior probability p(z) is assumed to be In formula (1) can be rewritten as:

[0023]

[0024] Where D represents the dimension of the latent space variable, J represents the number of Monte Carlo sampling, Is to use z t The d-th mean element of the decoder calculated from the j-th sample of can be absorbed by the learning rate, and the lower bound in formula (4) is maximized.

[0025] In another implementation, the acoustic variability coding optimization training technique used specifically includes average pooling, time delay, fixed decoder and context input;

[0026] Average pooling: Assuming that acoustic variation is stable in a short time, average pooling on the time axis can force the encoder to encode stable information in a short time at each moment. During the training of the autoencoder, average pooling is applied to the equation (2) When the temporal neural network is a BiLSTM network, it can be written as:

[0027]

[0028] Where τ is the radius of the time period;

[0029] Time delay: The time delay forces the decoder to use the latent variable at the previous time t-Δt as input at time t, where Δt is the delay time and the expected latent variable z t-Δt Coding and z t Similar information, the decoder y in formula (3) t Rewritten as:

[0030]

[0031] Fixed decoder: After pre-training the decoder, the parameters in the decoder can be fixed, and then only the encoder is trained in the fine-tuning step of the speech variability variational autoencoder to force the latent variables to encode the information about the complementarity of the phoneme sequence;

[0032] Contextual input: The encoder uses continuous frames of acoustic features as input, rather than a single frame at each time instant.

[0033] In another implementation, the method further includes sampling the latent variables generated by the encoder and concatenating them with the contextual acoustic input features as auxiliary features of the acoustic model in the speech recognition system, and using the auxiliary features to retrain the acoustic model, so that the speech recognition system is adapted to the pathological speech scene;

[0034] During the acoustic model training, the mean and variance calculated by the encoder are used to sample the latent variable z to update the acoustic model parameters. The auxiliary features at the tth time can be expressed as:

[0035] o′ t =Concatenation{o t-τ , ..., o t , ..., o t+τ , z t，j} (7)

[0036] Where τ is the radius of the context window, z t，j represents the latent variable sampled at the jth update at time t.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The speech variability variational autoencoder designed in the present invention is specifically designed for speech variability, and captures and features pathological speech variability through an encoding-decoding process.

[0039] (2) The acoustic variability coding optimization training of the present invention adopts a series of techniques to ensure that the encoded speech variability is independent of the phoneme information.

[0040] (3) The method of the present invention has strong versatility. It directly encodes the variability of pathological speech instead of targeting the specific characteristics of the language. In this way, the present invention can be applied to pathological speech caused by various diseases, and can also be personalized for speech recognition of a specific speech disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is an overall schematic diagram of the speech variability encoding method for pathological speech recognition of the present invention.

[0042] Figure 2 The present invention is a schematic diagram of the structure of a speech variability variational autoencoder for the speech variability encoding method for pathological speech recognition. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 The present invention provides a speech variability coding method for pathological speech recognition, comprising:

[0045] Construct a speech variability variational autoencoder model to capture and characterize the acoustic variability of pathological speech signals and reconstruct acoustic features;

[0046] The acoustic variability coding optimization training technique is used to make the speech variability variational autoencoder model avoid phoneme information and encode phoneme-independent acoustic variability.

[0047] Based on the reconstructed acoustic features, the speech variability variational autoencoder model is applied to the pathological speech recognition system and the normal speech synthesis system.

[0048] The present invention has the following beneficial effects:

[0049] (1) The speech variability variational autoencoder designed in the present invention is specifically designed for speech variability, and captures and features pathological speech variability through an encoding-decoding process.

[0050] (2) The acoustic variability coding optimization training of the present invention adopts a series of techniques to ensure that the encoded speech variability is independent of the phoneme information.

[0051] (3) The method of the present invention has strong versatility. It directly encodes the variability of pathological speech instead of targeting the specific characteristics of the language. In this way, the present invention can be applied to pathological speech caused by various diseases, and can also be personalized for speech recognition of a specific speech disease.

[0052] Optionally, the speech variability variational autoencoder model is constructed to capture and characterize the acoustic variability of pathological speech signals and reconstruct acoustic features, including: the speech variability variational autoencoder in the speech variability variational autoencoder model adopts an encoder-decoder structure, and deploys a timing neural network to extract timing information in the pathological speech signal, wherein the timing neural network is a BiLSTM network or a Transformer layer based on multi-head self-attention; based on the timing information, the speech variability variational autoencoder reconstructs the acoustic input features into acoustic features containing speech variability through phoneme information and acoustic information.

[0053] Optionally, the speech variability variational autoencoder based on time series information reconstructs the acoustic input features into acoustic features containing speech variability through phoneme information and acoustic information, including: converting the acoustic input features into low-dimensional latent space variables through the encoder of the speech variability variational autoencoder, constraining the encoder during the conversion process so that the encoder encodes acoustic variability that is independent of phoneme information, and the decoder reconstructs the low-dimensional latent space variables and phoneme information into acoustic features containing speech variability.

[0054] Specifically, the scheme of the present invention is further described according to the following examples:

[0055] The present invention aims to construct a speech variability autoencoder model based on encoding-decoding to capture and characterize the acoustic variability of pathological speech. The characterized acoustic variability is used as an auxiliary feature to improve the performance and robustness of the acoustic model in the speech recognition system, making it suitable for pathological speech recognition scenarios, thereby promoting special language groups to benefit from speech-driven assistive technology and smart devices.

[0056] Speech variability variational autoencoder model design:

[0057] The speech variability variational autoencoder adopts an encoder-decoder structure and deploys a temporal neural network such as a bidirectional long short-term memory network model (Bidirectional Long Short-Term Memory, BiLSTM) or a Transformer layer based on multi-head self-attention to extract temporal information from speech signals. The encoder converts the input features into low-dimensional latent space variables. The conversion process constrains the encoder to encode acoustic variability that is unrelated to phoneme information. The decoder reconstructs the low-dimensional latent space variables and phoneme information into acoustic features containing speech variability.

[0058] Speech variability variational autoencoder structure:

[0059] Acoustic features are considered as a joint representation of phoneme information and acoustic variation. When using a low-dimensional latent space, the information encoded from the acoustic features is limited. At the same time, the encoded information is jointly used with the phoneme information to generate the acoustic features. This will force the encoding to retain more variation information independent of the phonemes. The generation process is as follows Figure 2 As shown (the encoder part).

[0060] Optionally, in the speech variability variational autoencoder model, let o = {o t＝1：T} is the acoustic feature sequence, c={c t＝1：T} is a phoneme sequence, z={z t＝1：T} is the acoustic variation, and the generated speech variability variational autoencoder model can be expressed as o~p(o|c,z). In order to show the encoded acoustic variation, z is regarded as a latent variable generated by the encoder. The posterior distribution of z can be approximated by the probability encoder output distribution q(z|o,φ) using the stochastic gradient variational Bayes algorithm. The conditional probability of o is calculated as follows:

[0061]

[0062] where θ and φ represent the model parameters of the encoder and decoder, respectively, and KL(q||p) is the KL divergence between q(z|o, φ) and the prior probability p(z). By fixing θ, we maximize the lower bound It is equivalent to minimizing the KL divergence between the true posterior probability distribution p(z|o, c) and q(z|o, φ), so q(z|o, φ) represents an approximation of the true posterior probability distribution.

[0063] Encoder part:

[0064] Optionally, the encoder output probability distribution q(z|o, φ) can be calculated as a Gaussian distribution with diagonal covariance, where for time t in the sentence, The diagonal vectors of mean and standard deviation are implemented by a temporal neural network, such as BiLSTM, using the entire sentence as input in order to take into account temporal information, which is represented as follows:

[0065]

[0066] Then, the latent variable z t Sampled as in In time, the latent variable z t The dimension should be significantly lower than The structure of this encoder is as follows Figure 2 The left half of the diagram is shown.

[0067] Decoder part:

[0068] Optionally, in the decoder, p(o|c, z, θ) is modeled as an isotropic Gaussian distribution, and for time t, we have When the time series neural network is a BiLSTM network, the mean vector can be implemented as:

[0069]

[0070] Among them, c t represents the one-hot vector of the t-th phoneme label. Variance σ 2 is assumed to be an adjustable scalar constant. The decoder structure is as follows Figure 2 The right half of .

[0071] Encoder-decoder training and fine-tuning:

[0072] Optionally, the method further includes training and fine-tuning the speech variability variational autoencoder, and the specific steps include: first pre-training the decoder part of the speech variability variational autoencoder layer by layer until it converges, and then fine-tuning all parameters of the speech variability variational autoencoder together. During the training process of the speech variability variational autoencoder, the prior probability p(z) is assumed to be In formula (1) can be rewritten as:

[0073]

[0074] Where D represents the dimension of the latent space variable, J represents the number of Monte Carlo sampling, Is to use z t The d-th mean element of the decoder calculated by the d-th sample. During the training process, the previous term can be absorbed by the learning rate, and the lower bound in formula (4) is maximized.

[0075] Encoding Optimization Training Techniques for Acoustic Variability:

[0076] Autoencoder training uses a range of techniques to avoid encoding phoneme information, allowing the autoencoder to focus on acoustic variability; the techniques used include average pooling, time delays, fixed decoders, and contextual inputs. Although the encoder uses a low-dimensional latent space, it may also greedily try to model frame-level phoneme information, causing the autoencoder to degenerate into an identity transformation of acoustic features. Some operations can be applied to prevent the encoder from encoding frame-level phoneme information, thereby optimizing the encoding of acoustic variability.

[0077] Optionally, the acoustic variability coding optimization training technique used specifically includes average pooling, time delay, fixed decoder and context input;

[0078] Average pooling: Assuming that acoustic variation is stable in a short time, average pooling on the time axis can force the encoder to encode stable information in a short time at each moment. During the training of the autoencoder, average pooling is applied to the equation (2) When the temporal neural network is a BiLSTM network, it can be written as:

[0079]

[0080] Where τ is the radius of the time period;

[0081] Time delay: The time delay forces the decoder to use the latent variable at the previous time t-Δt as input at time t, where Δt is the delay time and the expected latent variable z t-Δt Coding and z t Similar information, the decoder y in formula (3) t Rewritten as:

[0082]

[0083] Fixed decoder: After pre-training the decoder, the parameters in the decoder can be fixed, and then only the encoder is trained in the fine-tuning step of the speech variability variational autoencoder to force the latent variables to encode the information about the complementarity of the phoneme sequence;

[0084] Contextual input: The encoder uses continuous frames of acoustic features as input, rather than a single frame at each time instant.

[0085] Variability variational autoencoder applied to pathological speech recognition:

[0086] (1) Pathological speech features

[0087] Optionally, the method further comprises sampling the latent variables generated by the encoder and concatenating them with the contextual acoustic input features as auxiliary features of the acoustic model in the speech recognition system, and using the auxiliary features to retrain the acoustic model, so that the speech recognition system is adapted to the pathological speech scene;

[0088] During the acoustic model training, the mean and variance calculated by the encoder are used to sample the latent variable z to update the acoustic model parameters. The auxiliary features at the tth time can be expressed as:

[0089] o′ t =Concatenation{o t-τ , ..., o t , ..., o t+τ , z t，j} (7)

[0090] Where τ is the radius of the context window, z t，j represents the latent variable sampled at the jth update at time t.

[0091] (2) Acoustic model training strategy

[0092] Considering that pathological speech data resources are limited, latent variables will inevitably capture some variation information that is irrelevant to the speech recognition target. Therefore, o′ is used t Training an acoustic model from scratch is prone to overfitting. One way to solve this problem is to use o′ t Retrain an already trained acoustic model for a small number of steps.

[0093] The training strategy of the acoustic model in the pathological speech recognition system is to update the trained model parameters with a small learning rate and update the parameters related to z with a large learning rate. t，j In the test phase, the latent variable z in formula (7) t，j The mean vector Replacement for final decoding and recognition.

[0094] In summary, the present invention adopts a speech variability encoder based on a variational autoencoder to explicitly encode the variation of pathological speech. The encoder uses phoneme information and low-dimensional latent variables to reconstruct the input acoustic features. In this way, the latent variables are forced to encode variations independent of phonemes. Temporal neural networks such as BiLSTM or Transformer layers are used in the encoder to capture the temporal information of pathological speech. The stochastic gradient variational Bayes algorithm is used to model the distribution of generated variation codes. In the acoustic modeling of pathological speech, these codes are further used as auxiliary input features of the acoustic model in the speech recognition system, thereby improving the performance of the speech recognition system in pathological speech recognition scenarios.

[0095] It should be noted that the present invention may be a method, a system, an apparatus and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0096] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0097] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0098] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0099] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0101] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0102] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0103] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A speech variability coding method for pathological speech recognition, characterized in that: include: Construct a speech variability variational autoencoder model to capture and characterize the acoustic variability of pathological speech signals and reconstruct acoustic features; The acoustic variability coding optimization training technique is used to make the speech variability variational autoencoder model avoid phoneme information and encode phoneme-independent acoustic variability. Based on the reconstructed acoustic features, the speech variability variational autoencoder model is applied to the pathological speech recognition system and the normal speech synthesis system.

2. The method according to claim 1, characterized in that The speech variability variational autoencoder model is constructed to capture and characterize the acoustic variability of pathological speech signals and reconstruct acoustic features, including: The speech variability variational autoencoder in the speech variability variational autoencoder model adopts an encoder-decoder structure and deploys a time series neural network to extract the time series information in the pathological speech signal, where the time series neural network is a BiLSTM network or a Transformer layer based on multi-head self-attention; Based on the temporal information, the speech variability variational autoencoder reconstructs the acoustic input features into acoustic features containing speech variability through phoneme information and acoustic information.

3. The method according to claim 2, characterized in that The speech variability variational autoencoder based on time sequence information reconstructs the acoustic input features into acoustic features containing speech variability through phoneme information and acoustic information, including: The encoder of the speech variability variational autoencoder converts the acoustic input features into low-dimensional latent space variables. During the conversion process, the encoder is constrained to encode acoustic variability that is unrelated to phoneme information. The decoder reconstructs the low-dimensional latent space variables and phoneme information into acoustic features containing speech variability.

4. The method according to claim 3, characterized in that In the speech variability variational autoencoder model, let o = {o t＝1：T } is the acoustic feature sequence, c={c t＝1：T } is a phoneme sequence, z={z t＝1：T } is the acoustic variation, and the generated speech variability variational autoencoder model can be expressed as o~p(o|c,z). In order to show the encoded acoustic variation, z is regarded as a latent variable generated by the encoder. The posterior distribution of z can be approximated by the probability encoder output distribution q(z|o,φ) using the stochastic gradient variational Bayes algorithm. The conditional probability of o is calculated as follows: where θ and φ represent the model parameters of the encoder and decoder, respectively, and KL(q||p) is the KL divergence between q(z|o, φ) and the prior probability p(z). By fixing θ, we maximize the lower bound It is equivalent to minimizing the KL divergence between the true posterior probability distribution p(z|o, c) and q(z|o, φ), so q(z|o, φ) represents an approximation of the true posterior probability distribution.

5. The method according to claim 4, characterized in that The encoder output probability distribution q(z|o, φ) can be calculated as a Gaussian distribution with diagonal covariance. For time t in the sentence, we have The diagonal vectors of mean and standard deviation are implemented by a temporal neural network, using the entire sentence as input in order to take into account temporal information, which is represented as follows: Then, the latent variable z t Sampled as in In time, the latent variable z t The dimension is lower than 6. The method according to claim 5, characterized in that In the decoder, p(o|c, z, θ) is modeled as an isotropic Gaussian distribution. For the tth time, we have When the time series neural network is a BiLSTM network, the mean vector can be implemented as: Among them, c t represents the one-hot vector of the t-th phoneme label. Variance σ 2 is assumed to be a tunable scalar constant.

7. The method according to claim 6, characterized in that The method further includes training and fine-tuning the speech variability variational autoencoder, and the specific steps include: First, the decoder part of the speech variability variational autoencoder is pre-trained layer by layer until it converges, and then all parameters of the speech variability variational autoencoder are fine-tuned together. During the training process of the speech variability variational autoencoder, the prior probability p(z) is assumed to be In formula (1) can be rewritten as: Where D represents the dimension of the latent space variable, J represents the number of Monte Carlo sampling, Is to use z t The d-th mean element of the decoder calculated from the j-th sample of can be absorbed by the learning rate, and the lower bound in formula (4) is maximized.

8. The method according to claim 7, characterized in that The acoustic variability coding optimization training techniques used specifically include average pooling, time delay, fixed decoder, and context input; Average pooling: Assuming that acoustic variation is stable in a short time, average pooling on the time axis can force the encoder to encode stable information in a short time at each moment. During the training of the autoencoder, average pooling is applied to the equation (2) When the temporal neural network is a BiLSTM network, it can be written as: Where τ is the radius of the time period; Time delay: The time delay forces the decoder to use the latent variable at the previous time t-Δt as input at time t, where Δt is the delay time and the expected latent variable z t-Δt Coding and z t Similar information, the decoder y in formula (3) t Rewritten as: Fixed decoder: After pre-training the decoder, the parameters in the decoder can be fixed, and then only the encoder is trained in the fine-tuning step of the speech variability variational autoencoder to force the latent variables to encode the information about the complementarity of the phoneme sequence; Contextual input: The encoder uses continuous frames of acoustic features as input, rather than a single frame at each time instant.

9. The method according to claim 8, characterized in that The method also includes sampling the latent variables generated by the encoder and concatenating them with the contextual acoustic input features as auxiliary features of the acoustic model in the speech recognition system, and using the auxiliary features to retrain the acoustic model, so that the speech recognition system is adapted to the pathological speech scene; During the acoustic model training, the mean and variance calculated by the encoder are used to sample the latent variable z to update the acoustic model parameters. The auxiliary features at the tth time can be expressed as: o′ t =Concatenation{o t-τ ,...,o t ,...,o t+τ ,z t，j } (7) Where τ is the radius of the context window, z t，j Represents the latent variable sampled at the tth update.

Images