Self-supervised learning speech recognition method and system based on brain activity information
By adopting a self-supervised learning speech recognition method based on brain activity information in the power grid field, the problem of poor speech recognition effect of unsupervised learning model when the data volume is insufficient is solved, and better generalization ability and recognition effect are achieved. It is suitable for power grids Small data volume scenarios in the field.
Patent Information
- Application Number
- CN202411644560.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-02-18
AI Technical Summary
In the field of power grid, the speech recognition model based on unsupervised learning leads to poor speech recognition effect when the amount of training data is insufficient.
Using a self-supervised learning speech recognition method based on brain activity information, a self-supervised speech recognition model is established by collecting speech signals and brain blood oxygen level changes, and a composite convolutional layer, a Transformer layer and a linear projection layer are used for preliminary training, and then through The convolution training unit and the standardization layer are further processed, and a self-supervised speech recognition model data set is constructed, and trained through the loss function layer to obtain the trained self-supervised speech recognition model.
In the case of low signal-to-noise ratio, it has better generalization ability than other voice models, can better adapt to the working scenarios of small data volume in the power grid field, and improve the accuracy and recognition effect of speech recognition.
Smart Images
Figure CN119479631A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition, and in particular to a self-supervised learning speech recognition method and system based on brain activity information. Background Art
[0002] With the rapid development of energy internet, the working mode of power grid is undergoing earth-shaking changes, and speech recognition technology is widely used in the field of power grid. At present, most speech recognition models used in the field of power grid are based on unsupervised learning, which reduces the amount of data labeling but often requires a large amount of training data, which is often difficult to achieve in the field of power grid.
[0003] With the development of technology in recent years, the addition of self-supervised learning has brought revolutionary changes to the field of speech processing. The models generated by it can extract effective and robust universal features from unlabeled sample data through adversarial learning, clustering and other methods. However, most of the current research is limited to theory, and there are few practical applications, especially in the field of power grids. The research and application are even rarer and cannot be promoted and applied.
[0004] Therefore, there is an urgent need for a speech recognition method for the power grid field where the amount of training data is insufficient. Summary of the invention
[0005] Purpose of the invention: The present invention aims to provide a self-supervised learning speech recognition method and system based on brain activity information, so as to solve the problem of poor speech recognition effect caused by insufficient training data in speech recognition algorithms based on unsupervised learning.
[0006] Technical solution: The present invention discloses a self-supervised learning speech recognition method based on brain activity information, comprising the following steps:
[0007] Step 1: Collect voice signals and scan brain blood oxygen level change signals generated simultaneously with the voice signals, wherein the brain blood oxygen level change signals represent brain activity information;
[0008] Step 2: Establish a self-supervised speech recognition model training unit, perform preliminary speech recognition model training on the collected speech signals and brain activity information, and obtain an output result Y, wherein the self-supervised speech recognition model training unit includes a composite convolutional layer, a Transformer layer, and a linear projection layer;
[0009] Step 3: Establish a convolution training unit and a standardization layer, wherein the convolution training unit performs a convolution operation on the output result Y, and then the standardization layer standardizes the Y after the convolution operation;
[0010] Step 4: Establish a linear regression layer and a loss function layer, perform linear modeling on the speech signal and the brain blood oxygen level change signal through the linear regression layer, and construct a self-supervised speech recognition model dataset Z; train the self-supervised speech recognition model dataset Z through the loss function layer to obtain the characteristic parameters Z* of the trained self-supervised speech recognition model;
[0011] Step 5: Output the characteristic parameter Z* of the trained self-supervised speech recognition model to obtain the trained self-supervised speech recognition model, and use the model to recognize the collected speech signal to be recognized.
[0012] Furthermore, the voice signal in step 1 is acquired by a microphone array, and the brain blood oxygen level change signal is obtained by scanning with a functional magnetic resonance imaging scanner, and each voice signal corresponds to a brain blood oxygen level change signal generated simultaneously therewith.
[0013] Furthermore, in step 2, the speech signal and the blood oxygen level change signal representing brain activity information are used as input signals and input into the self-supervised speech model training unit. The processing process is as follows:
[0014] First, the context representation C of the input signal is obtained through multiple layers of composite convolutional layers and Transformer layers.
[0015] Then it is projected to the output layer Y through the linear projection layer, which is expressed as:
[0016] Y=L m,n C
[0017] Among them, L mn Represents an m×n matrix, where m and n are the sizes of the set training units.
[0018] Furthermore, in step 3, the convolution training unit performs a convolution operation on the output result Y, and then performs batch normalization processing, sets the batch size t, and divides the convolution-operated Y into several groups of small batch data sets X on average, and calculates the mean μB and variance of each group of small batch data sets X. The formula is:
[0019]
[0020]
[0021] Among them, x i is the value of the i-th data point in this mini-batch dataset, and t is the size of this mini-batch dataset.
[0022] Furthermore, the several groups of small batch data sets X are normalized, and the formula is expressed as follows:
[0023]
[0024] Among them, x (k) is a data point in the mini-batch dataset X; ∈ is a very small natural number to avoid the denominator being zero; each data point x in X (k) The normalized Combine into a normalized dataset
[0025] Furthermore, the standardization operation of Y in step 3 adopts Z-score standardization, which is expressed as follows:
[0026]
[0027] Where s represents the data set Size, X mean It is a dataset The mean of the data set, standard deviatio The standard deviation of Y Z is the standardized Y.
[0028] Furthermore, a linear regression function is provided on the linear regression layer in step 4, and a linear regression function is used to perform linear modeling on the speech signal and the brain blood oxygen level change signal to construct a self-supervised speech recognition model dataset Z. The linear regression function is expressed as:
[0029]
[0030] Among them, x z is a data point in the self-supervised speech recognition model dataset Z.
[0031] Furthermore, the loss function is a smooth-L1 regularized mean square error function, which is expressed as follows:
[0032]
[0033] Furthermore, based on this loss function, an L2 regularization term is added to train the model parameters, which can be expressed as:
[0034]
[0035] Among them, λ is a preset hyperparameter, and the characteristic parameter Z* of the self-supervised speech recognition model is obtained after the loss function training is completed.
[0036] The present invention also discloses a self-supervised learning speech recognition system based on brain activity information, comprising:
[0037] An input layer, for collecting speech signals and scanning brain blood oxygen level change signals generated simultaneously with the speech signals;
[0038] A model training unit, used to perform preliminary speech recognition model training on the collected speech signals and brain activity information;
[0039] A convolution training unit, used to perform a convolution operation on the output result of the self-supervised speech model training unit;
[0040] The normalization layer is used to normalize the output results of the convolutional training unit;
[0041] A linear regression layer to predict the response relationship between speech signals and brain blood oxygen level change signals;
[0042] Loss function layer, used to train the parameters of the self-supervised speech recognition model;
[0043] The output layer is used to output the characteristic parameters of the trained self-supervised speech recognition model;
[0044] Self-supervised speech recognition model is used to realize the recognition of speech signals.
[0045] Beneficial effects: The present invention provides a self-supervised learning speech recognition method and system based on brain activity information. Compared with the prior art, the present invention integrates the brain activity information corresponding to the speech signal into the training process of the characteristic parameters of the speech recognition model by establishing a neural coding model, and has better generalization ability than other speech models under low signal-to-noise ratio conditions. By improving the recognition effect of the traditional speech recognition model, the present invention can better adapt to the working scenarios in the power grid field.
[0046] The present invention solves the problem of poor recognition effect of previous unsupervised learning speech recognition models and is a speech recognition method for the situation where the amount of training data is insufficient in the field of power grids. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A flowchart of a self-supervised learning speech recognition method based on brain activity information of the present invention;
[0048] Figure 2 This is a structural diagram of the self-supervised speech model training unit;
[0049] Figure 3 4 is a structural diagram of a convolutional training unit according to an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The following further describes the technical solution of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all embodiments. Figure 1 Shown is a flow chart of a self-supervised learning speech recognition method based on brain activity information of the present invention. Next, the method of the present invention is explained in conjunction with a specific implementation method.
[0051] Step 1: Collect voice signals and scan brain blood oxygen level change signals generated simultaneously with the voice signals, wherein the brain blood oxygen level change signals represent brain activity information.
[0052] The speech signal is acquired by a microphone array, and the brain blood oxygen level change signal is obtained by a functional magnetic resonance imaging scanner. Each speech signal corresponds to a brain blood oxygen level change signal generated simultaneously with it. The signal data obtained each time is projected onto a matrix area composed of 40962 three-dimensional pixels, representing all the regions of interest in the human brain. Subsequently, the brain area is divided into 180 regions of interest, and the pixel areas corresponding to the brain interest areas related to auditory and language processing are selected as the information areas for this experiment. After processing, each pre-processed blood oxygen level dependent comparison data is represented by a 5085-dimensional vector, corresponding to 2468 pixel areas in the right brain and 2617 pixel areas in the left brain.
[0053] Step 2: Establish a self-supervised speech recognition model training unit to perform preliminary speech recognition model training on the collected speech signals and brain activity information. The self-supervised speech model training unit includes a composite convolutional layer, a Transformer layer, and a linear projection layer.
[0054] like Figure 2 The flowchart of the self-supervised speech model training unit is shown. After the speech signal in step 2 is input into the self-supervised speech model training unit, it first passes through a 12-layer composite convolutional layer to obtain a potential speech representation, then passes through a Transformer layer to obtain a contextual representation C of the speech signal, and then passes through a linear projection layer to project it to the output layer Y, which can be expressed as:
[0055] Y=L m，n C
[0056] Among them, L m,n Represents an m×n matrix, where m and n are the sizes of the set training units, respectively, through the matrix L m,n Linear projection is performed on the above line text representation C to obtain output data Y. In this embodiment, m is set to 150 and n is set to 75.
[0057] Step 3: Establish a convolutional training unit and a standardization layer. The convolutional training unit performs a convolution operation on the output result of the self-supervised speech model training unit in step 2, and the output result of the convolutional training unit is standardized through the standardization layer.
[0058] like Figure 3 The flowchart of the convolution training unit of the present invention is shown, wherein the RELU function is an activation function. The convolution training unit in step 3 includes 4 convolution modules, each of which includes a one-dimensional convolution layer for batch normalization. In this embodiment, the step sizes of the 4 convolution modules are N÷n, 5, 5, and 3, respectively, where N refers to the number of frames.
[0059] After convolution of the input data, batch normalization is performed to make the output data distribution consistent. Set the batch size t, divide the convolution Y into several small batch data sets X, and calculate the mean μ of each small batch data set X. B and variance The formula is:
[0060]
[0061]
[0062] Among them, x i is the value of the i-th data point in this mini-batch dataset, and t is the size of this mini-batch dataset.
[0063] The plurality of groups of small batch data are normalized to adjust each feature of the small batch data to μB=0 and The processing formula is as follows:
[0064]
[0065] Among them, x (k) is a data point in the mini-batch dataset X; ∈ is a very small natural number to avoid the denominator being zero; each data point x in X (k) The normalized Combine into a normalized dataset
[0066] In this embodiment, an activation function is added here, and nonlinear characteristics are introduced by the activation function, so that the model can better process complex features in speech signals and blood oxygen level signals. The formula is as follows:
[0067] f(x)=x + =max(0,x)
[0068] In the above formula, x is the output data of the normalization process.
[0069] The standardization operation described in step 3 uses Z scoreStandardization: First, calculate the mean and standard deviation of the data set. Then, for each data point, subtract the mean to get the distance from the mean. Divide the distance from the mean by the standard deviation to get Z. score The standardized value is expressed as:
[0070]
[0071] Where s represents the data set Size, X mean It is a dataset The mean of the data set, standard deviatio The standard deviation of .
[0072] Step 4: Establish a linear regression layer and a loss function layer, perform linear modeling on the speech signal and the brain blood oxygen level change signal through the linear regression layer, and construct a self-supervised speech recognition model dataset Z; train the self-supervised speech recognition model dataset Z through the loss function layer to obtain the characteristic parameters Z* of the trained self-supervised speech recognition model.
[0073] The linear regression layer models the linear relationship between the speech signal and the brain blood oxygen level change signal through linear regression. The linear regression function is used to linearly model the speech signal and the brain blood oxygen level change signal to obtain the self-supervised speech recognition model dataset Z. The linear regression function is expressed as:
[0074]
[0075] Among them, x z is a data point in the self-supervised speech recognition model dataset Z.
[0076] The loss function is the smooth L1 regularized mean square error function, which is expressed as
[0077]
[0078] After training with the loss function, the feature parameters of the self-supervised speech recognition model are obtained. Feature parameters refer to the parameters extracted from the original speech signal data set to represent the characteristics of speech. As the input of the model, the quality of the feature parameters directly affects the learning effect of the model parameters. The feature parameters used in this example are Mel-frequency cepstral coefficients (MFCC) and filter bank feature coefficients (Fbank).
[0079] Based on this loss function, an L2 regularization term is added to train the model parameters. It can be expressed as:
[0080]
[0081] Among them, λ is a preset hyperparameter, and the characteristic parameter Z* of the self-supervised speech recognition model is obtained after the loss function training is completed.
[0082] Step 5: Output the characteristic parameter Z* of the trained self-supervised speech recognition model to obtain the trained self-supervised speech recognition model, and use the model to recognize the collected speech signal to be recognized.
[0083] The self-supervised learning speech recognition model obtained by the present invention automatically learns speech representation for unlabeled data, and captures key features by predicting certain parts of the speech signal, thereby achieving accurate recognition of the speech signal with strong generalization ability. In contrast, the traditional speech recognition model relies on manually labeled feature data, and performs speech recognition by training acoustic models and language models. It has good recognition effects in specific scenarios, but its generalization ability is weak. For work scenarios with small data volumes in the power grid field, the self-supervised learning speech recognition model involved in the present invention performs better in processing complex speech signals.
[0084] The present invention integrates the brain activity information corresponding to the speech signal into the training process of the characteristic parameters of the speech recognition model by establishing a neural coding model, and has better generalization ability than other speech models under low signal-to-noise ratio conditions. By improving the recognition effect of the speech recognition model, the present invention can better adapt to the working scenarios of small data volumes in the power grid field.
Claims
1. A self-supervised learning speech recognition method based on brain activity information, characterized in that: The specific steps include: Step 1: Collect voice signals and scan brain blood oxygen level change signals generated simultaneously with the voice signals, wherein the brain blood oxygen level change signals represent brain activity information; Step 2: Establish a self-supervised speech recognition model training unit, perform preliminary speech recognition model training on the collected speech signals and brain activity information, and obtain an output result Y, wherein the self-supervised speech recognition model training unit includes a composite convolutional layer, a Transformer layer, and a linear projection layer; Step 3: Establish a convolution training unit and a standardization layer, wherein the convolution training unit performs a convolution operation on the output result Y, and then the standardization layer standardizes the Y after the convolution operation; Step 4: Establish a linear regression layer and a loss function layer, perform linear modeling on the speech signal and the brain blood oxygen level change signal through the linear regression layer, and construct a self-supervised speech recognition model dataset Z; train the self-supervised speech recognition model dataset Z through the loss function layer to obtain the characteristic parameters Z* of the trained self-supervised speech recognition model; Step 5: Output the characteristic parameter Z* of the trained self-supervised speech recognition model to obtain the trained self-supervised speech recognition model, and use the model to recognize the collected speech signal to be recognized.
2. The self-supervised learning speech recognition method according to claim 1, characterized in that: In step 1, the speech signal is acquired by a microphone array, and the brain blood oxygen level change signal is obtained by scanning with a functional magnetic resonance imaging scanner. Each speech signal corresponds to a brain blood oxygen level change signal generated simultaneously with the speech signal.
3. The self-supervised learning speech recognition method according to claim 1, characterized in that: Step 2 uses the speech signal and the blood oxygen level change signal representing brain activity information as input signals and inputs them into the self-supervised speech model training unit. The processing process is as follows: First, the context representation C of the input signal is obtained through multiple layers of composite convolutional layers and Transformer layers. Then it is projected to the output layer Y through the linear projection layer, which is expressed as: Y=L m,n C Among them, L m,n Represents an m×n matrix, where m and n are the sizes of the set training units.
4. The self-supervised learning speech recognition method according to claim 1, characterized in that: Step 3: After the convolution training unit performs a convolution operation on the output result Y, it performs batch normalization processing, sets the batch size t, and divides the convolution-operated Y into several groups of small batch data sets X on average. For each group of small batch data sets X, the mean μ of the data set is calculated. B and variance The formula is: Among them, x i is the value of the i-th data point in this mini-batch dataset, and t is the size of this mini-batch dataset.
5. The self-supervised learning speech recognition method according to claim 4, characterized in that: The several groups of small batch data sets X are normalized, and the formula is expressed as: Among them, x (k) is a data point in the mini-batch dataset X; ∈ is a very small natural number to avoid the denominator being zero; each data point x in X (k) The normalized Combine into a normalized dataset 6. The self-supervised learning speech recognition method according to claim 5, characterized in that: The standardization operation for Y described in step 3 uses Z-score standardization, which is expressed as: Where s represents the data set Size, X mean It is a dataset The mean of the data set, standard deviatio The standard deviation of Y Z is the standardized Y.
7. The self-supervised learning speech recognition method according to claim 6, characterized in that: In step 4, a linear regression function is provided on the linear regression layer, through which the speech signal and the brain blood oxygen level change signal are linearly modeled to construct a self-supervised speech recognition model data set Z. The linear regression function is expressed as: Among them, x z is a data point in the self-supervised speech recognition model dataset Z.
8. The self-supervised learning speech recognition method according to claim 7, characterized in that: The loss function is the smooth-L1 regularized mean square error function, which is expressed as:
9. The self-supervised learning speech recognition method according to claim 8, characterized in that: On the basis of this loss function, an L2 regularization term is added to train the model parameters, which can be expressed as: Among them, λ is a preset hyperparameter, and the characteristic parameter Z* of the self-supervised speech recognition model is obtained after the loss function training is completed.
10. A self-supervised learning speech recognition system based on brain activity information, characterized in that: include: An input layer, for collecting speech signals and scanning brain blood oxygen level change signals generated simultaneously with the speech signals; A model training unit, used to perform preliminary speech recognition model training on the collected speech signals and brain activity information; A convolution training unit, used to perform a convolution operation on the output result of the self-supervised speech model training unit; The normalization layer is used to normalize the output results of the convolutional training unit; A linear regression layer to predict the response relationship between speech signals and brain blood oxygen level change signals; Loss function layer, used to train the parameters of the self-supervised speech recognition model; The output layer is used to output the characteristic parameters of the trained self-supervised speech recognition model; Self-supervised speech recognition model is used to realize the recognition of speech signals.