Self-adaptive audio-video speech recognition method and device

Abstract

The invention discloses a self-adaptive audio-video speech recognition method and device. The method comprises the following steps: constructing an audio-video speech recognition base model; training the audio-video speech recognition base model by using the video modal training data set and the audio modal training data set to obtain a parameter-cured audio-video speech recognition base model; accessing an audio and video input parameter fine-tuning model and an audio and video parameter adaptation model to the parameter-fixed audio and video voice recognition base model to obtain a fine-tuning audio and video voice recognition model; and processing a target speaker audio modal data set and a target speaker video modal data set by using the fine-tuning audio-video speech recognition model to obtain target speaker feature information. According to the invention, by introducing the audio and video input parameter fine tuning model and the fine tuning parameter adaptation model, the efficiency of capturing the feature information of the target speaker and the recognition accuracy of the data of the unseen speaker are improved, and the rapid adaptive audio and video speech recognition of the unseen speaker is realized.

Description

Technical Field

[0001] The present invention relates to the fields of speech recognition, multimodal fusion and human-computer interaction, and in particular to an adaptive audio-visual speech recognition method and device. Background Art

[0002] Audio-Visual Speech Recognition (AVSR) is a technology that combines audio and visual information for speech recognition. Unlike traditional audio-only speech recognition, AVSR uses the speech signal in combination with visual information such as the speaker's mouth shape and facial movements to improve speech recognition performance. This technology aims to more accurately capture the context of speech and the speaker's mouth shape by fusing multimodal information, thereby improving the robustness of the speech recognition system in noisy environments and ambiguous pronunciations.

[0003] Speaker adaptation is a task applied in the field of speech processing, which aims to enable a pre-trained speech recognition (ASR) model to effectively adapt to the speech characteristics of a new speaker. Usually, the pre-trained model is trained on large-scale speech data and has general speech recognition capabilities. However, since the speech characteristics of each speaker are different, the model may face performance degradation when facing a new speaker.

[0004] In speaker adaptation, the speech features of the new speaker are first extracted, such as spectral features or mel-spectrogram features. These features are then used to fine-tune the pre-trained model to adjust the model parameters to adapt to the speech features of the new speaker. This fine-tuning process usually requires less new speaker data and aims to improve the recognition performance of the model on a specific speaker while maintaining generalizability.

[0005] Recently, the concept of speaker adaptation has been gradually extended to areas such as voice and vision fusion (AVSR) and visual speech conversion (VSR). In these cross-modal tasks, the model needs to consider both speech and image information to better understand and process cross-modal data. Speaker adaptation in these scenarios allows the model to better adapt to the speech and visual features of new speakers to improve the understanding ability between different speakers and different modalities. This idea of ​​fast cross-modal adaptation reflects the research trend in multimodal human-computer interaction and cross-modal learning.

[0006] In some existing practical engineering applications, there is a continuous increase in data from unseen speakers, which requires continuous fine-tuning of the baseline model structure. However, practical engineering applications require reducing the time cost as much as possible, that is, it is hoped that the baseline model can be fine-tuned and adapted to the unseen speakers as soon as possible after obtaining a new batch of unseen speaker data. Therefore, it is necessary to choose to fine-tune only some parameters instead of adjusting the global parameters, and it is hoped that the adaptation effect after fine-tuning can be improved as much as possible.

[0007] Fast speaker adaptation requires not only enabling the baseline model to learn the features of unseen speakers, but also fast fine-tuning. The simplest way to fine-tune is to directly fine-tune the original base model after obtaining a batch of new data. This rough method will cause the model to update global parameters through the back propagation algorithm when fine-tuning, which will undoubtedly greatly increase computing power consumption and take a lot of time, and obviously does not meet the requirement of fast fine-tuning. "Fast" and "fine-tuning" are highly correlated, that is, after obtaining a small amount of data of the target speaker on the basis of the base model, a large number of parameters in the base model are frozen, so that during model fine-tuning training, the back propagation algorithm considers the gradient of these frozen parameters to be 0, so that only a small number of parameters are gradient updated to adjust the parameter values, and this small number of parameters may come from the model itself, that is, the small number of target fine-tuning parameters left after freezing a large number of model parameters; or it may come from a small number of additional parameters, that is, after obtaining the baseline model, the structure of the model is adjusted on a small scale or a new local network structure is introduced, so that these changed / introduced parameters need to be adjusted.

[0008] Among the fast adaptation methods that require additional introduction of model parameters, one is to introduce parameters in the input layer. This method is proposed to achieve fast adaptation of the VSR model for unseen speakers. That is, after the VSR pre-training obtains the benchmark model, a parameter matrix of the same size is added to its input, and then all the parameters of the backbone model are frozen during the formal fine-tuning, and only the parameters in the parameter matrix introduced at the input end are fine-tuned. This method introduces a small amount of parameters. By changing the input, the data distribution of the unseen speaker is fitted as much as possible to the data distribution of this part of the data used to train the benchmark model, so that the unseen speaker is approximated to the seen speaker in the base model, so as to achieve the purpose of fast adaptation. However, this method has two problems, namely: 1. If the data distribution range of the base model is narrow, that is, the data distribution of the unseen speaker and the existing speaker is quite different, it is difficult to help solve the problem of cross-subjects, and the effect of fine-tuning is relatively poor. 2. Because only the input end is fine-tuned, then for the subsequent frozen network layer parameters, the adjustment of this part of the parameters introduced at the input end will be limited by the subsequent frozen network parameters, so the fine-tuning effect will also be limited.

[0009] Another fast adaptive method that introduces additional model parameters is to improve the Adapter structure for fine-tuning, which is relatively simple: downsampling + activation function + upsampling, that is, first downsampling and reducing the dimension of the input data features in turn to obtain its more abstract high-level features, and then after passing through the activation function, use the upsampling linear layer to restore it to the specified feature resolution. This fine-tuning method places high demands on the baseline model structure to be introduced, the amount of parameters introduced, and the way of introduction. Different choices have a greater impact on the results of fine-tuning. The sparsity of the Adapter parameters introduced during fine-tuning may also cause the model to be overly adapted to the fine-tuning data, resulting in a decrease in generalization performance, that is, there is a risk of overfitting. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide an adaptive audio-visual speech recognition method and device. The present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, thereby improving the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizing rapid adaptive audio-visual speech recognition of unseen speakers.

[0011] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses an adaptive audio-visual speech recognition method, the method comprising:

[0012] S1, build the audio-visual speech recognition base model;

[0013] S2, obtaining a multi-person video modality training dataset and a multi-person audio modality training dataset;

[0014] S3, using the video modality training data set and the audio modality training data set to train the audio-visual speech recognition base model to obtain a parameter-fixed audio-visual speech recognition base model;

[0015] S4, obtaining a target speaker audio modality dataset and a target speaker video modality dataset;

[0016] S5, connecting the audio-visual input parameter fine-tuning model to the input layer of the parameter-fixed audio-visual speech recognition base model, and connecting the audio-visual parameter adaptation model to the feature fusion processing layer of the parameter-fixed audio-visual speech recognition base model to obtain a fine-tuned audio-visual speech recognition model;

[0017] S6, using the fine-tuned audio-visual speech recognition model, processing the target speaker audio modality dataset and the target speaker video modality dataset to obtain target speaker feature information.

[0018] As an optional implementation, in the first aspect of the embodiment of the present invention, the use of the multi-person video modality training data set and the multi-person audio modality training data set to train the audio-visual speech recognition base model to obtain a parameter-fixed audio-visual speech recognition base model includes:

[0019] S301, randomly initializing the initial values ​​of all parameters of the audio-visual speech recognition base model;

[0020] S302, setting the total number of training rounds n and the input parameter set of the audio-visual speech recognition base model;

[0021] S303, initializing the number of training rounds m=1, initializing the input parameter set of the audio-visual speech recognition base model;

[0022] S304, inputting the multi-person video modality training data set and the multi-person audio modality training data set into the audio-visual speech recognition base model;

[0023] The multi-person video modality training data set includes a plurality of multi-person video modality training data;

[0024] The multi-person audio modality training data set includes a plurality of multi-person audio modality training data;

[0025] S305: determining all of the multi-person video modality training data and the multi-person audio modality training data based on the multi-person video modality training data set and the multi-person audio modality training data set;

[0026] S306, using the weights and biases of the input layer of the audio-visual speech recognition base model, preprocessing any of the multi-person video modality training data and the multi-person audio modality training data to obtain multi-person preprocessed video modality training data and multi-person preprocessed audio modality training data;

[0027] S307, using a first processing function to process the multi-person pre-processed video modality training data and the multi-person pre-processed audio modality training data to obtain multi-person audio and video latent information and multi-person feature training label information;

[0028] S308, using a second processing function, processing the multi-person audio and video hidden information and multi-person feature training label information, updating the input parameter set, and obtaining an audio-visual speech recognition base model optimization parameter set;

[0029] The number of training rounds m=m+1;

[0030] S309, determining whether the number of training rounds m is greater than the total number of training rounds n, and obtaining a result of determining the number of training rounds;

[0031] When the result of the training times is no, executing S306 to S308;

[0032] When the result of the training times is yes, executing S310;

[0033] S310, performing optimization and selection processing on all the audio-visual-speech recognition base model parameter sets to obtain a fixed parameter set of the audio-visual-speech recognition base model;

[0034] S311, based on the audio-visual-speech recognition base model, solidify and freeze the solidified parameter set of the audio-visual-speech recognition base model to obtain a parameter-solidified audio-visual-speech recognition base model.

[0035] As an optional implementation, in the first aspect of the embodiment of the present invention, the use of the fine-tuned audio-visual speech recognition model to process the target speaker video modality dataset and the target speaker audio modality dataset to obtain target speaker feature information includes:

[0036] S61, initializing the audio-visual input fine-tuning parameters of the audio-visual input parameter fine-tuning model of the audio-visual speech recognition model;

[0037] Initializing adapter fine-tuning parameters of the audio-visual parameter adaptation model of the fine-tuning audio-visual speech recognition model;

[0038] S62, dividing the target speaker video modality data set and the target speaker audio modality data set according to a preset second ratio to obtain a target speaker video modality data training set, a target speaker video modality data test set, a target speaker audio modality data training set, and a target speaker audio modality data test set;

[0039] S63, fine-tuning the model based on the audio-visual input parameter, using the target speaker video modality data training set and the target speaker audio modality data training set for preprocessing to obtain a target speaker preprocessed video modality training data set and a target speaker preprocessed audio modality training data set;

[0040] S64, performing feature recognition processing on the target speaker preprocessed video modality training data set and the target speaker preprocessed video modality training data set to obtain target speaker video modality data feature information and target speaker audio modality data feature information;

[0041] S65, comprehensively processing the target speaker video modality data feature information and the target speaker audio modality data feature information to obtain a target fine-tuning parameter set;

[0042] S66, based on the audio-visual parameter adaptation model, the target speaker data fine-tuning parameter set is used to process the target speaker video modality data test set and the target speaker audio modality data test set to obtain target speaker feature information.

[0043] As an optional implementation, in the first aspect of the embodiment of the present invention, the fine-tuning model based on the audio-visual input parameter pre-processes the target speaker video modality data training set and the target speaker audio modality data training set to obtain a target speaker pre-processed video modality training data set and a target speaker pre-processed audio modality training data set, including:

[0044] S631, based on the audio-visual input parameter fine-tuning model, using the video modality data preprocessing model to process the target speaker video modality data training set to obtain a target speaker preprocessed video modality training data set;

[0045] S632: Based on the audio-visual input parameter fine-tuning model, the audio modal data training set of the target speaker is processed using an audio modal data preprocessing model to obtain a target speaker preprocessed audio modal training data set.

[0046] As an optional implementation, in the first aspect of the embodiment of the present invention, the performing feature recognition processing on the target speaker preprocessed video modality training data set and the target speaker preprocessed video modality training data set to obtain target speaker video modality data feature information and target speaker audio modality data feature information includes:

[0047] S641, using a video modality feature processing model, performing feature recognition processing on the target speaker preprocessed video modality training data set to obtain feature information of the target speaker video modality data;

[0048] S642: Using the audio modality feature processing model, perform feature recognition processing on the target speaker preprocessed audio modality training data set to obtain audio modality data feature information of the target speaker.

[0049] As an optional implementation, in the first aspect of the embodiment of the present invention, the audio modality feature processing model is used to perform feature recognition processing on the target speaker preprocessed video modality training data set to obtain the target speaker audio modality data feature information, including:

[0050] S6411, using a first video feature extraction model, processing the target speaker preprocessed video modality training data set to obtain first feature information of the target speaker video modality data and second feature information of the target speaker video modality data;

[0051] S6412, using a second video feature extraction model, processing the second feature information of the target speaker preprocessed video modality training data set to obtain third feature information of the target speaker video modality data;

[0052] The second extraction model expression of the video feature is:

[0053] y=F(x,W i )+x;

[0054] Where y is the third feature information of the target speaker video modality data; F(x,W i ) is the convolution operation inside the residual block of the deep residual neural network; x is the second feature information of the target speaker video modality data, W i is the learning parameter;

[0055] S6413: Fusing the first feature information of the target speaker's video modal data and the third feature information of the target speaker's video modal data to obtain feature information of the target speaker's video modal data.

[0056] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the comprehensive processing of the target speaker video modality data feature information and the target speaker audio modality data feature information to obtain a target fine-tuning parameter set includes:

[0057] S651, performing splicing and fusion processing on the target speaker's video modal data feature information and the target speaker's audio modal data feature information to obtain first comprehensive feature information of the target speaker;

[0058] S652, based on the bidirectional parameter acquisition model, processing the first comprehensive feature information of the target speaker to obtain a forward fine-tuning parameter set and a backward fine-tuning parameter set;

[0059] The bidirectional parameter acquisition model expression is:

[0060]

[0061] Among them, σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; ⊙ represents element-by-element multiplication; represents the forward hidden state parameters of the bidirectional parameter acquisition model; represents the backward hidden state parameters of the bidirectional parameter acquisition model; x t represents the forward time step t input of the bidirectional parameter acquisition model; x' t represents the backward time step t input of the bidirectional parameter acquisition model; represents the hidden state of the bidirectional parameter acquisition model before time step t; represents the hidden state of the bidirectional parameter acquisition model after time step t; represents the forward weight of the bidirectional parameter acquisition model; represents the forward bias of the bidirectional parameter acquisition model; represents the backward weight of the bidirectional parameter acquisition model; represents the backward bias of the bidirectional parameter acquisition model; represents the forward forget gate weight of the bidirectional parameter acquisition model; represents the forward forget gate bias of the bidirectional parameter acquisition model; Represents the backward forget gate weight of the bidirectional parameter acquisition model; Represents the backward forget gate bias of the bidirectional parameter acquisition model; represents the forward output gate weight of the bidirectional parameter acquisition model; represents the forward output gate bias of the bidirectional parameter acquisition model; represents the backward output gate weight of the bidirectional parameter acquisition model; Represents the backward output gate bias of the bidirectional parameter acquisition model; represents the forward memory update weight of the bidirectional parameter acquisition model; represents the forward memory update bias of the bidirectional parameter acquisition model; represents the backward memory update weight of the bidirectional parameter acquisition model; represents a backward memory update bias of the bidirectional parameter acquisition model; represents the new information acceptance coefficient of the forward time step t of the bidirectional parameter acquisition model; represents the new information acceptance coefficient of the backward time step t of the bidirectional parameter acquisition model; represents the forgetting coefficient of the old information at the forward time step t of the bidirectional parameter acquisition model; represents the backward time step t old information forgetting coefficient of the bidirectional parameter acquisition model; Indicates that the bidirectional parameter obtains the forward output gate coefficient of the model time step t; Represents the backward output gate coefficient of the bidirectional parameter acquisition model time step t; represents the forward memory update coefficient of the bidirectional parameter acquisition model; represents the backward memory update coefficient of the bidirectional parameter acquisition model; represents the forward time step t memory value of the bidirectional parameter acquisition model; represents the backward time step t memory update coefficient of the bidirectional parameter acquisition model;

[0062] S653: The forward fine-tuning parameter set and the backward fine-tuning parameter set are combined in order to obtain a target fine-tuning parameter set.

[0063] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the target speaker video modality data test set and the target speaker audio modality data test set based on the audio-visual parameter adaptation model using the target fine-tuning parameter set to obtain the target speaker feature information includes:

[0064] S661, based on the audio-visual input parameter fine-tuning model, using the target fine-tuning parameter set to process the target speaker audio modal data test set to obtain the target speaker audio feature information;

[0065] S662, based on the audio-visual input parameter fine-tuning model, using the target fine-tuning parameter set to process the target speaker video modal data test set to obtain the target speaker video feature information;

[0066] S663, fusing the target speaker's audio feature information and the target speaker's video feature information to obtain first audio and video comprehensive feature information of the target speaker;

[0067] S664, based on the fine-tuned audio-visual speech recognition model, the audio-visual parameter adaptation model and the target fine-tuning parameter set are used to process the first audio-visual comprehensive feature information of the target speaker to obtain the second audio-visual comprehensive feature information of the target speaker;

[0068] S665, classifying and upgrading the second audio and video comprehensive feature information of the target speaker to obtain third audio and video comprehensive feature information of the target speaker;

[0069] S666: Add the second audio and video comprehensive feature information of the target speaker and the third audio and video comprehensive feature information of the target speaker in time sequence to obtain feature information of the target speaker.

[0070] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the method of fine-tuning the audio-visual speech recognition model and processing the first audio-visual comprehensive feature information of the target speaker using the audio-visual parameter adaptation model and the target fine-tuning parameter set to obtain the second audio-visual comprehensive feature information of the target speaker includes:

[0071] S6641, using a parameter forward propagation model and a loss function model, processing the target fine-tuning parameter set and the first audio and video comprehensive feature information of the target speaker to obtain a forward propagation parameter set;

[0072] The parameter forward propagation model expression is:

[0073]

[0074] Among them, h k represents the forward output layer output of the parameter forward propagation model; activate represents the activation function; θ represents the forward input layer input of the parameter forward propagation model; j represents the forward hidden layer neuron index of the parameter forward propagation model; n represents the number of forward input layer neurons of the parameter forward propagation model; i represents the forward input layer neuron index of the parameter forward propagation model; μ represents the forward input layer weight of the parameter forward propagation model; a represents the forward input layer bias of the parameter forward propagation model; k represents the forward output layer neuron index of the parameter forward propagation model; m represents the number of forward hidden layer neurons of the parameter forward propagation model; w represents the forward output layer weight of the parameter forward propagation model; β represents the forward output layer bias of the parameter forward propagation model;

[0075] The loss function model expression is:

[0076]

[0077] Wherein, O represents the original output of the fine-tuned audio-visual speech recognition model; B represents the actual label of the first audio-visual comprehensive feature information of the target speaker, softmax represents the softmax function; j represents the category index of the first audio-visual comprehensive feature information of the target speaker;

[0078] S6642, using a parameter backpropagation model and a parameter update model, processing the forward propagation parameter set and the first audio and video comprehensive feature information of the target speaker to obtain a backpropagation parameter set;

[0079] The parameter back propagation model expression is:

[0080]

[0081] Wherein, Q represents the parameter update model; a represents the output value of the backward output layer neuron of the parameter backward propagation model; B represents the input value of the backward hidden layer neuron n of the parameter backward propagation model; q represents the number of backward output layer neurons of the parameter backward propagation model; represents the backward output layer gradient of the parameter back-propagation model; Represents the influence coefficient of the input value of the backward output layer neuron k of the parameter backward propagation model on the backward hidden layer neuron n; represents the activation function of the backward output layer neuron k of the parameter backward propagation model; n represents the backward hidden layer neuron index of the parameter backward propagation model; k represents the backward output layer neuron index of the parameter backward propagation model;

[0082] The parameter update model expression is:

[0083]

[0084] Among them, r ij represents the update parameters of the i-th neuron and the j-th neuron of the parameter back-propagation model; Q represents the parameter update model; μ represents the learning rate of the parameter back-propagation model;

[0085] S6643, using the back-propagation parameter set and the fine-tuned audio-visual speech recognition model, processing the first audio and video comprehensive feature information of the target speaker to obtain the second audio and video comprehensive feature information of the target speaker.

[0086] A second aspect of the present invention discloses another adaptive audio-visual speech recognition device, the device comprising:

[0087] A memory storing executable program code;

[0088] a processor coupled to the memory;

[0089] The processor calls the executable program code stored in the memory to execute part or all of the steps in the adaptive audio-visual speech recognition method disclosed in the first aspect of the embodiment of the present invention.

[0090] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0091] 1. In the embodiment of the present invention, a fine-tuning matcher structure is proposed, the downsampling layer is improved, and the matcher performance is effectively improved through a multi-scale feature encoding method;

[0092] 2. A general, easy-to-use, and low-parameter embedded fine-tuning structure is proposed. The structure includes adding data fine-tuning parameters and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model. The structure can be embedded in the commonly used audio-visual recognition model. After obtaining the new speaker data, the model can be quickly adapted to the speaker data by adjusting the parameters of the fine-tuning structure, thereby realizing model personalization. The problem that the data distribution of unseen speakers is easily affected by the frozen backbone model parameters, which hinders the data distribution of the training benchmark model weight parameters, is solved. Combined with the method of introducing a small number of parameters in the input data and deep feature dimensions at the same time, the purpose of fast fine-tuning is achieved, and the accuracy of the benchmark model for unseen speaker recognition is improved.

[0093] 3. The present invention proposes a new fine-tuning paradigm. After obtaining the baseline model, all its parameters are frozen. After introducing the fine-tuning structure proposed in this patent, the data fine-tuning parameter model is initialized to 0, and the fine-tuning parameter adaptation model is randomly initialized. During fine-tuning training, only a small part of the introduced parameters is updated through backpropagation, which improves the efficiency of capturing the characteristic information of the target speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0095] Figure 1 It is a flow chart of a speaker adaptive audio-visual speech recognition method disclosed in an embodiment of the present invention;

[0096] Figure 2 It is a structural schematic diagram of an audio-visual speech recognition base model of a speaker-adaptive audio-visual speech recognition method disclosed in an embodiment of the present invention;

[0097] Figure 3 It is a structural schematic diagram of a speaker adaptive audio-visual speech recognition device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0099] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0100] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0101] The present invention discloses a speaker adaptive audio-visual speech recognition method and device, which improves the efficiency of capturing target speaker feature information and the recognition accuracy of unseen speaker data by introducing an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model, thereby realizing rapid adaptive audio-visual speech recognition of unseen speakers. The following are detailed descriptions.

[0102] Embodiment 1

[0103] See also Figure 1 , Figure 1 1 is a flow chart of a method for speaker adaptive audio-visual speech recognition disclosed in an embodiment of the present invention. Figure 1 The speaker adaptive audio-visual speech recognition method described is applied to a speech recognition system, such as a local server or a cloud server for speech recognition, and the embodiments of the present invention do not limit this. Figure 1 As shown, the speaker-adaptive audio-visual speech recognition method may include the following operations:

[0104] S1, build the audio-visual speech recognition base model;

[0105] It should be noted that if Figure 2 As shown, the audio-visual speech recognition base model includes an audio modality data input layer 201, an audio modality data feature extraction layer 202, a video modality data input layer 203, a video modality data feature extraction layer 204 and an audio-video feature fusion processing layer 205;

[0106] It should be noted that the audio modality data input layer 201 is used to receive audio modality data; the audio modality data feature extraction layer 202 is used to process the audio modality data to obtain audio feature information;

[0107] It should be noted that the video modality data input layer 203 is used to receive video modality data; the video modality data feature extraction layer 204 is used to process the video modality data to obtain video feature information;

[0108] It should be noted that the audio and video feature fusion processing layer 205 is used to fuse the audio feature information and the video feature information to obtain speaker feature information;

[0109] S2, obtaining a multi-person video modality training dataset and a multi-person audio modality training dataset;

[0110] S3, using the video modality training data set and the audio modality training data set to train the audio-visual speech recognition base model to obtain a parameter-fixed audio-visual speech recognition base model;

[0111] S4, obtaining a target speaker audio modality dataset and a target speaker video modality dataset;

[0112] S5, connecting the audio-visual input parameter fine-tuning model to the input layer of the parameter-fixed audio-visual speech recognition base model, and connecting the audio-visual parameter adaptation model to the feature fusion processing layer of the parameter-fixed audio-visual speech recognition base model to obtain a fine-tuned audio-visual speech recognition model;

[0113] It should be noted that the input layer includes the audio modality data input layer and the video modality data input layer;

[0114] S6, using the fine-tuned audio-visual speech recognition model, processing the target speaker audio modality dataset and the target speaker video modality dataset to obtain target speaker feature information.

[0115] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0116] In an optional embodiment, in the above step S3, the audio-visual speech recognition base model is trained using the multi-person video modality training data set and the multi-person audio modality training data set to obtain a parameter-fixed audio-visual speech recognition base model, including:

[0117] S301, randomly initializing the initial values ​​of all parameters of the audio-visual speech recognition base model;

[0118] S302, setting the total number of training rounds n and the input parameter set of the audio-visual speech recognition base model;

[0119] S303, initializing the number of training rounds m=1, initializing the input parameter set of the audio-visual speech recognition base model;

[0120] S304, inputting the multi-person video modality training data set and the multi-person audio modality training data set into the audio-visual speech recognition base model;

[0121] It should be noted that the multi-person video modality training data set includes a plurality of multi-person video modality training data;

[0122] It should be noted that the multi-person audio modality training data set includes a plurality of multi-person audio modality training data;

[0123] S305: determining all of the multi-person video modality training data and the multi-person audio modality training data based on the multi-person video modality training data set and the multi-person audio modality training data set;

[0124] S306, using the weights and biases of the input layer of the audio-visual speech recognition base model, preprocessing any of the multi-person video modality training data and the multi-person audio modality training data to obtain multi-person preprocessed video modality training data and multi-person preprocessed audio modality training data;

[0125] It should be noted that the preprocessing of any of the multi-person video modality training data and the multi-person audio modality training data includes weighting and summing the weights and biases of the input layer to any of the multi-person video modality training data and the multi-person audio modality training data;

[0126] S307, using a first processing function to process the multi-person pre-processed video modality training data and the multi-person pre-processed audio modality training data to obtain multi-person audio and video latent information and multi-person feature training label information;

[0127] Optionally, the first processing function may be but is not limited to sigmoid or ReLU;

[0128] S308, using a second processing function, processing the multi-person audio and video hidden information and multi-person feature training label information, updating the input parameter set, and obtaining an audio-visual speech recognition base model optimization parameter set;

[0129] Optionally, the second processing function may be but is not limited to a loss function;

[0130] The loss function expression is:

[0131]

[0132] Wherein, L represents the output value of the loss function; i represents the classification category index of the multi-person audio and video hidden information; y represents the true label distribution of the multi-person audio and video hidden information; represents the predicted distribution of the multi-person audio and video hidden information; y i Represents the probability corresponding to the i-th category index in the true label; represents the probability corresponding to the i-th category index in the predicted distribution;

[0133] It should be noted that the use of the second processing function to process the multi-person audio and video hidden information and multi-person feature training label information includes: calculating the propagation error of each input parameter to the second processing function from the output layer to the input layer layer by layer, and using the propagation error to update the input parameter in the audio-visual speech recognition base model using a gradient descent algorithm;

[0134] The gradient descent algorithm expression is:

[0135]

[0136] Wherein, L represents the loss function; β represents the input parameter; σ represents the learning rate of the audio-visual speech recognition base model;

[0137] The number of training rounds m=m+1;

[0138] S309, determining whether the number of training rounds m is greater than the total number of training rounds n, and obtaining a result of determining the number of training rounds;

[0139] When the result of the training times is no, executing S306 to S308;

[0140] When the result of the training times is yes, executing S310;

[0141] S310, performing optimization and selection processing on all the audio-visual-speech recognition base model parameter sets to obtain a fixed parameter set of the audio-visual-speech recognition base model;

[0142] S311, based on the audio-visual-speech recognition base model, solidify and freeze the solidified parameter set of the audio-visual-speech recognition base model to obtain a parameter-solidified audio-visual-speech recognition base model.

[0143] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0144] In an optional embodiment, in the above step S6, the use of the fine-tuned audio-visual speech recognition model to process the target speaker video modality dataset and the target speaker audio modality dataset to obtain target speaker feature information includes:

[0145] S61, initializing the audio-visual input fine-tuning parameters of the audio-visual input parameter fine-tuning model of the audio-visual speech recognition model;

[0146] It should be noted that the initialization includes: initializing the input fine-tuning parameters of the audio modality data to 0, and initializing the input fine-tuning parameters of the video modality data to 0, so as to ensure that the data distribution of the unseen speaker can be better adapted during fine-tuning;

[0147] Initializing adapter fine-tuning parameters of the audio-visual parameter adaptation model of the fine-tuning audio-visual speech recognition model;

[0148] It should be noted that the adapter fine-tuning parameters of the audio-visual parameter adaptation model are randomly initialized;

[0149] S62, dividing the target speaker video modality data set and the target speaker audio modality data set according to a preset second ratio to obtain a target speaker video modality data training set, a target speaker video modality data test set, a target speaker audio modality data training set, and a target speaker audio modality data test set;

[0150] S63, fine-tuning the model based on the audio-visual input parameter, using the target speaker video modality data training set and the target speaker audio modality data training set for preprocessing to obtain a target speaker preprocessed video modality training data set and a target speaker preprocessed audio modality training data set;

[0151] S64, performing feature recognition processing on the target speaker preprocessed video modality training data set and the target speaker preprocessed video modality training data set to obtain target speaker video modality data feature information and target speaker audio modality data feature information;

[0152] S65, comprehensively processing the target speaker video modality data feature information and the target speaker audio modality data feature information to obtain a target fine-tuning parameter set;

[0153] S66, based on the audio-visual parameter adaptation model, the target speaker data fine-tuning parameter set is used to process the target speaker video modality data test set and the target speaker audio modality data test set to obtain target speaker feature information.

[0154] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0155] In an optional embodiment, in the above step S63, the fine-tuning model based on the audio-visual input parameter pre-processes the target speaker video modality data training set and the target speaker audio modality data training set to obtain a target speaker pre-processed video modality training data set and a target speaker pre-processed audio modality training data set, including:

[0156] S631, based on the audio-visual input parameter fine-tuning model, using the video modality data preprocessing model to process the target speaker video modality data training set to obtain a target speaker preprocessed video modality training data set;

[0157] It should be noted that the video modality data preprocessing model expression is:

[0158] f(x)=x·p s ;

[0159] Wherein, x is the target speaker video modality data training set; p s is the video data fine-tuning parameter, and f(x) is the target speaker pre-processed video modality training dataset;

[0160] S632, based on the audio-visual input parameter fine-tuning model, using the audio modality data preprocessing model to process the target speaker audio modality data training set to obtain a target speaker preprocessed audio modality training data set;

[0161] It should be noted that the audio modal data preprocessing model expression is:

[0162] f(y)=y·p y ;

[0163] Wherein, y is the target speaker audio modality data training set; p y is the audio data fine-tuning parameter, and f(y) is the pre-processed audio modality training dataset of the target speaker.

[0164] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0165] In an optional embodiment, in the above step S64, the feature recognition processing is performed on the target speaker preprocessed video modality training data set and the target speaker preprocessed video modality training data set to obtain the target speaker video modality data feature information and the target speaker audio modality data feature information, including:

[0166] S641, using a video modality feature processing model, performing feature recognition processing on the target speaker preprocessed video modality training data set to obtain feature information of the target speaker video modality data;

[0167] S642: Using the audio modality feature processing model, perform feature recognition processing on the target speaker preprocessed audio modality training data set to obtain audio modality data feature information of the target speaker.

[0168] In an optional embodiment, in the above step S641, the use of the audio modality feature processing model to perform feature recognition processing on the target speaker preprocessed video modality training data set to obtain the target speaker audio modality data feature information includes:

[0169] S6411, using a first video feature extraction model, processing the target speaker preprocessed video modality training data set to obtain first feature information of the target speaker video modality data and second feature information of the target speaker video modality data;

[0170] It should be noted that the first video feature extraction model includes: a data input layer, a 3D convolution layer, a 3D normalization layer, an activation function processing layer and a data output layer;

[0171] The output end of the data input layer is data-connected to the input end of the 3D convolution layer; the output end of the 3D convolution layer is data-connected to the input end of the 3D normalization layer; the output end of the 3D normalization layer is data-connected to the input end of the activation function processing layer; the output end of the activation function processing layer is data-connected to the input end of the data output layer;

[0172] The data input layer is used to receive the target speaker fine-tuned video modality data and send the target speaker fine-tuned video modality data to the first data processing layer;

[0173] The 3D convolution layer is used to perform convolution processing on the target speaker fine-tuning video modality data to obtain preliminary data of the first feature of the target speaker video modality data and preliminary data of the second feature of the target speaker video modality data;

[0174] It should be noted that the 3D convolution layer has 64 convolution kernels, the size of the convolution kernel is 3x3x3, and the stride of the convolution kernel is: one unit is moved in depth, height and width each time;

[0175] The 3D normalization layer is used to normalize the first feature preliminary data of the target speaker video modality data and the second feature preliminary data of the target speaker video modality data to obtain the first feature data of the target speaker fine-tuning video modality convergence and the second feature data of the target speaker fine-tuning video modality convergence;

[0176] The activation function processing layer is used to process the first feature data of the target speaker fine-tuning video modality convergence and the second feature data of the target speaker fine-tuning video modality convergence using a ReLU activation function to obtain first feature information of the target speaker video modality data and second feature information of the target speaker video modality data;

[0177] The data output layer is used to output the first feature information of the target speaker video modality data and the second feature information of the target speaker video modality data;

[0178] S6412, using a second video feature extraction model, processing the second feature information of the target speaker preprocessed video modality training data set to obtain third feature information of the target speaker video modality data;

[0179] The second extraction model expression of the video feature is:

[0180] y=F(x,W i )+x;

[0181] Where y is the third feature information of the target speaker video modality data; F(x,W i ) is the convolution operation inside the residual block of the deep residual neural network; x is the second feature information of the target speaker video modality data, W i is the learning parameter;

[0182] S6413: Fusing the first feature information of the target speaker's video modal data and the third feature information of the target speaker's video modal data to obtain feature information of the target speaker's video modal data.

[0183] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0184] In an optional embodiment, in the above step S642, the use of the audio modality feature processing model to perform feature recognition processing on the target speaker preprocessed audio modality training data set to obtain the target speaker audio modality data feature information includes:

[0185] S6421, traversing the target speaker preprocessed audio modality training data set to obtain all target speaker preprocessed audio modality training data;

[0186] S6422, performing frame processing on any of the target speaker preprocessed audio modal training data according to the sampling frequency of the target speaker preprocessed audio modal training data to obtain a target speaker preprocessed audio modal frame data set; the target speaker preprocessed audio modal frame data set includes a plurality of frames of target speaker preprocessed audio modal frame data;

[0187] S6423, performing windowing processing on any frame of the target speaker preprocessed audio modality frame data to obtain the target speaker preprocessed audio modality frame window data;

[0188] S6424, transform the target speaker preprocessed audio modal frame window data to obtain frequency spectrum information of the target speaker preprocessed audio modal frame data and power spectrum information of the target speaker preprocessed audio modal frame data;

[0189] S6425, processing the frequency spectrum information of the target speaker's preprocessed audio modal frame data to obtain feature information of the target speaker's audio modal data.

[0190] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0191] In an optional embodiment, in the above step S6425, the processing of the frequency spectrum information of the preprocessed audio modality frame data of the target speaker to obtain the audio modality data feature information of the target speaker includes:

[0192] S64251, based on the frequency acquisition model, processing the preprocessed audio modal frame window data of the target speaker to obtain Mel frequency;

[0193] The frequency acquisition model expression is:

[0194]

[0195] Wherein, f represents the actual frequency of the target speaker's audio modal data; M represents the adjustment coefficient; and F represents the reference frequency;

[0196] Preferably, the adjustment coefficient is 2595;

[0197] Preferably, the reference frequency is 700;

[0198] S64252. Based on the Mel frequency, obtain the Mel highest frequency, Mel lowest frequency, and Mel filter bank;

[0199] S64253. Based on the Mel filter bank, use a filtering processing model to process the spectral information of the preprocessed audio modal frame data of the target speaker, and obtain the preprocessed data of the first target speaker's preprocessed audio modal frame;

[0200] The expression of the filtering processing model is:

[0201]

[0202] C ld = f h(l-1) = f (l+1)d ;

[0203] k = fN / f s ;

[0204] Where L is the number of filters in the Mel filter bank; l is the l-th filter in the Mel filter bank, and 0 < l ≤ L; M(l) is the output value of the l-th Mel filter; f ld is the lower limit frequency of the l-th filter; f hl is the upper limit frequency of the l-th filter; C ld is the center frequency of the l-th filter; k is the k-th sampling point in the preprocessed audio modal frame data of the target speaker; f is the actual frequency of the target speaker's audio modal data; f s is the sampling frequency of the preprocessed audio modal frame data of the target speaker; n is the length of the preprocessed audio modal frame data of the target speaker; |X n (k)| 2 is the power spectrum of the data at the k-th sampling point;

[0205] S64254. Perform a logarithmic operation on the preprocessed data of the first target speaker's preprocessed audio modal frame to obtain the preprocessed data of the second target speaker's preprocessed audio modal frame;

[0206] S64255. Perform a discrete cosine transform on the preprocessed data of the second target speaker's preprocessed audio modal frame to obtain the preprocessed data of the third target speaker's preprocessed audio modal frame;

[0207] S64256. Based on an audio feature extraction model, process the preprocessed data of the fine-tuned audio modal frame of the third target speaker to obtain the feature information of the target speaker's audio modal data;

[0208] It should be noted that the expression of the audio feature extraction model is:

[0209]

[0210] Wherein, X is the preprocessed audio modal frame preprocessed data of the third target speaker; W is the convolution kernel for convolution; i is the output sequence element index of the audio feature extraction model; k is the convolution kernel weight index of the audio feature extraction model;

[0211] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0212] In an optional embodiment, in the above step S65, the target speaker video modality data feature information and the target speaker audio modality data feature information are comprehensively processed to obtain a target fine-tuning parameter set, including:

[0213] S651, performing splicing and fusion processing on the target speaker's video modal data feature information and the target speaker's audio modal data feature information to obtain first comprehensive feature information of the target speaker;

[0214] S652, based on the bidirectional parameter acquisition model, processing the first comprehensive feature information of the target speaker to obtain a forward fine-tuning parameter set and a backward fine-tuning parameter set;

[0215] The bidirectional parameter acquisition model expression is:

[0216]

[0217] Among them, σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; ⊙ represents element-by-element multiplication; represents the forward hidden state parameters of the bidirectional parameter acquisition model; represents the backward hidden state parameters of the bidirectional parameter acquisition model; x t represents the forward time step t input of the bidirectional parameter acquisition model; x' t represents the backward time step t input of the bidirectional parameter acquisition model; represents the hidden state of the bidirectional parameter acquisition model before time step t; represents the hidden state of the bidirectional parameter acquisition model after time step t; represents the forward weight of the bidirectional parameter acquisition model; represents the forward bias of the bidirectional parameter acquisition model; represents the backward weight of the bidirectional parameter acquisition model; represents the backward bias of the bidirectional parameter acquisition model; represents the forward forget gate weight of the bidirectional parameter acquisition model; represents the forward forget gate bias of the bidirectional parameter acquisition model; Represents the backward forget gate weight of the bidirectional parameter acquisition model; Represents the backward forget gate bias of the bidirectional parameter acquisition model; represents the forward output gate weight of the bidirectional parameter acquisition model; represents the forward output gate bias of the bidirectional parameter acquisition model; represents the backward output gate weight of the bidirectional parameter acquisition model; Represents the backward output gate bias of the bidirectional parameter acquisition model; represents the forward memory update weight of the bidirectional parameter acquisition model; represents the forward memory update bias of the bidirectional parameter acquisition model; represents the backward memory update weight of the bidirectional parameter acquisition model; represents a backward memory update bias of the bidirectional parameter acquisition model; represents the new information acceptance coefficient of the forward time step t of the bidirectional parameter acquisition model; represents the new information acceptance coefficient of the backward time step t of the bidirectional parameter acquisition model; represents the forgetting coefficient of the old information at the forward time step t of the bidirectional parameter acquisition model; represents the backward time step t old information forgetting coefficient of the bidirectional parameter acquisition model; Indicates that the bidirectional parameter obtains the forward output gate coefficient of the model time step t; Represents the backward output gate coefficient of the bidirectional parameter acquisition model time step t; represents the forward memory update coefficient of the bidirectional parameter acquisition model; represents the backward memory update coefficient of the bidirectional parameter acquisition model; represents the forward time step t memory value of the bidirectional parameter acquisition model; represents the backward time step t memory update coefficient of the bidirectional parameter acquisition model;

[0218] S653: The forward fine-tuning parameter set and the backward fine-tuning parameter set are combined in order to obtain a target fine-tuning parameter set.

[0219] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0220] In an optional embodiment, in the above step S66, the target speaker feature information is obtained by processing the target speaker video modality data test set and the target speaker audio modality data test set based on the audio-visual parameter adaptation model using the target fine-tuning parameter set, including:

[0221] S661, based on the audio-visual input parameter fine-tuning model, using the target fine-tuning parameter set to process the target speaker audio modal data test set to obtain the target speaker audio feature information;

[0222] S662, based on the audio-visual input parameter fine-tuning model, using the target fine-tuning parameter set to process the target speaker video modal data test set to obtain the target speaker video feature information;

[0223] S663, fusing the target speaker's audio feature information and the target speaker's video feature information to obtain first audio and video comprehensive feature information of the target speaker;

[0224] S664, based on the fine-tuned audio-visual speech recognition model, the audio-visual parameter adaptation model and the target fine-tuning parameter set are used to process the first audio-visual comprehensive feature information of the target speaker to obtain the second audio-visual comprehensive feature information of the target speaker;

[0225] S665, classifying and upgrading the second audio and video comprehensive feature information of the target speaker to obtain third audio and video comprehensive feature information of the target speaker;

[0226] S666: Add the second audio and video comprehensive feature information of the target speaker and the third audio and video comprehensive feature information of the target speaker in time sequence to obtain feature information of the target speaker.

[0227] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0228] In an optional embodiment, in the above step S664, the first audio and video comprehensive feature information of the target speaker is processed based on the fine-tuning audio-visual speech recognition model using the audio-visual parameter adaptation model and the target fine-tuning parameter set to obtain the second audio and video comprehensive feature information of the target speaker, including:

[0229] S6641, using a parameter forward propagation model and a loss function model, processing the target fine-tuning parameter set and the first audio and video comprehensive feature information of the target speaker to obtain a forward propagation parameter set;

[0230] It should be noted that the parameter forward propagation model includes a forward input layer, a forward hidden layer and a forward output layer;

[0231] The forward input layer, the forward hidden layer and the forward output layer are sequentially data-connected;

[0232] The parameter forward propagation model expression is:

[0233]

[0234] Among them, h k represents the forward output layer output of the parameter forward propagation model; activate represents the activation function; θ represents the forward input layer input of the parameter forward propagation model; j represents the forward hidden layer neuron index of the parameter forward propagation model; n represents the number of forward input layer neurons of the parameter forward propagation model; i represents the forward input layer neuron index of the parameter forward propagation model; μ represents the forward input layer weight of the parameter forward propagation model; a represents the forward input layer bias of the parameter forward propagation model; k represents the forward output layer neuron index of the parameter forward propagation model; m represents the number of forward hidden layer neurons of the parameter forward propagation model; w represents the forward output layer weight of the parameter forward propagation model; β represents the forward output layer bias of the parameter forward propagation model;

[0235] The loss function model expression is:

[0236] Loss(O,B)=-∑ j (B j log(softmax(Ox)));

[0237] Wherein, O represents the original output of the fine-tuned audio-visual speech recognition model; B represents the actual label of the first audio-visual comprehensive feature information of the target speaker, softmax represents the softmax function; j represents the category index of the first audio-visual comprehensive feature information of the target speaker;

[0238] S6642, using a parameter backpropagation model and a parameter update model, processing the forward propagation parameter set and the first audio and video comprehensive feature information of the target speaker to obtain a backpropagation parameter set;

[0239] It should be noted that the parameter backward propagation model includes a backward input layer, a backward hidden layer and a backward output layer;

[0240] The backward input layer, the backward hidden layer and the backward output layer are sequentially data-connected;

[0241] The parameter back propagation model expression is:

[0242]

[0243] Wherein, Q represents the weight of the parameter backpropagation model; a represents the output value of the backward output layer neuron of the parameter backpropagation model; B represents the input value of the backward hidden layer neuron n of the parameter backpropagation model; q represents the number of backward output layer neurons of the parameter backpropagation model; represents the backward output layer gradient of the parameter back-propagation model; Represents the influence coefficient of the input value of the backward output layer neuron k of the parameter backward propagation model on the backward hidden layer neuron n; represents the activation function of the backward output layer neuron k of the parameter backward propagation model; n represents the backward hidden layer neuron index of the parameter backward propagation model; k represents the backward output layer neuron index of the parameter backward propagation model;

[0244] It should be noted that Q represents the parameter backpropagation model weight, that is, the parameter update model;

[0245] The parameter update model expression is:

[0246]

[0247] Among them, r ij represents the update parameters of the i-th neuron and the j-th neuron of the parameter back-propagation model; Q represents the weight of the parameter back-propagation model, that is, the parameter update model; μ represents the learning rate of the parameter back-propagation model;

[0248] It should be noted that r ij Including: the weights of the i-th neuron and the j-th neuron of the parameter back-propagation model and the biases of the i-th neuron and the j-th neuron of the parameter back-propagation model;

[0249] S6643, using the back-propagation parameter set and the fine-tuned audio-visual speech recognition model, processing the first audio and video comprehensive feature information of the target speaker to obtain the second audio and video comprehensive feature information of the target speaker.

[0250] It can be seen that the present invention adds an audio-visual input parameter fine-tuning model and a fine-tuning parameter adaptation model to the fine-tuning audio-visual speech recognition model, which improves the efficiency of capturing the target speaker's feature information and the recognition accuracy of unseen speaker data, and realizes fast adaptive audio-visual speech recognition of unseen speakers.

[0251] Embodiment 2

[0252] See also Figure 3 , Figure 3 1 is a schematic diagram of the structure of a speaker adaptive audio-visual speech recognition device disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a speech recognition system, such as a local server or a cloud server for speech recognition, and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the device may include:

[0253] A memory 301 storing executable program codes;

[0254] a processor 302 coupled to the memory 301;

[0255] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the speaker adaptive audio-visual speech recognition method described in the first embodiment.

[0256] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0257] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0258] Finally, it should be noted that the speaker adaptive audio-visual speech recognition method and device disclosed in the embodiment of the present invention discloses only the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An adaptive audio-visual speech recognition method, characterized in that: The method comprises: S1, build the audio-visual speech recognition base model; S2, obtaining a multi-person video modality training dataset and a multi-person audio modality training dataset; S3, using the multi-person video modality training data set and the multi-person audio modality training data set to train the audio-visual speech recognition base model to obtain a parameter-fixed audio-visual speech recognition base model; S4, obtaining a target speaker audio modality dataset and a target speaker video modality dataset; S5, connecting the audio-visual input parameter fine-tuning model to the input layer of the parameter-fixed audio-visual speech recognition base model, and connecting the audio-visual parameter adaptation model to the feature fusion processing layer of the parameter-fixed audio-visual speech recognition base model to obtain a fine-tuned audio-visual speech recognition model; S6, using the fine-tuned audio-visual speech recognition model, processing the target speaker audio modality dataset and the target speaker video modality dataset to obtain target speaker feature information.

2. The adaptive audio-visual speech recognition method according to claim 1, characterized in that: The method of training the audio-visual speech recognition base model by using the multi-person video modality training data set and the multi-person audio modality training data set to obtain a parameter-fixed audio-visual speech recognition base model includes: S301, randomly initializing the initial values ​​of all parameters of the audio-visual speech recognition base model; S302, setting the total number of training rounds n and the training parameter set of the audio-visual speech recognition base model; S303, initializing the number of training rounds m=1, initializing the input parameter set of the audio-visual speech recognition base model; S304, inputting the multi-person video modality training data set and the multi-person audio modality training data set into the audio-visual speech recognition base model; The multi-person video modality training data set includes a plurality of multi-person video modality training data; The multi-person audio modality training data set includes a plurality of multi-person audio modality training data; S305: determining all of the multi-person video modality training data and the multi-person audio modality training data based on the multi-person video modality training data set and the multi-person audio modality training data set; S306, using the weights and biases of the input layer of the audio-visual speech recognition base model, preprocessing any of the multi-person video modality training data and the multi-person audio modality training data to obtain multi-person preprocessed video modality training data and multi-person preprocessed audio modality training data; S307, using a first processing function to process the multi-person pre-processed video modality training data and the multi-person pre-processed audio modality training data to obtain multi-person audio and video latent information and multi-person feature training label information; S308, using a second processing function, processing the multi-person audio and video hidden information and multi-person feature training label information, updating the input parameter set, and obtaining an audio-visual speech recognition base model optimization parameter set; The number of training rounds m=m+1; S309, determining whether the number of training rounds m is greater than the total number of training rounds n, and obtaining a result of determining the number of training rounds; When the result of the training times is no, executing S306 to S308; When the result of the training times is yes, executing S310; S310, performing optimization and selection processing on all the audio-visual-speech recognition base model parameter sets to obtain a fixed parameter set of the audio-visual-speech recognition base model; S311, based on the audio-visual-speech recognition base model, solidify and freeze the solidified parameter set of the audio-visual-speech recognition base model to obtain a parameter-solidified audio-visual-speech recognition base model.

3. The adaptive audio-visual speech recognition method according to claim 1, characterized in that: The method of using the fine-tuned audio-visual speech recognition model to process the target speaker video modality dataset and the target speaker audio modality dataset to obtain target speaker feature information includes: S61, initializing the audio-visual input fine-tuning parameters of the audio-visual input parameter fine-tuning model of the audio-visual speech recognition model; Initializing adapter fine-tuning parameters of the audio-visual parameter adaptation model of the fine-tuning audio-visual speech recognition model; S62, dividing the target speaker video modality data set and the target speaker audio modality data set according to a preset second ratio to obtain a target speaker video modality data training set, a target speaker video modality data test set, a target speaker audio modality data training set, and a target speaker audio modality data test set; S63, fine-tuning the model based on the audio-visual input parameter, using the target speaker video modality data training set and the target speaker audio modality data training set for preprocessing to obtain a target speaker preprocessed video modality training data set and a target speaker preprocessed audio modality training data set; S64, performing feature recognition processing on the target speaker preprocessed video modality training data set and the target speaker preprocessed video modality training data set to obtain target speaker video modality data feature information and target speaker audio modality data feature information; S65, comprehensively processing the target speaker video modality data feature information and the target speaker audio modality data feature information to obtain a target fine-tuning parameter set; S66, based on the audio-visual parameter adaptation model, the target speaker data fine-tuning parameter set is used to process the target speaker video modality data test set and the target speaker audio modality data test set to obtain target speaker feature information.

4. The adaptive audio-visual speech recognition method according to claim 3, characterized in that: The fine-tuning model based on the audio-visual input parameter pre-processes the target speaker video modality data training set and the target speaker audio modality data training set to obtain a target speaker pre-processed video modality training data set and a target speaker pre-processed audio modality training data set, including: S631, based on the audio-visual input parameter fine-tuning model, using the video modality data preprocessing model to process the target speaker video modality data training set to obtain a target speaker preprocessed video modality training data set; S632: Based on the audio-visual input parameter fine-tuning model, the audio modal data training set of the target speaker is processed using an audio modal data preprocessing model to obtain a target speaker preprocessed audio modal training data set.

5. The adaptive audio-visual speech recognition method according to claim 3, characterized in that: The performing feature recognition processing on the target speaker preprocessed video modality training data set and the target speaker preprocessed video modality training data set to obtain target speaker video modality data feature information and target speaker audio modality data feature information includes: S641, using a video modality feature processing model, performing feature recognition processing on the target speaker preprocessed video modality training data set to obtain feature information of the target speaker video modality data; S642: Using the audio modality feature processing model, perform feature recognition processing on the target speaker preprocessed audio modality training data set to obtain audio modality data feature information of the target speaker.

6. The adaptive audio-visual speech recognition method according to claim 5, characterized in that: The method of using the audio modality feature processing model to perform feature recognition processing on the target speaker preprocessed video modality training data set to obtain audio modality data feature information of the target speaker includes: S6411, using a first video feature extraction model, processing the target speaker preprocessed video modality training data set to obtain first feature information of the target speaker video modality data and second feature information of the target speaker video modality data; S6412, using a second video feature extraction model, processing the second feature information of the target speaker preprocessed video modality training data set to obtain third feature information of the target speaker video modality data; The second extraction model expression of the video feature is: y=F(x,W i )+x; Where y is the third feature information of the target speaker video modality data; F(x,W i ) is the convolution operation inside the residual block of the deep residual neural network; x is the second feature information of the target speaker video modality data, W i is the learning parameter; S6413: Fusing the first feature information of the target speaker's video modal data and the third feature information of the target speaker's video modal data to obtain feature information of the target speaker's video modal data.

7. The adaptive audio-visual speech recognition method according to claim 3, characterized in that: The step of comprehensively processing the target speaker video modality data feature information and the target speaker audio modality data feature information to obtain a target fine-tuning parameter set includes: S651, performing splicing and fusion processing on the target speaker's video modal data feature information and the target speaker's audio modal data feature information to obtain first comprehensive feature information of the target speaker; S652, based on the bidirectional parameter acquisition model, processing the first comprehensive feature information of the target speaker to obtain a forward fine-tuning parameter set and a backward fine-tuning parameter set; The bidirectional parameter acquisition model expression is: Among them, σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; ⊙ represents element-by-element multiplication; represents the forward hidden state parameters of the bidirectional parameter acquisition model; represents the backward hidden state parameters of the bidirectional parameter acquisition model; x t represents the forward time step t input of the bidirectional parameter acquisition model; x' t represents the backward time step t input of the bidirectional parameter acquisition model; represents the hidden state of the bidirectional parameter acquisition model before time step t; represents the hidden state of the bidirectional parameter acquisition model after time step t; represents the forward weight of the bidirectional parameter acquisition model; represents the forward bias of the bidirectional parameter acquisition model; represents the backward weight of the bidirectional parameter acquisition model; represents the backward bias of the bidirectional parameter acquisition model; represents the forward forget gate weight of the bidirectional parameter acquisition model; represents the forward forget gate bias of the bidirectional parameter acquisition model; Represents the backward forget gate weight of the bidirectional parameter acquisition model; Represents the backward forget gate bias of the bidirectional parameter acquisition model; represents the forward output gate weight of the bidirectional parameter acquisition model; represents the forward output gate bias of the bidirectional parameter acquisition model; represents the backward output gate weight of the bidirectional parameter acquisition model; Represents the backward output gate bias of the bidirectional parameter acquisition model; represents the forward memory update weight of the bidirectional parameter acquisition model; represents the forward memory update bias of the bidirectional parameter acquisition model; represents the backward memory update weight of the bidirectional parameter acquisition model; represents a backward memory update bias of the bidirectional parameter acquisition model; represents the new information acceptance coefficient of the forward time step t of the bidirectional parameter acquisition model; represents the new information acceptance coefficient of the backward time step t of the bidirectional parameter acquisition model; represents the forgetting coefficient of the old information at the forward time step t of the bidirectional parameter acquisition model; represents the backward time step t old information forgetting coefficient of the bidirectional parameter acquisition model; Indicates that the bidirectional parameter obtains the forward output gate coefficient of the model time step t; Represents the backward output gate coefficient of the bidirectional parameter acquisition model time step t; represents the forward memory update coefficient of the bidirectional parameter acquisition model; represents the backward memory update coefficient of the bidirectional parameter acquisition model; represents the forward time step t memory value of the bidirectional parameter acquisition model; represents the backward time step t memory update coefficient of the bidirectional parameter acquisition model; S653: The forward fine-tuning parameter set and the backward fine-tuning parameter set are combined in order to obtain a target fine-tuning parameter set.

8. The adaptive audio-visual speech recognition method according to claim 3, characterized in that: The step of processing the target speaker video modality data test set and the target speaker audio modality data test set based on the audio-visual parameter adaptation model using the target fine-tuning parameter set to obtain target speaker feature information includes: S661, based on the audio-visual input parameter fine-tuning model, using the target fine-tuning parameter set to process the target speaker audio modal data test set to obtain the target speaker audio feature information; S662, based on the audio-visual input parameter fine-tuning model, using the target fine-tuning parameter set to process the target speaker video modal data test set to obtain the target speaker video feature information; S663, fusing the target speaker's audio feature information and the target speaker's video feature information to obtain first audio and video comprehensive feature information of the target speaker; S664, based on the fine-tuned audio-visual speech recognition model, the audio-visual parameter adaptation model and the target fine-tuning parameter set are used to process the first audio-visual comprehensive feature information of the target speaker to obtain the second audio-visual comprehensive feature information of the target speaker; S665, classifying and upgrading the second audio and video comprehensive feature information of the target speaker to obtain third audio and video comprehensive feature information of the target speaker; S666: Add the second audio and video comprehensive feature information of the target speaker and the third audio and video comprehensive feature information of the target speaker in time sequence to obtain feature information of the target speaker.

9. The adaptive audio-visual speech recognition method according to claim 8, characterized in that: The method of processing the first audio and video comprehensive feature information of the target speaker based on the fine-tuning audio and video speech recognition model using the audio and video parameter adaptation model and the target fine-tuning parameter set to obtain the second audio and video comprehensive feature information of the target speaker includes: S6641, using a parameter forward propagation model and a loss function model, processing the target fine-tuning parameter set and the first audio and video comprehensive feature information of the target speaker to obtain a forward propagation parameter set; The parameter forward propagation model expression is: Among them, h k represents the forward output layer output of the parameter forward propagation model; act iv ate represents the activation function; θ represents the forward input layer input of the parameter forward propagation model; j represents the forward hidden layer neuron index of the parameter forward propagation model; n represents the number of neurons in the forward input layer of the parameter forward propagation model; i represents the forward input layer neuron index of the parameter forward propagation model; μ represents the forward input layer weight of the parameter forward propagation model; a represents the forward input layer bias of the parameter forward propagation model; k represents the forward output layer neuron index of the parameter forward propagation model; m represents the number of neurons in the forward hidden layer of the parameter forward propagation model; w represents the forward output layer weight of the parameter forward propagation model; β represents the forward output layer bias of the parameter forward propagation model; The loss function model expression is: Wherein, O represents the original output of the fine-tuned audio-visual speech recognition model; B represents the actual label of the first audio-visual comprehensive feature information of the target speaker, softmax represents the softmax function; j represents the category index of the first audio-visual comprehensive feature information of the target speaker; S6642, using a parameter backpropagation model and a parameter update model, processing the forward propagation parameter set and the first audio and video comprehensive feature information of the target speaker to obtain a backpropagation parameter set; The parameter back propagation model expression is: Wherein, Q represents the parameter update model; a represents the output value of the backward output layer neuron of the parameter backward propagation model; B represents the input value of the backward hidden layer neuron n of the parameter backward propagation model; q represents the number of backward output layer neurons of the parameter backward propagation model; represents the backward output layer gradient of the parameter back-propagation model; Represents the influence coefficient of the input value of the backward output layer neuron k of the parameter backward propagation model on the backward hidden layer neuron n; represents the activation function of the backward output layer neuron k of the parameter backward propagation model; n represents the backward hidden layer neuron index of the parameter backward propagation model; k represents the backward output layer neuron index of the parameter backward propagation model; The parameter update model expression is: Among them, r ij represents the update parameters of the i-th neuron and the j-th neuron of the parameter back-propagation model; Q represents the parameter update model; μ represents the learning rate of the parameter back-propagation model; S6643, using the back-propagation parameter set and the fine-tuned audio-visual speech recognition model, processing the first audio and video comprehensive feature information of the target speaker to obtain the second audio and video comprehensive feature information of the target speaker.

10. An adaptive audio-visual speech recognition device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the adaptive audio-visual speech recognition method as described in any one of claims 1-9.

Images