A lip reading recognition method and system based on phased cross-training

Abstract

The present invention relates to a lip reading recognition method and system based on staged cross-training. The method comprises: obtaining encoding network parameters and decoding network parameters in the first stage based on an end-to-end training method; in the second stage, training a model according to single identity data, generating initialization parameters of an encoding network and a decoding network with overfitting, adaptively generating weighted regularization constraints of a decoding network using the encoding network initialization parameters, and adaptively generating weighted regularization constraints of a decoding network using the decoding network initialization parameters. The method of the present invention ensures the independence of the encoding network and the decoding network by such cross-training method, thereby improving the generalization ability of the lip reading recognition network, and further, when the lip reading recognition model trained by the present invention is applied to lip reading recognition, the robustness and accuracy of the lip reading recognition of unknown speakers can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of lip reading recognition, and in particular to a lip reading recognition method and system based on staged cross-training. Background Art

[0002] As coupled modal information of speech signals, lip reading contains rich semantic content. Professionally trained lip reading experts can even analyze what the speaker said by relying on the single lip shape change information. By simulating this human cognitive mode, lip reading recognition technology aims to automatically identify the semantic content of lip image sequences through computer learning. Lip reading recognition technology starts from visual information and is not easily affected by environmental noise. It can well overcome the shortcomings in speech recognition. Lip reading recognition provides a new channel for human-computer semantic interaction. Lip reading recognition technology has broad application prospects. For example, hearing-impaired people cannot obtain the content expressed by the other party through voice information. Some deaf-mute people may only have hearing impairments. Due to the inability to obtain semantic content, they cannot form the ability to speak. Lip reading technology can provide these people with auxiliary semantic understanding channels, thereby helping hearing-impaired people to establish speaking ability. Outdoor security monitoring can usually only collect images without voice information. For example, if the video of the illegal person is collected, the intention of the illegal person can be analyzed by relying on lip reading recognition technology, thereby providing important clues for law enforcement personnel to solve the case. In view of this, the technology was successfully selected as one of the top ten black technologies that may change the way of life selected by the China International Big Data Industry Expo. Researching lip reading recognition methods that meet practical requirements has become an urgent need in today's society.

[0003] The learning method of the existing lip reading recognition model usually takes the prediction loss in the current training data as the learning target of the network. However, the test scenario of the lip reading system is for unknown users. Due to the differences in factors such as the lip appearance and speaking style of different speakers, the lip reading data has obvious distribution deviation characteristics. Using only training data as the basis for network optimization can easily cause the model to overfit. The model is only applicable to lip reading data with known speaker identities, and cannot guarantee its good generalization performance in unknown identity test scenarios. Existing strategies usually adopt the L1 norm, L2 norm, etc. of network parameters added to the network training loss to control the network complexity to avoid the overfitting problem of the model. However, the common problem of these strategies is that they rely too much on artificial experience and have no foresight for the generalization performance of the network in unknown data. In addition, limiting the complexity of the model will also cause the model representation ability to decline. Therefore, how to enhance the generalization performance of the network in unknown data distribution while ensuring the model representation ability is a technical problem that needs to be solved in this field. Summary of the invention

[0004] The purpose of the present invention is to provide a lip reading recognition method based on staged cross-training. From the perspective of training strategy, the method is based on the traditional lip reading recognition model architecture, with the goal of enhancing the generalization performance of the network in unknown data distribution. The method uses staged cross-training to optimize the network parameters of the lip reading recognition model, thereby improving the accuracy of lip reading recognition of unknown speakers.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A lip reading recognition method based on phased cross-training, comprising:

[0007] Acquire a set of lip reading training data for different identities; a single training data in the set of lip reading training data for different identities includes an image sequence of sample input and a category code of sample output;

[0008] Using all the training data in the lip reading training data set of different identities, a lip reading recognition model is trained in an end-to-end manner to obtain encoding network parameters and decoding network parameters in the end-to-end training stage; the lip reading recognition model includes a semantic feature encoding network and a semantic feature decoding network;

[0009] According to the encoding network parameters and the decoding network parameters in the end-to-end training phase, the lip reading recognition model is trained using the single identity training data in the different identity lip reading training data set to obtain the encoding network parameters and the decoding network parameters with overfitting;

[0010] According to the encoding network parameters and decoding network parameters with overfitting, adaptively generating the encoding network weighted regularization loss and the decoding network weighted regularization loss in the cross-training stage by using a cross-training method;

[0011] Determining an overall optimization loss based on the encoding network weighted regularization loss and the decoding network weighted regularization loss;

[0012] Taking the overall optimization loss as the optimization target, an Adam optimizer is used to optimize the encoding network parameters and the decoding network parameters of the lip reading recognition model;

[0013] Determining whether the lip reading recognition model has converged;

[0014] If the lip reading recognition model has not converged, returning to the step of adaptively generating a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase by using a cross-training method according to the encoding network parameters and the decoding network parameters with overfitting;

[0015] If the lip reading recognition model converges, generating a trained lip reading recognition model;

[0016] The trained lip reading recognition model is used to perform lip reading recognition.

[0017] Optionally, the step of using all the training data in the different identity lip reading training data set to train the lip reading recognition model in an end-to-end manner and obtaining encoding network parameters and decoding network parameters in the end-to-end training phase specifically includes:

[0018] Training data loss L with all identities all (θ, φ) is used as the optimization target, the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method, and the Adam optimizer is used to solve the network parameters of the optimization target to obtain the encoding network parameters θ and the decoding network parameters φ of the end-to-end training stage.

[0019] Optionally, the step of training the lip reading recognition model using single identity training data in the different identity lip reading training data set according to the encoding network parameters and decoding network parameters in the end-to-end training phase to obtain encoding network parameters and decoding network parameters with overfitting specifically includes:

[0020] Training data loss L with a single identity s (θ', φ') as the optimization target, based on the single identity training data D in the different identity lip reading training data set i , the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method to obtain encoding network parameters θ' and decoding network parameters φ' with overfitting.

[0021] Optionally, the step of adaptively generating a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase according to the overfitting encoding network parameters and decoding network parameters by cross-training specifically includes:

[0022] The semantic feature encoding network e with network parameters θ' is used θ' () Extract the semantic features of training and test data, and generate the first normalized weight based on the difference measurement of semantic features of different identities

[0023] According to the first normalized weight Calculate the decoding network weighted regularization loss L wf (φ);

[0024] The semantic feature encoding network e with network parameters θ is used θ () Extract the semantic features of training and test data, and generate the second normalized weights based on the difference measurement of semantic features of different identities

[0025] According to the second normalized weight Calculate the weighted regularization loss L of the encoding network we (θ).

[0026] Optionally, determining the overall optimization loss according to the encoding network weighted regularization loss and the decoding network weighted regularization loss specifically includes:

[0027] According to the encoding network weighted regularization loss L we (θ) and the decoding network weighted regularization loss L wf (φ), using the formula L(θ,φ)=L all (θ,φ)+λ 1 L we (θ)+λ 2 L wf (φ) determines the overall optimization loss L(θ,φ); where λ 1 , 2 is the regularization coefficient.

[0028] A lip reading recognition system based on phased cross-training, comprising:

[0029] A lip reading training data set acquisition module is used to acquire a lip reading training data set of different identities; a single training data in the lip reading training data set of different identities includes an image sequence of sample input and a category code of sample output;

[0030] An end-to-end network training module is used to train a lip reading recognition model in an end-to-end manner using all the training data in the lip reading training data set of different identities, and obtain encoding network parameters and decoding network parameters in the end-to-end training stage; the lip reading recognition model includes a semantic feature encoding network and a semantic feature decoding network;

[0031] A single identity training module is used to train the lip reading recognition model using the single identity training data in the different identity lip reading training data set according to the encoding network parameters and the decoding network parameters in the end-to-end training stage, so as to obtain the encoding network parameters and the decoding network parameters with overfitting;

[0032] A network cross-training module, used to adaptively generate a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase according to the encoding network parameters and the decoding network parameters with overfitting by using a cross-training method;

[0033] An overall optimization loss determination module, used to determine the overall optimization loss according to the encoding network weighted regularization loss and the decoding network weighted regularization loss;

[0034] A network parameter optimization module, used to optimize the encoding network parameters and decoding network parameters of the lip reading recognition model using the Adam optimizer with the overall optimization loss as the optimization target;

[0035] A network convergence judgment module, used to judge whether the lip reading recognition model has converged;

[0036] A cross-training return module, used for returning the step of adaptively generating the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network in the cross-training stage by cross-training according to the encoding network parameters and the decoding network parameters with overfitting if the lip reading recognition model has not converged;

[0037] A lip reading recognition model generation module, used for generating a trained lip reading recognition model if the lip reading recognition model converges;

[0038] The lip reading recognition module is used to perform lip reading recognition using the trained lip reading recognition model.

[0039] Optionally, the end-to-end network training module specifically includes:

[0040] End-to-end network training unit for training data with all identities with loss L all (θ, φ) is used as the optimization target, the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method, and the Adam optimizer is used to solve the network parameters of the optimization target to obtain the encoding network parameters θ and the decoding network parameters φ of the end-to-end training stage.

[0041] Optionally, the single identity training module specifically includes:

[0042] Single identity training unit, used to train data with a single identity loss L s (θ', φ') as the optimization target, based on the single identity training data D in the different identity lip reading training data set i , the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method to obtain encoding network parameters θ' and decoding network parameters φ' with overfitting.

[0043] Optionally, the network cross-training module specifically includes:

[0044] The first normalized weight generation unit is used to adopt the semantic feature encoding network e with network parameters θ' θ' () Extract the semantic features of training and test data, and generate the first normalized weight based on the difference measurement of semantic features of different identities

[0045] A decoding network weighted regularization loss calculation unit is used to calculate the loss according to the first normalized weight Calculate the decoding network weighted regularization loss L wf (φ);

[0046] The second normalized weight generation unit is used to adopt the semantic feature encoding network e with network parameters θ θ () Extract the semantic features of training and test data, and generate the second normalized weights based on the difference measurement of semantic features of different identities

[0047] The encoding network weighted regularization loss calculation unit is used to calculate the loss according to the second normalized weight Calculate the weighted regularization loss L of the encoding network we (θ).

[0048] Optionally, the overall optimization loss determination module specifically includes:

[0049] The overall optimization loss determination unit is used to weight the regularization loss L according to the encoding network. we (θ) and the decoding network weighted regularization loss L wf (φ), using the formula L(θ,φ)=L all (θ,φ)+λ 1 L we (θ)+λ 2 L wf (φ) determines the overall optimization loss L(θ,φ); where λ 1 , 2 is the regularization coefficient.

[0050] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0051] The present invention provides a lip reading recognition method and system based on staged cross-training, the method comprising: obtaining a lip reading training data set of different identities; using all the training data in the lip reading training data set of different identities to train a lip reading recognition model in an end-to-end manner, and obtaining encoding network parameters and decoding network parameters in the end-to-end training stage; the lip reading recognition model comprises a semantic feature encoding network and a semantic feature decoding network; according to the encoding network parameters and decoding network parameters in the end-to-end training stage, using a single identity training data in the lip reading training data set of different identities to train the lip reading recognition model, and obtaining encoding network parameters and decoding network parameters with overfitting. number; according to the encoding network parameters and decoding network parameters with overfitting, the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network in the cross-training stage are adaptively generated by cross-training; the overall optimization loss is determined according to the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network; with the overall optimization loss as the optimization target, the Adam optimizer is used to optimize the encoding network parameters and the decoding network parameters of the lip reading recognition model; it is determined whether the lip reading recognition model converges, if not, the network parameter cross-training is continued, if so, a trained lip reading recognition model is generated; and the trained lip reading recognition model is used for lip reading recognition. The method of the present invention starts from the perspective of training strategy, is based on the traditional lip reading recognition model architecture, aims to enhance the generalization performance of the network in the unknown data distribution, and optimizes the network parameters of the lip reading recognition model by means of staged cross-training. The lip reading recognition model trained by the present invention is used for lip reading recognition, which can improve the accuracy of lip reading recognition of unknown speakers. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A schematic diagram of the architecture of the traditional lip reading recognition model provided by the present invention;

[0054] Figure 2 A schematic diagram of the principle of a lip reading recognition method based on staged cross-training according to the present invention;

[0055] Figure 3 The present invention is a flow chart of a lip reading recognition method based on staged cross-training. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The purpose of the present invention is to provide a lip reading recognition method based on staged cross-training. From the perspective of training strategy, the method is based on the traditional lip reading recognition model architecture, with the goal of enhancing the generalization performance of the network in unknown data distribution. The method uses staged cross-training to optimize the network parameters of the lip reading recognition model, thereby improving the accuracy of lip reading recognition of unknown speakers.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Lip reading recognition is the process of decoding the speaker's lip shape information. Figure 1 This is a schematic diagram of the traditional lip reading recognition model architecture provided by the present invention. Figure 1 In the training phase of the lip reading recognition model, the lip reading recognition model is trained based on the input image sequence and the corresponding output text information, and the cross entropy loss between the model's predicted output and the actual output is used as the basis for model learning (optimization target). In the testing phase, semantic prediction is performed on the input lip image sequence based on the trained model.

[0060] The lip reading recognition model consists of two parts: the semantic feature encoding network and the semantic feature decoding network. The semantic feature encoding network is responsible for mining the corresponding minimum semantic unit from the lip image, that is, the minimum semantic feature used to distinguish different pronunciation units. The semantic feature decoding network is responsible for decoding the text information of the speech content from the continuous sequence of minimum semantic units. The commonly used lip reading recognition framework usually adopts the convolutional neural network (CNN) + recurrent neural network (RNN) structure to implement the above process. The CNN network encodes the minimum semantic unit of the lip image, and the RNN decodes the continuous minimum semantic unit features into text output.

[0061] In theory, the output of the semantic encoding network is the basic syllables that constitute the speech content, which is not affected by the speaker's identity; while the semantic decoding network refers to a unified language model to convert continuous syllables into text output, which should also be independent of the speaker's identity information. The two links should be independent of each other, and the semantic features mined from the lip reading data of any speaker can be translated into language through a specific decoding network. Similarly, the semantic decoding model obtained from the lip reading data of any speaker should be applicable to the semantic space generated by lip reading data of multiple identities. However, the training method of the traditional lip reading recognition model adopts an end-to-end learning mode, and the encoding network and the decoding network are adjusted simultaneously according to the final cross entropy loss. In the process of model learning, there is a lack of constraints on the identity independence of the encoding network and the decoding network, which will inevitably introduce information coupling, resulting in mutual dependence and restriction between the encoding network and the decoding network, and the dependence on the speaker's identity. It is impossible to obtain good recognition results in the unknown speaker test, that is, the model is overfitted to the training data. Therefore, under the premise of ensuring that the network achieves good performance on the training data, the mutual independence of each link of the network (semantic encoding, semantic decoding) is the key to restricting the generalization performance of the network in the unknown speaker test scenario.

[0062] Based on the above considerations, the present invention provides a phased cross-training method, providing an evaluation index of the network generalization performance based on known identity lip reading data, so as to improve the independence of the lip reading recognition encoding and decoding networks. The specific principle is as follows Figure 2 As shown in Figure 2, model learning is divided into two stages. The first stage is the end-to-end model training stage, and the second stage is the cross-model training stage.

[0063] Figure 3 This is a flow chart of a lip reading recognition method based on phased cross-training of the present invention. Figure 2 and Figure 3 The present invention provides a lip reading recognition method based on staged cross-training, comprising:

[0064] Step 101: Obtain a lip reading training data set of different identities.

[0065] Get different identity lip reading training data set Φ = {D 1 , D 2 …, D N}, where D i represents the set of training data with speaker identity i, and |Φ| represents the number of samples in the set. A single training data (x, y) (also called a training sample) in the different identity lip reading training data set Φ includes an image sequence x of the sample input and a category code y of the sample output. Different category codes y correspond to different text information of the speech content.

[0066] Step 102: using all the training data in the different identity lip reading training data set, training the lip reading recognition model in an end-to-end manner, and obtaining encoding network parameters and decoding network parameters in the end-to-end training phase.

[0067] See also Figure 1 The lip reading recognition model includes a semantic feature encoding network and a semantic feature decoding network. The semantic feature encoding network is responsible for mining the corresponding minimum semantic unit from the lip image x, that is, the minimum semantic feature used to distinguish different pronunciation units. The semantic feature decoding network is responsible for decoding the text information of the speech content from the continuous sequence of minimum semantic units (output in the form of category code y).

[0068] In the first stage of network model training of the present invention, the lip reading recognition model is trained using all data and an end-to-end method to obtain semantic encoding network parameters θ and decoding network parameters φ. Specifically, step 102 uses all identity training data loss L all (θ, φ) is used as the optimization target, and the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously in an end-to-end manner. The Adam optimizer is used to solve the network parameters of the optimization target, and the encoding network parameters θ and the decoding network parameters φ of the end-to-end training stage are obtained.

[0069] Specifically, calculate the supervision loss L of the lip reading recognition network in all training data all (θ, φ) is used as the network optimization target in the first stage:

[0070]

[0071] Among them, L all (θ, φ) is also referred to as the total identity training data loss in the present invention, loss represents the cross entropy loss of the training sample (x, y), Φ = {D 1 , D 2 …, D N} represents the set of lip reading training data of different identities, D i represents the set of training data with speaker identity i, and |Φ| represents the number of samples in the set. (x,y) represents a single training sample (training data) in the set Φ, where x represents the image sequence of the sample input and y represents the category code of the sample; Encode the predicted category of the lip reading recognition model prediction output. Represents the predicted output of the model The cross entropy loss with the true output y. The calculation method is as follows:

[0072]

[0073] where e θ () represents a semantic feature encoding network with a network parameter of θ, whose input is the lip image sequence x and output is the minimum semantic unit sequence. φ () represents a semantic decoding network with a network parameter of φ, whose input is the minimum semantic unit sequence e θ (x), the output is the predicted semantic category encoding

[0074] In the first stage of the method of the present invention, an end-to-end training method is used to simultaneously optimize the semantic encoding network and the decoding network of the lip reading recognition model, so as to make full use of a large amount of lip reading data of different identities, explore the unified law of semantic features and the semantic decoding method, and use the diversity of data to improve the generalization ability of the network model. The Adam optimizer is used to solve the network parameters of the optimization target, and the encoding network parameters θ and the decoding network parameters φ of the first stage are obtained.

[0075] Step 103: According to the encoding network parameters and decoding network parameters of the end-to-end training phase, the lip reading recognition model is trained using single-identity training data in the different-identity lip reading training data set to obtain encoding network parameters and decoding network parameters with overfitting.

[0076] See also Figure 2 In the second stage of network model training (i.e., cross-training stage), the lip reading recognition model is trained using single identity data to obtain the encoding network parameter θ' and the decoding network parameter φ' with overfitting. Specifically, step 103 uses the single identity training data loss L s (θ', φ') as the optimization target, based on the single identity training data D in the different identity lip reading training data set i , the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method to obtain encoding network parameters θ' and decoding network parameters φ' with overfitting.

[0077] Specifically, the lip reading recognition network is trained with single identity data, and its optimization objective is defined as follows:

[0078]

[0079] Where L s (θ',φ') is called the single identity training data loss. |D i | represents the number of morning samples in the training data set with speaker identity i. θ' and φ' represent the corresponding parameters of the lip reading recognition encoding network and decoding network trained using single identity data. e θ' () represents the semantic feature encoding network with network parameters θ', f φ'() represents the semantic feature decoding network with network parameters φ'. The above loss L is optimized using Adam optimizer s (θ', φ'), obtain the encoding network and decoding network parameters. Since there is only a single identity lip reading data, the network is at risk of overfitting to the current data distribution characteristics.

[0080] Step 104: According to the encoding network parameters and decoding network parameters with overfitting, a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase are adaptively generated by using a cross-training method.

[0081] In the cross-training stage of the network model training of the present invention, the semantic coding network regularization loss L is adaptively generated by cross-training. e (θ) and the decoding network regularization loss L f (φ), thereby constraining the independence of the encoding network and the decoding network, thereby improving the model's generalization ability for unknown identities. The basis for the generation of regularization loss is the tolerance of network parameters to data with unknown distribution differences.

[0082] See also Figure 2 , to encode the network e θ' () Extract the semantic features of the remaining test data and obtain the decoding network f φ () Semantic data with distribution offset, input the offset data into the semantic decoding network f φ (), calculate the prediction loss, and adaptively generate the decoding network regularization loss L f (φ). With decoding network e θ () Extract the semantic features of the remaining test data and obtain the decoding network f φ' () Semantic data with distribution offset, input the offset data into the semantic decoding network f φ' (), calculate the prediction loss, and adaptively generate the encoding network regularization loss L e (θ).

[0083] Specifically, we first fix the encoding network parameters θ' to extract the training data except the current identity data D i The remaining data D j ,j=1,2,…,N∩j≠i, due to the overfitting of network θ', the generated semantic features usually contain a larger distribution offset compared to the data extracted by the first stage network θ. θ' () Generate offset data, using the decoding network f φ () to decode and calculate its prediction loss:

[0084]

[0085] Among them, e θ' () represents the semantic encoding network with network parameters θ'; L f (φ) i Indicates that the i-th group of identity data D i Initialize the prediction loss corresponding to the network parameters. i , i=1,2,…,N initialize network parameters, N represents the number of speakers with different identities, and calculate the corresponding prediction loss L f (φ) i . Then the final adaptive decoding network regularization loss L f (φ) is expressed as:

[0086]

[0087] If the training loss L f The smaller (φ) is, the more robust the decoding network is to unknown offset data, and the better the generalization ability of the decoding network is.

[0088] Similarly, using the encoding network e θ () Extract the remaining data D except the current identity in the training data j ,j=1,2,…,N∩j≠i semantic features, fixed decoding network f φ' (), due to the overfitting problem in φ', the decoding network tends to the semantic space of the previous single identity, and the data distribution applicable to the network input is different from that of the network e θ () The extracted features have obvious differences. θ () Generate offset data, using network f φ' () to decode and calculate its prediction loss:

[0089]

[0090] Among them, f φ' () represents the semantic decoding network with network parameters φ'; L e (θ) i Indicates that the i-th group of identity data D i Initialize the prediction loss corresponding to the network parameters. i , i=1,2,…,N initialize network parameters, N represents the number of speakers with different identities, and calculate the corresponding prediction loss L e (θ) i . Then the final adaptive encoding network regularization loss L e (θ) is expressed as:

[0091]

[0092] If the training loss L e The smaller (θ) is, the less the semantic information represented by the feature extraction network is affected by identity interference, and the stronger the generalization ability of the encoding network is.

[0093] In the encoding network regularization loss L e (θ) and the decoding network regularization loss L f (φ), in order to avoid the unbalanced learning problem caused by similar identity data (causing the model to tend to a certain identity), the present invention measures the difference in the identity feature encoding distribution of data in different domains and uses this to weight the above regularization loss. Specifically, the encoding network e θ' () Extract the semantic features of training and test data, and generate normalized weights based on the difference measurement of semantic features of different identities And use this to calculate the decoding network weighted regularization loss L wf (φ); with encoding network e θ () Extract the semantic features of training and test data, and generate normalized weights based on the difference measurement of semantic features of different identities And use this to calculate the weighted regularization loss L of the encoding network we (θ).

[0094] Specifically, in the cross-training stage, using the single identity data D i Initialize the lip reading recognition network, and its encoding network and decoding network parameters are θ' and φ'. When calculating the regularization loss of the decoding network, the overfitting encoding network e θ' () Calculate the remaining different identity domain data D j ,j=1,2,…,N∩j≠i, generating semantic features for the decoding network f φ () For data with distribution shift, the distribution difference metric between the test identity and the training identity data is calculated as follows:

[0095]

[0096] Represents an overfitting encoding network e θ' () The similarity between the extracted training identity i and the test identity j. The following normalized similarity is used as the weight of the regularization loss in different domains:

[0097]

[0098] in It is called the first normalized weight.

[0099] During cross-training, the weighted regularization loss that considers the similarity between the test identity and the training identity is defined as:

[0100]

[0101]

[0102] Where L wf (φ) is the weighted regularization loss of the decoding network.

[0103] When calculating the regularization loss of the encoding network, the encoding network e θ () Calculate the remaining different identity data D j ,j=1,2,…,N∩j≠i, generating semantic features for the overfitting decoding network f φ' () For data with distribution shift, the distribution difference metric between the test identity and the training identity data is calculated as follows:

[0104]

[0105] Represented by the encoding network e θ () The similarity between the extracted training identity i and the test identity j. The following normalized similarity is used as the weight of the regularization loss in different domains:

[0106]

[0107] in It is called the second normalization weight.

[0108] During the cross-training process, the weighted regularization loss considering the similarity between the test domain and the training domain is defined as:

[0109]

[0110]

[0111] Where L we (θ) is the weighted regularization loss of the encoding network.

[0112] When testing domain identity data D j With the training domain identity data D i The greater the similarity, the smaller the weight. Smaller weights can reduce the impact of this type of identity on model training during cross-training and prevent the model from paying too much attention to similar identity data.

[0113] Therefore, the step 104 adaptively generates the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network in the cross-training stage according to the encoding network parameters and the decoding network parameters with overfitting by using a cross-training method, specifically including:

[0114] The semantic feature encoding network e with network parameters θ' is used θ' () Extract the semantic features of training and test data, and generate the first normalized weight based on the difference measurement of semantic features of different identities

[0115] According to the first normalized weight Calculate the decoding network weighted regularization loss L wf (φ);

[0116] The semantic feature encoding network e with network parameters θ is used θ () Extract the semantic features of training and test data, and generate the second normalized weights based on the difference measurement of semantic features of different identities

[0117] According to the second normalized weight Calculate the weighted regularization loss L of the encoding network we (θ).

[0118] Step 105: Determine the overall optimization loss according to the encoding network weighted regularization loss and the decoding network weighted regularization loss.

[0119] The cross-training method is used to adaptively generate the semantic encoding network weighted regularization loss L we (θ) and the decoding network weighted regularization loss L wf (φ). Then the final optimization goal of the network is:

[0120] L(θ,φ)=L all (θ,φ)+λ 1 L we (θ)+λ 2 L wf (φ) (16)

[0121] Among them, L(θ,φ) is the overall optimization loss, L all (θ, φ) is the total identity training data loss, λ 1 , 2 is the regularization coefficient.

[0122] After obtaining the loss L(θ,φ), the Adam optimizer is used to optimize the encoding network parameters θ and the decoding network parameters φ in the lip reading recognition model to obtain the final trained network model.

[0123] Step 106: Taking the overall optimization loss as the optimization target, an Adam optimizer is used to optimize the encoding network parameters and the decoding network parameters of the lip reading recognition model.

[0124] Step 107: Determine whether the lip reading recognition model has converged.

[0125] Step 108: If the lip reading recognition model has not converged, return to the step of adaptively generating the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network in the cross-training phase by cross-training according to the overfitting encoding network parameters and decoding network parameters.

[0126] Taking the overall optimization loss L(θ,φ) as the goal, the Adam optimizer is used to optimize the encoding network and decoding network parameters θ, φ, and the steps 104 to 106 are repeated until the network converges. The encoding and decoding networks with the optimized encoding network and decoding network parameters θ, φ together constitute a trained lip reading recognition model.

[0127] Step 109: If the lip reading recognition model converges, a trained lip reading recognition model is generated.

[0128] Step 110: Use the trained lip reading recognition model to perform lip reading recognition.

[0129] In practical applications, it is only necessary to input the lip image sequence to be recognized into the trained lip reading recognition model, and the category code corresponding to the text information of the speech content can be output, so that the speech content of the unknown speaker can be recognized.

[0130] Four different unknown speaker test scenarios are used to test the recognition results of the method of the present invention. The comparison method is an end-to-end learning method. The test results are shown in Table 1.

[0131] Table 1 Test results under different test scenarios with unknown speakers

[0132]

[0133] It can be seen from the test results shown in Table 1 that in different unknown speaker test scenarios, the staged cross-training method used by the method of the present invention achieves a higher recognition accuracy. At the same time, the staged cross-training method used by the method of the present invention has a smaller standard deviation, indicating that the staged cross-training method can achieve similar recognition accuracy for different users and is less affected by the identity of the speaker.

[0134] The present invention provides a phased cross-training method for lip reading recognition, the core of which includes: obtaining the encoding network parameters θ and the decoding network parameters φ in the first phase based on an end-to-end training method; in the second phase, training the model according to the single identity data to generate the initialization parameters θ' and φ' of the encoding network and the decoding network with overfitting, and using the encoding network initialization parameter θ' to adaptively generate the weighted regularization constraint L of the decoding network wf (φ), using the decoding network initialization parameter φ' to adaptively generate the encoding network weighted regularization constraint Lwe (θ). The present invention ensures the independence of the encoding network and the decoding network through such cross-training, thereby improving the generalization ability of the lip reading recognition network. Furthermore, when the lip reading recognition model trained by the present invention is applied to lip reading recognition, the robustness and accuracy of the lip reading recognition of unknown speakers can be improved.

[0135] Based on the lip reading recognition method based on staged cross-training provided by the present invention, the present invention also provides a lip reading recognition system based on staged cross-training, the system comprising:

[0136] A lip reading training data set acquisition module is used to acquire a lip reading training data set of different identities; a single training data in the lip reading training data set of different identities includes an image sequence of sample input and a category code of sample output;

[0137] An end-to-end network training module is used to train a lip reading recognition model in an end-to-end manner using all the training data in the lip reading training data set of different identities, and obtain encoding network parameters and decoding network parameters in the end-to-end training stage; the lip reading recognition model includes a semantic feature encoding network and a semantic feature decoding network;

[0138] A single identity training module is used to train the lip reading recognition model using the single identity training data in the different identity lip reading training data set according to the encoding network parameters and the decoding network parameters in the end-to-end training stage, so as to obtain the encoding network parameters and the decoding network parameters with overfitting;

[0139] A network cross-training module, used to adaptively generate a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase according to the encoding network parameters and the decoding network parameters with overfitting by using a cross-training method;

[0140] An overall optimization loss determination module, used to determine the overall optimization loss according to the encoding network weighted regularization loss and the decoding network weighted regularization loss;

[0141] A network parameter optimization module, used to optimize the encoding network parameters and decoding network parameters of the lip reading recognition model using the Adam optimizer with the overall optimization loss as the optimization target;

[0142] A network convergence judgment module, used to judge whether the lip reading recognition model has converged;

[0143] A cross-training return module, used for returning the step of adaptively generating the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network in the cross-training stage by cross-training according to the encoding network parameters and the decoding network parameters with overfitting if the lip reading recognition model has not converged;

[0144] A lip reading recognition model generation module, used for generating a trained lip reading recognition model if the lip reading recognition model converges;

[0145] The lip reading recognition module is used to perform lip reading recognition using the trained lip reading recognition model.

[0146] Wherein, the end-to-end network training module specifically includes:

[0147] End-to-end network training unit for training data with all identities with loss L all (θ, φ) is used as the optimization target, the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method, and the Adam optimizer is used to solve the network parameters of the optimization target to obtain the encoding network parameters θ and the decoding network parameters φ of the end-to-end training stage.

[0148] The single identity training module specifically includes:

[0149] Single identity training unit, used to train data with a single identity loss L s (θ', φ') as the optimization target, based on the single identity training data D in the different identity lip reading training data set i , the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method to obtain encoding network parameters θ' and decoding network parameters φ' with overfitting.

[0150] The network cross-training module specifically includes:

[0151] The first normalized weight generation unit is used to adopt the semantic feature encoding network e with network parameters θ' θ' () Extract the semantic features of training and test data, and generate the first normalized weight based on the difference measurement of semantic features of different identities

[0152] A decoding network weighted regularization loss calculation unit is used to calculate the loss according to the first normalized weight Calculate the decoding network weighted regularization loss L wf (φ);

[0153] The second normalized weight generation unit is used to adopt the semantic feature encoding network e with network parameters θθ () Extract the semantic features of training and test data, and generate the second normalized weights based on the difference measurement of semantic features of different identities

[0154] The encoding network weighted regularization loss calculation unit is used to calculate the loss according to the second normalized weight Calculate the weighted regularization loss L of the encoding network we (θ).

[0155] The overall optimization loss determination module specifically includes:

[0156] The overall optimization loss determination unit is used to weight the regularization loss L according to the encoding network. we (θ) and the decoding network weighted regularization loss L wf (φ), using the formula L(θ,φ)=L all (θ,φ)+λ 1 L we (θ)+λ 2 L wf (φ) determines the overall optimization loss L(θ,φ); where λ 1 , 2 is the regularization coefficient.

[0157] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0158] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A lip reading recognition method based on staged cross-training, characterized in that: include: Acquire a set of lip reading training data for different identities; a single training data in the set of lip reading training data for different identities includes an image sequence of sample input and a category code of sample output; Using all the training data in the lip reading training data set of different identities, a lip reading recognition model is trained in an end-to-end manner to obtain encoding network parameters and decoding network parameters in the end-to-end training stage; the lip reading recognition model includes a semantic feature encoding network and a semantic feature decoding network; According to the encoding network parameters and the decoding network parameters in the end-to-end training phase, the lip reading recognition model is trained using the single identity training data in the different identity lip reading training data set to obtain the encoding network parameters and the decoding network parameters with overfitting; According to the encoding network parameters and decoding network parameters with overfitting, adaptively generating the encoding network weighted regularization loss and the decoding network weighted regularization loss in the cross-training stage by using a cross-training method; Determining an overall optimization loss based on the encoding network weighted regularization loss and the decoding network weighted regularization loss; Taking the overall optimization loss as the optimization target, an Adam optimizer is used to optimize the encoding network parameters and the decoding network parameters of the lip reading recognition model; Determining whether the lip reading recognition model has converged; If the lip reading recognition model has not converged, returning to the step of adaptively generating a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase by using a cross-training method according to the encoding network parameters and the decoding network parameters with overfitting; If the lip reading recognition model converges, generating a trained lip reading recognition model; The trained lip reading recognition model is used to perform lip reading recognition.

2. The method according to claim 1, characterized in that The method of using all the training data in the lip reading training data set of different identities to train the lip reading recognition model in an end-to-end manner and obtaining the encoding network parameters and the decoding network parameters in the end-to-end training phase specifically includes: Training data loss L with all identities all (θ, φ) is used as the optimization target, the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method, and the Adam optimizer is used to solve the network parameters of the optimization target to obtain the encoding network parameters θ and the decoding network parameters φ of the end-to-end training stage.

3. The method according to claim 2, characterized in that The method of training the lip reading recognition model using the single identity training data in the different identity lip reading training data set according to the encoding network parameters and the decoding network parameters in the end-to-end training stage to obtain the encoding network parameters and the decoding network parameters with overfitting specifically includes: Training data loss L with a single identity s (θ', φ') as the optimization target, based on the single identity training data D in the different identity lip reading training data set i , the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method to obtain encoding network parameters θ' and decoding network parameters φ' with overfitting.

4. The method according to claim 3, characterized in that The method of adaptively generating a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase according to the overfitting encoding network parameters and the decoding network parameters by cross-training specifically includes: The semantic feature encoding network e with network parameters θ' is used θ' () Extract the semantic features of training and test data, and generate the first normalized weight based on the difference measurement of semantic features of different identities According to the first normalized weight Calculate the decoding network weighted regularization loss L wf (φ); The semantic feature encoding network e with network parameters θ is used θ () Extract the semantic features of training and test data, and generate the second normalized weights based on the difference measurement of semantic features of different identities According to the second normalized weight Calculate the weighted regularization loss L of the encoding network we (θ).

5. The method according to claim 4, characterized in that The determining of the overall optimization loss according to the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network specifically includes: According to the encoding network weighted regularization loss L we (θ) and the decoding network weighted regularization loss L wf (φ), using the formula L(θ,φ)=L all (θ,φ)+λ1L we (θ)+λ2L wf (φ) determines the overall optimization loss L(θ,φ); where λ1 and λ2 are regularization coefficients.

6. A lip reading recognition system based on staged cross-training, characterized in that: include: A lip reading training data set acquisition module is used to acquire lip reading training data sets of different identities; A single training data in the different identity lip reading training data set includes an image sequence of sample input and a category code of sample output; An end-to-end network training module is used to train a lip reading recognition model in an end-to-end manner using all the training data in the lip reading training data set of different identities, and obtain encoding network parameters and decoding network parameters in the end-to-end training stage; the lip reading recognition model includes a semantic feature encoding network and a semantic feature decoding network; A single identity training module is used to train the lip reading recognition model using the single identity training data in the different identity lip reading training data set according to the encoding network parameters and the decoding network parameters in the end-to-end training stage, so as to obtain the encoding network parameters and the decoding network parameters with overfitting; A network cross-training module, used to adaptively generate a weighted regularization loss of the encoding network and a weighted regularization loss of the decoding network in a cross-training phase according to the encoding network parameters and the decoding network parameters with overfitting by using a cross-training method; An overall optimization loss determination module, used to determine the overall optimization loss according to the encoding network weighted regularization loss and the decoding network weighted regularization loss; A network parameter optimization module, used to optimize the encoding network parameters and decoding network parameters of the lip reading recognition model using the Adam optimizer with the overall optimization loss as the optimization target; A network convergence judgment module, used to judge whether the lip reading recognition model has converged; A cross-training return module, used for returning the step of adaptively generating the weighted regularization loss of the encoding network and the weighted regularization loss of the decoding network in the cross-training stage by cross-training according to the encoding network parameters and the decoding network parameters with overfitting if the lip reading recognition model has not converged; A lip reading recognition model generation module, used for generating a trained lip reading recognition model if the lip reading recognition model converges; The lip reading recognition module is used to perform lip reading recognition using the trained lip reading recognition model.

7. The system according to claim 6, characterized in that The end-to-end network training module specifically includes: End-to-end network training unit for training data with all identities with loss L all (θ, φ) is used as the optimization target, the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method, and the Adam optimizer is used to solve the network parameters of the optimization target to obtain the encoding network parameters θ and the decoding network parameters φ of the end-to-end training stage.

8. The system according to claim 7, characterized in that The single identity training module specifically includes: Single identity training unit, used to train data with a single identity loss L s (θ', φ') as the optimization target, based on the single identity training data D in the different identity lip reading training data set i , the semantic feature encoding network and the semantic feature decoding network of the lip reading recognition model are optimized simultaneously by using the end-to-end method to obtain encoding network parameters θ' and decoding network parameters φ' with overfitting.

9. The system according to claim 8, characterized in that The network cross-training module specifically includes: The first normalized weight generation unit is used to adopt the semantic feature encoding network e with network parameters θ' θ' () Extract the semantic features of training and test data, and generate the first normalized weight based on the difference measurement of semantic features of different identities A decoding network weighted regularization loss calculation unit is used to calculate the loss according to the first normalized weight Calculate the decoding network weighted regularization loss L wf (φ); The second normalized weight generation unit is used to adopt the semantic feature encoding network e with network parameters θ θ () Extract the semantic features of training and test data, and generate the second normalized weights based on the difference measurement of semantic features of different identities The encoding network weighted regularization loss calculation unit is used to calculate the loss according to the second normalized weight Calculate the weighted regularization loss L of the encoding network we (θ).

10. The system according to claim 9, characterized in that The overall optimization loss determination module specifically includes: The overall optimization loss determination unit is used to weight the regularization loss L according to the encoding network. we (θ) and the decoding network weighted regularization loss L wf (φ), using the formula L(θ,φ)=L all (θ,φ)+λ1L we (θ)+λ2L wf (φ) determines the overall optimization loss L(θ,φ); where λ1 and λ2 are regularization coefficients.

Images