Visual language recognition method based on spatio-temporal local harmonic neural network and application

Abstract

The application discloses a visual language recognition method based on a space-time local harmonic neural network and application, and steps of the method comprise the following steps: 1, data preprocessing; 2, constructing a visual language recognition model based on a space-time local harmonic neural network; 3, training of the network model.The method can solve the problems of the existing visual language recognition methods, such as insufficient extraction ability of time characteristics and space characteristics, single method, and not paying attention to the difference of different information, so that the method can accurately recognize the word content in the scene where the speaker posture and the speech speed frequently change, and further provides a new solution for visual language recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a visual language recognition method based on a spatiotemporal local harmonic neural network. BACKGROUND

[0002] With the development of deep learning, visual language recognition has also attracted more and more attention. Visual language recognition specifically refers to "reading out" or "partly reading out" the text content spoken by a speaker only through the visual image of the speaker without audio information of the speaker. In real life, people also have demands for visual language tasks in many aspects, such as assisting audio recognition, keyword recognition in public security, and language recognition for the hearing impaired, etc. With the development of machine learning, visual language recognition has now exceeded human beings.

[0003] For visual language recognition, it is important to extract spatiotemporal features with higher recognition degree, and the existing visual language recognition network still has some defects in this respect. At present, most visual language recognitions are performed by sequence convolution structure for feature extraction. This sequence structure lacks reasonable correlation of global and local information of the lips in the time and space dimensions, and the extracted features cannot balance the relationship between the whole and the local in time and space, affecting the recognition degree of the spatiotemporal features. SUMMARY

[0004] In order to solve the above-mentioned problems of the prior art, the present application provides a visual language recognition method based on a spatiotemporal local harmonic neural network and an application, so as to solve the problems of insufficient ability of the existing visual language recognition method to extract time features and space features, single method, and lack of attention to the difference of different information, so as to accurately recognize the word content in the scene where the speaker's posture and speech speed change frequently, and thus provide a new solution for visual language recognition.

[0005] In order to achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0006] The visual language recognition method based on a spatiotemporal local harmonic neural network has the following steps:

[0007] Step 1: data preprocessing:

[0008] Step 1.1: obtain a visual language data set with word labels, and perform gray scale processing on the video image data, then calibrate and align the face key points, so that the face is in the middle position of the image, thereby obtaining a face image sequence, denoted as P={P1, P2,...P i ,...P n}, P idenotes the i-th frame of face image; n denotes the total number of face images;

[0009] Step 1.2: cropping the lip region of each frame of face image in the face image sequence P by a size of HxH, and randomly cropping the cropped lip region, so as to obtain a lip reading sequence with a size of WxW, denoted as P s1 s1 s2 si sn};wherein, P si denotes the lip reading region of the i-th frame of face image, and W<H;

[0010] Step 2: constructing a visual language recognition model based on a spatio-temporal local harmonic neural network, including a 3D spatio-temporal feature extraction module, a spatial harmonic module, a temporal harmonic module, a 2D spatial feature extraction module, a time sequence convolution module, and a classifier;

[0011] Step 2.1: the 3D spatio-temporal feature extraction module is composed of N k 3D blocks connected in sequence, and each 3D block is composed of a convolution layer, a BN layer and a ReLU activation function connected in sequence;

[0012] The lip reading sequence P s1 is input into the 3D spatio-temporal feature extraction module, and after being processed by N k blocks in sequence, the spatio-temporal feature F d ={F d1 , F d2 ,... F di ,... F dn} is obtained, F di is the 3D spatio-temporal feature of the i-th face image;

[0013] Step 2.2: the spatial harmonic module is composed of N l 2D spatial blocks and a deformable convolution, wherein any l-th 2D spatial block is composed of a convolution layer with a convolution kernel of X l x X l , a BN layer and a ReLU activation function connected in sequence; N l is an odd number greater than or equal to 3;

[0014] Step 2.2.1: the 3D spatio-temporal feature F d is input into the spatial harmonic module, and after being processed in the first two 2D spatial blocks, the corresponding spatio-temporal feature F 1c ={F 1c1 , F 1c2 ,... F 1ci ,... F​​​​1cn} and F 2c = {F 2c1 , F 2c2， ... F 2ci ,... F 2cn}, where F 1ci , F 2ci represent the 2D spatio-temporal features after the i-th 3D spatio-temporal feature passes through the 1st and 2nd 2D spatial blocks, respectively;

[0015] After connecting F 1c , F 2c in the time dimension, the spliced spatio-temporal features F c = {F c1 , F c2 ,... F ci ,... F cn} are obtained, where F ci is the i-th spliced spatio-temporal feature;

[0016] Step 2.2.2: If j = l, the spliced spatio-temporal feature F c is input into the N l -th 2D spatial block for processing to obtain the spatially harmonized offset, denoted as O f = {O f1 , O f2 ,... Of i ,... O fn}, where O fi represents the i-th spatially harmonized offset; otherwise, the spliced spatio-temporal feature F c continues the process of step 2.2.1 to input the subsequent 2D spatial block for processing;

[0017] The spatio-temporal feature F d is then input into the deformable convolution DCN together with the offset O f for processing, and the spatially harmonized spatio-temporal feature F o = {F o1 , F o2 ,... F oi ,... F on} is obtained, where F oi is the i-th spatially harmonized spatio-temporal feature;

[0018] Step 2.3: The temporal harmonization module is composed of N z 2D temporal blocks and deformable convolution, where any z-th 2D temporal block is composed of a convolution layer with a convolution kernel of X z × X z , a BN layer, and a ReLU activation function connected in sequence;

[0019] Step 2.3.1: Spatiotemporal characteristics of spatial harmony Spatiotemporal characteristics F o The input is given to the time harmonization module, and the i-th context spatiotemporal feature F is obtained using equation (1). ai Thus, the spatiotemporal features F are obtained. a ={F a1 F a2 , ...F ai , ...F an}:

[0020]

[0021] In equation (1), F o，i-1 F represents the (i-1)th harmonic spatiotemporal feature; o，i+1 This represents the (i+1)th harmonic spatiotemporal feature;

[0022] Step 2.3.2: Connect the spatiotemporal context features F obtained after concatenation. a The first N in sequence z After processing one 2D time block, a cascaded time harmonic is obtained, which is then processed through the Nth time block. z After processing each 2D time block, the time-harmonic offset is obtained, denoted as S. f ={S f1 S f2 ,...S fi ,...S fn}, where S fi Represents the offset of the i-th time harmonization;

[0023] Then the spatiotemporal features F o With offset S f The data are fed into a deformable convolutional neural network (DCN) for processing, and the temporally harmonized spatiotemporal features F are obtained. s ={F s1 F s2 , ...F si , ...Fs n}, where F si The spatiotemporal features after harmonization at the i-th time;

[0024] Step 2.4: The 2D spatial feature extraction consists of the first four residual blocks of ResNet-18, and each residual block includes two convolutional layers, one BN layer and one ReLU activation function in sequence;

[0025] The spatiotemporal feature F of time harmony s The residual spatiotemporal feature F is obtained by inputting into the 2D spatial feature module and processing it sequentially through four layers of residual network blocks. m ={F m1 F m2..., F mi ..., F mn ..., F mi is the i-th residual spatio-temporal feature.

[0026] Step 2.5: the timing convolution module is composed of N v residual blocks, wherein any v-th residual block contains two layers of timing convolution layers and a nonlinear mapping layer, and each layer of timing convolution layer is connected with a BN layer and a ReLU activation function; wherein the convolution kernel size of the timing convolution layer in any v-th residual block is X v × X v ;

[0027] The spatio-temporal feature F m is input into the timing convolution module and sequentially processed by N v residual blocks to obtain timing convolution spatio-temporal features F t ={F t1 , F t2 ,... Fx i , … F tn}, F ti is the i-th timing convolution spatio-temporal feature.

[0028] Step 2.6: the classifier is composed of a linear layer and a SoftMax function;

[0029] The timing feature F t is input into the classifier and processed by the linear layer to obtain a distribution feature F q , and then the probability distribution Pre is obtained by the SoftMax function.

[0030] Step 3: training of the network model:

[0031] The KL divergence is calculated according to the probability distribution Pre and the real word label, and is used as a loss function. The visual language recognition model is trained by a gradient descent method, and the loss function is calculated to update the model parameters until the model gradually converges, thereby obtaining a trained visual language recognition model for recognizing the lip speech words of any input face image.

[0032] The electronic device of the present application comprises a memory and a processor, characterized in that the memory is used to store a program supporting the processor to execute the visual language recognition method, and the processor is configured to execute the program stored in the memory.

[0033] The computer readable storage medium of the present application stores a computer program thereon, characterized in that the computer program is executed by the processor to perform the steps of the visual language recognition method.

[0034] Compared with the prior art, the present application has the beneficial effects that:

[0035] 1. The present application, by constructing a spatial harmonic module and a temporal harmonic module in the mixed convolution feature extraction stage, reasonably associates the global and local information of the lips from the two dimensions of time and space, extracts the spatio-temporal features that can balance the overall and local relationship in space and time, enhances the recognition of the features, and is conducive to improving the accuracy and robustness of the audio-visual network model.

[0036] 2. In the spatial harmonic module, the advantages of deformable convolution are effectively utilized, the correlation between the overall and local information in space is obtained through the traditional convolution method, and the Offset offset is converted into a way to guide the deformation of the convolution kernel, so that the spatial features with balanced overall and local information in space are extracted, the ability of spatial information extraction in the front-end feature extraction is effectively improved, and the accuracy of the audio-visual network model is improved in the scene where the speaker's posture changes frequently.

[0037] 3. In the temporal harmonic module, the short-time correlation of the time sequence information is effectively utilized, the correlation between the current time and the upper and lower times in time is obtained through the traditional convolution method, and the Offset offset is converted into a way to guide the deformation of the convolution kernel, so that the time features with balanced overall and local information in short time are extracted, the ability of time information extraction in the front-end feature extraction is effectively improved, and the audio-visual network model is adapted to the speaking speed of the speaker, and the recognition accuracy is increased. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the flow chart of the visual language recognition method in the present application;

[0039] Figure 2 is the structure diagram of the spatial harmonic module constructed in the present application;

[0040] Figure 3 is the schematic diagram of the temporal harmonic module constructed in the present application. DETAILED DESCRIPTION

[0041] In this embodiment, a visual language recognition method based on a spatio-temporal local harmonic neural network is a method for recognizing visual language through deep learning, which enhances the correlation between the local and overall visual features in space through a spatial local harmonic module, and enhances the short-time correlation of the visual features in time through a temporal local harmonic module, thereby providing a new solution for visual language recognition. The visual language recognition method based on the spatio-temporal local harmonic neural network specifically includes data set preprocessing, spatio-temporal harmonic module construction, visual language neural network construction, network model training, network model testing and application.

[0042] As Figure 1 shown, it is a visual language recognition method framework based on spatio-temporal local harmonic neural network. First, the network is inputted with data, and then the 3D feature extraction is performed in the front end, followed by spatial and temporal harmonics, and then 2D feature extraction is performed, completing the front-end extraction process of video features. Then the extracted features are inputted into the time series convolution network for time series modeling, and finally the results are outputted through the classifier. The specific steps are as follows:

[0043] Step 1: data preprocessing:

[0044] Step 1.1: obtain the visual language dataset with word labels, and perform grayscale processing on the video image data, then calibrate and align the face key points, so that the face is in the middle position of the image, thereby obtaining the face image sequence, denoted as P = {P1, P2,... P i ,... P n}, P i represents the i-th frame face image; n represents the total number of face images, in this example, the two commonly used data sets in the visual language recognition task, LRW and RLW-1000, are used. According to the video information of the data set, the input of the network is uniformly processed as 29 frames, that is, n is set to 29.

[0045] Step 1.2: crop the lip region of each frame of face image in the face image sequence P with a size of HxH, and randomly crop the cropped lip region, thereby obtaining a lip reading sequence with a size of WxW, denoted as P s1 = {P s1 , P s2 ,... P si ,... P sn}; wherein P si represents the i-th frame face image lip reading region, W < H; in this example, the size of H is set to 99 pixel points, in order to enhance the robustness of the model, the image is randomly cropped, and W is set to 88 pixel points according to the size of the lip region.

[0046] Step 2: construct a visual language recognition model based on spatio-temporal local harmonic neural network, as shown in Figure 1 , including: 3D spatio-temporal feature extraction module, spatial harmonic module, temporal harmonic module, 2D spatial feature extraction module, time series convolution module and a classifier;

[0047] Step 2.1: the 3D spatio-temporal feature extraction module is composed of N k 3D blocks connected in turn, and each 3D block is composed of a convolution layer, a BN layer and a ReLU activation function connected in turn; in this example, N k= 2, the first 3D block convolution kernel size is 5x7x7, the convolution step is 1x2x2, and the padding is zero; the second 3D block convolution kernel size is 3x3x3, the convolution step is 1x1x1, and the padding is zero;

[0048] Lip reading sequence P s1 The 3D spatio-temporal feature extraction module is input, and after being processed by N k blocks, in this example, N k = 2, the spatio-temporal feature F d = {F d1 , F d2 ,... F di ,... F dn} is obtained, where F di is the 3D spatio-temporal feature of the lip reading area of the i-th face image;

[0049] Step 2.2: The spatial harmonics module is composed of N l 2D spatial blocks and deformable convolution, wherein any l-th 2D spatial block is composed of a convolution layer with a convolution kernel of X l x X l , a BN layer and a ReLU activation function connected in sequence; N l is an odd number greater than or equal to 3; in this example, N l = 3, the first 2D block convolution kernel size is 5x5, the convolution step is 1, and the padding is zero; the second 2D block convolution kernel size is 3x3, the convolution step is 1, and the padding is zero; the third 2D block convolution kernel size is 3x3, the convolution step is 1, and the padding is zero; the specific structure is shown in Figure 2 .

[0050] Step 2.2.1: The 3D spatio-temporal feature Fd 输 is input into the spatial harmonics module, and after being processed in the first two 2D spatial blocks, the corresponding spatio-temporal features F 1c = {F 1c1 , F 1c2 ,... F 1ci ,... F 1cn} and F 2c = {F 2c1 , F 2c2 ,... F 2ci ,... F 2cn} are obtained, wherein F 1ci and F 2ci represent the 2D spatio-temporal features of the i-th 3D spatio-temporal feature after passing through the first and second 2D spatial blocks, respectively;

[0051] After connecting F 1c and F 2c in the time dimension, the spliced spatio-temporal feature Fc = {F c1 c2 ,... F ci ,... F cn}, F ci is the i-th spatio-temporal feature;

[0052] Step 2.2.2: if j = 1, then the spatio-temporal feature F c is input into the N l -th 2D spatial block for processing to obtain a spatial harmonized offset, denoted as O f = {O f1 , O f2 ,... O fi ,... O fn}, in this example, N l = 3, where O fi represents the i-th spatial harmonized offset; otherwise, the spatio-temporal feature F c is input into the subsequent 2D spatial block for processing, and the process continues in step 2.2.1;

[0053] The spatio-temporal feature F d is input into the deformable convolution DCN together with the offset O f , and a spatial harmonized spatio-temporal feature F o = {F o1 , F o2 ,... F oi ,.. F on} is obtained, where F oi is the i-th spatial harmonized spatio-temporal feature;

[0054] Step 2.3: the temporal harmonization module is composed of N z 2D temporal blocks and deformable convolution, where any z-th 2D temporal block is composed of a convolution layer with a convolution kernel of X z X z , a BN layer, and a ReLU activation function in series; in this example, N z = 3, the first 2D block has a convolution kernel size of 5x5, a convolution step size of 1, and padding of zero; the second 2D block has a convolution kernel size of 3x3, a convolution step size of 1, and padding of zero; the third 2D block has a convolution kernel size of 3x3, a convolution step size of 1, and padding of zero; the specific structure is shown in Figure 3 .

[0055] Step 2.3.1: the spatial harmonized spatio-temporal feature F o is input into the temporal harmonization module, and the i-th contextual spatio-temporal feature F ai is obtained using equation (1), thereby obtaining the contextual spatio-temporal feature F a ​={F a1 F a2 , ...F ai , ...F an}:

[0056]

[0057] In equation (1), F o，i-1 F represents the (i-1)th harmonic spatiotemporal feature; o，i+1 This represents the (i+1)th harmonic spatiotemporal feature;

[0058] Step 2.3.2: Connect the spatiotemporal context features F obtained after concatenation. a The first N in sequence z After processing one 2D time block, a cascaded time harmonic is obtained, which is then processed through the Nth time block. z After processing each 2D time block, the time-harmonic offset is obtained, denoted as S. f ={S f1 S f2 ,...S fi ,...S fn In this example, N z =3, where S fi Represents the offset of the i-th time harmonization;

[0059] Then the spatiotemporal features F o With offset S f The data are fed into a deformable convolutional neural network (DCN) for processing, and the temporally harmonized spatiotemporal features F are obtained. s ={F s1 F s2 , ...F si , ...F sn}, where F si The spatiotemporal features after harmonization at the i-th time;

[0060] Step 2.4: 2D spatial feature extraction consists of the first four residual blocks of ResNet-18, and each residual block includes two convolutional blocks, one BN layer and one ReLU activation function in sequence; each convolutional block is composed of two convolutional layers. In this example, in the first residual block, the kernel size of the convolutional layers in both convolutional blocks is 3×3, and the number of channels is 64. Only the stride of the first convolutional layer in the first convolutional block is 2, while the stride of the other convolutional layers is 1. In the second residual block, the kernel size of the convolutional layers in both convolutional blocks is 3×3, and the number of channels is 128. Only the stride of the first convolutional layer in the first convolutional block is 2, while the stride of the other convolutional layers is 1. In the third residual block, the kernel size of the convolutional layers in both convolutional blocks is 3×3, and the number of channels is 256. Only the stride of the first convolutional layer in the first convolutional block is 2, while the stride of the other convolutional layers is 1. In the fourth residual block, the kernel size of the convolutional layers in both convolutional blocks is 3×3, and the number of channels is 512. Only the stride of the first convolutional layer in the first convolutional block is 2, while the stride of the other convolutional layers is 1.

[0061] The spatiotemporal characteristics F of time harmony s The input is processed by the 2D spatial feature module and then by four layers of residual network blocks to obtain the residual spatiotemporal feature F. m ={F m1 F m2 , ...F mi , ...F mn}, F mi Let i be the spatiotemporal feature of the i-th residual.

[0062] Step 2.5: The temporal convolution module consists of N v The system consists of *v* residual blocks, where any *v*-th residual block contains two layers of temporal convolutional layers and a nonlinear mapping layer. Each temporal convolutional layer is connected to a BN layer and a ReLU activation function. The kernel size of the temporal convolutional layer in any *v*-th residual block is *X*. v ×X v In this example, N v =2, the temporal convolution kernel size in the first residual block is 3, the number of holes is 1, and the stride is 1; the temporal convolution kernel size in the second residual block is 5, the number of holes is 1, and the stride is 1.

[0063] Spatiotemporal characteristics F m The input is processed through the temporal convolution module and sequentially through N... v After processing the residual blocks, in this example, N v =2, thus obtaining the temporal convolutional spatiotemporal features F t ={F t1 F t2 , ...F ti , ...Ftn},F ti is the i-th time sequence convolutional feature;

[0064] Step 2.6: the classifier is composed of a linear layer and a SoftMax function;

[0065] time sequence feature F t is input into the classifier and obtained after a linear layer to obtain a distribution feature F q , and then a probability distribution Pre is obtained through a SoftMax function;

[0066] Step 3: training of the network model:

[0067] The KL divergence is calculated according to the probability distribution Pre and the real word label, and is used as a loss function. The visual language recognition model is trained through a gradient descent method, and the loss function is calculated to update the model parameters until the model gradually converges, thereby obtaining a trained visual language recognition model for recognizing the lip speech words of any input face image.

[0068] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above visual language recognition method, and the processor is configured to execute the program stored in the memory.

[0069] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above visual language recognition method.

Claims

1. A visual language recognition method based on a spatio-temporal local harmonic neural network, The features include the following steps: Step 1: data preprocessing: Step 1.1: Obtain a visual language dataset with word labels, and after converting the video image data to grayscale, label and align the facial key points to center the face in the image, thus obtaining a facial image sequence, denoted as . , Indicates the first A frame of face images; n represents the total number of face images; Step 1.2: cropping the mouth region of each frame of the face image sequence to obtain a lip reading sequence of size , denoted as ; wherein, the mouth region of each frame of the face image sequence is cropped to obtain a mouth region of size representing a first frame face image, a lip reading region, ; Step 2: constructing a visual language recognition model based on a spatiotemporal local harmonic neural network, comprising: a 3D spatiotemporal feature extraction module, a spatial harmonic module, a temporal harmonic module, a 2D spatial feature extraction module, a time sequence convolution module, and a classifier; Step 2.1: the 3D space-time feature extraction module is composed of three 3D blocks connected in sequence, and each 3D block is composed of a convolution layer, a BN layer and a ReLU activation function connected in sequence; The lip reading sequence The data is input into the 3D spatiotemporal feature extraction module and sequentially processed. After processing each block, the spatiotemporal features are obtained. , For the first 3D spatiotemporal features of the lip-reading region in an individual's facial image; Step 2.2: The spatial harmonics module consists of a 2D spatial block and a deformable convolution, wherein any i-th 2D spatial block consists of a convolution layer with a kernel of × , a BN layer, and a ReLU activation function connected in sequence; is an odd number greater than or equal to 3. Step 2.2.1: the 3D spatio-temporal features The input is input into the spatial harmonic module, and after processing in the first two 2D spatial blocks, the corresponding spatio-temporal features are obtained And Wherein, , Respectively represent the first 3D spatio-temporal features through the 2D spatio-temporal features after the first and second 2D spatial blocks Will , After connection in the time dimension, a spatio-temporal feature is obtained , The first spatio-temporal feature Step 2.2.2: If , the spatiotemporal feature is spliced; otherwise, the spatiotemporal feature is not spliced. Step 2.2.3: The spatially harmonized offset in the first 2D spatial block is obtained by processing, and is denoted as , where represents the first spatially harmonized offset; otherwise, the spatiotemporal feature continues the process of Step 2.2.1 to input the subsequent 2D spatial block for processing. spatiotemporal features with the offset are input into a deformable convolution DCN for processing and obtaining spatially harmonized spatiotemporal features wherein, is the spatially harmonized spatiotemporal feature Step 2.3: The time harmonization module consists of a 2D temporal block and a deformable convolution, wherein any i-th 2D temporal block consists of a convolution layer with a kernel of , a BN layer, and a ReLU activation function connected in sequence; Step 2.3.1: Spatially harmonized spatio-temporal features The time harmonized module is inputted with the time harmonized features and the equation (1) to obtain the first contextual spatio-temporal features , thus obtaining the contextual spatio-temporal features : (1) In formula (1), is the first harmonized spatiotemporal feature; is the second harmonized spatiotemporal feature; is the first harmonized spatiotemporal feature; is the second harmonized spatiotemporal feature; Step 2.3.2: Connect the spatiotemporal features of the context obtained after concatenation. in sequence After processing one 2D time block, a cascaded time harmonic is obtained, which is then processed by the... After processing each 2D time block, the time-harmonic offset is obtained, denoted as . ,in, Representing the An offset that is reconciled over time; spatiotemporal features with the offset are input into a deformable convolution DCN for processing and obtain time harmonics spatiotemporal features wherein, is the th time harmonics spatiotemporal feature Step 2.4: the 2D spatial feature extraction is composed of the first four residual blocks of ResNet-18, and each residual block sequentially includes two convolution layers, a BN layer and a ReLU activation function; The time-harmonized spatiotemporal features The 2D spatial feature extraction module is inputted and processed by four layers of residual network blocks in sequence to obtain residual spatiotemporal features , is the first residual spatiotemporal feature Step 2.5: The temporal convolution module is composed of... It consists of n residual blocks, where any nth residual block... Each residual block contains two layers of temporal convolutional layers and a nonlinear mapping layer. Each temporal convolutional layer is connected to a BN layer and a ReLU activation function; where, any _th _ The kernel size of the temporal convolutional layer in each residual block is ; The spatio-temporal features The spatio-temporal features are input into the time sequence convolution module and sequentially pass through After the processing of the residual block, the spatio-temporal features of the time sequence convolution are obtained The spatio-temporal features of the first time sequence convolution are obtained The spatio-temporal features of the first time sequence convolution​ Step 2.6: the classifier is composed of a linear layer and a SoftMax function; The temporal convolution spatio-temporal features The classifier is inputted and distribution features are obtained after the linear layer The probability distribution is obtained through the SoftMax function ; Step 3: training of the network model: According to the probability distribution The KL divergence is calculated with the real word label and taken as a loss function, the visual language recognition model is trained by a gradient descent method, and the loss function is calculated to update the model parameters until the model gradually converges, thereby obtaining a trained visual language recognition model for recognizing the lip speech words of any input face image.

2. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the visual language recognition method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the visual language recognition method of claim 1.

Images