Face recognition method, device and storage medium based on lip reading password
Through the combination of dlib+Resnet and two-way LSTM models, facial recognition based on lip passwords is realized, solving the problems of forgetting and fingerprint facial recognition copy attacks in traditional password recognition methods, and providing low-cost and high-accuracy security Identify the solution.
Patent Information
- Application Number
- CN201910885930.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-09-19
AI Technical Summary
Traditional password recognition methods are easy to forget and cumbersome to operate. Fingerprint and face recognition have the risk of copy attacks and the equipment costs are high.
The face recognition method based on lip password is adopted, and the lip image is filtered through the dlib+Resnet face key point detection model, and the lip password is recognized using a two-way LSTM model to realize the recognition technology of lip language as a password.
It realizes low-cost and high-accuracy security identification, avoiding the risks of traditional password memory and copy attacks.
Smart Images

Figure CN110717407B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of biometric identification technology, and in particular to a face recognition method, device and storage medium based on lip reading password. Background Art
[0002] In traditional password recognition methods, the password used is a combination of numbers, characters or other symbols. The user must accurately remember the password used and enter it correctly in order to be successfully recognized. Therefore, traditional password recognition methods have disadvantages such as easy forgetting, misremembering, and cumbersome operation.
[0003] The fingerprint recognition, face recognition, and iris recognition methods that have become popular in recent years can overcome the shortcomings of the above-mentioned traditional passwords that need to be remembered and entered one by one, but they also have the following defects: 1. Simple fingerprint recognition and face recognition face the risk of attacks by copied fingerprints and photos. Copied fingerprints and static photos can deceive fingerprint recognition and face recognition; 2. Iris recognition is more secure, but it has the problem of expensive equipment and high economic cost.
[0004] In view of the existence of the above problems, there is an urgent need for a safe identification method with low detection cost and no loss of detection accuracy. Summary of the invention
[0005] The present invention provides a face recognition method based on lip reading password, an electronic device and a computer-readable storage medium, which mainly screen lip reading images through a dlib+Resnet face key point detection model; classify lip reading images through a lip reading recognition model based on a bidirectional LSTM, and achieve the technical effect of using lip reading as a password for face recognition.
[0006] To achieve the above-mentioned purpose, the present invention provides a face recognition method based on lip reading password, which is applied to electronic devices, and the method comprises: S110, obtaining a password reading video of a subject to be tested; S120, detecting the video frame by frame through a face detection model based on Resnet, and obtaining lip reading images of continuous frames in a time period from the starting moment to the end moment of the subject to be tested reading the password in the video; S130, determining a lip reading feature set in the time period according to the lip reading images of continuous frames in the time period; S140, inputting the lip reading feature set into a trained lip reading recognition model based on bidirectional LSTM to obtain a lip reading password prediction value; S150, if the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested has passed the lip reading password recognition.
[0007] Preferably, the method for constructing the bidirectional LSTM-based lip reading recognition model includes:
[0008] S210, constructing an initial network layer for obtaining the lip features of the subject to be tested; the initial network layer is a 2D convolutional network; S220, constructing a bidirectional LSTM layer on the initial network layer for extracting time features in the training set data; S230, constructing a Softmax layer on the bidirectional LSTM layer for outputting lip reading password prediction values; S240, constructing an optimized network layer on the Softmax layer; wherein the optimized network layer is used to input the lip reading password prediction values into a loss function for iterative training until the value of the loss function reaches a set threshold.
[0009] Preferably, step S140 includes, in the Softmax layer, calculating the time feature data extracted by the bidirectional LSTM layer as a lip reading password prediction value through a lip reading password prediction value formula, and the lip reading password prediction value formula is as follows: P=W*X+b; wherein P is the lip reading password prediction value, X is the time feature data, W is the weight, and b is the offset.
[0010] Preferably, the step S130 includes: S310, normalizing the lip reading image; S320, storing the normalized lip reading image as a data set sample in a specified format; the data set sample includes: number of samples, data sequence length, image length, image width, and image depth.
[0011] Preferably, in step S120, the lip reading image is parsed by a feature point model of a dlib database to obtain lip feature information; the time period from the starting moment to the end moment of the subject to be tested reading the password in the video is obtained by parsing the sound waveform of the password reading video.
[0012] To achieve the above-mentioned purpose, the present invention also provides an electronic device, which includes: a memory and a processor, wherein the memory stores a face recognition program based on a lip reading password, and when the face recognition program based on a lip reading password is executed by the processor, the following steps are implemented: S110, obtaining a password reading video of the subject to be tested; S120, performing frame-by-frame detection on the video through a face detection model based on Resnet, and obtaining lip reading images of continuous frames in a time period from the starting moment to the end moment of the subject to be tested reading the password in the video; S130, determining a lip reading feature set in the time period based on the lip reading images of continuous frames in the time period; S140, inputting the lip reading feature set into a trained lip reading recognition model based on bidirectional LSTM to obtain a lip reading password prediction value; S150, if the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested has passed the lip reading password recognition.
[0013] Preferably, the method for constructing the bidirectional LSTM-based lip reading recognition model includes:
[0014] S210, constructing an initial network layer for obtaining the lip features of the subject to be tested; the initial network layer is a 2D convolutional network; S220, constructing a bidirectional LSTM layer on the initial network layer for extracting time features in the training set data; S230, constructing a Softmax layer on the bidirectional LSTM layer for outputting lip reading password prediction values; S240, constructing an optimized network layer on the Softmax layer; wherein the optimized network layer is used to input the lip reading password prediction values into the loss function for iterative training until the value of the loss function reaches a set threshold.
[0015] Preferably, step S140 includes: in the Softmax layer, calculating the time feature data extracted by the bidirectional LSTM layer as a lip reading password prediction value through a lip reading password prediction value formula, and the lip reading password prediction value formula is as follows: P=W*X+b; wherein P is the lip reading password prediction value, X is the time feature data, W is the weight, and b is the offset.
[0016] Preferably, in step S120, the lip reading image is parsed by a feature point model of a dlib database to obtain lip feature information; the time period from the starting moment to the end moment of the subject to be tested reading the password in the video is obtained by parsing the sound waveform of the password reading video.
[0017] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program includes a face recognition program based on lip reading password. When the face recognition program based on lip reading password is executed by a processor, the steps of the above-mentioned face recognition method based on lip reading password are implemented.
[0018] The face recognition method based on lip reading password, electronic device and computer-readable storage medium proposed in the present invention classify lip reading images and train a model by adopting a 2D convolution model + a bidirectional LSTM model + a Softmax layer + an optimized network layer; wherein the 2D convolution model is used to extract the lip features of a person; the bidirectional LSTM model is used to connect multiple lip pictures in a video in series to extract time features; the Softmax layer is used to output a lip reading password prediction value; the optimized network layer is used to input the lip reading password prediction value into a loss function for iterative training until the value of the loss function reaches a set threshold; finally, a face recognition model based on lip reading password is obtained; the face recognition model based on lip reading password is used to perform lip reading password recognition, which can achieve a technical effect of low cost and high recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flow chart of a preferred embodiment of the face recognition method based on lip reading password of the present invention;
[0020] Figure 2 It is a flowchart of a preferred embodiment of a method for constructing a lip reading recognition model based on a bidirectional LSTM of the present invention;
[0021] Figure 3 The schematic diagram of the principle of the face recognition method based on lip reading password of the present invention;
[0022] Figure 4 FIG. 4 is a schematic structural diagram of a preferred embodiment of the electronic device of the present invention.
[0023] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0024] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0025] The invention provides a face recognition method based on lip reading password. Figure 1 The flowchart of a preferred embodiment of the face recognition method based on lip reading password according to an embodiment of the present invention is shown. The method can be executed by a device, and the device can be implemented by software and / or hardware.
[0026] This application uses the dlib+Resnet facial key point detection model to screen lip reading images; and uses a bidirectional LSTM model to classify lip reading images, thereby realizing face recognition using lip reading as a password.
[0027] It should be noted that Resnet (Residual Network) is a new type of network structure with powerful functions. The Resnet network can be regarded as a combination of multiple parallel and serial modules. Resnet is a relatively good network for classification problems in the ImageNet competition. It has multiple structural forms, including Resnet-34, Resnet-50, Resnet-101, and Resnet-152. The design of the Resnet network structure follows two design rules: (1) For the same output feature map size, the layers have the same number of filters; (2) If the feature map size is halved, the number of filters is doubled to maintain the time complexity of each layer. The first is the skip connection method, and the second is the use of the Batch Normalization layer. The face detection algorithm based on Resnet needs to slide windows of different sizes and positions in the image, and then determine whether there is a face in the window.
[0028] In addition, dlib uses the HOG (histogram of oriented gradient) + regression tree method. The detection effect is much better when using the model trained by dlib. dlib also uses convolutional neural networks for face detection.
[0029] The face recognition method based on lip reading password of the present invention uses the interface of the Resnet model that has been trained in dlib, and this interface returns a 128-dimensional face feature vector.
[0030] like Figure 1 As shown, in this embodiment, the face recognition method based on lip reading password includes steps S110 to S150.
[0031] S110, obtaining a password reading video of the subject to be tested.
[0032] The subject to be tested is the user who wants to perform a password test. The user who needs to perform a password test needs to read out a pre-set voice password to pass the password test. Specifically, the detection device needs to obtain a video of the subject to be tested reading the password.
[0033] In a specific embodiment, the user randomly selects 4-6 digits as the password, faces the camera, and reads the above digits at a constant speed, about one digit per second. The reading can be in Mandarin, dialect, English, etc. When collecting information, it can be repeated 4-10 times. The user must remember the 4-6 digits set by himself as the password. The next time he enters, he can read the above password to complete the recognition.
[0034] S120, performing frame-by-frame detection on the video using a Resnet-based face detection model to obtain lip reading images of consecutive frames in a time period from the starting moment to the end moment when the subject to be tested reads the password in the video.
[0035] In a specific embodiment, the lip reading image is parsed by a feature point model of a dlib database to obtain lip feature information; and the time period from the starting moment to the end moment of the subject to be tested reading the password in the video is obtained by parsing the sound waveform of the password reading video.
[0036] Specifically, the video is detected frame by frame through a Resnet-based face detection model, the lip position is determined using a feature point model from the dlib database, and several frames of lip images are cut out. The sound start time and sound end time in the digital password video are obtained by analyzing the sound waveform, and the lip reading images of consecutive frames are screened out based on the above start time and end time.
[0037] In a specific embodiment, the face detection model detects faces frame by frame in the video, and then confirms the position of lips based on the feature point model of dlib, so as to cut out the lip images in the above video. An exemplary explanation is as follows: for example, if the user reads 1234 in the video, the sample is a number of frames of lip images cut out, and the label is the pinyin corresponding to 1234: yi er san si, and different pinyins are separated by spaces.
[0038] For example, if the number 6 is read out in the user video, the sample will be cut out into several frames of lip pictures, and the label is the pinyin corresponding to 6: liu; the starting point and key time of the sound of each number are obtained according to the sound waveform, and the lip reading recognition model based on the bidirectional LSTM determines the image of a section of continuous frames of the face in the video according to the starting and ending time, so as to recognize the number through lip movement, and compare the recognized number with the password pre-stored in the background. If they are consistent, it passes.
[0039] S130: Determine a lip reading feature set within the time period according to the lip reading images of consecutive frames within the time period.
[0040] The method for preprocessing the lip reading image in step S130 includes: S310, normalizing the lip reading image; S320, storing the normalized lip reading image as a data set sample in a specified format; the storage format of the data set sample is [number of samples, data sequence length, image length, image width, image depth].
[0041] S140, inputting the lip reading feature set into a trained bidirectional LSTM-based lip reading recognition model to obtain a lip reading password prediction value.
[0042] S150: If the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested has passed the lip reading password recognition.
[0043] In summary, the use of a face recognition model based on lip reading passwords for lip reading password recognition can achieve a technical effect of low cost and high recognition accuracy.
[0044] Figure 2 FIG. 4 is a flowchart of a preferred embodiment of a method for constructing a lip reading recognition model based on a bidirectional LSTM according to an embodiment of the present invention. Figure 2 As shown, the method for constructing a lip reading recognition model based on a bidirectional LSTM includes: steps S210 to S240.
[0045] S210, constructing an initial network layer for acquiring lip features of the subject to be tested; the initial network layer is a 2D convolutional network.
[0046] Specifically, a masking layer is first established to filter out data padded to zero in the input sample; then, the number of convolution kernels is designed to construct the initial network layer of the 2D convolution network, and the 2D convolution network of the initial network layer is used to extract the feature value of each frame in the input data; the convolution kernel is set according to the sample lip size in the training set data. Each frame of the input training set sample is multiplied and summed with the convolution kernel in the network, and after three layers of convolution, the feature value is obtained.
[0047] S220. Construct a bidirectional LSTM layer on the initial network layer for extracting time features in the training set data.
[0048] Mouth sound production is a dynamic process, and we need to look at the previous and next frames of the current frame to more accurately obtain the hidden layer information of the current frame. In other words, the bidirectional LSTM layer considers the impact of the "past" and "future" on the current frame.
[0049] Among them, the bidirectional LSTM layer has a step size equal to the number of lip images extracted from the video.
[0050] S230: construct a Softmax layer on the bidirectional LSTM layer for outputting lip reading password prediction values.
[0051] Specifically, the Softmax layer is used to classify a lip movement image into 0-9, a total of 10 categories. It should be noted that the output layer using Softmax has multiple units. In fact, there are as many units as there are categories. In this example, we have 10 categories, so there are 10 neural units, which represent these 10 categories. Under the action of Softmax, each neural unit will calculate the probability that the current sample belongs to this category.
[0052] The time value X obtained by the neural network is passed through the Softmax layer to obtain the lip reading password prediction value of the sample; in other words, the Softmax layer calculates the time feature data extracted by the bidirectional LSTM layer as the lip reading password prediction value through the formula, and the formula is as follows: P=W*X+b; wherein P is the lip reading password prediction value, X is the time feature data, W is the weight, and b is the offset.
[0053] In general, the neural network of the lip reading recognition model includes: three layers of 2D convolutional networks, two layers of bidirectional LSTM networks, and a fully connected layer with the activation function of Softmax, and finally a logical prediction layer (i.e., the optimized network layer).
[0054] S240, constructing an optimized network layer on the Softmax layer; wherein the optimized network layer is used to input the lip reading password prediction value into a loss function for iterative training until the value of the loss function reaches a set threshold.
[0055] The training process of the neural network is to input the training set data into the neural network, and after layer-by-layer calculation, the lip reading password prediction value is obtained, and the lip reading password prediction value and the real label are input into the loss function to calculate the loss, and the model parameters are corrected through back propagation. In other words, the above neural network model is iteratively trained using the loss function until the loss reaches the set threshold.
[0056] In general, the lip reading recognition model based on bidirectional LSTM of the present invention is a bidirectional recurrent neural network, in which the forward layer and the backward layer are connected to the output layer together, which contains 6 shared weights w1-w6. The connection weight w between neurons and the bias b of each neuron itself. The parameter adjustment method is to use the gradient descent algorithm to adjust the parameter size along the gradient direction. The larger the gradient of the activation function, the faster the size of w and b is adjusted, and the faster the training converges. The activation function commonly used in neural networks is the sigmoid function.
[0057] The forward layer performs forward calculations from time 1 to time t, and obtains and saves the output of the forward hidden layer at each time. The backward layer performs reverse calculations from time t to time 1, and obtains and saves the output of the backward hidden layer at each time. Finally, at each time, the output results of the corresponding time of the forward layer and the backward layer are combined to obtain the final output.
[0058] In summary, the bidirectional LSTM network performs better in the time series classification task, and uses the historical and future information of the time series, combined with the contextual information, to make a comprehensive judgment, and the judgment result is more accurate.
[0059] Figure 3 FIG. 1 is a schematic diagram showing the principle of a face recognition method based on lip reading password according to an embodiment of the present invention. Figure 3 As shown, first, continuous frames of lip reading images are obtained through a video of the subject to be tested reading the password, and then the obtained lip reading images are normalized, and the normalized lip reading images are stored as data set samples in a specified format; the specified format is the storage format of the data set samples, and in a specific embodiment of the present invention, the specified storage format is [number of samples, data sequence length, image length, image width, image depth].
[0060] The above data set is divided into two parts in a ratio of eight to two, one part is used as a training set and the other part is used as a test set; wherein, the training set is used to train a lip reading recognition model based on a bidirectional LSTM. The steps of constructing a lip reading recognition model based on a bidirectional LSTM include: first designing the number of convolution kernels, constructing a 2D convolution network, then designing the number of hidden layer cells and the step size, and constructing a bidirectional LSTM network; then constructing a Softmax layer, which is used to classify the temporal features of the password reading video of the subject to be tested using the Softmax function, and finally selecting the node with the largest probability (that is, the largest corresponding value) as the lip reading password prediction value output. In the last step, the loss function is used to optimize the network, and finally the bidirectional LSTM-based lip reading recognition model with the best recognition effect is obtained through iterative training of the loss function.
[0061] The samples in the test set are used to test the bidirectional LSTM-based lip reading recognition model. After the test, the bidirectional LSTM-based lip reading recognition model can be used to predict lip reading videos.
[0062] The present invention provides a face recognition method based on lip reading code, which is applied to an electronic device 4. Figure 4 A schematic diagram of the application environment of a preferred embodiment of the face recognition method based on lip reading password according to the present invention is shown.
[0063] In this embodiment, the electronic device 4 may be a terminal device with computing functions, such as a server, a smart phone, a tablet computer, a portable computer, or a desktop computer.
[0064] like Figure 4 As shown, the electronic device 4 includes a processor 42 , a memory 41 , a communication bus 43 and a network interface 44 .
[0065] The memory 41 includes at least one type of readable storage medium. The at least one type of readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory 41, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 4, such as a hard disk of the electronic device 4. In other embodiments, the readable storage medium may also be an external memory 41 of the electronic device 4, such as a plug-in hard disk equipped on the electronic device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc.
[0066] In this embodiment, the readable storage medium of the memory 41 is generally used to store the face recognition program 40 based on lip reading password installed in the electronic device 4. The memory 41 can also be used to temporarily store data that has been output or is to be output.
[0067] In some embodiments, the processor 42 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 41, such as executing a face recognition program 40 based on lip reading password.
[0068] The communication bus 43 is used to realize the connection and communication between these components.
[0069] The network interface 44 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the electronic device 4 and other electronic devices.
[0070] Figure 4 Only the electronic device 4 is shown with components 41 - 44 , but it should be understood that implementation of all of the components shown is not required, and more or fewer components may alternatively be implemented.
[0071] Optionally, the electronic device 4 may also include a user interface, which may include an input unit such as a keyboard, a voice input device such as a microphone and other devices with voice recognition function, a voice output device such as a speaker, headphones, etc. Optionally, the user interface may also include a standard wired interface or a wireless interface.
[0072] Optionally, the electronic device 4 may further include a display, which may also be referred to as a display screen or a display unit. In some embodiments, it may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (OLED) touch device. The display is used to display information processed in the electronic device 4 and to display a visual user interface.
[0073] Optionally, the electronic device 4 may also include a radio frequency (RF) circuit, a sensor, an audio circuit, etc., which will not be described in detail here.
[0074] exist Figure 4In the device embodiment shown, the memory 41 as a computer storage medium can store an operating system and a face recognition program 40 based on lip reading password; when the processor 42 executes the face recognition program 40 based on lip reading password stored in the memory 41, the following steps are implemented: S110, obtaining a password reading video of the subject to be tested; S120, performing frame-by-frame detection on the obtained password reading video through a Resnet-based face detection model, and obtaining lip reading images of continuous frames in the time period from the starting moment to the end moment of the password reading of the subject to be tested in the obtained video; S130, determining the lip reading feature set within the time period based on the lip reading images of continuous frames within the time period; S140, inputting the lip reading feature set into a trained lip reading recognition model based on bidirectional LSTM to obtain a lip reading password prediction value; S150, if the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested has passed the lip reading password recognition.
[0075] In other embodiments, the face recognition program 40 based on lip reading password can also be divided into one or more modules, one or more modules are stored in the memory 41 and executed by the processor 42 to complete the present invention. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions.
[0076] The face recognition program 40 based on lip reading password can be divided into: a password video acquisition unit, a lip reading feature set acquisition unit, a lip reading recognition model training unit based on bidirectional LSTM, and a lip reading recognition model testing unit based on bidirectional LSTM; wherein the password video acquisition unit is used to obtain the password reading video of the subject to be tested; the lip reading feature set acquisition unit is used to detect the video frame by frame through a face detection model based on Resnet, and obtain the lip reading images of continuous frames in the time period from the starting moment to the end moment of the subject to be tested reading the password in the video; according to the lip reading images of continuous frames in the time period, determine the lip reading feature set in the time period; the lip reading recognition model training unit based on bidirectional LSTM is a neural network model trained using the lip reading feature set arranged in time series; the lip reading recognition model testing unit based on bidirectional LSTM is a lip reading recognition model based on bidirectional LSTM that is trained using the lip reading feature set arranged in time series. The functions or operation steps implemented are similar to those above and will not be described in detail here.
[0077] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, wherein the computer-readable storage medium includes a face recognition program based on a lip reading password, and the face recognition program based on a lip reading password implements the following operations when executed by a processor: S110, obtaining a password reading video of the subject to be tested; S120, performing frame-by-frame detection on the video through a face detection model based on Resnet, and obtaining lip reading images of continuous frames in a time period from the starting moment to the end moment of the subject to be tested reading the password in the video; S130, determining a lip reading feature set within the time period based on the lip reading images of continuous frames within the time period; S140, inputting the lip reading feature set into a trained lip reading recognition model based on bidirectional LSTM to obtain a lip reading password prediction value; S150, if the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested has passed the lip reading password recognition.
[0078] The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the above-mentioned face recognition method and electronic device based on lip reading password, and will not be described in detail here.
[0079] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.
[0080] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. Through the description of the above implementation modes, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation mode. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0081] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A face recognition method based on lip reading password, applied to electronic devices, characterized in that: The method comprises: S110, obtaining a password reading video of the subject to be tested; S120, performing frame-by-frame detection on the video using a Resnet-based face detection model to obtain lip reading images of consecutive frames in a time period from the start time to the end time of the subject to be tested reading the password in the video; S130, determining a lip reading feature set within the time period according to the lip reading images of the continuous frames within the time period; S140, inputting the lip reading feature set into a trained bidirectional LSTM-based lip reading recognition model to obtain a lip reading password prediction value; S150: If the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested passes the lip reading password recognition; The method for constructing the lip reading recognition model based on bidirectional LSTM includes: S210, constructing an initial network layer for obtaining lip features of the subject to be tested; the initial network layer is a 2D convolutional network; wherein the convolution kernel is set according to the sample lip size in the training set data; S220, constructing a bidirectional LSTM layer for extracting temporal features in the training set data on the initial network layer; wherein the step size is the number of lip images extracted from the video; S230, constructing a Softmax layer for outputting a lip reading password prediction value on the bidirectional LSTM layer; S240, constructing an optimized network layer on the Softmax layer; wherein the optimized network layer is used to input the lip reading password prediction value into a loss function for iterative training until the value of the loss function reaches a set threshold.
2. The face recognition method based on lip reading password according to claim 1 is characterized in that: Step S140 includes: In the Softmax layer, the time feature data extracted by the bidirectional LSTM layer is calculated as the lip reading password prediction value by the lip reading password prediction value formula, and the lip reading password prediction value formula is as follows: P = W*X+b; Among them, P is the lip reading password prediction value, X is the time feature data, W is the weight, and b is the offset.
3. The face recognition method based on lip reading password according to claim 1, characterized in that: The step S130 of determining the lip reading feature set within the time period includes: S310, normalizing the lip reading image; S320, storing the normalized lip reading image as a data set sample in a specified format; the data set sample includes: sample number, data sequence length, image length, image width, and image depth.
4. The face recognition method based on lip reading password according to claim 1, characterized in that: In the step S120, the lip reading image is parsed by the feature point model of the dlib database to obtain lip feature information; the time period from the starting moment to the end moment of the subject to be tested reading the password in the video is obtained by parsing the sound waveform of the password reading video.
5. An electronic device, characterized in that: The electronic device comprises: a memory and a processor, wherein a face recognition program based on a lip reading password is stored in the memory, and when the face recognition program based on a lip reading password is executed by the processor, the following steps are implemented: S110, obtaining a password reading video of the subject to be tested; S120, performing frame-by-frame detection on the video using a Resnet-based face detection model to obtain lip reading images of consecutive frames in a time period from the start time to the end time of the subject to be tested reading the password in the video; S130, determining a lip reading feature set within the time period according to the lip reading images of the continuous frames within the time period; S140, inputting the lip reading feature set into a trained bidirectional LSTM-based lip reading recognition model to obtain a lip reading password prediction value; S150: If the lip reading password prediction value is consistent with the password stored in the lip reading recognition model, it is confirmed that the subject to be tested passes the lip reading password recognition; The method for constructing the lip reading recognition model based on bidirectional LSTM includes: S210, constructing an initial network layer for obtaining lip features of the subject to be tested; the initial network layer is a 2D convolutional network; wherein the convolution kernel is set according to the sample lip size in the training set data; S220, constructing a bidirectional LSTM layer for extracting temporal features in the training set data on the initial network layer; wherein the step size is the number of lip images extracted from the video; S230, constructing a Softmax layer for outputting a lip reading password prediction value on the bidirectional LSTM layer; S240, constructing an optimized network layer on the Softmax layer; wherein the optimized network layer is used to input the lip reading password prediction value into a loss function for iterative training until the value of the loss function reaches a set threshold.
6. The electronic device according to claim 5, characterized in that: Step S140 includes: in the Softmax layer, calculating the time feature data extracted by the bidirectional LSTM layer as a lip reading password prediction value through a lip reading password prediction formula, and the lip reading password prediction value formula is as follows: P=W*X+b; Among them, P is the lip reading password prediction value, X is the time feature data, W is the weight, and b is the offset.
7. The electronic device according to claim 5, characterized in that: In the step S120, the lip reading image is parsed by the feature point model of the dlib database to obtain lip feature information; the time period from the starting moment to the end moment of the subject to be tested reading the password in the video is obtained by parsing the sound waveform of the password reading video.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes a face recognition program based on lip reading password. When the face recognition program based on lip reading password is executed by a processor, the steps of the face recognition method based on lip reading password as described in any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
Lip language unlocking realization method
CN107122646A
Identity authentication method, device, and equipment, and computer readable storage medium
CN107977559A
A lip recognition method and system based on neural network
CN109409195A
Identity authentication method, terminal device, and computer-readable storage medium
CN109726624A
