Unlocking method and apparatus based on facial expression, and computer device and storage medium

Abstract

This disclosure relates to a facial expression-based unlocking method and apparatus, a computer device, and a storage medium. The method includes: displaying an expression unlock page (S202); displaying an unlock node sequence on the expression unlock page (S204); creating, at a to-be-processed unlock node in the unlock node sequence, an unlock status identification based on a facial expression in a facial image captured in real time (S206); and completing unlocking based on the unlock status identification and match between the facial expression in the facial image and a corresponding target expression (S208).

Description

Unlocking method, device, computer equipment and storage medium based on facial expression

[0001] This application claims priority to Chinese patent application number 2020109161381, filed with the Patent Office of China on September 3, 2020, entitled “Unlocking method, device, computer device and storage medium based on facial expression”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a facial expression-based unlocking method, apparatus, computer device, and storage medium. Background Art

[0003] With the popularization of smart devices and the continuous development of information security, users have higher and higher requirements for information security. Therefore, the application of security locks has become more and more popular among users, such as device power-on password locks, application login security locks, and payment password locks, etc.

[0004] Common security lock unlocking methods typically require entering a password, entering a fingerprint, or performing facial recognition. However, facial recognition allows third parties to unlock the device using photos or facial models, posing a security risk.

[0005] Summary of the Invention

[0006] According to various embodiments of the present application, a facial expression-based unlocking method, apparatus, computer device, and storage medium are provided.

[0007] A facial expression-based unlocking method, executed by a terminal, comprising:

[0008] Display the emoticon unlock page;

[0009] Display the unlock node sequence on the expression unlock page;

[0010] At an unlocking node to be processed in the unlocking node sequence, generating an unlocking state identifier based on a facial expression in a facial image collected in real time;

[0011] Unlocking is completed based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

[0012] A facial expression-based unlocking device, comprising:

[0013] Display module, used to display the expression unlocking page;

[0014] A first display module, configured to display an unlocking node sequence on the expression unlocking page;

[0015] a generating module for generating an unlocking state identifier based on a facial expression in a facial image collected in real time at an unlocking node to be processed in the unlocking node sequence;

[0016] The unlocking module is used to complete the unlocking based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

[0017] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Display the emoticon unlock page;

[0019] Display the unlock node sequence on the expression unlock page;

[0020] At an unlocking node to be processed in the unlocking node sequence, generating an unlocking state identifier based on a facial expression in a facial image collected in real time;

[0021] Unlocking is completed based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0023] Display the emoticon unlock page;

[0024] Display the unlock node sequence on the expression unlock page;

[0025] At an unlocking node to be processed in the unlocking node sequence, generating an unlocking state identifier based on a facial expression in a facial image collected in real time;

[0026] Unlocking is completed based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

[0027] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned facial expression-based unlocking method.

[0028] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a diagram illustrating an application environment of an unlocking method based on facial expression according to an embodiment;

[0030] FIG2 is a schematic diagram of a flow chart of a facial expression-based unlocking method according to an embodiment;

[0031] FIG3 is a schematic diagram of an expression unlocking page in one embodiment;

[0032] FIG4 is a schematic diagram of entering the expression unlocking page by triggering the face detection control in one embodiment;

[0033] FIG5 is a schematic diagram of prompting adjustment of the acquisition orientation in one embodiment;

[0034] FIG6 is a schematic diagram of recognizing a facial image and superimposing an expression model obtained by recognition on the facial expression in one embodiment;

[0035] FIG7 is a schematic diagram of facial feature points in one embodiment;

[0036] FIG8 is a schematic diagram of a flowchart of unlocking steps using facial expressions and faces in one embodiment;

[0037] FIG9 is a schematic diagram of a flowchart of unlocking steps using expressions and gestures in accordance with an embodiment;

[0038] FIG10 is a schematic diagram of a process for unlocking each unlocking node using at least two facial expressions in one embodiment;

[0039] FIG11 is a schematic diagram of a process for entering an expression image in one embodiment;

[0040] FIG12 is a schematic diagram of entering an expression entry page through a face entry control in one embodiment;

[0041] FIG13 is a schematic diagram of entering an expression image after expression recognition in one embodiment;

[0042] FIG14 is a schematic diagram of combining and sorting emoticon identifiers in one embodiment;

[0043] FIG15 is a schematic diagram of expression combination and expression recognition in one embodiment;

[0044] FIG16 is a flow chart of an unlocking method based on facial expression in another embodiment;

[0045] FIG17 is a block diagram of a facial expression-based unlocking device according to one embodiment;

[0046] FIG18 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] The facial expression-based unlocking method provided in this application can be applied to the application environment shown in Figure 1. In this application environment, a terminal 102 and a server 104 are included. The terminal 102 can capture the facial image of the object to be tested in real time through a built-in camera or an external camera, and then recognize the facial expression of the facial image. When the recognition is completed, an unlocking state mark is generated at the corresponding unlocking node to indicate the unlocking state of the unlocking node. When an unlocking state mark is generated at each unlocking node in the unlocking node sequence and the facial expression in each facial image matches the corresponding target expression, the entire unlocking node sequence is unlocked successfully.

[0049] Among them, the object to be tested can refer to the user to be tested or other objects to be tested (such as animals). In the subsequent embodiments, the object to be tested is taken as an example of a user to be tested. The unlocking node sequence can be regarded as a security lock with a multi-digit password. Each unlocking node corresponds to a password, and the password is decoded by facial expression. The unlocking state includes: a state in which the unlocking operation has not been performed and a state in which the unlocking operation has been performed. The state in which the unlocking operation has been performed includes: a state in which the unlocking operation has been performed and the unlocking node has been successfully unlocked, and a state in which the unlocking operation has been performed but the unlocking node has not been successfully unlocked.

[0050] In addition, the terminal 102 can also pre-record the target object's facial expression. Different unlocking nodes can record different facial expressions (i.e., corresponding to different target expressions), or two or three different unlocking nodes can record the same facial expression (i.e., two or three different unlocking nodes correspond to the same target expression). Each unlocking node corresponds to a password, and the unlocking node is unlocked (i.e., decoded) by the corresponding facial expression. The target object's facial expression pre-recorded by the terminal 102 can be saved locally or saved on the server 104. During the unlocking process, the facial expression recognized from the facial image of the object to be tested is compared with the saved facial expression by the server 104 to determine whether the facial expression of the object to be tested is consistent with the target expression of the target object.

[0051] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc.; in addition, it may also be an access control device, a gate, etc., but is not limited thereto.

[0052] The server 104 may be an independent physical server or a server cluster composed of multiple physical servers. It may be a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud storage, and content delivery networks (CDN).

[0053] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus), or a network, and this application does not impose any limitation thereto.

[0054] In one embodiment, as shown in FIG2 , a facial expression-based unlocking method is provided. The method is described by applying it to the terminal 102 in FIG1 as an example, and includes the following steps:

[0055] S202: Display the expression unlocking page.

[0056] The expression unlocking page can refer to a page that uses facial expressions to unlock, or a page that verifies the facial expression of the subject being tested. For example, the expression unlocking page is used to unlock the terminal to enter the user interface, or to enter the fund management page of an application or other pages with private information (such as the chat history page in a social application), or the expression unlocking page is used to swipe facial expressions to make payments in payment scenarios.

[0057] For example, as shown in Figure 2, the expression unlocking page includes a face preview area, an unlocking progress area, and an expression prompt area. The face preview area is used to display a real-time facial image; the unlocking progress area is used to display the unlocking node sequence. When an unlocking operation is performed on a certain unlocking node, an unlocking status indicator indicating that the unlocking operation has been performed is displayed at the location of the unlocking node. The expression prompt area can be used to display a prompt image corresponding to the unlocking node. For example, if the first unlocking node is unlocked, the expression prompt area can display a prompt image associated with the first unlocking node. For example, if the pre-recorded expression corresponding to the first unlocking node is a smiling expression, the prompt image can be a blue sky; if the pre-recorded expression corresponding to the first unlocking node is an open mouth expression, the prompt image can be a sunflower.

[0058] In one embodiment, the terminal may display the expression unlock page upon detecting an unlock command, an operation command to enter a page containing private information, an operation command to enter a fund management page, or a payment command. Furthermore, upon receiving a facial recognition command triggered on a face management page, the expression unlock page is accessed. The term "face" may refer to a person's face or the face of another object.

[0059] For example, if the subject wants to access the terminal's operation page, the terminal will enter the expression unlock page upon receiving the unlock command. Alternatively, if the subject wants to make an online payment, the terminal will enter the expression unlock page upon detecting the payment command. Alternatively, as shown in Figure 4, if the subject clicks or touches the face recognition control on the face management page, the expression unlock page will be entered.

[0060] S204, displaying the unlocking node sequence on the expression unlocking page.

[0061] The unlocking node sequence may refer to a node sequence consisting of nodes that need to be unlocked (i.e., unlocking nodes). The unlocking nodes in the unlocking node sequence may be nodes with a sequential order. In other words, when unlocking, each unlocking node in the unlocking node sequence is unlocked in sequence. The unlocking node sequence may correspond to a security lock or a password, and each unlocking node in the unlocking node sequence is unlocked by a corresponding facial expression.

[0062] In one embodiment, when the emoticon unlocking page is displayed, a sequence of unlocking nodes is displayed in the unlocking progress area. In the unlocking progress area, a pointer, such as an arrow or a ">" symbol, may be displayed between each unlocking node in the unlocking node sequence, as shown in FIG4 .

[0063] In one embodiment, after S204 , the terminal may further display the facial image captured in real time in the facial preview area of ​​the expression unlocking page.

[0064] The face can generally refer to the face, chin, lips, eyes, nose, eyebrows, forehead, ears, etc. of the subject to be tested. The face preview area can display a face capture frame, which can be used to capture facial tiles in the facial image, thereby avoiding the problem of high computational complexity caused by recognizing the entire facial image.

[0065] In one embodiment, the terminal captures a real-time facial image of the subject through a built-in camera or an external camera connected to the terminal, and then displays the real-time captured facial image in the face preview area of ​​the expression unlocking page. The facial image may include only the face of the subject or both the face and hands of the subject.

[0066] In one embodiment, after the step of displaying the real-time captured facial image in the facial preview area of ​​the expression unlocking page, the method may further include: the terminal detecting whether the facial key points in the facial image are located within the facial capture frame; if so, executing S206; if not, issuing a prompt message to adjust the capture orientation.

[0067] The key points of the face may be the ears, chin, and forehead of the subject to be measured. If the ears, chin, and forehead in the facial image are within the facial acquisition frame, it can be indicated that the entire face is within the facial acquisition frame.

[0068] For example, as shown in FIG5 , when the facial key points in the detected facial image are not located within the facial acquisition frame (such as the black rectangular wireframe in FIG5 ), a prompt message “Please adjust the acquisition direction” is displayed near the facial acquisition frame so that the entire face in the facial image is located within the facial acquisition frame.

[0069] S206 , at the unlocking node to be processed in the unlocking node sequence, generating an unlocking state identifier based on the facial expression in the facial image collected in real time.

[0070] The pending unlocked node may refer to a node in the unlocked node sequence that has not been unlocked. In this embodiment, the pending unlocked node may also refer to a node in the unlocked node sequence that has not been unlocked and currently needs to be unlocked.

[0071] The unlock status indicator can be used to indicate that the unlock node to be processed has been successfully unlocked, or to indicate that the unlock node to be processed has been unlocked but it is not certain whether the unlocking has been successfully completed. Based on the above two meanings of the unlock status indicator, S206 can be divided into the following two scenarios:

[0072] Scenario 1: The unlock status indicator indicates that the unlock node to be processed has been unlocked, but it is not certain whether the unlocking has been completed successfully.

[0073] In one embodiment, the unlock node sequence is displayed in the unlock progress area of ​​the expression unlock page. The step of displaying the real-time collected facial image in the face preview area of ​​the expression unlock page may specifically include: the terminal performs facial expression recognition on the facial images corresponding to the unlock nodes to be processed in the unlock node sequence in sequence according to the order of the unlock nodes in the unlock node sequence; when the facial expression recognition is completed each time, an unlock status identifier is generated at the corresponding unlock node in the unlock progress area. The unlock status identifier can be used to indicate that the unlock node to be processed has been unlocked. The unlock status identifier includes: a status identifier that the unlock operation has been performed and the unlock node has been successfully unlocked, or a status identifier that the unlock operation has been performed but the unlock node has not been successfully unlocked.

[0074] The unlocking steps for a pending unlocking node in a sequence of unlocking nodes may include: when unlocking the currently pending unlocking node in the sequence of unlocking nodes, the terminal performs facial expression recognition on a currently captured facial image, compares the recognized facial expression with a target expression corresponding to the currently pending unlocking node, and, upon obtaining a comparison result, generates an unlocking status indicator for the currently pending unlocking node. After the unlocking status indicator is generated, the currently pending unlocking node is converted to a processed unlocking node, and then the remaining pending unlocking nodes are unlocked.

[0075] For example, as shown in Figure 6, in the figure on the left, none of the unlock nodes in the unlock node sequence (i.e., nodes 1-6) have been unlocked. When it is detected that the entire face in the facial image is within the facial capture frame, the first unlock node in the unlock node sequence is first unlocked. The unlocking process includes: performing facial expression recognition on the facial image displayed in the face preview area, that is, performing facial expression recognition on the facial image block within the facial capture frame to obtain a facial expression recognition result; wherein the facial expression is an expression of opening the mouth and squinting the left eye. After obtaining the facial expression recognition result, the recognized facial expression is compared with the target expression corresponding to the first unlock node. When the comparison result is obtained, an unlock status indicator is generated at the position of the first unlock node, indicating that the unlock operation has been performed on the first unlock node. Similarly, the other unlock nodes in the unlock node sequence are unlocked.

[0076] In one embodiment, each time a terminal completes facial expression recognition, it generates an expression model corresponding to the facial expression; displays the expression model overlaid on the corresponding facial image in the face preview area; then compares the expression model with the expression image of the corresponding unlock node. Upon obtaining the comparison result, an unlock status indicator is generated at the unlock node to indicate that the facial image corresponding to the unlock node has been recognized. The expression model can be seen as the black dot in Figure 6.

[0077] Scenario 2: The unlock status indicator indicates that the pending unlock node has been successfully unlocked.

[0078] In one embodiment, each time the terminal completes facial expression recognition, it generates an expression model diagram corresponding to the facial expression; the expression model diagram is superimposed and displayed on the corresponding facial image in the face preview area; when the expression model diagram is consistent with the expression diagram of the corresponding unlocking node, it is determined that the facial expression in the facial image matches the corresponding target expression, and then an unlocking state identifier is generated at the unlocking node to be processed in the unlocking node sequence.

[0079] Among them, the expression model diagram can refer to a graphic generated based on the recognized facial expression, which can be used to represent the facial expression obtained by recognizing the object to be tested. In addition, the expression model diagram can also be used to indicate that the facial image corresponding to the unlocking node to be processed has undergone facial expression recognition.

[0080] When the expression model graph is consistent with the expression graph of the corresponding unlocking node, it means that the facial expression in the facial image matches the expression graph pre-recorded in the unlocking node to be processed, and the unlocking node to be processed can be successfully unlocked.

[0081] For example, as shown in Figure 6, when unlocking the first unlock node, the facial expression in the facial image is recognized. After identifying the facial key points in the facial image, an expression model image matching the facial expression of open mouth and squinting left eye is generated and superimposed on the facial expression in the facial image. This expression model image is then compared with the pre-registered expression image corresponding to the first unlock node. If they match, an unlock status indicator is generated at the location of the first unlock node.

[0082] During the process of unlocking each unlock node, a prompt image may be used to indicate the expression corresponding to the unlock node that currently needs to be unlocked.

[0083] In one embodiment, the expression unlocking page includes an expression prompt area; the method further includes: during the unlocking process of a pending unlocking node in the unlocking node sequence, in response to an expression prompt operation triggered in the expression prompt area, displaying a prompt image corresponding to the pending unlocking node in the expression prompt area. Thus, if the user forgets the facial expression corresponding to the unlocking solution, they can associate it with the corresponding facial expression through the prompt image. It should be noted that for the target user who entered the target expression, the prompt image can be used to associate images of joy, anger, sorrow, and happiness. For other users, the specific meaning of the prompt image may not be clear, such as using an image of a sunflower to associate an open mouth.

[0084] As shown in FIG5 , when unlocking the first unlocking node, if the subject forgets what facial expression it is, the sunflower image can be used to determine what facial expression the first unlocking node corresponds to, and then the facial expression can be made to unlock the first unlocking node.

[0085] S208 , completing the unlocking based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

[0086] In one embodiment, unlocking is successful when an unlock status indicator is generated at each unlock node in the unlock node sequence and the facial expression in the corresponding facial image matches the corresponding target expression. For example, referring to FIG6 , unlocking is successful when the positions of unlock nodes 1-6 all change from colorless (or white) to gray and the facial expression in the corresponding facial image matches the corresponding target expression. Alternatively, unlocking is successful when unlock status indicators are generated at at least two unlock nodes in the unlock node sequence and the facial expression in the corresponding facial image matches the corresponding target expression. For example, referring to FIG6 , unlocking is successful when the positions of unlock nodes 1-3 all change from colorless (or white) to gray and the facial expression in the facial image corresponding to unlock nodes 1-3 matches the corresponding target expression.

[0087] The target expression may refer to an expression in a pre-recorded expression diagram. For example, for a security lock with 6 unlocking nodes, one or more expression diagrams are pre-recorded in each unlocking node, and the expression in the expression diagram is the target expression.

[0088] When the recognized facial expression matches the expression graph corresponding to the corresponding unlocking node to be processed, it means that the facial expression matches the corresponding target expression; conversely, when the recognized facial expression matches the corresponding target expression, it means that the facial expression matches the expression graph corresponding to the corresponding unlocking node to be processed.

[0089] In addition, when the generated expression model graph matches the expression graph corresponding to the corresponding unlocking node to be processed, it means that the expression model graph matches the corresponding target expression; conversely, when the generated expression model graph matches the corresponding target expression, it means that the expression model graph matches the expression graph corresponding to the corresponding unlocking node to be processed.

[0090] In one embodiment, as the terminal unlocks each unlocking node, it records the time interval corresponding to the completion of unlocking at each unlocking node, and then calculates the total time interval. Unlocking is successful when an unlocking status indicator is generated at each unlocking node in the unlocking node sequence, the facial expression in each facial image matches the corresponding target expression, and the total time interval is less than a preset time interval.

[0091] If an unlock status indicator is generated at each unlock node in the unlock node sequence, but the facial expression in one or more facial images does not match the corresponding target expression, the terminal fails to unlock. In this case, the terminal can be unlocked again, that is, the process returns to S204 to S208 until the unlock is successful or the cumulative number of unlock failures reaches a preset number, at which point the unlock is suspended.

[0092] In one embodiment, when an unlock status identifier is generated at each unlock node in the unlock node sequence, but the facial expression in at least one facial image does not match the corresponding target expression, the terminal issues a prompt message indicating that the unlocking has failed; the unlock status identifier is canceled, and execution returns to S204 to S208.

[0093] In one embodiment, when unlocking fails, the cumulative number of unlocking failures is obtained; when the cumulative number reaches a preset number, the unlocking process is paused; a reserved communication signal is obtained, and an alarm message is sent to the reserved communication signal.

[0094] Among them, the above-mentioned cumulative number may refer to the total number of unlocking failures in this round or the unlocking failures within a preset time period. The preset time period can be set according to actual conditions and is not specifically limited in this embodiment. The above-mentioned reserved communication signal may refer to: a communication identifier reserved by the target object (such as the user who entered the emoticon) in the application account of the terminal for receiving alarm information or emergency contact, such as a reserved mobile phone number, email account or other instant messaging account. It should be noted that the cumulative number of unlocking failures refers to the total number of consecutive unlocking failures. If there is one successful unlocking, the cumulative number of unlocking failures is set to zero.

[0095] For example, taking the reserved communication number as the reserved mobile phone number, if the cumulative number of unlocking failures within 5 minutes reaches the preset number of 5, the unlocking process will be paused, and an alarm message will be sent to the reserved mobile phone number to alert the target user corresponding to the reserved mobile phone number, so as to inform the target user to reset the password to avoid malicious unlocking by others, or to inform the target user to obtain the password again.

[0096] For example, performing face recognition on a facial image can obtain the recognition results of facial feature points as shown in Figure 7. For the convenience of explanation below, each facial feature point obtained by recognition is marked with a number. For example, as shown in Figure 7, 1 to 17 represent facial edge feature points, 18 to 22 and 23 to 27 represent the user's left eyebrow feature points and right eyebrow feature points respectively, 28 to 36 represent the user's nose feature points, 37 to 42 represent the user's left eye feature points, 43 to 48 represent the user's right eye feature points, and 49 to 68 represent the user's lip feature points. It should be noted that the above is only an example. In optional embodiments, only some or more feature points can be recognized among the above facial feature points, or each feature point can be marked in other ways, all of which fall within the scope of the embodiments of the present invention.

[0097] Facial feature point recognition technology is usually divided into two categories according to the different criteria it adopts and the different features it recognizes:

[0098] (1) Methods based on local features

[0099] In one embodiment, a local feature-based approach can utilize local geometric features of the face, such as the relative positions and distances of facial features (eyes, nose, mouth, etc.), to describe the face. Its feature components typically include Euclidean distances, curvatures, and angles between feature points, enabling efficient description of prominent facial features.

[0100] For example, the integral projection method is used to locate facial landmarks, and the Euclidean distance between landmarks is used as a feature component to identify a multidimensional facial landmark vector for classification. These feature components primarily include: the vertical distance between the eyebrows and the center of the eyes; multiple descriptive data points for the arch of the eyebrows; nose width and vertical position; nostril position; and face width. By identifying these facial landmarks, a 100% accuracy rate can be achieved during the recognition process.

[0101] In an optional embodiment of the present invention, the local feature-based method can also be an empirical description of the general characteristics of facial feature points. For example, facial images have some obvious basic characteristics, such as the facial region generally includes facial feature points such as the eyes, nose, and lips, whose brightness is generally lower than that of the surrounding areas; the eyes are roughly symmetrical, and the nose and mouth are located on the axis of symmetry.

[0102] (2) Holistic approach

[0103] Here, the holistic method treats the facial image as a whole and performs some transformation processing on it to identify features. This method takes into account the overall attributes of the face and also retains the topological relationship between facial parts and the information of each part itself.

[0104] Because facial images are typically very high in dimensionality and their distribution in high-dimensional space is not compact, classification is difficult and computationally complex. Subspace analysis can be used to find a linear or nonlinear spatial transformation based on a specific objective, compressing the original high-dimensional data into a low-dimensional subspace. This makes the data distribution in this subspace more compact and reduces computational complexity.

[0105] Alternatively, a set of rectangular grid nodes can be placed on the facial image. The characteristics of each node are described using the multi-scale wavelet feature at that node, and the connection relationship between nodes is represented by geometric distance, thus forming a facial representation based on a two-dimensional topological graph. During the face recognition process, identification is based on the similarity between the nodes and connections in the two images.

[0106] In addition to the above-mentioned subspace analysis method and elastic graph matching method, the overall method also includes a neural network-based method. In the embodiment of the present invention, there is no restriction on the type of the overall method.

[0107] In the above embodiment, an unlocking node sequence consisting of multiple unlocking nodes is configured on the expression unlocking page. When unlocking each unlocking node, it is necessary to perform expression recognition on the facial image collected in real time, and each unlocking node corresponds to a specific target expression. Only when the facial expressions in the facial images corresponding to all unlocking nodes match the target expression corresponding to the corresponding unlocking node, the entire unlocking process is completed, thereby effectively avoiding unlocking due to stolen images or face models, and effectively improving information security.

[0108] In one embodiment, in addition to unlocking by facial expression, unlocking can also be performed by combining facial expression and face recognition. As shown in Figure 8, the method may also include:

[0109] S802 , performing facial expression recognition on facial images corresponding to the unlocked nodes to be processed in the unlocked node sequence in sequence according to the order of the unlocked nodes in the unlocked node sequence.

[0110] Among them, facial expressions can be expressions presented by different parts of the face performing corresponding actions or being in corresponding postures, such as the expression of opening the mouth and squinting the left eye as shown in Figures 5 and 6.

[0111] In one embodiment, the step of facial expression recognition includes: the terminal extracts eye feature points from the facial image; among the eye feature points, determines a first distance between the upper eyelid feature point and the lower eyelid feature point, and determines a second distance between the left eye corner feature point and the right eye corner feature point; determines the eye posture based on the relationship between the ratio between the first distance and the second distance and at least one preset interval.

[0112] Among them, the left corner feature point and the right corner feature point refer to the left corner feature point and the right corner feature point of the same eye respectively. For example, for the left eye, the left corner feature point refers to the left corner feature point of the left eye, and the right corner feature point refers to the right corner feature point of the left eye.

[0113] For example, as shown in Figure 7, the terminal calculates the first distance between the upper eyelid feature point 38 and the lower eyelid feature point 42, and the second distance between the left eye corner feature point 37 and the right eye corner feature point 40 based on the facial feature points acquired by face recognition technology. When the ratio between the first distance and the second distance is 0, it is determined that the subject is squinting. When the ratio between the first distance and the second distance is less than 0.2, it is determined that the subject is blinking to the left. When the ratio between the first distance and the second distance is greater than 0.2 and less than 0.6, it is determined that the subject is glaring with the left eye.

[0114] For lip gestures, the following methods can be used for recognition:

[0115] Method 1: Recognize lip gestures based on the height of lip feature points.

[0116] In one embodiment, the facial expression recognition step further includes: the terminal extracting lip feature points from the facial image; and determining the lip posture among the lip feature points based on the height difference between the lip center feature point and the lip corner feature point.

[0117] For example, as shown in Figure 7, the terminal determines the height of the lip center feature point 63 and the height of the lip corner feature point 49 (or 55), and then calculates the height difference between the lip center feature point 63 and the lip corner feature point 49 (or 55). If the height difference is positive (that is, the height of the lip center feature point 63 is higher than the height of the lip corner feature point 49), the lip posture is judged to be a smile.

[0118] Method 2: Recognize lip gestures based on the distance between the upper and lower lip feature points.

[0119] In one embodiment, among the lip feature points, the terminal determines the lip gesture according to a third distance between the upper lip feature point and the lower lip feature point.

[0120] For example, based on facial feature points acquired by face recognition technology, the third distance between the upper lip feature point and the lower lip feature point is compared with a distance threshold. When the third distance reaches the distance threshold, the lip posture is determined to be open. As shown in Figure 7, the third distance between the upper lip feature point 63 and the lower lip feature point 67 is compared with the distance threshold. If it is greater than or equal to the distance threshold, it is indicated that the subject is opening their mouth.

[0121] In another embodiment, the terminal calculates a fourth distance between the left lip corner feature point and the right lip corner feature point, and when the third distance is greater than or equal to the fourth distance, determines that the lip posture is open.

[0122] Method 3: Recognize lip gestures based on the ratio of the distance between the upper and lower lips to the distance between the left and right lip corner feature points.

[0123] In one embodiment, among the lip feature points, the lip posture is determined based on the relationship between the ratio of the third distance to the fourth distance and at least one preset interval; wherein the fourth distance is the distance between the left lip corner feature point and the right lip corner feature point.

[0124] For example, as shown in Figure 7, the terminal determines the third distance between upper lip feature point 63 and lower lip feature point 67, and the fourth distance between left lip corner feature point 49 and right lip corner feature point 55. The lip posture is determined based on the relationship between the ratio of the third and fourth distances and at least one preset interval. For example, when the ratio of the third and fourth distances is within a first preset interval, the lip posture is determined to be open; furthermore, when the ratio of the third and fourth distances is within a second preset interval, the lip posture is determined to be closed. The values ​​in the first preset interval are always greater than the values ​​in the second preset interval.

[0125] In one embodiment, the step of facial expression recognition also includes: extracting eyebrow feature points and eyelid feature points from the facial image; determining a fifth distance between the eyebrow feature points and the eyelid feature points; and determining the eyebrow posture based on the size relationship between the fifth distance and the preset distance.

[0126] For example, as shown in FIG7 , the distance between the eyebrow feature point and the eyelid feature point 38 is calculated. When the distance is greater than a preset distance, it is determined that the subject is raising his eyebrows.

[0127] S804 , performing face recognition on the facial image corresponding to each unlocked node according to the order of the unlocked nodes in the unlocked node sequence to obtain a face recognition result.

[0128] For the face recognition process, reference may be made to S208 of the above embodiment.

[0129] S806: When an unlock status identifier is generated at each unlocking node, and the facial expression in each facial image matches the corresponding target expression, and it is determined based on the face recognition result that the object to be tested is consistent with the target object, the unlocking is successful.

[0130] When using facial expressions for unlocking, facial recognition can also be performed to determine whether the face matches the pre-registered face of the target object. If a match occurs, it indicates that the target object and the target object are the same person. The facial image is an image captured from the target object.

[0131] Unlocking fails when an unlock status indicator is generated at each unlocking node, the facial expression in each facial image matches the corresponding target expression, and the face recognition results indicate that the subject under test and the target object are inconsistent. If the face recognition results indicate that the subject under test and the target object are not the same, unlocking fails.

[0132] In the above embodiment, combining facial expressions with human faces for unlocking can further improve the security of the unlocking node sequence and enhance information security.

[0133] In one embodiment, in addition to unlocking by facial expression, unlocking can also be performed in combination with facial expression and gesture, as shown in FIG9 , and the method may further include:

[0134] S902 , performing facial expression recognition on facial images corresponding to the unlocked nodes to be processed in the unlocked node sequence in sequence according to the order of the unlocked nodes in the unlocked node sequence.

[0135] Among them, facial expressions can be expressions presented by different parts of the face performing corresponding actions or being in corresponding postures, such as the expression of opening the mouth and squinting the left eye as shown in Figures 5 and 6.

[0136] For the recognition of facial expressions, reference may be made to S802 in the above embodiment.

[0137] S904: During the facial expression recognition process, gesture recognition is performed on the hands in the facial image.

[0138] In one embodiment, the terminal performs convolution processing on the facial image through a neural network model, thereby extracting gesture features in the facial image and determining a specific gesture based on the gesture features.

[0139] The neural network model may be a network model for extracting gesture features, specifically a two-dimensional convolutional neural network model. The two-dimensional network model may be one of the network branches of a machine learning model.

[0140] S906 , when facial expression recognition and gesture recognition are completed each time, an unlocking status mark is generated at the corresponding unlocking node in the unlocking progress area.

[0141] In one embodiment, the terminal can also determine whether the facial expression is consistent with the corresponding target expression after each facial expression recognition is completed; in addition, the terminal can also determine whether the gesture is consistent with the corresponding target gesture after each gesture recognition is completed. If the facial expression is consistent with the corresponding target expression and the gesture is consistent with the corresponding target gesture, an unlock status mark is generated at the corresponding unlock node in the unlock progress area.

[0142] S908, when an unlock status identifier is generated at each unlocking node, the facial expression in each facial image matches the corresponding target expression, and each recognized gesture is consistent with the corresponding target gesture, the unlocking is successful.

[0143] In the above embodiment, unlocking by combining facial expressions with hand gestures can further improve the security of the unlocking node sequence and enhance information security.

[0144] In one embodiment, each unlocking node can be unlocked by at least two facial expressions, and each unlocking node corresponds to at least two different target expressions; as shown in FIG10 , S206 may specifically include:

[0145] S1002: Perform facial expression recognition on at least two facial images corresponding to the unlocked node to be processed.

[0146] When unlocking the unlocking node to be processed, facial expression recognition may be performed on at least two facial images corresponding to the unlocking node to be processed.

[0147] For example, when unlocking node 1, facial image 1 is first collected, and then facial expression recognition is performed on facial image 1 to obtain facial expression 1; then facial expression recognition is performed on the collected facial image 1 to obtain facial expression 2.

[0148] The process of facial expression recognition may refer to S802 in the above embodiment.

[0149] S1004: When the facial expressions in at least two facial images match the corresponding target expressions, an unlock state identifier is generated at the unlock node to be processed.

[0150] The facial expressions in at least two facial images may be the same or different.

[0151] In one embodiment, when the facial expressions in at least two facial images match the corresponding target expressions, the terminal determines the acquisition time interval between the at least two facial images; or determines the unlocking time interval between the unlocking node to be processed and the last unlocking node processed. When the acquisition time interval or the unlocking time interval meets the corresponding time interval condition, S1004 is executed. This avoids wasting excessive time when unlocking the same unlocking node or unlocking different unlocking nodes, thereby improving unlocking efficiency. It also prevents others from trying to unlock a particular unlocking node by multiple attempts with different facial expressions.

[0152] S1006: When an unlock status identifier is generated at each unlocking node and the facial expression in each facial image matches the corresponding target expression, the unlocking is successful.

[0153] In the above embodiment, each unlocking node is unlocked using at least two facial expressions, which can further improve the security of the unlocking node sequence and enhance information security.

[0154] In one embodiment, a specific expression image may be pre-recorded for each unlocking node. The step of recording the expression image may include:

[0155] S1102, displaying the expression entry page.

[0156] The expression entry page may include an expression identifier or may not include an expression identifier. The expression identifier is used to indicate the corresponding expression type, and different expression identifiers correspond to different expression types.

[0157] In one embodiment, when the terminal receives an expression entry command triggered on the expression management page, it switches the page to the expression entry page containing the expression identifier. As shown in Figure 12, when a click or touch operation is detected on the face entry control on the expression management page, the expression entry page containing the expression identifier is displayed. The upper half of the expression entry page is used to display the collected expression image, and the lower half is used to display the expression identifier.

[0158] S1104, sequentially inputting expression images corresponding to the expression identifiers in the expression input page through the expression input page.

[0159] In one embodiment, when the expression entry page includes expression identifiers, the terminal collects expression images of the target object in the order of the expression identifiers, and then records the collected expression images in sequence. As shown in Figure 12, in the expression entry page, the expression image is first entered for the first expression identifier, and then the expressions of subsequent expression identifiers are entered in sequence.

[0160] In one embodiment, when the expression entry page does not contain an expression identifier, the terminal recognizes the captured expression image to obtain a target expression; based on the target expression, the corresponding expression identifier is displayed in the expression identifier preview area of ​​the expression entry page. As shown in Figure 13, an open-mouth facial image is recognized to obtain an open-mouth facial expression; an expression identifier is then generated based on the open-mouth facial expression and displayed in the expression identifier preview area of ​​the expression entry page.

[0161] S1106, enter the expression combination page.

[0162] The expression combination page may be a page for combining expression identifiers.

[0163] S1108 , in response to a combination operation triggered on the expression combination page, the expression identifiers are combined to obtain an expression unlocking sequence.

[0164] In one embodiment, in the expression combination page, the terminal can choose to sort the expression identifiers in the expression entry page and construct an expression unlocking sequence based on the sorted expression identifiers; or, the terminal first combines the expression identifiers in the expression entry page and then sorts them, and constructs an expression unlocking sequence based on the combined and sorted expression identifiers.

[0165] For example, as shown in Figure 14, the target object can combine the expression symbols in the expression combination page in pairs to obtain the corresponding expression combination. As shown in the expression combination page on the right side of Figure 14, in the first row, the normal expression and the left-glaring expression are combined.

[0166] In the above embodiment, the corresponding expression images are pre-recorded according to the expression identifiers, and then the expression identifiers are combined and sorted to generate an expression unlocking sequence, so that when unlocking, the expression unlocking node is unlocked according to the corresponding facial expression. Moreover, when unlocking each unlocking node, it is necessary to compare the currently recognized facial expression with the pre-recorded corresponding expression image. Only when the two match can the unlocking node be unlocked, thereby effectively improving the security of the unlocking node.

[0167] As an example, let's take facial images as an example. As shown in Figures 15 and 16, in this embodiment, the smart password lock function is primarily implemented through facial recognition + combined expressions + expression sequences + prompt images. This effectively maintains user privacy. Combining expressions increases the difficulty of cracking expression recognition, using expression sequences increases the complexity of unlocking, and using prompt images helps users remember password combinations while preventing password leaks. The expression sequence corresponds to the unlocking node sequence described above.

[0168] In this embodiment, users can customize the combination of different expressions, set expression sequences, add prompt images and pictures (or animations) to create a combination lock that can meet the user's own encryption and decryption needs and increase personal privacy protection. The specific implementation steps for defining and decrypting the password lock include:

[0169] S1, recognize facial feature points through expression recognition to obtain corresponding facial expressions.

[0170] S2, redefine and implement the logic of expression triggering.

[0171] 1) Blinking: Based on the facial feature points acquired by facial recognition technology, the distance between the center feature points of the upper and lower eyelids is divided by the distance between the feature points of the left and right eye corners to obtain a ratio. If this ratio is less than a set distance threshold, it is considered a blink. As shown in Figure 7, the distance between the center feature point 38 of the left upper eyelid and the center feature point 42 of the left lower eyelid is divided by the distance between the feature points 37 and 40 of the left and right eye corners. If the resulting ratio is less than 0.2, it is considered a blink.

[0172] 2) Staring: Based on the facial feature points acquired by face recognition technology, the distance between the center feature points of the upper and lower eyelids is divided by the distance between the feature points of the left and right corners to obtain a ratio. If the ratio is greater than a set ratio threshold, it is considered a staring event. For example, in Figure 7, the distance between the center feature point 38 of the left upper eyelid and the center feature point 42 of the left lower eyelid is divided by the distance between the feature points 37 and 40 of the left and right corners. If the resulting ratio is less than 0.6, it is considered a left wink.

[0173] 3) Pouting: Based on the facial feature points acquired by face recognition technology, a pouting ratio threshold is set. The distance between the upper and lower lip feature points is divided by the distance between the left and right lip feature points. When the resulting quotient reaches the set pouting ratio threshold, the lip is identified as pouting. As shown in Figure 7, the distance between the upper and lower center lip feature points 67 and 63 is divided by the distance between the left and right lip feature points 49 and 55. If the resulting quotient is greater than 1.0, the lip is identified as pouting.

[0174] 4) Smiling: Based on the facial feature points acquired by face recognition technology, a smile is detected when the center feature point of the lips is lower than the height of the left and right feature points of the lips. As shown in Figure 7, a smile is detected when the center feature point 63 of the lips is lower than the positions of the left and right feature points 49 and 55 of the lips.

[0175] 5) Angry: Based on the facial feature points acquired by face recognition technology, anger is detected when the center feature point of the lips is higher than the left and right feature points. As shown in the figure, anger is detected when the center feature point 63 of the lips is higher than the left and right feature points 49 and 55.

[0176] 6) Mouth opening: Based on the facial feature points acquired by face recognition technology, the unit distance between the upper and lower lips is calculated as a weighted average, and a distance threshold for mouth opening is set. When the distance between the feature points of the upper and lower lips is greater than this distance threshold, the mouth is considered open. As shown in Figure 7, the mouth is considered open when the distance between the center feature point 63 of the upper lip and the terminal feature point 67 of the lower lip is greater than 0.2 times the distance between the left and right feature points 49 and 55 of the lips.

[0177] 7) Eyebrow raising: Based on the facial feature points obtained by face recognition technology, a distance threshold for eyebrow raising is set. When the distance between the eyebrow feature point and the upper eyelid feature point is greater than the distance threshold, it is judged as eyebrow raising.

[0178] 8) Nodding: Based on the facial feature points obtained by face recognition technology, the angle of the head rotation in the previous frame of the face image and the angle of the head rotation in the current frame of the face image are calculated, and the angle difference between the two is calculated. When the angle difference exceeds 10 degrees, it is judged as a nod.

[0179] S3, combine the expressions.

[0180] Users can combine the above expressions in pairs as needed, such as blinking the left eye + opening the mouth to form an expression combination, blinking the right eye + pouting to form an expression combination.

[0181] S4, sort different expression combinations.

[0182] Users can arrange the set expression combinations in a certain order and build a password group based on the sorted expression combinations, such as: Expression combination 1 (wink + nod): Expression combination 2 (wink left eye + open mouth): Expression combination 3 (wink + pout): Expression combination 4 (smile): Expression combination 5 (raise eyebrow): Expression combination 6 (stare).

[0183] S5, set prompt image and animation.

[0184] For each emoji combination, you can set a reminder image or animation that helps with memorization without revealing the emoji. For example, you can set a blue sky to remember a smile, rain to remember anger, and a palm to remember a nod. This association scheme is set according to the user's preference. Try to avoid too many obvious reminders to prevent leaking the password combination.

[0185] S6, set the over-limit protector.

[0186] Set the verification time for each expression combination, such as 5s, and set the total number of recognition errors. If the recognition error operation is performed five times, the password lock will be locked and cannot be opened for a certain period of time, and an early warning message will be sent to a fixed email address and mobile phone number.

[0187] S7, unlock.

[0188] First, facial recognition is performed, followed by expression recognition. If an unlocking node doesn't have a corresponding expression, a prompt image is displayed. After recognizing a single facial expression, the system then recognizes the combination of expressions to determine whether it corresponds to the correct unlocking node. After each expression combination is correctly recognized, the entire expression sequence is recognized to determine whether the order of each expression combination is correct. Finally, the system determines the cumulative number of timeouts or failures. If the cumulative number of timeouts or failures reaches five, unlocking is suspended and an alert is sent to a fixed email address and mobile phone number.

[0189] The solution of this embodiment can help users flexibly set password combinations for access control, file encryption, entering smart terminal operation pages and payment needs. It has low technical difficulty, strong privacy, and is easy to operate and remember.

[0190] It should be understood that although the various steps in the flowcharts of Figures 2, 8-11 are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in Figures 2, 8-11 may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0191] In one embodiment, as shown in FIG17 , a facial expression-based unlocking device is provided. The device may be implemented as a software module or a hardware module, or a combination of both, as part of a computer device. The device specifically includes: a display module 1702 , a first display module 1704 , a generation module 1706 , and an unlocking module 1708 , wherein:

[0192] Display module 1702, used to display the expression unlocking page;

[0193] The first display module 1704 is used to display the unlock node sequence on the expression unlock page;

[0194] A generating module 1706 is configured to generate an unlocking state identifier based on a facial expression in a facial image collected in real time at an unlocking node to be processed in the unlocking node sequence;

[0195] The unlocking module 1708 is configured to complete the unlocking based on the unlocking status identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

[0196] In the above embodiment, an unlocking node sequence consisting of multiple unlocking nodes is configured on the expression unlocking page. When unlocking each unlocking node, it is necessary to perform expression recognition on the facial image collected in real time, and each unlocking node corresponds to a specific target expression. Only when the facial expressions in the facial images corresponding to all unlocking nodes match the target expression corresponding to the corresponding unlocking node, the entire unlocking process is completed, thereby effectively avoiding unlocking due to stolen images or face models, and effectively improving information security.

[0197] In one embodiment, the unlocking node sequence is displayed in the unlocking progress area of ​​the emoticon unlocking page. The device also includes:

[0198] The second display module is used to display the real-time collected facial image in the face preview area of ​​the expression unlocking page;

[0199] The generation module is also used to perform facial expression recognition on the facial images corresponding to the unlocking nodes to be processed in the unlocking node sequence in sequence according to the order of the unlocking nodes in the unlocking node sequence; when the facial expression recognition is completed each time, an unlocking status mark is generated at the corresponding unlocking node in the unlocking progress area.

[0200] In one embodiment, the apparatus further includes: a generating module, a superposition module, and a determining module; wherein:

[0201] A generation module, configured to generate an expression model graph corresponding to the facial expression each time facial expression recognition is completed;

[0202] An overlay module, configured to overlay and display an expression model diagram on the corresponding facial image in the facial preview area;

[0203] The determination module is used to determine whether the facial expression in the facial image matches the corresponding target expression when the expression model graph is consistent with the expression graph of the corresponding unlocked node.

[0204] In one embodiment, the facial image includes a face and a hand. The apparatus further comprises: a recognition module; wherein:

[0205] A recognition module for performing hand gesture recognition on a facial image during facial expression recognition;

[0206] The generating module is further configured to generate an unlocking status identifier at a corresponding unlocking node in the unlocking progress area each time facial expression recognition and gesture recognition are completed.

[0207] In the above embodiment, unlocking by combining facial expressions with hand gestures can further improve the security of the unlocking node sequence and enhance information security.

[0208] In one embodiment, each unlocked node corresponds to at least two different target expressions. The generation module is further configured to perform facial expression recognition on at least two facial images corresponding to the unlocked node to be processed; when the facial expressions in the at least two facial images match the corresponding target expressions, an unlocked state indicator is generated at the unlocked node to be processed.

[0209] In one embodiment, the determination module is further configured to determine, when the facial expressions in the at least two facial images match the corresponding target expressions, a capture time interval between the at least two facial images; or determine an unlocking time interval between the unlocking node to be processed and the last unlocking node processed;

[0210] The generating module is further configured to generate an unlocking state identifier at the unlocking node to be processed when the collection time interval or the unlocking time interval meets the corresponding time interval condition.

[0211] In the above embodiment, each unlocking node is unlocked using at least two facial expressions, which can further improve the security of the unlocking node sequence and enhance information security.

[0212] In one embodiment, the recognition module is further used to extract eye feature points from the facial image; among the eye feature points, determine the first distance between the upper eyelid feature point and the lower eyelid feature point, and determine the second distance between the left eye corner feature point and the right eye corner feature point; and determine the eye posture based on the relationship between the ratio between the first distance and the second distance and at least one preset interval.

[0213] In one embodiment, the recognition module is also used to extract lip feature points from the facial image; among the lip feature points, the lip posture is determined based on the height difference between the lip center feature point and the lip corner feature point; or, among the lip feature points, the lip posture is determined based on the third distance between the upper lip feature point and the lower lip feature point; or, among the lip feature points, the lip posture is determined based on the relationship between the ratio of the third distance to the fourth distance and at least one preset interval; wherein the fourth distance is the distance between the left lip corner feature point and the right lip corner feature point.

[0214] In one embodiment, the recognition module is further used to extract eyebrow feature points and eyelid feature points from the facial image; determine the fifth distance between the eyebrow feature points and the eyelid feature points; and determine the eyebrow posture based on the size relationship between the fifth distance and the preset distance.

[0215] In one embodiment, a face collection frame is displayed in the face preview area. The device further includes: a detection module and a prompt module; wherein:

[0216] A detection module is used to detect whether the facial key points in the facial image are located within the facial acquisition frame;

[0217] The generating module is further configured to generate an unlock state identifier based on the facial expression in the corresponding facial image at each unlock node in the order of the unlock nodes in the unlock node sequence if the facial key points in the facial image are within the facial capture frame;

[0218] The prompt module is used to issue a prompt message to adjust the acquisition direction if the facial key points in the facial image are not located within the facial acquisition frame.

[0219] In one embodiment, the facial image is an image captured from the subject to be tested. The recognition module is further configured to perform facial recognition on the facial image corresponding to each unlocked node in the unlocked node sequence according to the order of the unlocked nodes, thereby obtaining a facial recognition result.

[0220] The unlocking module is also used to successfully unlock the device when an unlocking status identifier is generated at each unlocking node, the facial expression in each facial image matches the corresponding target expression, and the object to be tested is determined to be consistent with the target object based on the face recognition result.

[0221] In the above embodiment, combining facial expressions with human faces for unlocking can further improve the security of the unlocking node sequence and enhance information security.

[0222] In one embodiment, the apparatus further comprises: a cancellation module; wherein:

[0223] The prompt module is further configured to issue a prompt message indicating unlocking failure when an unlocking status identifier is generated at each unlocking node in the unlocking node sequence but the facial expression in each facial image does not match the corresponding target expression;

[0224] The cancellation module is used to cancel the display of the unlock state identifier and return to the step of generating the unlock state identifier based on the facial expression in the corresponding facial image at the unlock node to be processed in the unlock node sequence.

[0225] In one embodiment, the device further includes: an acquisition module, a pause module, and an alarm module; wherein:

[0226] The acquisition module is used to obtain the cumulative number of unlocking failures when unlocking fails;

[0227] A pause module is used to pause the unlocking process when the cumulative number of times reaches a preset number;

[0228] The alarm module is used to obtain the reserved communication signal and send alarm information to the reserved communication signal.

[0229] In one embodiment, the device further comprises: an input module and a combination module; wherein:

[0230] The second display module is used to display the expression entry page;

[0231] An input module, used to input expression images corresponding to expression identifiers in the expression input page in sequence through the expression input page;

[0232] Entry module, used to enter the expression combination page;

[0233] The combination module is used to combine the expression identifiers in response to the combination operation triggered on the expression combination page to obtain the expression unlocking sequence.

[0234] In one embodiment, the expression unlocking page includes an expression prompt area. The second display module is further configured to display a prompt image corresponding to the pending unlocking node in the unlocking node sequence in response to an expression prompt operation triggered in the expression prompt area during the unlocking process of the pending unlocking node in the unlocking node sequence.

[0235] In the above embodiment, the corresponding expression images are pre-recorded according to the expression identifiers, and then the expression identifiers are combined and sorted to generate an expression unlocking sequence, so that when unlocking, the expression unlocking node is unlocked according to the corresponding facial expression. Moreover, when unlocking each unlocking node, it is necessary to compare the currently recognized facial expression with the pre-recorded corresponding expression image. Only when the two match can the unlocking node be unlocked, thereby effectively improving the security of the unlocking node.

[0236] The specific limitations of the facial expression-based unlocking device can be found in the limitations of the facial expression-based unlocking method described above and will not be further elaborated here. Each module in the facial expression-based unlocking device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each of the modules described above may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0237] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in Figure 18. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication, and the wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. When the computer program is executed by the processor, it implements a facial expression-based unlocking method. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or may be a key, trackball, or touchpad provided on the computer device housing, or may be an external keyboard, touchpad, or mouse.

[0238] Those skilled in the art will understand that the structure shown in Figure 18 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0239] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0240] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0241] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0242] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0243] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0244] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A facial expression-based unlocking method, executed by a terminal, characterized in that: The method comprises: Display the emoticon unlock page; Display the unlock node sequence on the expression unlock page; At an unlocking node to be processed in the unlocking node sequence, generating an unlocking state identifier based on a facial expression in a facial image collected in real time; Unlocking is completed based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

2. The method according to claim 1, characterized in that The unlocking node sequence is displayed in the unlocking progress area of ​​the emoticon unlocking page; the method further includes: Displaying the real-time collected facial image in the face preview area of ​​the expression unlocking page; The step of generating an unlocking state identifier based on a facial expression in a real-time collected facial image at an unlocking node to be processed in the unlocking node sequence includes: performing facial expression recognition on the facial images corresponding to the unlocked nodes to be processed in the unlocked node sequence in sequence according to the order of the unlocked nodes in the unlocked node sequence; Each time facial expression recognition is completed, an unlocking status identifier is generated at the corresponding unlocking node in the unlocking progress area.

3. The method according to claim 2, characterized in that The method further comprises: When facial expression recognition is completed each time, an expression model diagram corresponding to the facial expression is generated; Overlaying and displaying the expression model diagram on the corresponding facial image in the facial preview area; When the expression model graph is consistent with the expression graph of the corresponding unlocked node, it is determined that the facial expression in the facial image matches the corresponding target expression.

4. The method according to claim 1, wherein Each of the unlocking nodes corresponds to at least two different target expressions; Generating an unlock state identifier based on a facial expression in the corresponding facial image at an unlock node to be processed in the unlock node sequence includes: Performing facial expression recognition on at least two facial images corresponding to the unlocked node to be processed; When the facial expressions in the at least two facial images match the corresponding target expressions, an unlock state identifier is generated at the unlock node to be processed.

5. The method according to any one of claims 2 to 4, characterized in that The sequentially performing facial expression recognition on the facial images corresponding to the unlocking nodes to be processed in the unlocking node sequence includes: Extracting eye feature points from the facial images corresponding to the unlocking nodes to be processed in the unlocking node sequence; Among the eye feature points, determining a first distance between an upper eyelid feature point and a lower eyelid feature point, and determining a second distance between a left eye corner feature point and a right eye corner feature point; The eye gesture is determined according to a relationship between a ratio of the first distance to the second distance and at least one preset interval.

6. The method according to any one of claims 2 to 4, characterized in that The sequentially performing facial expression recognition on the facial images corresponding to the unlocking nodes to be processed in the unlocking node sequence includes: Extracting lip feature points in the facial image corresponding to the unlocking nodes to be processed in the unlocking node sequence; Among the lip feature points, the lip posture is determined according to the height difference between the lip center feature point and the lip corner feature point; or, Among the lip feature points, determining the lip posture according to a third distance between the upper lip feature point and the lower lip feature point; or, Among the lip feature points, the lip posture is determined based on the relationship between the ratio between the third distance and the fourth distance and at least one preset interval; wherein the fourth distance is the distance between the left lip corner feature point and the right lip corner feature point.

7. The method according to claim 2, characterized in that A face collection frame is displayed in the face preview area; after the face image collected in real time is displayed in the face preview area of ​​the expression unlocking page, the method further includes: Detecting whether facial key points in the facial image are located within the facial acquisition frame; If so, executing the step of generating an unlock state identifier based on the facial expression in the real-time collected facial image at the unlock node to be processed in the unlock node sequence; If not, a prompt message is issued to adjust the collection direction.

8. The method according to claim 1, characterized in that The facial image is an image obtained by capturing the object to be tested; the method further includes: performing face recognition on the facial image corresponding to each unlocked node according to the order of the unlocked nodes in the unlocked node sequence to obtain a face recognition result; The completing the unlocking based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression includes: When an unlocking status identifier is generated at each unlocking node, and the facial expression in each facial image matches the corresponding target expression, and it is determined based on the face recognition result that the object to be tested is consistent with the target object, the unlocking is successful.

9. The method according to claim 1, characterized in that The method further comprises: When an unlock status identifier is generated at each unlock node in the unlock node sequence, but the facial expression in each facial image does not match the corresponding target expression, a prompt message indicating unlock failure is issued; The display of the unlock status identifier is canceled, and the process returns to executing the step of generating the unlock status identifier based on the facial expression in the real-time collected facial image at the unlock node to be processed in the unlock node sequence.

10. The method according to claim 9, characterized in that The method further comprises: When unlocking fails, get the cumulative number of unlocking failures; When the accumulated number of times reaches a preset number of times, the unlocking process is paused; Acquire a reserved communication signal, and send an alarm message to the reserved communication signal.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Display the expression entry page; Entering expression images corresponding to the expression identifiers in the expression entry page in sequence through the expression entry page; Enter the emoticon combination page; In response to a combination operation triggered on the expression combination page, the expression identifiers are combined to obtain the expression unlocking sequence.

12. The method according to any one of claims 1 to 10, characterized in that The expression unlocking page includes an expression prompt area; the method further includes: In a process of unlocking an unlock node to be processed in the unlock node sequence, in response to an expression prompt operation triggered in the expression prompt area, a prompt image corresponding to the unlock node to be processed is displayed in the expression prompt area.

13. The method according to claim 2 or 3, characterized in that The facial image includes a face and a hand; the method further includes: During the facial expression recognition process, performing gesture recognition on the hands in the facial image; The step of generating an unlock status indicator at a corresponding unlock node in the unlock progress area each time facial expression recognition is completed includes: Each time facial expression recognition and gesture recognition are completed, an unlocking status identifier is generated at the corresponding unlocking node in the unlocking progress area.

14. The method according to claim 4, characterized in that The method further comprises: When the facial expressions in the at least two facial images match the corresponding target expressions, determining the acquisition time interval between the at least two facial images; or Determine an unlocking time interval between the unlocking node to be processed and the last unlocking node processed; When the collection time interval or the unlocking time interval meets the corresponding time interval condition, the step of generating an unlocking state identifier at the unlocking node to be processed is performed.

15. The method according to any one of claims 2 to 4, characterized in that The sequentially performing facial expression recognition on the facial images corresponding to the unlocking nodes to be processed in the unlocking node sequence includes: Extracting eyebrow feature points and eyelid feature points from the facial image corresponding to the unlocking nodes to be processed in the unlocking node sequence; determining a fifth distance between the eyebrow feature point and the eyelid feature point; The eyebrow posture is determined according to the magnitude relationship between the fifth distance and the preset distance.

16. A facial expression-based unlocking device, characterized in that: The device comprises: Display module, used to display the expression unlocking page; A first display module, configured to display an unlocking node sequence on the expression unlocking page; a generating module configured to generate an unlocking state identifier based on a facial expression in the facial image collected in real time at an unlocking node to be processed in the unlocking node sequence; The unlocking module is used to complete the unlocking based on the unlocking state identifier and the matching between the facial expression in the corresponding facial image and the corresponding target expression.

17. The device according to claim 16, characterized in that The unlocking node sequence is displayed in the unlocking progress area of ​​the emoticon unlocking page; the device further comprises: A second display module is used to display the real-time collected facial image in the face preview area of ​​the expression unlocking page; The generating module is further configured to perform facial expression recognition on the facial images corresponding to the unlocking nodes to be processed in the unlocking node sequence in sequence according to the order of the unlocking nodes in the unlocking node sequence; and generate an unlocking status identifier at the corresponding unlocking node in the unlocking progress area each time facial expression recognition is completed.

18. The device according to claim 17, characterized in that The device further comprises: A generation module, configured to generate an expression model diagram corresponding to the facial expression each time facial expression recognition is completed; an overlay module, configured to overlay and display the expression model diagram on the corresponding facial image in the facial preview area; A determination module is configured to determine whether the facial expression in the facial image matches the corresponding target expression when the expression model graph is consistent with the expression graph of the corresponding unlocked node.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.