System and method for multi-person simultaneous face recognition and motion recognition verification

Abstract

The present invention discloses a system and method for multi-person face recognition and motion recognition verification at the same time, and relates to the field of recognition and verification systems. The system and method for multi-person face recognition and motion recognition verification at the same time includes a user entering a login page, users who need to be verified log in to the system separately, and the system can log in multiple people according to business scenarios. The system and method for multi-person face recognition and motion recognition verification at the same time can verify faces and actions at the same time, and realize multi-person cooperative verification, which can better represent the attitude of cooperation between two people and effectively reduce the situation of regret afterwards. For example, in a bank counter card opening system, after a user opens a card by himself, the lobby manager is generally required to enter a password on the machine or confirm on an iPad. With the present invention, the lobby manager can make a corresponding confirmation action to confirm during confirmation, and retain image records at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of identification and verification systems, and in particular to a system and method for multi-person simultaneous face recognition and action recognition and verification. Background Art

[0002] Face recognition is a biometric technology that identifies people based on their facial features. It uses a camera or camcorder to capture images or video streams containing faces, and automatically detects and tracks faces in the images, and then uses a series of related technologies to perform facial recognition on the detected faces. It is also commonly called portrait recognition or facial recognition. Face recognition technology is common in existing mobile apps and is mostly used in scenarios such as mobile payments and transfers.

[0003] Existing face recognition verification technology extracts the features of two faces for similarity comparison and finally returns the corresponding confidence score. The system decides "reject" or "accept" based on the confidence score to determine whether the two input faces belong to the same person. It is suitable for application queries such as identity recognition and similar face query.

[0004] Existing face recognition verification technology is generally a single-person face recognition, which compares the face captured by the recognition camera with the face information of the person in the database. The verification method is relatively simple and cannot support scenarios with two or more people. It is only suitable for personal business. When encountering scenarios that require confirmation by multiple people, there is usually no simultaneous record retention;

[0005] The existing face recognition technology is generally for single-person face recognition, which is fine for the verification of single-person business, but it cannot be used for business that requires multiple people to verify. The present invention can solve business scenarios that require unified verification by multiple people, such as the transfer of shared property between husband and wife, and has higher security. Summary of the invention

[0006] 1. Technical issues to be solved

[0007] In view of the deficiencies in the prior art, the present invention discloses a system and method for simultaneous face recognition and motion recognition verification of multiple people, so as to solve the problem in the above background technology that services requiring verification by multiple people cannot be performed.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a system for multi-person face recognition and action recognition verification, comprising the following steps:

[0010] S1: The user enters the login page and starts using the system. The page displays the login and registration buttons.

[0011] S2: The users who need to be verified log in to the system respectively. The system checks whether each user has been registered. If there is an unregistered user, it enters process S2.1. The user enters the registration page and enters the face information and actions according to the prompts. The user agrees on the meaning of each action. For example, the user sets a thumbs up to represent "agree";

[0012] S3: The system can log in multiple people according to the business scenario. The logged-in users are displayed in a list. After multiple users log in separately, click Next to enter the verification page;

[0013] S4: The camera is turned on. According to the number of logins, the face information is identified through image recognition technology, and the positions of all faces, the number of faces, the approximate positions of several faces, the facial contours are provided, and the user is prompted to place the face within the contours;

[0014] S5: Use the number of detected faces to compare whether the number is consistent with the number of logged-in people. If they are consistent, the images are temporarily stored. If they are inconsistent, proceed to process S5.1: the system prompts the user that the recognition cannot be successfully performed and re-detection is required;

[0015] S6: Cut out the face of the outline, and use face recognition technology to determine whether the face information is consistent with that of each login person. If all are consistent, proceed to the next step. If not, return to the login page and let the user check whether the login person information is correct.

[0016] S7: Prompt the user to make corresponding actions (such as agree, refuse, etc.) in text form on the screen, record the customer's action video, collect action information, save the video to the file server, wait for action comparison, and retain the verification record;

[0017] S8: Read the action meaning list stored in the registration stage according to the registration record, and use the action recognition technology to determine whether each user has made the correct action. If all actions are consistent with the registration record, the verification is passed. If the action made by any user does not meet the registration record, the system prompts that the verification has failed (Note: the action recognition technology is an existing technology, such as the algorithm: openposeplus);

[0018] S9: Jump to S4 or end verification according to business needs.

[0019] Preferably, the method for simultaneous face recognition and motion recognition verification for multiple people includes motion recognition and face recognition.

[0020] Preferably, the action recognition algorithm comprises the following steps:

[0021] (1) Extract the position coordinates of the human joint points in each frame of the video:

[0022] (2) Use the Open-pose method to estimate the posture of the human body in each frame of the video, and obtain the position coordinates of 15 joints of the human body, including the neck, chest, head, right shoulder, left shoulder, right hip, left hip, right elbow, left elbow, right knee, left knee, right wrist, left wrist, right ankle and left ankle. The coordinates of the kth joint point are expressed as Lk = (xk, yk), where k ranges from 1 to 15;

[0023] (3) Normalize the position coordinates of each joint point;

[0024] (4) Use the normalized coordinates of the 15 joint points to form a coordinate matrix P, P = [(x1, y1), (x2, y2), ..., (xk, yk), ..., (x15, y15)], where (xk, yk) represents the normalized coordinates of the kth joint point;

[0025] (5) Calculate the distance change matrix of human joint points in two adjacent frames;

[0026] (6) According to the coordinate matrices Pn and Pn-1 of two adjacent frames, the coordinate change matrix of the joint point position between two adjacent frames is calculated;

[0027] (7) Calculate the joint point distance variation matrix D according to the joint point position coordinate variation matrix;

[0028] (8) Generate video features;

[0029] (9) Divide the video into 4 segments according to its time length, add the distance change matrix D generated by two adjacent frames in each segment, and obtain the cumulative distance change matrix Di of each segment, where i ranges from 1 to 4;

[0030] (10) Perform L2 normalization on Di to obtain the normalized Di';

[0031] (11) Connect the cumulative distance change matrix Di' in series as the feature of the entire video;

[0032] F = [D1', D2', D3', D4'];

[0033] (12) Train the classifier to classify the video;

[0034] (13) Divide the videos of the sub-JHMDB dataset into a training set and a test set, input the features of the training set videos into a support vector machine for training, and obtain a trained support vector machine;

[0035] (14) Input the features of the test set video into the trained support vector machine to obtain the classification result.

[0036] Preferably, the face recognition comprises the following steps:

[0037] (1) The camera is turned on, a face is captured and the face selection box is verified. The face area is detected from the input image, and the coordinates of the face bounding box are returned. The face information is obtained and temporarily stored. If the number of faces is inconsistent, a prompt is given to re-detect. When the number of faces is consistent, the next process is entered.

[0038] (2) Acquisition of facial information and temporary storage. Detection of facial feature points from the face area, and normalization of the face based on the feature points to make the scale and angle of the face area consistent, which is convenient for feature extraction and face matching. The ultimate goal of face alignment is to locate the precise shape of the face in a known face box. It is mainly divided into two categories: optimization-based methods and regression-based methods. The regression tree-based face alignment algorithm here is a facial feature point recognition method published by Vahid Kazemi and Josephine Sullivan in CVPR 2014. It is a regression tree-based face alignment method. This method establishes a cascaded residual regression tree (GBDT) to make the face regress from the current shape to the true shape little by little.

[0039] (3) Feature extraction: extract features from the normalized face area to obtain a feature vector. For example, some deep neural network methods use 128 features to represent faces. The ideal situation is that the feature vectors extracted from photos of different people are different, while similar feature vectors can be extracted from different photos of the same person.

[0040] (4) Face matching: Compare the feature vectors calculated from the two pictures to obtain the similarity score of the two pictures. Based on the similarity score, the one with a high score can be judged as the same person, and the one with a low score can be judged as different people.

[0041] Preferably, the existing face recognition technology is used to identify whether the person performing the action is the person himself.

[0042] The present invention discloses a system and method for multi-person face recognition and action recognition verification, which has the following beneficial effects:

[0043] The system and method for simultaneous face recognition and motion recognition verification of multiple people can simultaneously verify faces and motions, and realize multi-person cooperative verification, which can better represent the attitude of cooperation between two people and effectively reduce the situation of regret afterwards. For example, in a bank counter card opening system, after a user opens a card by himself, the lobby manager is generally required to enter a password on the machine or confirm on an iPad. With the present invention, the lobby manager can make a corresponding confirmation action to confirm during confirmation, and retain image records at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a system diagram of the present invention;

[0045] Figure 2 The user face data table of the present invention;

[0046] Figure 3 The user action data table of the present invention;

[0047] Figure 4 This is the user verification record table of the present invention. DETAILED DESCRIPTION

[0048] The embodiment of the present invention discloses a system and method for multi-person simultaneous face recognition and action recognition verification. Figure 1-4 , a system for multi-person face recognition and action recognition verification includes the following steps:

[0049] like Figure 1 As shown:

[0050] S1: The user enters the login page and starts using the system. The page displays the login and registration buttons.

[0051] S2: The users who need to be verified log in to the system respectively. The system checks whether each user has been registered. If there is an unregistered user, it enters process S2.1. The user enters the registration page and enters the face information and actions according to the prompts. The user agrees on the meaning of each action. For example, the user sets a thumbs up to represent "agree";

[0052] S3: The system can log in multiple people according to the business scenario. The logged-in users are displayed in a list. After multiple users log in separately, click Next to enter the verification page;

[0053] S4: The camera is turned on. According to the number of logins, the face information is identified through image recognition technology, and the positions of all faces, the number of faces, the approximate positions of several faces, the facial contours are provided, and the user is prompted to place the face within the contours;

[0054] S5: Use the number of detected faces to compare whether the number is consistent with the number of logged-in people. If they are consistent, the images are temporarily stored. If they are inconsistent, proceed to process S5.1: the system prompts the user that the recognition cannot be successfully performed and re-detection is required;

[0055] S6: Cut out the face of the outline, and use face recognition technology to determine whether the face information is consistent with that of each login person. If all are consistent, proceed to the next step. If not, return to the login page and let the user check whether the login person information is correct.

[0056] S7: Prompt the user to make corresponding actions (such as agree, refuse, etc.) in text form on the screen, record the customer's action video, collect action information, save the video to the file server, wait for action comparison, and retain the verification record;

[0057] S8: Read the action meaning list stored in the registration stage according to the registration record, and use the action recognition technology to determine whether each user has made the correct action. If all actions are consistent with the registration record, the verification is passed. If the action made by any user does not meet the registration record, the system prompts that the verification has failed (Note: the action recognition technology is an existing technology, such as the algorithm: openposeplus);

[0058] S9: Jump to S4 or end verification according to business needs.

[0059] The method for simultaneous face recognition and motion recognition verification for multiple people includes motion recognition and face recognition.

[0060] The action recognition algorithm comprises the following steps:

[0061] (1) Extract the position coordinates of the human joint points in each frame of the video:

[0062] (2) Use the Open-pose method to estimate the posture of the human body in each frame of the video, and obtain the position coordinates of 15 joints of the human body, including the neck, chest, head, right shoulder, left shoulder, right hip, left hip, right elbow, left elbow, right knee, left knee, right wrist, left wrist, right ankle and left ankle. The coordinates of the kth joint point are expressed as Lk = (xk, yk), where k ranges from 1 to 15;

[0063] (3) Normalize the position coordinates of each joint point;

[0064] (4) Use the normalized coordinates of the 15 joint points to form a coordinate matrix P, P = [(x1, y1), (x2, y2), ..., (xk, yk), ..., (x15, y15)], where (xk, yk) represents the normalized coordinates of the kth joint point;

[0065] (5) Calculate the distance change matrix of human joint points in two adjacent frames;

[0066] (6) According to the coordinate matrices Pn and Pn-1 of two adjacent frames, the coordinate change matrix of the joint point position between two adjacent frames is calculated;

[0067] (7) Calculate the joint point distance variation matrix D according to the joint point position coordinate variation matrix;

[0068] (8) Generate video features;

[0069] (9) Divide the video into 4 segments according to its time length, add the distance change matrix D generated by two adjacent frames in each segment, and obtain the cumulative distance change matrix Di of each segment, where i ranges from 1 to 4;

[0070] (10) Perform L2 normalization on Di to obtain the normalized Di';

[0071] (11) Connect the cumulative distance change matrix Di' in series as the feature of the entire video;

[0072] F = [D1', D2', D3', D4'];

[0073] (12) Train the classifier to classify the video;

[0074] (13) Divide the videos of the sub-JHMDB dataset into a training set and a test set, input the features of the training set videos into a support vector machine for training, and obtain a trained support vector machine;

[0075] (14) Input the features of the test set video into the trained support vector machine to obtain the classification result.

[0076] It can perform face and motion verification simultaneously and realize multi-person collaborative verification, which can better represent the attitude of cooperation between two people and effectively reduce the situation of regret afterwards.

[0077] The face recognition comprises the following steps:

[0078] (1) The camera is turned on, a face is captured and the face selection box is checked. The face area is detected from the input image, and the coordinates of the face bounding box are returned. The face information is obtained and temporarily stored. If the number of faces is inconsistent, a prompt is given to re-detect. When the number of faces is consistent, the next process is entered.

[0079] (2) Acquisition of facial information and temporary storage. Detection of facial feature points from the face area, and normalization of the face based on the feature points to make the scale and angle of the face area consistent, which is convenient for feature extraction and face matching. The ultimate goal of face alignment is to locate the precise shape of the face in a known face box. It is mainly divided into two categories: optimization-based methods and regression-based methods. The regression tree-based face alignment algorithm here is a facial feature point recognition method published by Vahid Kazemi and Josephine Sullivan in CVPR 2014. It is a regression tree-based face alignment method. This method establishes a cascaded residual regression tree (GBDT) to gradually regress the face from the current shape to the true shape.

[0080] (3) Feature extraction: extract features from the normalized face region to obtain a feature vector. For example, some deep neural network methods use 128 features to represent faces. The ideal situation is that the feature vectors extracted from photos of different people are different, while similar feature vectors can be extracted from different photos of the same person.

[0081] (4) Face matching: Compare the feature vectors calculated from the two images to obtain the similarity scores of the two photos. Based on the similarity scores, the ones with higher scores can be judged as the same person, while the ones with lower scores can be judged as different people.

[0082] The existing face recognition technology is used to identify whether the person doing the action is the person himself.

[0083] The face and action verification can be performed simultaneously, and multi-person cooperative verification can be achieved, which can better represent the attitude of cooperation between two people and effectively reduce the situation of regret afterwards. For example, in the bank counter card opening system, after the user opens the card by himself, the lobby manager generally needs to enter the password on the machine or confirm on the iPad. With the present invention, the lobby manager can make the corresponding confirmation action to confirm, and keep the image record at the same time.

[0084] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for multi-person face recognition and action recognition verification, characterized in that: The steps include: S1: The user enters the login page and starts using the system. The page displays the login and registration buttons. S2: The users who need to be verified log in to the system respectively. The system checks whether each user has been registered. If there is an unregistered user, it enters process S2.

1. The user enters the registration page and enters the face information and actions according to the prompts. The user agrees on the meaning of each action. The user sets the thumbs up to represent "agree"; S3: The system can log in multiple people according to the business scenario. The logged-in users are displayed in a list. After multiple users log in separately, click Next to enter the verification page; S4: The camera is turned on. According to the number of logins, the face information is identified through image recognition technology, and the positions of all faces, the number of faces, the approximate positions of several faces, the facial contours are provided, and the user is prompted to place the face within the contours; S5: Use the number of detected faces to compare whether the number is consistent with the number of logged-in people. If they are consistent, the images are temporarily stored. If they are inconsistent, proceed to process S5.1: the system prompts the user that the recognition cannot be successfully performed and re-detection is required; S6: Cut out the face of the outline, and use face recognition technology to determine whether the face information is consistent with that of each login person. If all are consistent, proceed to the next step. If not, return to the login page and let the user check whether the login person information is correct. S7: Prompt the user to take corresponding actions in text form on the screen, agree or refuse, record the customer's action video, collect action information, save the video to the file server, wait for action comparison, and retain the verification record; S8: Read the action meaning list stored in the registration stage according to the registration record, and use the action recognition technology to determine whether each user has made the correct action. If all actions are consistent with the registration record, the verification is passed. If the action made by any user does not meet the registration record, the system prompts that the verification has failed. The action recognition technology is the openpose plus algorithm; S9: Jump to S4 or end verification according to business needs.

2. The method for simultaneous face recognition and motion recognition verification of multiple persons according to claim 1, characterized in that: The method for simultaneous face recognition and motion recognition verification of multiple people includes motion recognition and face recognition.

3. The method for simultaneous face recognition and action recognition verification of multiple persons according to claim 2, characterized in that: The action recognition algorithm comprises the following steps: (1) Extract the position coordinates of the human joint points in each frame of the video: (2) Use the Open-pose method to estimate the posture of the human body in each frame of the video, and obtain the position coordinates of 15 joints of the human body, including the neck, chest, head, right shoulder, left shoulder, right hip, left hip, right elbow, left elbow, right knee, left knee, right wrist, left wrist, right ankle and left ankle. The coordinates of the kth joint point are expressed as Lk = (xk, yk), where k ranges from 1 to 15; (3) Normalize the position coordinates of each joint point; (4) Use the normalized coordinates of the 15 joint points to form a coordinate matrix P, P = [(x1, y1), (x2, y2), ..., (xk, yk), ..., (x15, y15)], where (xk, yk) represents the normalized coordinates of the kth joint point; (5) Calculate the distance change matrix of human joint points in two adjacent frames; (6) According to the coordinate matrices Pn and Pn-1 of two adjacent frames, the coordinate change matrix of the joint point position between two adjacent frames is calculated; (7) Calculate the joint point distance variation matrix D according to the joint point position coordinate variation matrix; (8) Generate video features; (9) Divide the video into 4 segments according to its time length, add the distance change matrix D generated by two adjacent frames in each segment, and obtain the cumulative distance change matrix Di of each segment, where i ranges from 1 to 4; (10) Perform L2 normalization on Di to obtain the normalized Di'; (11) Connect the cumulative distance change matrix Di' in series as the feature of the entire video; F = [D1', D2', D3', D4']; (12) Train the classifier to classify the video; (13) Divide the videos of the sub-JHMDB dataset into a training set and a test set, input the features of the training set videos into a support vector machine for training, and obtain a trained support vector machine; (14) Input the features of the test set video into the trained support vector machine to obtain the classification result.

4. The method for simultaneous face recognition and action recognition verification of multiple persons according to claim 2, characterized in that: The face recognition comprises the following steps: (1) The camera is turned on, a face is captured and the face selection box is verified. The face area is detected from the input image, and the coordinates of the face bounding box are returned. The face information is obtained and temporarily stored. If the number of faces is inconsistent, a prompt is given to re-detect. When the number of faces is consistent, the next process is entered. (2) Obtaining facial information and temporarily storing it. Detecting facial feature points from the face area, and normalizing the face based on the feature points to make the scale and angle of the face area consistent, which is convenient for feature extraction and face matching. The ultimate goal of face alignment is to locate the precise shape of the face in a known face box. It is mainly divided into two categories: optimization-based methods and regression-based methods. The regression tree-based face alignment algorithm here is a facial feature point recognition method published by Vahid Kazemi and Josephine Sullivan in CVPR 2014. It is a regression tree-based face alignment method. This method establishes a cascaded residual regression tree (GBDT) to make the face regress from the current shape to the true shape little by little; (3) Feature extraction: extract features from the normalized face region to obtain a feature vector. For example, some deep neural network methods use 128 features to represent faces. The ideal situation is that the feature vectors extracted from photos of different people are different, while similar feature vectors can be extracted from different photos of the same person. (4) Face matching: Compare the feature vectors calculated from the two pictures to obtain the similarity scores of the two pictures. Based on the similarity scores, the one with a high score can be judged as the same person, and the one with a low score can be judged as different people.

5. The method for simultaneous face recognition and action recognition verification of multiple persons according to claim 4, characterized in that: The existing face recognition technology is used to identify whether the person doing the action is the person himself.

Images