Interview cheating detection method and system based on human facial motion unit

Abstract

A method and system for detecting cheating in an interview based on human facial motion units, the method comprising the steps of: performing face recognition on video data collected during an interview, extracting facial features and encoding facial motion units, head motion units and / or eye motion units; using a trained classifier to identify the combination of the facial features and facial motion units, head motion units and / or eye motion units, and determining whether cheating exists. The cheating detection method of the present invention obtains detection results by detecting, extracting features and classifying facial motion units, and using a machine learning classification algorithm, which is not affected by the personal relationships or emotions of the invigilators, and ensures the relative fairness of the monitoring results; the present invention improves the efficiency of interview judgment, and the error rate is also better than the original manual judgment method.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, in particular to the field of machine learning assisted technology, and more specifically to an interview cheating detection method and system based on human facial motion units. Background Art

[0002] Face recognition, also known as facial recognition system, refers to computer technology that uses analysis and comparison of facial visual feature information to identify individuals. It is a type of biometric recognition technology. In a broad sense, face recognition includes a series of related technologies for building a face recognition system, including face image acquisition, face positioning, face recognition preprocessing, identity confirmation, and identity search; while in a narrow sense, face recognition specifically refers to the technology or system for identity confirmation or identity search through the face.

[0003] The research on face recognition system began in the 1960s, when researcher Bledsoe established a semi-automatic face recognition system. At that time, face recognition research aroused great interest from researchers in many disciplines. After the 1980s, it was improved with the development of computer technology and optical imaging technology. After entering the 1990s, with the urgent demand for face recognition systems in various industries, face recognition research became a hot topic again, and finally it was mainly realized by technology from the United States, Germany and Japan. The key to the success of a face recognition system lies in whether it has a cutting-edge core algorithm and whether the recognition results have a practical recognition rate and speed. After more than 30 years of research, face recognition has become one of the most successful applications in the field of image analysis and image understanding.

[0004] At present, facial recognition technology is commonly used in personal identity confirmation, such as used as a power-on password, for personal payment, and personal identity search, such as online pursuit, public place monitoring and other technical fields. In application scenarios such as interviews and written tests, facial recognition technology is only used to determine whether the person taking the exam is the real person. Facial recognition technology is used to determine whether there is cheating based on the expressions and movements of the interviewees. No related reports have been found so far. Summary of the invention

[0005] In view of this, the main purpose of the present invention is to propose an interview cheating detection method and system based on human facial motion units, in order to at least partially solve at least one of the above-mentioned technical problems.

[0006] In order to achieve the above object, as a first aspect of the present invention, a method for detecting cheating in an interview based on a human facial motion unit is provided, comprising the following steps:

[0007] Performing face recognition on the video data collected during the interview, extracting facial features and encoding facial motion units, head motion units and / or eye motion units;

[0008] The trained classifier is used to identify the combination of the above-mentioned facial features and facial motion units, head motion units and / or eye motion units to determine whether cheating occurs.

[0009] As a second aspect of the present invention, there is also provided an interview cheating detection system based on human facial motion unit, comprising:

[0010] A face recognition unit, used to perform face recognition on the video data collected during the interview, extract facial features and encode facial motion units;

[0011] The classification and discrimination unit is used to use the trained classifier to identify the above-mentioned facial features and facial motion units to determine whether there is cheating.

[0012] A third aspect of the present invention proposes an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer executable program is executed by the processor, the processor executes the interview cheating detection method based on human facial motion units as described above.

[0013] The fourth aspect of the present invention further proposes a computer-readable medium storing a computer executable program, which, when executed, implements the interview cheating detection method based on human facial motion units as described above.

[0014] Based on the above technical solution, it can be seen that the interview cheating detection method and system based on human facial motion unit of the present invention has at least one of the following beneficial effects compared with the prior art:

[0015] 1. The present invention solves the problem of low efficiency and high misjudgment rate in the prior art of manually judging the cheating behavior of interviewees;

[0016] 2. The present invention obtains the detection results by detecting, extracting features and classifying facial motion units, and using a machine learning classification algorithm. The detection results are not affected by the personal relationships or emotions of the invigilators, thus ensuring the relative fairness of the monitoring results.

[0017] 3. The present invention combines a neural network and a classifier trained with massive data to automatically screen interview videos, improving the efficiency of interview judgment. The speed is much faster than that of manual methods, and with the support of sufficient labeled data, the misjudgment rate can also be reduced to a very low level, which is better than the original manual judgment method. According to the inventor's experimental research, the current accuracy of the manual method is about 50%, and the judgment time for a single video is 3 minutes. After adopting the method of the present invention, the accuracy can be increased to 82%, and the judgment time for a single video is less than 1 second, which greatly improves the efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a list of specific manifestations of the motion features of the expressive face;

[0019] Figure 2 is a flow chart of the interview cheating detection method based on human facial motion unit of the present invention;

[0020] Figure 3 It is a network structure diagram of a neural network model of an interview cheating detection system based on human facial motion units of the present invention;

[0021] Figure 4 It is a simplified framework diagram of the network structure adopted in the video classification step of the interview cheating detection system based on human facial motion unit of the present invention;

[0022] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention;

[0024] Figure 7 It is the main facial action coding diagram involved in the present invention;

[0025] Figure 8 It is the main head movement unit coding diagram involved in the present invention;

[0026] Fig. 9 It is the main eye movement unit coding diagram involved in the present invention.

[0027] Fig.10 It is the basic expression unit coding diagram involved in the present invention. DETAILED DESCRIPTION

[0028] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0029] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0030] Frames in the attached figure Figure 1Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0031] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0032] Face recognition, also known as facial recognition system, refers to computer technology that uses analysis and comparison of facial visual feature information to identify individuals. It is a type of biometric recognition technology. After more than 30 years of research, face recognition has become one of the most successful applications in the field of image analysis and image understanding.

[0033] Generally speaking, face recognition systems include image acquisition, face location, image preprocessing, and face recognition (identity confirmation or identity search). The system input is generally one or a series of face images with undetermined identities, as well as several face images of known identities in the face database or corresponding codes, and its output is a series of similarity scores indicating the identity of the face to be recognized.

[0034] At present, face recognition algorithms can be classified into: recognition algorithms based on facial feature points (feature-basedrecognition algorithms), recognition algorithms based on the entire face image (appearance-basedrecognition algorithms), template-based recognition algorithms (template-based recognitionalgorithms), recognition algorithms using neural networks (recognition algorithms using neuralnetwork) and recognition algorithms using support vector machines (recognition algorithms using SVM), etc.

[0035] Although the research and development of face recognition technology is in the ascendant, it is rare to see face recognition technology applied to the identification of cheating. The reasons are that cheating actions are usually small in amplitude and fast in speed. The general video acquisition system has a low frame rate and cannot detect related actions; another is that cheating actions are often confused with other unconscious small movements, which are difficult to train using machine learning and the recognition rate is not high. After a long period of research, the inventor has taken a different approach. On the basis of existing face recognition technology, the inventor has proposed a method for detecting cheating in interviews based on human facial motion units in order to solve the problem of low efficiency and high error rate in the application scenario of traditional manual methods for screening cheating in interview videos. This method combines neural networks and classifiers trained with massive data to automatically screen interview videos at a speed far exceeding that of manual methods. With the support of sufficient labeled data, its error rate can also be reduced to a very low level, which is significantly better than manual methods.

[0036] Specifically, the present invention uses a convolutional neural network to detect faces in interview videos. After detecting faces, various movements of the faces, including facial motion units, head movements, eye movements, etc., are detected separately. Unlike conventional machine learning, which directly trains with overall facial movements and expressions, the present invention encodes the motion units after detecting them, and annotates the partial combined codes to obtain a large amount of sample data, and then uses machine learning classification methods to train a classifier, which is a classifier that can identify whether a person in the video is cheating. These combined codes are obtained through a lot of practice and machine learning by the inventor.

[0037] Therefore, the training method of the neural convolutional network adopted includes:

[0038] Constructing a neural network model for identifying cheating, wherein the neural network model is preferably a convolutional neural network model;

[0039] The neural network model is used to perform face detection on interview videos, especially high frame rate interview videos (conventionally 24 frames per second, preferably greater than or equal to 36 frames per second, for example 50 frames per second, 200 frames per second). After the face is detected, the face including facial motion units, head motion units and / or eye motion units are detected separately, and special data annotation is performed based on the combined coding of specific different parts obtained in the study to obtain a large amount of sample data, and then a machine learning classification method is used to train a classifier.

[0040] Therefore, the corresponding identification method steps specifically include the following steps:

[0041] Perform face recognition on the video data collected during the interview, extract facial features and encode facial motion units;

[0042] The classifier trained as above is used to identify the above-mentioned facial features and facial motion units, head motion units and / or eye motion units to determine whether there is cheating.

[0043] Among them, facial features, namely facial key points, are important feature points of various parts of the face, usually contour points and corner points. Facial key point detection is a key step in the field of face recognition and analysis. It is the premise and breakthrough for other face-related problems such as automatic face recognition, expression analysis, 3D face reconstruction and 3D animation. In recent years, deep learning methods have been successfully applied to many fields such as image recognition and analysis, speech recognition and natural language processing due to their automatic learning and continuous learning capabilities, and have brought significant improvements in these aspects. Facial key point detection can locate the key points of a face on a given face image, including eyebrows, eyes, nose, mouth, and points in the facial contour area. Due to the influence of factors such as posture and occlusion, facial key point detection is a challenging task. It currently has the following main applications: (1) Face posture alignment. Algorithms such as face recognition need to align the posture of the face to improve the accuracy of the model; (2) Face beautification and editing. Based on key points, the face shape, eye shape, nose shape, etc. can be accurately analyzed, so as to modify and process specific positions of the face, realize special effects beautification, patch and other entertainment functions of the face, and can also assist some face editing algorithms to play a better role; (3) Facial expression analysis. Based on key points, facial expressions of people can be analyzed, so as to be used in interactive entertainment, behavior prediction and other scenarios. The present invention is designed and developed based on the application of point (3).

[0044] The detection of facial features may use, for example, image pyramid technology to further optimize the recognition of facial features and improve classification accuracy.

[0045] For example, a multi-task cascade convolutional network (MTCNN) can be used to build a neural network model to perform face region detection and face key point detection simultaneously. The multi-task cascade convolutional network model mainly uses three cascade networks, adopts the idea of ​​candidate boxes plus classifiers, and performs fast and efficient face detection. It can be generally divided into a three-layer network structure of P-Net, R-Net and O-Net, which are P-Net for fast generation of candidate windows, R-Net for high-precision candidate window filtering and selection, and O-Net for generating the final bounding box and face key points. It uses technologies such as picture pyramid, bounding-box regression, and non-maximum suppression (NMS).

[0046] Among them, P-Net stands for Proposal Network. Its basic structure is a fully convolutional network. It uses a full neural network (FCN) to perform preliminary feature extraction and border calibration on the image pyramid built in the previous step, and performs border regression to adjust the window and NMS to filter most windows. The basic idea is to use a shallower and simpler CNN to quickly generate face candidate windows.

[0047] R-Net is short for Refine Network. Its basic structure is a convolutional neural network. Compared with the first layer of P-Net, it adds a fully connected layer, so the screening of input data is more stringent. The basic idea is to use a more complex network structure than P-Net to further select and adjust the possible face area windows generated by P-Net, so as to achieve high-precision filtering and face area optimization.

[0048] Among them, O-Net stands for Output Network. Its basic structure is a more complex convolutional neural network with one more convolutional layer than R-Net. The difference between O-Net and R-Net is that this layer of structure will identify the facial area through more supervision, and will regress the facial feature points of the person, and finally output five facial feature points of the person. The basic idea of ​​O-Net is similar to that of R-Net, using a more complex network to optimize the model performance.

[0049] The detailed reasoning process of face recognition using a multi-task cascade convolutional network includes:

[0050] Step 1: Continuously resize the test image to obtain an image pyramid. Resize the test image according to resize_factor (usually between 0.70-0.80) until it is larger than or equal to the 12*12 size required by Pnet, thereby obtaining images of different sizes such as original image, original image*resize_factor, original image*resize_factor^2..., original image*resize_factor^n, etc.

[0051] Step 2: Input the obtained image pyramid into Pnet to obtain a large number of candidates, and the output map shape is (m, n, 16). According to the classification score, most of the candidates are filtered out, and then the bbox is calibrated according to the 4 offsets to obtain the upper left and lower right coordinates of the bbox. These candidates are subjected to non-maximum suppression (NMS) according to the IOU value to filter out most of the candidates. Repeat this operation and finally obtain (num_left_after_nms, 16) candidates.

[0052] Step 3: According to the coordinates output by Pnet, cut out the image from the original image, and cut out the image using the square with the maximum side length to avoid deformation and retain more details. Then resize it to 24*24 and input it to Rnet for fine-tuning. Rnet will still output 2 binary one-hot outputs, 4 bbox coordinate offset outputs, and 10 landmark outputs. Most of the candidates that are not faces are eliminated based on the binary classification scores, and the bbox of the screenshot is adjusted for offset. Repeat the IOU NMS described in Pnet again to eliminate most of the candidates. Finally, Rnet also outputs (num_left_after_Rnet,16) candidates. Cut out the image from the original image based on the bbox coordinates and input it to Onet. It also cuts out the image using the square with the maximum side length to avoid deformation and retain more details.

[0053] Step 4: Repeat the Pnet process roughly, except that in addition to paying attention to the coordinates of the bbox, the coordinates of the landmarks must also be output. After classification screening and NMS screening after frame adjustment, accurate face bbox coordinates and landmark points are obtained, and the task is successfully completed.

[0054] Among them, for example, the Facial Action Coding System (FAC) and the Facial Action Coding System (FACS) are used to construct training samples. The FACS includes at least 6 basic expressions (i.e., happiness, sadness, surprise, fear, anger, and disgust), and systematically establishes thousands of different facial expression image libraries; and the FACS divides the human face into about 46 independent and interconnected motion units (AUs) according to the anatomical characteristics of the human face, and analyzes the motion characteristics of these motion units, the main areas they control, and the expressions related thereto, and provides a large number of photo illustrations. In the present invention, facial movements, head movements, and eye movements are further optimized and encoded, such as Figure 7 , Figure 8 , Fig. 9 As shown. Basic expression coding is as follows Fig.10 As shown in the figure, for example, "6+12" is used to represent "happy", and the corresponding table shows "raised cheeks" + "upturned corners of mouth". For cheating actions, they are also decomposed into corresponding small expressions or actions to avoid the overall action video being easily confused with other inadvertent small actions and reducing the recognition accuracy.

[0055] It should be noted that the human face has 42 muscles, which are controlled by different areas of the brain. Some muscles can be directly controlled by consciousness, while others are not easy to control by consciousness. Muscles that can be directly controlled by consciousness are called "voluntary muscles". Muscles that most people cannot control by consciousness are called "involuntary muscles". According to the characteristics of human muscles, facial behaviors can be divided into 46 facial movement units (Action Unit), 8 head movement units and 4 eye movement units ( Figure 7-9 Through the combination of these facial movement units, head movement units and eye movement units, people's emotions and facial movements can be recognized, so as to make judgments on their behaviors.

[0056] By using the facial behavior coding system and the facial movement coding system, the source of model training samples can be richer, the training samples can be more accurate, and the results can be more precise.

[0057] Among them, in the encoding step of the facial motion unit, it is necessary to locate the 68 key points of the face through the landmark obtained in the previous step, and then independently detect them according to different area ROIs. Finally, the detection of each ROI is summarized to obtain the detection result of the whole face.

[0058] Among them, the above encoding step also includes grouping different AUs according to the prior knowledge of facial motion units (Action Units). AUs occurring in the same position area are grouped together, and faces can be divided into 9 groups such as left eye, right eye, and nose wing. Then, based on the correspondence between facial motion units and facial key points, the AU result of a single face is obtained.

[0059] The classifier is trained using the following method:

[0060] A two-channel convolutional neural network is constructed, with one channel input as the video stream and the other channel input as the AU-encoded matrix obtained above. The spatial semantic information of the face, head, and eyes is captured by encoding the facial motion unit, head motion unit, and eye motion unit, and the motion information at fine temporal resolution is captured by high frame rate video. Then, at the decoding layer, the feature maps obtained from the two channels are fused, and finally the classification result of whether the video interviewee cheats is obtained.

[0061] The present invention also discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer executable program, wherein when the computer executable program is executed by the processor, the processor executes the interview cheating detection method based on human facial motion units as described above.

[0062] The electronic device may be, for example, in the form of a general-purpose computing device. The processor may be one or more than one processor working in coordination. The present invention does not exclude distributed processing, that is, the processor may be distributed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0063] The memory stores a computer executable program, usually a machine-readable code, which can be executed by the processor to enable the electronic device to execute the method of the present invention, or at least some of the steps in the method.

[0064] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0065] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0066] The electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0067] The present invention also discloses a computer-readable medium on which a computer executable program is stored, wherein when the computer executable program is executed, the interview cheating detection method based on human facial motion units as described above is implemented.

[0068] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0069] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Python, Java, C++, etc., as well as conventional procedural programming languages ​​such as C, assembly language, or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0070] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0071] Example 1

[0072] In this embodiment, the training data used is self-labeled video data, totaling about 5,000,000 videos, with a total video length of about 2,500,000 hours.

[0073] 1. Face recognition and feature point extraction

[0074] System flow chart as follows Figure 2 As shown, the present invention first performs face recognition on the input video and detects the coordinates of the face position in the video. The face recognition adopts a multi-task cascade convolutional network.

[0075] The reasoning process of this multi-task cascade convolutional network is as follows:

[0076] Step 1: First, continuously resize the test image to obtain an image pyramid.

[0077] resize_factor can be determined according to the face size distribution of the dataset, and is usually set between 0.70-0.80. If it is set too large, it is easy to prolong the inference time, and if it is too small, it is easy to miss some small and medium-sized faces. Resize the test image according to the set resize_factor until it is greater than or equal to the 12*12 size required by Pnet. In this way, a series of pictures of different sizes such as original image, original image*resize_factor, original image*resize_factor^2..., original image*resize_factor^n can be obtained. Stacked together, it looks like a pyramid, so it is called a picture pyramid. In the above resizing step, the size of the last picture will be greater than or equal to 12. It should be noted that these images must be input into Pnet one by one to obtain candidates.

[0078] Input the obtained image pyramid into Pnet to obtain a large number of candidates. According to the image pyramid obtained in step 1 above, input all images into Pnet, and the output map shape is (m, n, 16). According to the classification score, filter out a large part of the candidates, and then calibrate the bbox according to the 4 offsets to obtain the upper left and lower right coordinates of the bbox (according to the offset correction, I will bury a pit first, and fill it in when describing the training stage), and perform non-maximum suppression (NMS) on these candidates according to the IOU value to filter out a large part of the candidates. Specifically, it is to sort from large to small according to the classification score, and obtain a tensor of (num_left, 4), that is, the upper left and lower right absolute coordinates of num_left bboxes. Each time, the IOU is calculated with the bbox coordinates of the maximum score value in the queue and the remaining coordinates, and the box with an IOU greater than 0.6 is removed, and the maximum score value is moved to the final result; among them, the threshold is set in advance. Repeating this operation will remove many bboxes with a lot of overlap, and finally get (num_left_after_nms, 16) candidates. These candidates need to be cut out of the original image according to the bbox coordinates, resized to 24*24 and input into Rnet.

[0079] The candidate images selected by Pnet are fine-tuned by Rnet. According to the coordinates output by Pnet, the image is cut out from the original image. When cutting out the image, the bbox needs to be cut out as a square with the maximum side length to ensure that no deformation occurs during resizing and retain more details around the face frame. The image is resized to 24*24 and input to Rnet for fine-tuning. Rnet will still output 2 binary one-hot outputs, 4 bbox coordinate offset outputs, and 10 landmark outputs. Most of the candidates that are not faces are removed according to the binary classification scores, and the bbox of the screenshot is offset, that is, the x and y coordinates of the upper left and lower right are adjusted up and down, and the IOU NMS described in Pnet is repeated again to remove most of the candidates. Finally, Pnet outputs (num_left_after_Rnet,16). According to the coordinates of the bbox, the image is cut out from the original image and input into Onet. The method of cutting out the image according to the square with the maximum side length is also used to avoid deformation and retain more details.

[0080] After Rnet eliminates many candidates, the image is input to Onet, which outputs accurate bbox coordinates and landmark coordinates. The process of Pnet can be roughly repeated, but the difference is that in addition to paying attention to the bbox coordinates, the landmark coordinates must also be output. After classification screening, frame adjustment and NMS screening, accurate face bbox coordinates and landmark points are obtained, and the task is successfully completed.

[0081] 2. Facial motion unit encoding

[0082] The 68 key points of the face are located through the landmarks obtained in the previous step, and then independent detection is performed according to different area ROIs. Finally, the detection of each ROI is summarized to obtain the detection result of the whole face.

[0083] According to the prior knowledge of facial action units, different AUs are grouped. AUs occurring in the same position area are grouped together, and faces can be divided into 9 groups such as left eye, right eye, nose wing, etc. Then, according to the correspondence between facial action units and facial key points, the AU results of a single face are obtained.

[0084] 3. Video Classification

[0085] Construct a two-channel convolutional neural network such as Figure 4, one channel input is the video stream, and the other channel input is the matrix obtained after AU encoding. The facial spatial semantic information is captured by encoding the facial motion unit, and the motion information at fine temporal resolution is captured by high frame rate video. Then in the decoding layer, the feature maps obtained from the two channels are fused, and finally it is concluded whether the video interviewee cheats. Among them, Figure 4 The specific parameters of the neural network model are obtained through training, and they are all hidden layers.

[0086] The dual-channel convolutional neural network trained in this embodiment can be used in interviews or examinations. The micro-expressions of the interviewer or the examinee can be monitored in real time through video acquisition equipment, and the slight changes in their expressions and movements can be used to determine whether they have committed cheating. Compared with the traditional classifier trained purely by machine learning of movements, the dual-channel convolutional neural network trained in this embodiment has a faster computing speed and higher accuracy.

[0087] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0088] It can be seen that the present invention aims at the problem that the existing interview system mainly judges the interviewee's cheating behavior manually with low efficiency and high error rate, and proposes an interview cheating detection method and system based on facial motion units. It obtains the interviewee's current emotions and facial movements through the detection, feature extraction and classification of facial motion units, and then obtains the cheater's feature information through a large amount of data training, and obtains the result using the classification algorithm of machine learning. The practical results show that the present invention improves the efficiency of interview judgment, and the error rate is better than the original manual judgment method.

[0089] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting cheating in an interview based on human facial motion units, characterized in that: include: Perform face recognition on the video data collected during the interview to determine the face coordinates. Use the multi-task cascade convolutional network MTCNN to build a neural network model. Use the image pyramid to perform face region detection and face key point detection based on facial expression analysis on the recognized face at the same time. Locate the detected face key points and perform independent detection of different regions. Aggregate the detection of each region to obtain all the face key points and extract them as face features. The neural network model has three cascaded networks, which are divided into a three-layer network structure of P-Net, R-Net and O-Net. P-Net performs preliminary feature extraction and border calibration on the input image pyramid through the full neural network FCN, R-Net selects and adjusts the window estimated to be the face area generated by P-Net, and O-Net identifies the face area through more supervision and regresses the facial feature points of the person and finally outputs the facial feature points of the face, i.e., the facial key points; The facial motion units are grouped according to their positions and the motion unit results of a single face are obtained according to the correspondence between the facial motion units and the key points of the face. The facial motion units are encoded to obtain an encoded matrix, and the spatial semantic information of the face, head and eyeballs is captured by encoding the facial motion units, head motion units and eyeball motion units. The trained classifier is used to identify the coding combination of facial features and facial motion units, head motion units and / or eye motion units. The video stream and the coding combination are input respectively. The facial spatial semantic information is captured by encoding and the motion information with fine temporal resolution is captured by high-resolution video. The obtained feature maps are fused to determine whether the interviewee in the video is cheating.

2. The method according to claim 1, characterized in that Also includes Before detecting facial key points, the test image needs to be continuously resized to obtain the image pyramid.

3. The method according to claim 1, characterized in that Also includes: In the process of detecting facial key points, a border regression algorithm is used to implement border regression adjustment window and / or non-maximum suppression NMS is used to perform most window filtering.

4. The method according to claim 1, characterized in that: Also includes: The trained classifier uses a facial behavior coding system and a facial movement coding system to construct training samples.

5. The method according to claim 1, characterized in that Also includes: In the face recognition step, facial behavior is divided into 46 facial movement units, 8 head movement units and 4 eye movement units.

6. The method according to claim 4, characterized in that Also includes: When encoding the facial motion unit, it is necessary to locate the 68 key points of the face through the landmark obtained in the previous step, and then perform independent detection according to different area ROIs. Finally, the detection of each ROI is summarized to obtain the detection result of the whole face.

7. The method according to claim 4, characterized in that Also includes: The training of the classifier also includes grouping different facial motion units according to prior knowledge of facial motion units; and / or Facial motion units occurring in the same position area are grouped together, and the face is divided into nine groups including the left eye, right eye, and nose wing.

8. The method according to claim 1, characterized in that The classifier is trained using the following method: A dual-channel convolutional neural network is constructed, with one channel input as the video stream and the other channel input as the matrix of facial motion unit encoding obtained above. After capturing the facial spatial semantic information through the encoding of facial motion units, head motion units and eye motion units, high frame rate video is used to capture motion information with fine temporal resolution. Then, at the decoding layer, the feature maps obtained from the two channels are fused, and finally a classification result is obtained as to whether the video interviewee has cheated.

9. An interview cheating detection system based on human facial motion units, characterized in that: include: The face recognition unit is used to perform face recognition on the video data collected during the interview, determine the face coordinate position, use the multi-task cascade convolutional network MTCNN to build a neural network model, and use the image pyramid to simultaneously perform face region detection and face key point detection based on face expression analysis on the recognized face, so as to locate the detected face key points and independently detect different regions, and summarize the detection of each region to obtain all the face key points and extract them as face features; The facial motion units are grouped according to the positions and the motion unit results of a single face are obtained according to the correspondence between the facial motion units and the key points of the face, the facial motion units are encoded to obtain the encoded matrix, and the spatial semantic information of the face, head and eyeballs is captured by encoding the facial motion units, head motion units and eyeball motion units; wherein the neural network model has three cascaded networks, which are divided into a three-layer network structure of P-Net, R-Net and O-Net, P-Net performs preliminary feature extraction and border calibration on the constructed image pyramid through the full neural network FCN, R-Net selects and adjusts the window estimated to be the face area generated by P-Net, and O-Net identifies the face area through more supervision and regresses the facial feature points of the person and finally outputs the facial feature points of the face, i.e., the facial key points; The classification and discrimination unit is used to use the trained classifier to identify the above-mentioned facial features and facial motion unit coding combinations, input the video stream and the coding combination respectively, capture the facial spatial semantic information through coding and the motion information at fine time resolution captured by high-resolution video, fuse the obtained feature maps, and judge whether the interviewee in the video is cheating.

10. The system according to claim 9, characterized in that Also includes: Before detecting facial key points, the test image needs to be continuously resized to obtain an image pyramid.

11. The system according to claim 9, characterized in that In the process of detecting facial key points, a border regression algorithm is used to implement border regression adjustment window and / or a non-maximum suppression technique NMS is used to perform most window filtering.

12. The system according to claim 9, characterized in that The trained classifier uses the Facial Action Coding System and the Facial Movement Coding System FACS to construct training samples.

13. The system according to claim 9, characterized in that In the face recognition unit, facial behavior is divided into 46 facial movement units, 8 head movement units and 4 eye movement units.

14. The system according to claim 12, characterized in that In the encoding step of the facial motion unit, it is necessary to locate the 68 key points of the face through the landmark obtained in the previous step, and then perform independent detection according to different area ROIs. Finally, the detection of each ROI is summarized to obtain the detection result of the whole face.

15. The system according to claim 12, characterized in that The training of the classifier further includes grouping different facial motion units according to prior knowledge of facial motion units; and / or Facial movement units occurring in the same position area are grouped together, and the face is divided into nine groups including the left eye, right eye, and nose wing.

16. The system according to claim 9, characterized in that The classifier is trained using the following method: A dual-channel convolutional neural network is constructed, with one channel input as the video stream and the other channel input as the matrix of facial motion unit encoding obtained above. After capturing the facial spatial semantic information through the encoding of facial motion units, head motion units and eye motion units, high frame rate video is used to capture motion information with fine temporal resolution. Then, at the decoding layer, the feature maps obtained from the two channels are fused, and finally a classification result is obtained as to whether the video interviewee has cheated.

17. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor executes the interview cheating detection method based on human facial motion unit as described in any one of claims 1-8.

18. A computer-readable medium storing a computer-executable program, characterized in that: When the computer executable program is executed, the interview cheating detection method based on human facial motion unit as described in any one of claims 1-8 is implemented.

Images