Online and remote systems attendance and attendance prevention systems and methods thereof

Abstract

The present invention aims to provide a system and method that can verify normal class attendance and work attendance by receiving video and using facial recognition AI when attending and working online and remotely, in order to solve the problems of the prior art.

Description

Technology Field

[0001] The present invention relates to a system and method that can verify normal class attendance and work attendance by receiving video and using facial recognition AI when attending and working online and remotely. Background Technology

[0002] In the case of online / remote classes and work, since it is impossible to know whether work is being done diligently when classes and work are conducted in a fixed space, most people gather at a specific location of their choice to attend classes or work, even though it requires a lot of space and cost.

[0003] In cases where mandatory education, such as government-implemented safety and food safety training, was required, online and remote classes were unavoidable during the COVID-19 pandemic; however, it is difficult to achieve the intended educational objectives because the content is often displayed while the students are engaged in other activities.

[0004] In the case of online attendance, since it is impossible to know whether the employee is working in front of a PC, there has been an increase in cases where employees are sent back to work after being sent online or remotely. Prior art literature

[0005] Republic of Korea Published Patent 10-2016-0053155 (Published May 13, 2016) The problem to be solved

[0006] The present invention aims to provide a system and method that can verify normal class attendance and work attendance by receiving video and using facial recognition AI when attending and working online and remotely, in order to solve the problems of the prior art. means of solving the problem

[0007] The attendance and work attendance fraud prevention system of the online and remote system of the present invention may include: a sleep prevention unit that detects eye blinking and normalizes personalized eye size; an absence prevention unit that recognizes absence when there is no human face through face recognition or when the detected human face is different from the face of the person attending the class; a photo usage fraud prevention unit that distinguishes as a photo when there is no eye blinking within one minute through eye blink analysis; a video usage fraud prevention unit that detects video fraud by detecting a video playback device of a smartphone or tablet laptop using an object detection method, and detects whether an action has been performed by making a random action request when a user is in front using an action request method; and a proxy attendance fraud prevention unit that recognizes attendance when there is no human face through face recognition or when the detected human face is different from the face of the person attending the class. Effects of the invention

[0008] According to the present invention, there is an advantage of providing a system that can verify normal classes and attendance by matching images read through a camera in real time, reading eye blinks, lip movements, eyebrow movements, etc. appearing in the images in real time to confirm that they are not photographs, and preventing attempts to fraudulently attend classes and work through videos and repetitive videos. Brief explanation of the drawing

[0009] Figure 1 is an example of 68 Facial Landmark Detections provided by Dlib. Figure 2 shows the change in eye landmarks according to eye blinking. Figure 3 is a graph of the change in EOS when you close your eyes and then open them for a moment. Figure 4 shows three types of head movements. Figure 5 is an attendance and clock-in fraud prevention system for online and remote systems. Figure 6 illustrates the relationship between the management server and the control server during attendance of the online and remote systems of the present invention. FIG. 7 is a module for preventing fraudulent attendance in the online and remote system of the present invention. Figure 8 is a method for preventing fraudulent attendance using the online and remote systems of the present invention. Figure 9 illustrates the relationship between the management server and the control server of the online attendance fraud prevention system of the present invention. FIG. 10 is a fraudulent attendance prevention module of the online attendance fraud prevention system of the present invention. Specific details for implementing the invention

[0010] It is known that humans blink their eyes up to 20 times per minute on average, and the average number of blinks when drowsy is about 43.91 times per minute, which increases as the stage of drowsiness deepens. Therefore, drowsiness prevention using a method that detects eye blinking assumes that drowsiness is occurring when the number of times the eyes close increases and performs a drowsiness warning.

[0012] 1. Drowsiness prevention

[0013] Drowsiness prevention can detect eye blinking and utilize a personalized eye size normalization method.

[0014] (1) Eye blink detection method

[0015] The latest facial landmark detection method is a technique that identifies key feature points such as eyes, nose, mouth, and jawline in a facial image and provides more than 68 landmarks (which can be used for eye blink detection by providing feature points around the eyes).

[0016] When you close your eyes, the figure above distance and The distance approaches 0, distance and The distance increases. Taking note of this point Using distance as a reference point distance and The ratio of the distances can be calculated using the following mathematical formula 1.

[0017]

[0018] Here EOS represents the Eye Open Strength, indicating the size of the open eye. The calculated EOS will have a value close to 1 when the eyes are open and a value close to 0 when the eyes are closed.

[0019] When measuring EOS in actual face images, the value is close to 1 when the eyes are open and close to 0 when the eyes are closed. At this time, if a specific threshold is applied, it is possible to determine whether the eyes are open or closed, and the number of blinks can be counted by defining the change from Eye-Open -> Eye-Close -> Eye-Open as one blink.

[0020] In this case, a single threshold cannot be applied because eye size varies from person to person; depending on eye size This is because the values ​​differ significantly. Therefore, to compensate for eye sizes that vary from person to person, and because some people have very small eyes when open, there may not be a significant difference between the open and closed eye sizes. Thus, a personalized eye size normalization method is additionally applied.

[0022] (2) Personalized eye size normalization and threshold application method

[0023] As the simplest method, measure EOS for a certain period of time (10 seconds or more), measure the maximum and minimum values ​​of EOS, and use this to normalize as follows.

[0024]

[0026] The second method is normalization by the eye-distance ratio, as shown in the equation below.

[0027]

[0028]

[0030] The third method is a normalization method based on eye area, which is normalized as shown in the equation below.

[0031]

[0032] Here Is The x-coordinate value of the nth eye Landmark, Is It represents the y-coordinate value of the i-th eye Landmark, and A is the area value of those coordinates, representing the size of the eye as an area. The calculated value is applied to EOS as follows to perform eye area-based normalization.

[0033]

[0034] Normalized by one of the three methods above Since the eye characteristics that differ from person to person have been eliminated through normalization, a threshold is determined using a machine learning model (SVM, KNN, Decision Tree) or a deep learning model (CNN, RNN, LSTM, Transformer, etc.).

[0036] 2. Prevent Away From Seat

[0037] Through face recognition, when a person's face is missing or when the detected face differs from the face of a student attending the class, it can be recognized as being out of place.

[0039] 3. Prevention of unauthorized use of photos

[0040] Unauthorized use of photos is prevented by utilizing the fact that there is no facial movement when inserting a photo, whereas real human faces exhibit some movement. First, through eye blink analysis, if there is no blinking within one minute, it is identified as a photo.

[0041] Secondly, taking into account that when using a photograph, only Roll head movement is possible, Head Pose Estimation determines that it is a photograph if there is no change in Yaw or Pitch for more than 1 minute.

[0043] 4. Prevention of Unauthorized Use of Video Recordings

[0044] First, video usage fraud prevention is achieved by detecting video playback devices, such as smartphones, tablets, and laptops, using an Object Detection method. Second, an action request method is used to make random action requests when a user is present and to detect whether the action has been performed. Examples include "Close your eyes for one second" or "Open your mouth."

[0046] 5. Prevention of fraud through proxy attendance

[0047] Through face recognition, when a person's face is missing or the detected face differs from that of a student attending the class, it can be recognized as proxy attendance.

Claims

Claim 1 An attendance and clock-in fraud prevention system for online and remote systems comprising: a drowsiness prevention unit that detects blinking and normalizes personalized eye size; an absence prevention unit that recognizes absence through face recognition when no human face is present or when the detected human face differs from the face of the person attending the class; a photo usage fraud prevention unit that distinguishes as a photo if no blinking occurs within one minute through blink analysis; a video usage fraud prevention unit that detects video fraud by detecting video playback devices such as smartphones, tablets, or laptops using an object detection method, and detects whether an action was performed by making a random action request when a user is present using an action request method; and a proxy attendance fraud prevention unit that recognizes attendance as proxy attendance when no human face is present through face recognition or when the detected human face differs from the face of the person attending the class.

Images