Concentration degree evaluation method and device, electronic equipment and storage medium
By segmenting and sampling video clips from remote videos, and performing face and pose detection, the accuracy of attention assessment in remote video interviews is solved, the risk of misjudgment is reduced, and the security of financial transactions is improved.
Patent Information
- Application Number
- CN202411197622.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-29
Smart Images

Figure CN119048962A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a concentration assessment method and device, an electronic device, and a storage medium. Background Art
[0002] In a remote video interview or video audit, participants are often required to answer questions attentively without looking around, so as to avoid being prompted by others or being suspected of cheating. Therefore, how to accurately assess the concentration of participants has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] The main purpose of the embodiments of the present application is to propose a concentration assessment method and device, electronic device and storage medium, which can accurately assess the concentration of participants. In remote video interviews or video face-to-face audits, it can well assess the situation where the participants in the video do not look directly at the screen, so as to accurately assess the concentration of participants, achieve effective assessment of participants, and reduce the risk of cheating. When applied in the field of financial technology, it can help financial risk control and improve the security of financial services.
[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a concentration evaluation method, the method comprising:
[0005] Get the recorded video of the target object;
[0006] Dividing the recorded video into multiple video clips of fixed length;
[0007] Extracting a plurality of target object video frames from the video clip according to a preset frame sampling rate;
[0008] Performing face detection on a plurality of the target object video frames, determining continuous target state abnormal frames in the plurality of the target object video frames, wherein the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than a preset continuous length;
[0009] The concentration evaluation result of the target object is determined according to the ratio of the target state abnormal frames to the length of the video clip.
[0010] In some embodiments, performing face detection on the plurality of target object video frames to determine consecutive target state abnormal frames in the plurality of target object video frames includes:
[0011] Performing face detection on a plurality of the target object video frames to determine continuous frames in which the face appears in the plurality of the target object video frames, wherein the length of the continuous frames in which the face appears is greater than the preset continuous length;
[0012] Determine a target face whose value is greater than a face detection threshold in a plurality of the target object video frames;
[0013] Extracting a face detection area based on the target face;
[0014] Extracting facial key points according to the face detection area;
[0015] Determine the eye state detection result of the target object according to the facial key points;
[0016] Determining, according to the eye state detection result, the continuous eye state abnormal frames in the plurality of target object video frames, wherein the length of the continuous eye state abnormal frames is greater than the preset continuous length;
[0017] Calculating a sight line estimation result of the target object according to the face detection area;
[0018] Determining, according to the sight line estimation result, continuous sight line abnormal frames in a plurality of the target object video frames, wherein the length of the continuous sight line abnormal frames is greater than the preset continuous length;
[0019] Calculate the head posture result of the target object according to the face detection area;
[0020] Determine the continuous abnormal head posture frames in a plurality of the target object video frames according to the head posture result, wherein the length of the continuous abnormal head posture frames is greater than the preset continuous length.
[0021] In some embodiments, determining the eye state detection result of the target object according to the facial key points includes:
[0022] Extracting a target eye area of the target object according to the facial key points;
[0023] The target eye region is input into a pre-trained open-and-closed eye model to obtain an eye state detection result of the target object, wherein the eye state detection result includes an open-eye state and a closed-eye state.
[0024] In some embodiments, calculating the sight line estimation result of the target object according to the face detection area includes:
[0025] Inputting the face detection area into a pre-trained sight line estimation model to obtain a gaze pitch angle and a gaze yaw angle of the target object;
[0026] Calculating the sight line length and the sight line angle of the target object according to the gaze pitch angle and the gaze yaw angle;
[0027] A sight line estimation result of the target object is determined according to the sight line length and the sight line angle.
[0028] In some embodiments, determining the continuous sight line abnormal frames in the plurality of target object video frames according to the sight line estimation result includes:
[0029] When the sight line length of the target object is greater than a preset length threshold and the sight line angle is outside the range of 40° to 140°, determining that the sight line of the target object is abnormal;
[0030] When the continuous length of the target object video frames where the target object sight line is abnormal exceeds a preset continuous length, the continuous target object video frames are determined to be the sight line abnormality frames.
[0031] In some embodiments, calculating the head posture result of the target object according to the face detection area includes:
[0032] Inputting the face detection area into a pre-trained head posture model to obtain the head pitch angle, head yaw angle and head roll angle of the target object;
[0033] A head posture result of the target object is determined according to the head pitch angle, the head yaw angle and the head roll angle.
[0034] In some embodiments, determining the continuous abnormal head posture frames in the plurality of target object video frames according to the head posture result includes:
[0035] When the pitch angle or the yaw angle of the head of the target object is greater than a preset angle threshold, determining that the head posture of the target object is abnormal;
[0036] When the continuous length of the target object video frames where the target object's head posture is abnormal exceeds a preset continuous length, the continuous target object video frames are determined to be the head posture abnormal frames.
[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a concentration evaluation device, the device comprising:
[0038] An acquisition module, used to acquire recorded video of a target object;
[0039] A segmentation module, used for segmenting the recorded video into a plurality of video segments of fixed length;
[0040] A frame extraction module, used for extracting a plurality of target object video frames from the video clip according to a preset frame extraction sampling rate;
[0041] A detection module is used to perform face detection on a plurality of the target object video frames, and determine continuous target state abnormal frames in the plurality of the target object video frames, wherein the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames, and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than a preset continuous length;
[0042] An evaluation module is used to determine a concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip.
[0043] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0045] The concentration evaluation method and device, electronic device and storage medium proposed in the present application obtain a recorded video of a target object; divide the recorded video into multiple video clips of fixed length; extract multiple target object video frames from the video clip according to a preset frame sampling rate; perform face detection on the multiple target object video frames, and determine continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, line of sight abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than the preset continuous length; determine the concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip. By dividing the real-time recording or recorded video into video segments, and then sampling the video segments to obtain multiple target object video frames, face detection is performed on the multiple target object video frames to determine the continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, by reasonably calculating the state of the target object in the video, detecting the appearance state of the target object, long-term eye closure, abnormal sight and abnormal head posture, it is possible to well evaluate the situation that the target object in the video does not look directly at the screen, so as to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as the judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate. Based on this, the embodiment of the present application can accurately evaluate the concentration of the participants. In the remote video interview or video face-to-face review scenario, it can well evaluate the situation that the participants in the video do not look directly at the screen, so that the concentration of the participants can be accurately evaluated, and the effective evaluation of the participants is achieved, and the risk of cheating is reduced. The focus assessment method of this application can also be applied in the field of financial technology to assist in financial risk control and improve the security of financial services. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of a concentration evaluation method provided in an embodiment of the present application;
[0047] Figure 2 yes Figure 1 Flow chart of step S104 in FIG.
[0048] Figure 3 yes Figure 2 Flow chart of step S205 in FIG.
[0049] Figure 4 yes Figure 2 Flow chart of step S207 in FIG.
[0050] Figure 5 yes Figure 4 Flow chart of step S403 in FIG.
[0051] Figure 6 yes Figure 2 Flow chart of step S209 in FIG.
[0052] Figure 7 yes Figure 6 Flowchart of step S602 in FIG.
[0053] Figure 8 is a flow chart of a concentration evaluation method provided by another embodiment of the present application;
[0054] Fig. 9 is a structural schematic diagram of a concentration evaluation device provided in an embodiment of the present application;
[0055] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0059] First, some nouns involved in this application are analyzed:
[0060] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0061] Facial key point detection: It is an important branch in the field of computer vision. It determines the location of key feature points on the face by analyzing the face image. These key points include not only the basic outline of the face, such as the location of the eyes, nose, and mouth, but also more detailed feature points, such as the corners of the eyes and mouth. This technology is crucial to improving the accuracy of face recognition, realizing expression analysis, beauty applications and other functions.
[0062] Euler angles: A method used to describe the rotation of an object in three-dimensional space. It consists of three angles, usually called roll, pitch, and yaw. The Euler angle system can be used to represent the direction or attitude of an object, especially in the fields of flight, robot control, 3D computer graphics, and aerospace.
[0063] Pitch: Pitch is the angle of rotation around the side axis of an object. Pitch changes when an object is tilted up or down. Like roll, pitch is usually expressed in radians or degrees, and usually ranges from -π / 2 to π / 2 or from -90° to 90°.
[0064] Yaw: Yaw is the angle of rotation around an object's vertical axis. When an object rotates around its vertical axis to change its orientation, the yaw angle changes. Yaw is usually expressed in radians or degrees, and the range is usually from -π to π or from -180° to 180°.
[0065] Roll Angle: The roll angle is the angle of rotation around the longitudinal axis of an object. The roll angle changes when an object rotates around its own forward direction. It is usually expressed in radians or degrees, and the range is usually from -π to π or from -180° to 180°.
[0066] In a remote video interview or video audit, participants are often required to answer questions attentively without looking around, so as to avoid being prompted by others or being suspected of cheating. Therefore, how to accurately assess the concentration of participants has become a technical problem that needs to be solved urgently.
[0067] Based on this, the embodiments of the present application provide a concentration assessment method and device, an electronic device and a storage medium, which obtain a recorded video of a target object; divide the recorded video into multiple video segments of fixed length; extract multiple target object video frames from the video segment according to a preset frame sampling rate; perform face detection on the multiple target object video frames, and determine continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, line of sight abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than the preset continuous length; determine the concentration assessment result of the target object according to the ratio of the target state abnormal frames to the length of the video segment. By dividing the real-time recording or recorded video into video segments, and then sampling the video segments to obtain multiple target object video frames, face detection is performed on the multiple target object video frames to determine the continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, by reasonably calculating the state of the target object in the video, detecting the appearance state of the target object, long-term eye closure, abnormal sight and abnormal head posture, it is possible to well evaluate the situation that the target object in the video does not look directly at the screen, so as to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as the judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate. Based on this, the embodiment of the present application can accurately evaluate the concentration of the participants. In the remote video interview or video face-to-face review scenario, it can well evaluate the situation that the participants in the video do not look directly at the screen, so that the concentration of the participants can be accurately evaluated, and the effective evaluation of the participants is achieved, and the risk of cheating is reduced. The focus assessment method of this application can also be applied in the field of financial technology to assist in financial risk control and improve the security of financial services.
[0068] The concentration evaluation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the concentration evaluation method in the embodiments of the present application is described.
[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0071] The concentration evaluation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The concentration evaluation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the concentration evaluation method, etc., but is not limited to the above forms.
[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0074] Figure 1 is an optional flow chart of the concentration evaluation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.
[0075] Step S101, obtaining a recorded video of a target object;
[0076] Step S102, dividing the recorded video into a plurality of video segments of fixed length;
[0077] Step S103, extracting a plurality of target object video frames from the video clip according to a preset frame sampling rate;
[0078] Step S104, performing face detection on multiple target object video frames, determining continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames including face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames, and the length of continuous target state abnormal frames is greater than a preset continuous length;
[0079] Step S105, determining the focus evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip.
[0080] In step S101 of some embodiments, a recorded video of the target object is obtained. The recorded video may be recorded in real time or may be already recorded. The recorded video of the target object may be obtained through a camera or an electronic device having a camera. In a remote video interview or video face-to-face review scenario, the concentration of the participant may be assessed through the recorded video.
[0081] In step S102 of some embodiments, the recorded video is divided into a plurality of video segments of fixed length. Assuming that the length of a recorded video is T, it is divided into small video segments of fixed length N = {1, 2, ..., N}, and the total length of the divided video segments is Based on this, by evaluating the concentration of each video clip, we can accurately know at which stage of the video recording the target object is not paying attention.
[0082] In step S103 of some embodiments, multiple target object video frames are extracted from the video clip according to a preset frame sampling rate. Since each video clip contains a large number of video frames, detecting all video frames in the video clip will consume more resources and time. Therefore, multiple target object video frames can be extracted from the video clip according to a preset frame sampling rate. For example, the frame sampling rate can be set to r = {1, 2, 4}, that is, face detection is performed at intervals of r frames. This can save time and resource consumption, speed up the efficiency of face detection, and thus achieve the concentration assessment of the participants in a shorter time.
[0083] In step S104 of some embodiments, face detection is performed on multiple target object video frames to determine continuous target state abnormal frames in multiple target object video frames, and target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames, and the length of continuous target state abnormal frames is greater than the preset continuous length. Face detection is performed on multiple target object video frames to determine continuous target state abnormal frames in multiple target object video frames, and target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames. By reasonably calculating the state of the target object in the video, the appearance state, long-term eye closure, abnormal sight line and abnormal head posture of the target object are detected, and the situation that the target object in the video does not look directly at the screen can be well evaluated to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as a judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate.
[0084] In step S105 of some embodiments, the concentration evaluation result of the target object is determined according to the ratio of the target state abnormal frames to the length of the video clip. The concentration evaluation result of the target object can be reflected by a concentration score, which is determined by calculating the proportion of the video clip in which the participant looks directly at the camera and does not look around. Specifically, it can be calculated by 1-(face appearance ratio ∪ closed eyes ratio ∪ abnormal sight ratio ∪ abnormal head posture ratio).
[0085] Steps S101 to S105 shown in the embodiment of the present application are to obtain a recorded video of the target object; divide the recorded video into multiple video clips of fixed length; extract multiple target object video frames from the video clips according to a preset frame sampling rate; perform face detection on the multiple target object video frames, and determine continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, line of sight abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than the preset continuous length; determine the concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip. By dividing the real-time recording or recorded video into video segments, and then sampling the video segments to obtain multiple target object video frames, face detection is performed on the multiple target object video frames to determine the continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, by reasonably calculating the state of the target object in the video, detecting the appearance state of the target object, long-term eye closure, abnormal sight and abnormal head posture, it is possible to well evaluate the situation that the target object in the video does not look directly at the screen, so as to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as the judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate. Based on this, the embodiment of the present application can accurately evaluate the concentration of the participants. In the remote video interview or video face-to-face review scenario, it can well evaluate the situation that the participants in the video do not look directly at the screen, so that the concentration of the participants can be accurately evaluated, and the effective evaluation of the participants is achieved, and the risk of cheating is reduced. The focus assessment method of this application can also be applied in the field of financial technology to assist in financial risk control and improve the security of financial services.
[0086] See also Figure 2 In some embodiments, step S104 may include but is not limited to steps S201 to S210:
[0087] Step S201, performing face detection on a plurality of target object video frames, and determining continuous frames in which a face appears in the plurality of target object video frames, wherein the length of the continuous frames in which a face appears is greater than a preset continuous length;
[0088] Step S202, determining a target face whose value is greater than a face detection threshold in a plurality of target object video frames;
[0089] Step S203, extracting a face detection area based on the target face;
[0090] Step S204, extracting facial key points according to the face detection area;
[0091] Step S205, determining the eye state detection result of the target object according to the key points of the face;
[0092] Step S206, determining continuous eye state abnormal frames in a plurality of target object video frames according to the eye state detection result, wherein the length of the continuous eye state abnormal frames is greater than a preset continuous length;
[0093] Step S207, calculating the sight line estimation result of the target object according to the face detection area;
[0094] Step S208, determining continuous sight line abnormal frames in a plurality of target object video frames according to the sight line estimation result, wherein the length of the continuous sight line abnormal frames is greater than a preset continuous length;
[0095] Step S209, calculating the head posture result of the target object according to the face detection area;
[0096] Step S210, determining continuous head posture abnormal frames in multiple target object video frames according to the head posture results, wherein the length of the continuous head posture abnormal frames is greater than a preset continuous length.
[0097] In some embodiments, face detection is performed on multiple target object video frames, and video frames where the participant's head leaves the video clip area with a continuous length of L can be determined. These video frames are continuous face appearance frames. A target face greater than the face detection threshold is selected from multiple target object video frames, and a face detection area is extracted based on the target face. The eye state detection result of the target object can be determined by the open and closed eyes model. The video frames where the participant's eyes are closed with a continuous length of L are continuous eye state abnormal frames. The sight estimation result of the target object is calculated based on the face detection area. The video frames where the sight abnormality is located with a continuous length of L can be determined by the sight estimation model. These video frames are continuous sight abnormal frames. The head posture result of the target object is calculated based on the face detection area. The video frames where the head posture abnormality is located with a continuous length of L can be determined by the head posture model. These video frames are continuous head posture abnormal frames. It should be noted that the specific value of the preset continuous length L can be set according to the actual requirements of the concentration evaluation, and the embodiments of the present application are not limited to this.
[0098] See also Figure 3 In some embodiments, step S205 may include but is not limited to steps S301 to S302:
[0099] Step S301, extracting the target eye area of the target object according to the key points of the face;
[0100] Step S302: input the target eye region into a pre-trained open-eye-closed model to obtain an eye state detection result of the target object, where the eye state detection result includes an open-eye state and a closed-eye state.
[0101] In some embodiments, the face key points of the target object are extracted according to the face detection area, and the target eye area with a larger left eye and right eye area is extracted according to the face key points of the target object, and then the target eye area is input into the pre-trained open and closed eye model to calculate the eye state E state ={0,1}, 0 means eyes closed, 1 means eyes open.
[0102] See also Figure 4 In some embodiments, step S207 may include but is not limited to steps S401 to S403:
[0103] Step S401, inputting the face detection area into a pre-trained gaze estimation model to obtain the gaze pitch angle and gaze yaw angle of the target object;
[0104] Step S402, calculating the sight length and sight angle of the target object according to the gaze pitch angle and the gaze yaw angle;
[0105] Step S403: determining a sight line estimation result of the target object according to the sight line length and the sight line angle.
[0106] In some embodiments, the face detection area is input into a pre-trained line of sight estimation model to obtain the gaze pitch angle and gaze yaw angle of the target object. The gaze pitch angle is used to characterize the up and down angle deviation of the line of sight direction of the target object around the horizontal axis, and the gaze yaw angle is used to characterize the left and right angle deviation of the line of sight direction of the target object around the vertical axis. According to the gaze pitch angle and the gaze yaw angle, the line of sight length and line of sight angle of the target object can be calculated. The line of sight length refers to the distance from the observer to the observed object, which can be a straight-line distance or an actually measured distance. The line of sight angle refers to the angle between the line of sight and a reference direction (such as a horizontal line or a vertical line), which can be used to describe the viewing angle of the observer when observing the object. According to the line of sight length and the line of sight angle, the line of sight estimation result of the target object can be determined, that is, whether the line of sight of the target object is facing the screen, which can be used to determine whether the target object is looking around.
[0107] See also Figure 5 In some embodiments, step S403 may include but is not limited to steps S501 to S502:
[0108] Step S501, when the sight line length of the target object is greater than a preset length threshold and the sight line angle is outside the range of 40° to 140°, determining that the sight line of the target object is abnormal;
[0109] Step S502 : when the continuous length of the target object video frames where the target object sight line is abnormal exceeds a preset continuous length, determining that the continuous target object video frames are sight line abnormality frames.
[0110] In some embodiments, when the target object's sight length is greater than a preset length threshold and the sight angle is outside the range of 40° to 140°, the target object's sight is considered abnormal. When the continuous length of the video frame where the target object's sight is abnormal exceeds the preset continuous length, the continuous target object video frame is determined to be a sight abnormal frame.
[0111] See also Figure 6 In some embodiments, step S209 may include but is not limited to steps S601 to S602:
[0112] Step S601, inputting the face detection area into a pre-trained head posture model to obtain the head pitch angle, head yaw angle and head roll angle of the target object;
[0113] Step S602: determining the head posture result of the target object according to the head pitch angle, the head yaw angle and the head roll angle.
[0114] In some embodiments, the face detection area is input into a pre-trained head posture model, and the head pitch angle, head yaw angle and head roll angle of the target object can be obtained. It should be noted that the head posture estimation angle analysis can include three main angles: head pitch angle, head yaw angle and head roll angle. These angles describe the rotation of the head in three-dimensional space. Among them, the head pitch angle is used to describe the rotation of the head on the vertical plane, that is, the angle of upside down flipping, positive value indicates upside flipping, and negative value indicates down flipping. The head yaw angle is used to describe the rotation of the head on the horizontal plane, that is, the angle of turning the head left and right, positive value indicates turning the head right, and negative value indicates turning the head left. The head roll angle is used to describe the rotation of the head in the front and back directions, that is, the angle of shaking the head, positive value indicates shaking the head forward, and negative value indicates shaking the head backward. Based on this, the head posture result of the target object can be determined according to the head pitch angle, head yaw angle and head roll angle.
[0115] See also Figure 7 In some embodiments, step S602 may include but is not limited to steps S701 to S702:
[0116] Step S701, when the pitch angle or yaw angle of the head of the target object is greater than a preset angle threshold, determining that the head posture of the target object is abnormal;
[0117] Step S702: when the continuous length of the target object video frames where the target object's head posture is abnormal exceeds a preset continuous length, determine that the continuous target object video frames are head posture abnormal frames.
[0118] In some embodiments, when the pitch angle or yaw angle of the target object's head is greater than a preset angle threshold, the target object's head posture is considered abnormal. When the continuous length of the video frame where the target object's head posture is abnormal exceeds the preset continuous length, the continuous target object video frame is determined to be a head posture abnormal frame.
[0119] The following is a further description of the concentration evaluation method of the present application in conjunction with the accompanying drawings and specific embodiments. The specific steps are as follows:
[0120] like Figure 8 As shown, the main process of the concentration evaluation method of the embodiment of the present application includes the following:
[0121] Assume that a recorded video is of length T, and divide it into small video segments of fixed length N = {1, 2, ..., N}. The total length of the divided video segments is Set the frame sampling rate to r = {1, 2, 4}, that is, perform face detection, face key point detection, eye opening and closing judgment, line of sight estimation and head posture estimation at intervals of r frames. The extraction process of the above information estimation is:
[0122] 1. Input an image to the face detection model, extract the faces in the image, and select the largest face that is larger than the face detection threshold;
[0123] 2. If there is a face detection result F conf , extract the face area; if it does not exist, end;
[0124] 3. According to the face detection area, input it into the face key point model to extract the face key points, extract the larger area of the left and right eyes, and input it into the open and closed eye model to calculate the eye state E state ={0,1}, 0 means eyes closed, 1 means eyes open;
[0125] 4. According to the face detection area, input it into the line of sight estimation model to calculate the line of sight estimation result. The predicted value is G pitch , G yaw ;
[0126] 5. According to the face detection area, input it into the head posture model to calculate the head posture result. The predicted value is
[0127] H pitch , H yaw ,H roll ; See the specific flow chart Figure 1 shown.
[0128] For a video clip of fixed length N, after frame sampling rate r, the state estimation result is: Each S i Contains F conf ,E state ,G pitch , G yaw ,H pitch , H yaw ,H roll According to the detection results, the frames where the head is not in the video clip area for a continuous length of L, the frames where the eye state is abnormal for a continuous length of L, the frames where the line of sight is abnormal for a continuous length of L, and the frames where the head posture is abnormal for a continuous length of L are calculated. Assuming that the head is not in the video clip for a continuous length of L, the face abnormality of the Lth frame is recorded as 1, and the first L-1 frames are recorded as 0.
[0129] Among them, the judgment logic of abnormal vision is based on G pitch , G yaw The sight line length and sight line angle are calculated. When the sight line angle is not between 40° and 140° and the sight line length is greater than the threshold, the sight line is considered abnormal.
[0130] Among them, the judgment logic of abnormal head posture is H pitch or H yaw If it is greater than the threshold, the head posture is considered abnormal.
[0131] Based on the above calculation results, we get the face appearance frame, eye state abnormal frame, sight abnormal frame and head posture abnormal frame. Find the union of these frames, divide it by the length of the state estimation result, and subtract this value from 1 to get the final concentration score.
[0132] The embodiment of the present application combines a face detection model, a face key point model, an eye opening and closing model, a sight line estimation model, and a head posture model to detect the state of the participant. By reasonably calculating the state of the participant in the video, abnormal sight lines, abnormal head postures, long-term eye closure, and out-of-camera states are detected, and the concentration of the participant is calculated. This method is suitable for real-time recording scenarios and is also suitable for evaluation after recording.
[0133] The concentration evaluation result of the target object can be reflected by the concentration score. The concentration score is determined by calculating the proportion of the video clip in which the participant looks directly at the camera and does not look around. Specifically, it can be calculated by 1-(face appearance ratio ∪ closed eyes ratio ∪ abnormal line of sight ratio ∪ abnormal head posture ratio).
[0134] Based on this, the concentration evaluation method of the embodiment of the present application can well evaluate the situation where the person in the video is not looking directly at the screen. And the use of multi-frame anomalies as the final anomaly can effectively reduce the state misjudgment caused by model misdetection, making the output concentration evaluation result more stable. When used in remote video interviews or video audit scenarios, real-time performance is also taken into account. In the recorded real-time clips, the concentration of the person being audited can also be effectively evaluated, which is conducive to risk prompts. After the recording, the scene can also be effectively evaluated to reduce risks. When applied in the field of financial technology, it can assist in financial risk control and improve the security of financial services.
[0135] See also Fig. 9 The embodiment of the present application also provides a concentration evaluation device, which can implement the above-mentioned concentration evaluation method, and the device includes:
[0136] An acquisition module 910 is used to acquire a recorded video of a target object;
[0137] A segmentation module 920, used to segment the recorded video into a plurality of video segments of fixed length;
[0138] A frame extraction module 930 is used to extract a plurality of target object video frames from the video clip according to a preset frame extraction sampling rate;
[0139] A detection module 940 is used to perform face detection on multiple target object video frames, and determine continuous target state abnormal frames in the multiple target object video frames, wherein the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames, and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than a preset continuous length;
[0140] The evaluation module 950 is used to determine the concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip.
[0141] In some embodiments of the present application, an acquisition module 910 acquires a recorded video of a target object; a segmentation module 920 segments the recorded video into multiple video segments of fixed lengths; a frame extraction module 930 extracts multiple target object video frames from the video segment according to a preset frame extraction sampling rate; a detection module 940 performs face detection on the multiple target object video frames, and determines continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames including face appearance frames, eye state abnormal frames, line of sight abnormal frames and head posture abnormal frames, and the length of continuous target state abnormal frames is greater than a preset continuous length; an evaluation module 950 determines a concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video segment.
[0142] In some embodiments of the present application, a recorded video of the target object is obtained. The recorded video can be recorded in real time or already recorded. The recorded video of the target object can be obtained through a camera or an electronic device with a camera. In a remote video interview or video face-to-face review scenario, the concentration of the participant can be assessed through the recorded video.
[0143] In some embodiments of the present application, the recorded video is divided into multiple video segments of fixed length. Assuming that the length of a recorded video is T, it is divided into small video segments of fixed length N = {1, 2, ..., N}, and the total length of the divided video segments is Based on this, by evaluating the concentration of each video clip, we can accurately know at which stage of the video recording the target object is not paying attention.
[0144] In some embodiments of the present application, multiple target object video frames are extracted from a video clip according to a preset frame sampling rate. Since each video clip contains a large number of video frames, detecting all video frames in the video clip will consume more resources and time. Therefore, multiple target object video frames can be extracted from a video clip according to a preset frame sampling rate. For example, the frame sampling rate can be set to r = {1, 2, 4}, that is, face detection is performed at intervals of r frames. This can save time and resource consumption, speed up the efficiency of face detection, and thus enable the concentration assessment of participants to be achieved in a shorter time.
[0145] In some embodiments of the present application, face detection is performed on multiple target object video frames to determine continuous target state abnormal frames in multiple target object video frames, and the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames, and the length of continuous target state abnormal frames is greater than the preset continuous length. Face detection is performed on multiple target object video frames to determine continuous target state abnormal frames in multiple target object video frames, and the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames. By reasonably calculating the state of the target object in the video, the appearance state, long-term eye closure, abnormal sight line and abnormal head posture of the target object are detected, and the situation that the target object in the video does not look directly at the screen can be well evaluated to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as a judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate.
[0146] In some embodiments of the present application, the concentration evaluation result of the target object is determined according to the ratio of the length of the target state abnormal frame to the video clip. The concentration evaluation result of the target object can be reflected by the concentration score, which is determined by calculating the proportion of the video clip in which the participant looks directly at the camera and does not look around. Specifically, it can be calculated by 1-(face appearance ratio ∪ closed eyes ratio ∪ abnormal sight ratio ∪ abnormal head posture ratio).
[0147] Based on this, in the concentration evaluation device of the embodiment of the present application, the acquisition module 910 acquires the recorded video of the target object; the segmentation module 920 segments the recorded video into multiple video segments of fixed lengths; the frame extraction module 930 extracts multiple target object video frames from the video segment according to a preset frame extraction sampling rate; the detection module 940 performs face detection on the multiple target object video frames, and determines continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, line of sight abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than the preset continuous length; the evaluation module 950 determines the concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video segment. The present application obtains a recorded video of a target object; divides the recorded video into multiple video clips of fixed length; extracts multiple target object video frames from the video clips according to a preset frame sampling rate; performs face detection on multiple target object video frames to determine continuous target state abnormal frames in multiple target object video frames, target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, and the length of continuous target state abnormal frames is greater than the preset continuous length; according to the ratio of target state abnormal frames to the length of the video clip, the concentration evaluation result of the target object is determined. By dividing the real-time recorded or recorded video into video clips, and then sampling multiple target object video frames from the video clips, face detection is performed on multiple target object video frames to determine continuous target state abnormal frames in multiple target object video frames, target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, by reasonably calculating the state of the target object in the video, detecting the appearance state of the target object, long-term eye closure, abnormal sight and abnormal head posture, it is possible to well evaluate the situation in which the target object in the video does not look directly at the screen, so as to confirm the concentration of the target object. And the length of the continuous target state abnormal frame greater than the preset continuous length is used as the judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate. Based on this, the embodiment of the present application can accurately evaluate the concentration of the participants. In the scenario of remote video interview or video face-to-face review, it can well evaluate the situation where the participants in the video do not look directly at the screen, so as to accurately evaluate the concentration of the participants, realize effective evaluation of the participants, and reduce the risk of cheating. The concentration evaluation method of the present application can also be applied in the field of financial technology, which can assist financial risk control and improve the security of financial services.
[0148] The specific implementation of the concentration evaluation device is substantially the same as the specific implementation of the above-mentioned concentration evaluation method, and will not be described in detail herein.
[0149] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned concentration evaluation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0150] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0151] The processor 1001 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.
[0152] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 is used to call and execute the concentration evaluation method of the embodiment of this application, that is, by obtaining a recorded video of the target object; dividing the recorded video into a plurality of video clips of fixed length; extracting a plurality of target object video frames from the video clip according to a preset frame sampling rate; performing face detection on the plurality of target object video frames, determining continuous target state abnormal frames in the plurality of target object video frames, the target state abnormal frames including face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than the preset continuous length; determining the concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip. By dividing the real-time recording or recorded video into video segments, and then sampling the video segments to obtain multiple target object video frames, face detection is performed on the multiple target object video frames to determine the continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, by reasonably calculating the state of the target object in the video, detecting the appearance state of the target object, long-term eye closure, abnormal sight and abnormal head posture, it is possible to well evaluate the situation that the target object in the video does not look directly at the screen, so as to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as the judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate. Based on this, the embodiment of the present application can accurately evaluate the concentration of the participants. In the remote video interview or video face-to-face review scenario, it can well evaluate the situation that the participants in the video do not look directly at the screen, so that the concentration of the participants can be accurately evaluated, and the effective evaluation of the participants is achieved, and the risk of cheating is reduced. The focus assessment method of this application can also be applied in the field of financial technology to assist in financial risk control and improve the security of financial services.
[0153] The input / output interface 1003 is used to implement information input and output.
[0154] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.).
[0155] The bus transmits information between various components of the device (eg, the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 ).
[0156] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via a bus.
[0157] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned concentration evaluation method is implemented.
[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0159] The concentration evaluation method, concentration evaluation device, electronic device and storage medium provided in the embodiments of the present application obtain a recorded video of the target object; divide the recorded video into multiple video clips of fixed length; extract multiple target object video frames from the video clips according to a preset frame sampling rate; perform face detection on the multiple target object video frames, and determine continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, line of sight abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than the preset continuous length; determine the concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip. By dividing the real-time recording or recorded video into video segments, and then sampling the video segments to obtain multiple target object video frames, face detection is performed on the multiple target object video frames to determine the continuous target state abnormal frames in the multiple target object video frames, the target state abnormal frames include face appearance frames, eye state abnormal frames, sight abnormal frames and head posture abnormal frames, by reasonably calculating the state of the target object in the video, detecting the appearance state of the target object, long-term eye closure, abnormal sight and abnormal head posture, it is possible to well evaluate the situation that the target object in the video does not look directly at the screen, so as to confirm the concentration of the target object. And the length of the continuous target state abnormal frame is greater than the preset continuous length as the judgment basis, that is, it will be judged as the final abnormality only if it meets the multi-frame abnormality, thereby effectively reducing the state misjudgment caused by false detection, making the evaluation result more stable and accurate. Based on this, the embodiment of the present application can accurately evaluate the concentration of the participants. In the remote video interview or video face-to-face review scenario, it can well evaluate the situation that the participants in the video do not look directly at the screen, so that the concentration of the participants can be accurately evaluated, and the effective evaluation of the participants is achieved, and the risk of cheating is reduced. The focus assessment method of this application can also be applied in the field of financial technology to assist in financial risk control and improve the security of financial services.
[0160] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as a computer-readable program, a data structure, a program module or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer-readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0161] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0162] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0164] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0165] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0166] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0168] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0171] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A concentration evaluation method, characterized in that: The method comprises: Get the recorded video of the target object; Dividing the recorded video into multiple video clips of fixed length; Extracting a plurality of target object video frames from the video clip according to a preset frame sampling rate; Performing face detection on a plurality of the target object video frames, determining continuous target state abnormal frames in the plurality of the target object video frames, wherein the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than a preset continuous length; The concentration evaluation result of the target object is determined according to the ratio of the target state abnormal frames to the length of the video clip.
2. The method according to claim 1, characterized in that The performing face detection on the plurality of target object video frames to determine continuous target state abnormal frames in the plurality of target object video frames comprises: Performing face detection on a plurality of the target object video frames to determine continuous frames in which the face appears in the plurality of the target object video frames, wherein the length of the continuous frames in which the face appears is greater than the preset continuous length; Determine a target face whose value is greater than a face detection threshold in a plurality of the target object video frames; Extracting a face detection area based on the target face; Extracting facial key points according to the face detection area; Determine the eye state detection result of the target object according to the facial key points; Determining, according to the eye state detection result, the continuous eye state abnormal frames in the plurality of target object video frames, wherein the length of the continuous eye state abnormal frames is greater than the preset continuous length; Calculating a sight line estimation result of the target object according to the face detection area; Determining, according to the sight line estimation result, continuous sight line abnormal frames in a plurality of the target object video frames, wherein the length of the continuous sight line abnormal frames is greater than the preset continuous length; Calculate the head posture result of the target object according to the face detection area; Determine the continuous abnormal head posture frames in a plurality of the target object video frames according to the head posture result, wherein the length of the continuous abnormal head posture frames is greater than the preset continuous length.
3. The method according to claim 2, characterized in that The step of determining the eye state detection result of the target object according to the facial key points includes: Extracting a target eye area of the target object according to the facial key points; The target eye region is input into a pre-trained open-and-closed eye model to obtain an eye state detection result of the target object, wherein the eye state detection result includes an open-eye state and a closed-eye state.
4. The method according to claim 2, characterized in that: The calculating the sight line estimation result of the target object according to the face detection area includes: Inputting the face detection area into a pre-trained sight line estimation model to obtain a gaze pitch angle and a gaze yaw angle of the target object; Calculating the sight line length and the sight line angle of the target object according to the gaze pitch angle and the gaze yaw angle; A sight line estimation result of the target object is determined according to the sight line length and the sight line angle.
5. The method according to claim 4, characterized in that The step of determining the continuous sight line abnormal frames in the plurality of target object video frames according to the sight line estimation result comprises: When the sight line length of the target object is greater than a preset length threshold and the sight line angle is outside the range of 40° to 140°, determining that the sight line of the target object is abnormal; When the continuous length of the target object video frames where the target object sight line is abnormal exceeds a preset continuous length, the continuous target object video frames are determined to be the sight line abnormality frames.
6. The method according to claim 2, characterized in that The step of calculating the head posture result of the target object according to the face detection area includes: Inputting the face detection area into a pre-trained head posture model to obtain the head pitch angle, head yaw angle and head roll angle of the target object; A head posture result of the target object is determined according to the head pitch angle, the head yaw angle and the head roll angle.
7. The method according to claim 6, characterized in that The step of determining the continuous abnormal head posture frames in a plurality of target object video frames according to the head posture result comprises: When the pitch angle or the yaw angle of the head of the target object is greater than a preset angle threshold, determining that the head posture of the target object is abnormal; When the continuous length of the target object video frames where the target object's head posture is abnormal exceeds a preset continuous length, the continuous target object video frames are determined to be the head posture abnormal frames.
8. A concentration assessment device, characterized in that: The device comprises: An acquisition module, used to acquire recorded video of a target object; A segmentation module, used for segmenting the recorded video into a plurality of video segments of fixed length; A frame extraction module, used for extracting a plurality of target object video frames from the video clip according to a preset frame extraction sampling rate; A detection module is used to perform face detection on a plurality of the target object video frames, and determine continuous target state abnormal frames in the plurality of the target object video frames, wherein the target state abnormal frames include face appearance frames, eye state abnormal frames, sight line abnormal frames, and head posture abnormal frames, and the length of the continuous target state abnormal frames is greater than a preset continuous length; An evaluation module is used to determine a concentration evaluation result of the target object according to the ratio of the target state abnormal frames to the length of the video clip.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the concentration assessment method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the concentration evaluation method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Prisoner-oriented active transformation behavior evaluation method
CN111507592A
Online classroom student concentration evaluation method and system based on multi-feature fusion
CN114663734A
Online learning concentration degree monitoring method and system based on machine vision and medium
CN115205764A
Concentration degree monitoring method and system based on continuous video frames and memory
CN115965893A
Behavior detection method and electronic equipment
CN118351590A
Cited By
Recruitment system based on Internet of Things
CN119991062A
An Internet of Things-based recruitment system
CN119991062B
