Concentration monitoring method and system

Abstract

The concentration monitoring method and system relate to the field of automated monitoring technology. The aforementioned concentration monitoring method includes the following steps: 1. Collecting image information of the person to be tested and the equipment using an image acquisition device; 2. Micro-behavior recognition detection; 3. Obtaining the line of sight trajectory of the person to be tested on the display screen, and combining it with the attention heat map of the display screen to compare the similarity between the two; 4. Analyzing the movement trend of the line of sight trajectory to determine whether it is consistent with prior knowledge; 5. Detecting the eye features of the person to be tested, obtaining the blinking frequency of the person to be tested, and determining whether it is normal; 6. Combining the above-mentioned test results and performing weighted calculations to determine whether the person to be tested is in a working state. The aforementioned concentration monitoring method can be applied to personnel distraction detection in environments such as aviation dispatch and nuclear power safety control, and improves concealment and convenience, while also improving the accuracy of detection.

Description

Technical Field

[0001] The present invention relates to the field of automated monitoring technology, and in particular to a concentration monitoring method and system. Background Art

[0002] At present, when performing operations such as aviation dispatch and nuclear power safety control, workers usually need to use auxiliary devices such as human-computer interaction output devices (such as monitors) and human-computer interaction input devices (such as keyboards and mice) to view the status and operate the host (such as computers). When working, the concentration of workers is often affected by their emotions, mental state and other factors. They are easily distracted, absent-minded, and dazed when they are in a bad mood or tired, resulting in poor concentration and affecting the timeliness and accuracy of operations. Therefore, how to identify the concentration of personnel is particularly important.

[0003] Chinese patent document CN109878527A discloses a distraction sensing system, which uses a contact sensor to collect skin electrical signals to analyze whether a person is distracted. This method requires the person to be detected to wear equipment, and lacks concealment and convenience. Chinese patent document CN104706366A discloses a distraction detection method, device and system, which collects the electrocardiogram data of the person to be detected, and also has the same problems as mentioned above. Chinese patent document CN115246405A discloses a method and system for detecting driver distraction behavior, which collects the driver's eye features and converts them into a range of visual attention. When it exceeds the preset normal range, the person to be detected is judged to be distracted. The detection granularity of this method is too coarse and cannot distinguish whether it is normal driving or distracted. Chinese patent document CN114026611A discloses a method for detecting driver attention using heat maps, which generates a scene attention heat map and a driver gaze heat map respectively, and determines whether the driver is distracted by analyzing the difference between the two heat maps. This method can only identify whether the line of sight of the person to be detected is consistent with the area of ​​interest, and lacks time series information. It is also difficult to distinguish situations such as the person to be detected being in a daze.

[0004] The above-mentioned distraction detection methods for automobile drivers still have many shortcomings, and the automobile driving environment is quite different from the operating environment such as aviation dispatching and nuclear power safety control. Therefore, they are not suitable for personnel distraction detection in working environments such as aviation dispatching and nuclear power safety control. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for detecting personnel distraction which can be applied to working environments such as aviation dispatching and nuclear power safety control.

[0006] In order to solve the above problems, the technical solution adopted by the present invention is as follows:

[0007] The concentration monitoring method includes the following steps:

[0008] 1. Use the image acquisition device to collect image information of the person and equipment to be tested;

[0009] Second, the area where the human-computer interaction input device is located in the image is used as the micro-behavior recognition detection area, and whether the hand of the person to be tested appears in the area is detected. The hand that appears in the area is subjected to micro-behavior recognition detection to obtain the state of the hand of the person to be tested and determine whether the hand is active or not;

[0010] Specifically, the area where the human-computer interaction input device is located in the image is used as the micro-behavior recognition detection area, and whether the hand features of the person to be tested appear in the micro-behavior recognition detection area is detected. If not, proceed to the next step. If yes, perform micro-behavior recognition detection on the hand features appearing therein to determine whether the hand of the person to be tested is active or stationary. If active, the person to be tested is determined to be in a working state, and if stationary, proceed to the next step.

[0011] 3. Obtain the eye track of the person under test on the monitor screen, and compare the similarity between the two in combination with the attention heat map on the monitor screen;

[0012] 4. Analyze the movement trend of the sight trajectory and determine whether it is consistent with prior knowledge;

[0013] 5. Detect the eye features of the person to be tested, obtain the blinking frequency of the person to be tested, and determine whether it is normal;

[0014] 6. Combining the above test results and performing weighted calculation, determine whether the person to be tested is in working condition.

[0015] Among them, in step one, the image acquisition device includes a camera, and the collected image information of the person to be tested includes image information of the eye features, hand features and facial features of the person to be tested, and the collected image information of the device includes image information of the human-computer interaction input device and the display screen.

[0016] Among them, in step 2, the human-computer interaction input device is a keyboard and a mouse, and the step of determining the micro-behavior recognition detection area includes:

[0017] a. Use the target detection technology in image processing to detect and determine the regional positions R of the keyboard and mouse in the image respectively. keyboard and R mouse , where R keyboard and R mouse for:

[0018] R keyboard =((x left ,y left ), (x right ,yright ));

[0019] R mouse =((x left ,y left ), (x right ,y right ));

[0020] Among them, (x left ,y left ) represents region R * The coordinate position of the upper left corner, (x right ,y right ) represents region R * The coordinate position of the lower right corner.

[0021] b. Determine the micro-behavior recognition detection area R through the regional positions of the keyboard and mouse detection , the regional coordinate parameters are:

[0022]

[0023] Among them, in step 2, the steps of micro-behavior recognition and detection include:

[0024] The key point information on the hand features appearing in the micro-behavior recognition detection area, including the fingertips, finger joints, palm center and wrist joints, is detected, and then the trajectory of the key points of the hand features in the video sequence is used to determine whether the hand of the person being tested is active or still.

[0025] Among them, in step three, the step of obtaining the sight track includes:

[0026] a. Extract and analyze the eye features of the person to be tested to obtain the direction of sight;

[0027] b. Detecting several feature values ​​of the face and comparing them with the feature values ​​of the face in a standard posture to obtain the posture relationship between the face and the image acquisition device;

[0028] c. Based on the positional relationship between the face of the person to be tested and the display screen, calculate the line of sight of the person to be tested on the display screen:

[0029] (1) Obtaining the transformation matrix R between the face and the screen through the relationship between the spatial coordinates of the image acquisition device and the plane position of the display screen;

[0030] (2) The position of the face relative to the screen is calculated through the transformation matrix R, and the position of the line of sight on the screen P is calculated in real time by calculating the distance between the face and the screen t ;

[0031] (3) Obtain the position sequence of the line of sight on the screen by accumulating over time (P i ), where i≥0.

[0032] Among them, in step 3, the distribution function of the screen attention heat map is Prob S :

[0033]

[0034] Among them, Bright(i,j) is the brightness value of the screen coordinate position ij,

[0035]

[0036] Among them, width and height are the resolution of the screen, and (i, j) is the coordinate position of the sampling point on the screen.

[0037] Among them, in step three, the step of detecting the similarity between the eye track and the screen attention heat map includes:

[0038] Normalize the gaze trajectory and the screen attention heat map, and sample according to the screen resolution to obtain the distribution function Prob of the gaze trajectory Eye , the distribution function of the screen attention heat map is Prob Screen , the similarity between the two is calculated by cross entropy:

[0039]

[0040] Where width and height are the screen resolutions, and (i, j) is the coordinate position of the sampling point on the screen;

[0041] The calculated gaze trajectory is compared with the similarity of the screen attention heat map and the pre-set threshold to determine whether the person being tested is in a working state.

[0042] Among them, in step 4, the prior knowledge is that the line of sight of the person being measured starts from the middle of the display screen and moves from the upper left position to the lower right position; the step of determining the direction of movement of the line of sight trajectory includes:

[0043] By calculating the gradient of the sight trajectory, we can determine the direction in which the sight moves on the screen:

[0044]

[0045] Among them, Trail i (x, y) is the coordinate information of the i-th point of the sight track;

[0046] pass The positive and negative signs of the two dimensions determine the direction of the trajectory movement.

[0047] Among them, in step 5, the step of detecting the blinking frequency of the person to be tested includes:

[0048] a. Use image processing technology to extract the face position in the image and segment the eye area;

[0049] b. Detect the positions of several key points of the eyes from the eye area, including the inner corner of the eye, the outer corner of the eye, the upper eyelid and the lower eyelid;

[0050] c. Calculate the ratio of the distance between the upper and lower eyelids to the distance between the inner and outer corners of the eye, and combine the time information to calculate the blinking frequency and the duration of eye closure;

[0051] d. The calculation method of blink discrimination parameters is:

[0052]

[0053] Among them, Dis W Dis is the distance between the inner corner of the eye P1 and the outer corner of the eye P4, H is the distance between the upper eyelid P2 and the lower eyelid P6;

[0054] Periodically calculate the blink discrimination parameter Blink to obtain the time-based change trend;

[0055] e. According to the duration of closing eyes close , determine whether the person being tested is working fatigued and make a judgment on the abnormal state of the person being tested:

[0056]

[0057] Here, threshold is the threshold of the duration of eye closure.

[0058] Another object of the present invention is to provide a concentration monitoring system, which includes a storage unit and a computing unit, wherein the storage unit stores a concentration monitoring program, and the concentration monitoring program is run by the computing unit to execute the steps in the concentration monitoring method.

[0059] The concentration monitoring method provided by the present invention is different from the common contact distraction detection method in the past. It adopts a non-contact detection method, which does not require the person to be tested to wear equipment. It only needs to use non-contact equipment such as cameras to collect image information, which has better concealment and convenience. It determines the state of the person to be tested through micro-behavior recognition, eye trajectory and screen attention heat map detection, blinking feature detection and other methods. Through weighted calculation (the weight value of each indicator can be determined by experiment), the concentration of the person to be tested is comprehensively analyzed. The detection result obtained based on this is more accurate than a single judgment method. Compared with the existing automobile driver distraction detection method, the concentration monitoring method provided by the present invention is better in concealment and convenience, and has higher detection accuracy. It can be applied to personnel distraction detection in operating environments such as aviation dispatch and nuclear power safety control. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the process of monitoring concentration in an embodiment;

[0061] Figure 2 is a schematic diagram of eye features in an embodiment;

[0062] Figure 3 is a schematic diagram of several feature points of a human face in an embodiment;

[0063] Figure 4 Schematic diagram of facial features in an embodiment;

[0064] Figure 5 A schematic diagram of calculating the sight line trajectory of the eyes of the person to be tested on the screen in an embodiment;

[0065] Figure 6 This is a screen attention heat map in the embodiment;

[0066] Figure 7 Schematic diagram of a micro-behavior recognition detection area determined in the embodiment;

[0067] Figure 8 Schematic diagram of the key point positions of the hand in the embodiment;

[0068] Fig. 9 Schematic diagram of the trajectory of the hand key points in the video sequence during implementation;

[0069] Fig.10 Schematic diagram of the positions of the inner canthus, outer canthus, upper eyelid and lower eyelid of the eye area in the embodiment;

[0070] Fig.11 Schematic diagram of blinking state changes in the embodiment, where (a) is eyes open, (b) is eyes closed, and (c) is the change trend of blink identification parameters based on time. DETAILED DESCRIPTION

[0071] In order to facilitate those skilled in the art to better understand the improvements of the present invention relative to the prior art, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0072] In view of the extremely high requirements for the concentration of workers in operating environments such as aviation dispatching and nuclear power safety control, this embodiment provides a concentration monitoring method to monitor the distraction of workers during operation, and eliminate the unstable factors caused by the distraction of workers as much as possible, so as to ensure the safe operation and operation of the equipment. However, many of the existing methods for detecting distraction of car drivers use contact sensors to detect the distraction of drivers, which are less concealed and convenient. If they are directly applied to operating environments such as aviation dispatching and nuclear power safety control, the contact sensors worn on the body will obviously have a certain impact on the workers, and may even cause distraction due to discomfort caused by wearing these devices. Therefore, the contact collection method is not suitable for this operating environment. Although some of the existing methods for detecting distraction of car drivers use non-contact detection methods, most of these methods still have the problem of low judgment accuracy due to too coarse detection granularity, and the problem of difficulty in distinguishing the situation of the tested person being dazed due to the lack of time series information. In order to solve these problems, this embodiment provides a non-contact concentration monitoring method and improves the accuracy of detection. Through weighted analysis of multiple detection methods such as micro-behavior recognition, eye trajectory and screen attention heat map detection, blink feature detection, etc., it can more accurately determine whether the person being tested is in a distracted state, thereby reducing misjudgments and missed judgments.

[0073] The present invention adopts a non-contact method, using preset non-contact sensors such as cameras and a distraction detection and analysis system. Without contacting the person being detected, it can perform non-contact detection of the distraction state of office workers in front of the computer, thereby achieving the purpose of automatically detecting the distraction of office workers when using computers. The concentration monitoring method therein uses multiple detection sub-functions, and after the results are aggregated, they are analyzed through weighted calculation (the weight value of each indicator can be determined through experiments) to analyze the distraction state of the person to be detected. The distraction of office workers (including not only operators of aviation dispatch and nuclear power safety control, but also other office workers who work with computers) has the following characteristics: static behavior, abnormal eye trajectory, and abnormal blinking characteristics, so the detection method includes the following detection systems: 1. Eye trajectory and screen attention heat map detection; 2. Micro-behavior recognition; 3. Blinking feature detection and other parts. The entire detection process is as shown in the attached. Figure 1 The following example uses an office worker sitting at a desk and working in front of a computer as an example. Figures 1 to 11 .

[0074] 1. Collect image information.

[0075] The image acquisition device is mainly used to collect image information of the person to be tested and the equipment. The image acquisition device includes a camera and may also include a video camera. The collected image information of the person to be tested includes image information of the eye features, hand features and facial features of the person to be tested. The collected image information of the equipment includes image information of the human-computer interaction input device and the display screen. The human-computer interaction input device mainly refers to the keyboard and mouse, and of course it can also be other devices such as a control joystick and a handle.

[0076] 2. Micro-behavior recognition and detection.

[0077] When office workers use computers to work, their actions are relatively small, involving only operations such as typing on the keyboard and operating the mouse. Therefore, micro-behavior recognition detection can be performed on the office workers' hands, and the desk area can be photographed with a camera to detect the status of the office workers' hands.

[0078] The area where the human-computer interaction input device is located in the image is taken as the micro-behavior recognition detection area, and whether the hand features of the person to be tested appear in the micro-behavior recognition detection area is detected. If not, proceed to the next step and perform other detection methods. If yes, perform micro-behavior recognition detection on the hand features appearing therein to determine whether the hands of the person to be tested are active or still. If active, the person to be tested is determined to be in a working state. If still, proceed to the next step and perform other detection methods.

[0079] The step of determining the micro-behavior recognition detection area includes:

[0080] a. Use the target detection technology in image processing to detect and determine the regional positions R of the keyboard and mouse in the image respectively. keyboard and R mouse , where R keyboard and R mouse is defined as:

[0081] R keyboard =((x left ,y left ), (x right ,y right ));

[0082] R mouse =((x left ,y left ), (x right ,y right ));

[0083] Among them, (x left ,y left ) represents region R* The coordinate position of the upper left corner, (x right ,y right ) represents region R * The coordinate position of the lower right corner.

[0084] b. Use the keyboard and mouse to determine the image area R to be detected for micro-behavior recognition detection , the regional coordinate parameters are calculated as follows:

[0085]

[0086] Attached Figure 7 The white rectangular area in the middle is the calculated micro-behavior recognition detection area.

[0087] The steps of micro-behavior recognition and detection include:

[0088] The key point information on the hand features that appear in the micro-behavior recognition detection area (including the fingertips, finger joints, palm center positions and wrist joint positions) is detected, and then the trajectory of the key points of the hand features in the video sequence is used to determine whether the hands of the person being tested are active or still.

[0089] Specifically, it detects whether there are two hands in the area, and detects the key point information of the two hands. The key point information of the two hands includes the fingertips and joints of each finger, the center position of the palm and the position of the wrist joint; see the attached Figure 8 ;

[0090] Through the trajectory of the key points of both hands in the video sequence, it can be determined whether the hands are still, typing on the keyboard or controlling the mouse. If both hands are operating the mouse and keyboard, it means that the person being tested is working and is not distracted. Fig. 9 .

[0091] 3. Detection of the line of sight of the person being tested on the monitor screen and the screen attention heat map.

[0092] Gaze trajectory and screen attention heat map detection refers to obtaining the gaze trajectory of the person to be tested on the screen respectively, and combining it with the attention area on the computer screen, and judging whether the person to be tested is distracted by comparing the similarity between the two.

[0093] The step of obtaining the sight track includes:

[0094] a. Extract and analyze the eye features of the person to be tested, and infer the line of sight through the eye features; a deep neural network can be selected as a tool to fit two angle parameters of the eyeball: pitch (vertical direction) and yaw (horizontal direction) through the detected eye pictures; see the attached Figure 2 .

[0095] b. Detect several feature values ​​of the face (such as the 68 feature points commonly used at present) and compare them with the feature values ​​of the face in the standard posture, so as to estimate the posture relationship between the face and the camera; see the attached Figure 3 , 4 ;

[0096] In the real facial image, several unobstructed feature points are selected and the head posture is estimated using the PnP method to obtain the posture information of the face relative to the camera imaging plane.

[0097] c. Combine the positional relationship between the face of the person to be tested and the display screen to calculate the line of sight of the person to be tested on the display screen. Figure 5 ;

[0098] (1) The transformation matrix R between the face and the screen can be calculated by the relationship between the spatial coordinates of the installed camera and the plane position of the computer screen.

[0099] (2) The position of the face relative to the screen can be calculated through the transformation matrix, and the position of the line of sight on the screen can be calculated in real time by estimating the distance between the face and the screen. t ;

[0100] (3) By accumulating over time, we can obtain the position sequence of the line of sight on the screen (P i ), where i ≥ 0, and the sequence length is related to the sampling frequency and sampling duration.

[0101] The detection program detects the screen content and uses the attention detection network to analyze the screen content and generate a screen attention heat map. Figure 6 .

[0102] The heat map has the same size as the screen, and the brightness of each position is proportional to the importance of the corresponding content. The brightness information of each position can be converted into a two-dimensional distribution Prob S , the conversion method is:

[0103]

[0104] Among them, Bright(i,j) is the brightness value of the screen coordinate position ij. It is easy to know from the calculation results that

[0105]

[0106] Among them, width and height are the resolution of the screen, and (i, j) is the coordinate position of the sampling point on the screen.

[0107] Then, the degree of consistency between the attention heat map and the eye trajectory is calculated and compared with the set threshold. When the threshold is exceeded, it is considered that the person being tested is working seriously, otherwise there is a possibility of distraction.

[0108] The calculation method is to treat the eye's sight track on the screen and the screen's attention heat map as a two-dimensional distribution. After normalizing the two distributions and sampling them according to the screen resolution, the distribution function Prob of the sight track is obtained. Eye , the distribution of the screen attention heat map is Prob Screen , then the similarity between the two can be calculated by cross entropy:

[0109]

[0110] Among them, width and height are the resolution of the screen, and (i, j) is the coordinate position of the sampling point on the screen.

[0111] 4. Testing of the trend of the line of sight of the person being tested on the display screen and prior knowledge.

[0112] After checking the degree of fit, it is also necessary to determine the directionality of the sight trajectory. Due to the particularity of computer operation, the operator's sight usually starts from the middle and moves from the upper left to the lower right. Analyze the movement trend of the sight trajectory to see if it is consistent with prior knowledge.

[0113] Let the coordinate information of the i-th point of the sight track be Trail i (x, y), the direction in which the line of sight moves on the screen can be determined by calculating the gradient of the line of sight trajectory.

[0114]

[0115] Then through The positive and negative signs of the two dimensions can determine the direction of the trajectory movement.

[0116] 5. Blink feature detection and analysis.

[0117] Blinking features refer to some blink-related information analyzed based on the feature point parameters after the computer vision technology collects the eye feature points of the person to be tested.

[0118] 1. Use image processing technology to extract the face position in the image and segment the eye area;

[0119] 2. From the eye area, detect the positions of several key points of the eye, including the inner corner of the eye, the outer corner of the eye, the upper eyelid and the lower eyelid; see the attached Fig.10 ;

[0120] 3. Calculate the ratio of the distance between the upper and lower eyelids to the distance between the inner and outer corners of the eye. Because the distance between the left and right corners of the eye remains unchanged, and the distance between the upper and lower eyelids is related to the state of opening and closing the eyes, the blinking frequency and the duration of closing the eyes can be calculated based on the ratio of the distance between the upper and lower eyelids to the distance between the inner and outer corners of the eye, combined with time information;

[0121] Let the distance between the inner corner of the eye P1 and the outer corner of the eye P4 be Dis W , the distance between the upper eyelid P2 and the lower eyelid P6 is Dis H , then the calculation method of the blink discrimination parameter is:

[0122]

[0123] By periodically calculating the blink discrimination parameter Blink (e.g., 10 Hz) and plotting the time-based change trend, a blink state change graph can be obtained; see the attached Fig.11 .

[0124] 4. According to the duration of closing eyes close , it can determine whether the monitored person is working fatigued and make a judgment on the abnormal state of the person being monitored.

[0125]

[0126] 6. After aggregating the test results, the distraction state of the person to be tested can be analyzed through weighted calculation.

[0127] In addition, let the sight track detection result be Result praj , the hand micro-behavior detection result is Result aetion , the blink detection result is Result blink , then the detection result of the whole system is

[0128] Result = α × Result praj +β×Result action +(1-α-β)×Result blmk ;

[0129] Among them, 0≤α≤1, 0≤β≤1, 0≤α+β≤1.

[0130] The concentration monitoring method provided by the present invention is different from the common contact distraction detection method in the past. It adopts a non-contact detection method, which does not require the person to be tested to wear equipment. It only needs to use non-contact equipment such as cameras to collect image information, which has better concealment and convenience. It determines the state of the person to be tested through micro-behavior recognition, eye trajectory and screen attention heat map detection, blinking feature detection and other methods. Through weighted calculation (the weight value of each indicator can be determined by experiment), the concentration of the person to be tested is comprehensively analyzed. The detection result obtained based on this is more accurate than a single judgment method. Compared with the existing automobile driver distraction detection method, the concentration monitoring method provided by the present invention is better in concealment and convenience, and has higher detection accuracy. It can be applied to personnel distraction detection in operating environments such as aviation dispatch and nuclear power safety control.

[0131] It should be noted that the concentration monitoring method of the present invention utilizes non-contact devices such as cameras to collect required image information, such as collecting images of the hands and keyboard and mouse of the personnel to be tested for micro-behavior recognition detection, images of the eyes of the personnel to be tested for blink detection, images of faces for detecting sight line tracks, etc., and then making preliminary judgments according to whether the hands of the personnel to be tested appear in the micro-behavior recognition detection area, and then making further judgments according to the activity state of the hands appearing in the micro-behavior recognition detection area, and then making further judgments by comparing the sight line tracks of the personnel to be tested on the display screen with the screen attention heat map, and then making further judgments by comparing the sight line tracks with prior knowledge, and then making further judgments according to the blinking frequency of the personnel to be tested, and finally collecting various test results and performing weighted calculations to judge the current state (working state, distracted state) of the personnel to be tested. The present invention can avoid many situations of misjudgment and missed judgment under the combination of multiple detection methods, and effectively improves the accuracy of detection. Among them, in the process of micro-behavior recognition, the person may be distracted or not distracted if both hands are not operating the keyboard and mouse. If the traditional single judgment method is used to directly judge the status of the person (for example, judging whether the person is working or distracted based on whether both hands are operating the keyboard and mouse), there will obviously be misjudgment. The present invention takes into account multiple possibilities, and through multi-stage detection and multiple optimizations at each level, distraction detection is made more accurate.

[0132] In addition, this embodiment further provides a concentration monitoring system, which includes a storage unit and a computing unit. The storage unit stores a concentration monitoring program, and the concentration monitoring program is run through the computing unit to execute the steps in the above-mentioned concentration monitoring method.

[0133] The above embodiments are preferred implementation schemes of the present invention. In addition, the present invention may also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the protection scope of the present invention.

[0134] In order to make it easier for ordinary technicians in the field to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified, and for the sake of clarity, some other elements are omitted in this application document. Ordinary technicians in the field should realize that these omitted elements may also constitute the content of the present invention.

Claims

1. A concentration monitoring method, characterized in that: The following steps are involved:

1. Use the image acquisition device to collect image information of the person and equipment to be tested; Second, the area where the human-computer interaction input device is located in the image is used as the micro-behavior recognition detection area, and whether the hand of the person to be tested appears in the area is detected. The hand that appears in the area is subjected to micro-behavior recognition detection to obtain the state of the hand of the person to be tested and determine whether the hand is active or not; 3. Obtain the eye track of the person under test on the monitor screen, and compare the similarity between the two in combination with the attention heat map of the monitor screen; 4. Analyze the movement trend of the sight trajectory and determine whether it is consistent with prior knowledge; 5. Detect the eye features of the person to be tested, obtain the blinking frequency of the person to be tested, and determine whether it is normal; 6. Combining the above test results and performing weighted calculation can determine whether the person to be tested is in working condition; In step three, the steps of obtaining the sight track include: a. Extract and analyze the eye features of the person to be tested to obtain the direction of sight; b. Detecting several feature values ​​of the face and comparing them with the feature values ​​of the face in a standard posture to obtain the posture relationship between the face and the image acquisition device; c. Calculate the sight track of the person under test on the display screen based on the positional relationship between the face of the person under test and the display screen, the steps include: (1) The transformation matrix between the face and the screen is calculated by the relationship between the spatial coordinates of the installed camera and the plane position of the computer screen; (2) The position of the face relative to the screen is calculated using the transformation matrix, and the position of the line of sight on the screen is calculated using the estimated distance between the face and the screen; (3) Obtain the position sequence of the gaze on the screen by accumulating it over time; In step 3, the distribution function of the screen attention heat map is Prob s : Among them, Bright(i, j) is the brightness value of the screen coordinate position ij, Where width and height are the screen resolutions, and (i, j) is the coordinate position of the sampling point on the screen; In step three, the steps of detecting the similarity between the eye track and the screen attention heat map include: Normalize the gaze trajectory and the screen attention heat map, and sample according to the screen resolution to obtain the distribution function Prob of the gaze trajectory Eye , the distribution function of the screen attention heat map is Prob Screen , the similarity between the two is calculated by cross entropy: Where width and height are the screen resolutions, and (i, j) is the coordinate position of the sampling point on the screen; The calculated gaze trajectory is compared with the similarity of the screen attention heat map and the pre-set threshold to determine whether the person being tested is in a working state.

2. The concentration monitoring method according to claim 1, characterized in that: In step one, the image acquisition device includes a camera, and the collected image information of the person to be tested includes image information of the eye features, hand features and facial features of the person to be tested, and the collected image information of the device includes image information of the human-computer interaction input device and the display screen.

3. The concentration monitoring method according to claim 1, characterized in that: In step 2, the human-computer interaction input device includes a keyboard and a mouse, and the steps of determining the micro-behavior recognition detection area include: a. Detect and determine the area position R of the keyboard and mouse in the image keyboard and R mouse , where R keyboard and R mouse for: R keyboard =((x left ,y left ),(x right ,y right )); R mouse =((x left ,y left ),(x right ,y right )); Among them, (x left ,y left ) represents region R * The coordinate position of the upper left corner, (x right ,y right ) represents region R * The coordinate position of the lower right corner; b. Determine the micro-behavior recognition detection area R through the regional positions of the keyboard and mouse detection , the regional coordinate parameters are:

4. The concentration monitoring method according to claim 1, characterized in that: In step 2, the steps of micro-behavior recognition and detection include: The key point information on the hand features appearing in the micro-behavior recognition detection area, including the fingertips, finger joints, palm center and wrist joints, is detected, and then the trajectory of the key points of the hand features in the video sequence is used to determine whether the hand of the person being tested is active or still.

5. The concentration monitoring method according to claim 1, characterized in that: In step 4, the prior knowledge is that the line of sight of the person being measured starts from the middle of the display screen and moves from the upper left position to the lower right position; the steps of determining the direction of movement of the line of sight trajectory include: By calculating the gradient of the sight trajectory, we can determine the direction in which the sight moves on the screen: Among them, Trail i (x, y) is the coordinate information of the i-th point of the sight track; pass The positive and negative signs of the two dimensions determine the direction of the trajectory movement.

6. The concentration monitoring method according to claim 1, characterized in that: In step 5, the step of detecting the blinking frequency of the person to be tested includes: a. Extract the face position in the image and segment the eye area; b. Detect the positions of several key points of the eyes from the eye area, including the inner corner of the eye, the outer corner of the eye, the upper eyelid and the lower eyelid; c. Calculate the ratio of the distance between the upper and lower eyelids to the distance between the inner and outer corners of the eye, and combine the time information to calculate the blinking frequency and the duration of eye closure; d. Blink identification parameter Blink is: Among them, Dis W Dis is the distance between the inner and outer corners of the eye, H It is the distance between the upper eyelid and the lower eyelid; The blink discrimination parameter Blink is calculated periodically to obtain the time-based change trend; e. According to the duration of closing eyes close , determine whether the person being tested is working fatigued and make a judgment on the abnormal state of the person being tested: Among them, threshold is the threshold.

7. A concentration monitoring system, comprising a storage unit and a computing unit, wherein the storage unit stores a concentration monitoring program, and the concentration monitoring program is run by the computing unit to execute the steps of the concentration monitoring method according to any one of claims 1 to 6.

Images