Controller fatigue state detection method based on infrared visible light image fusion

Abstract

The embodiment of the invention provides a controller fatigue state detection method based on infrared visible light image fusion. The method is used in the air-traffic-control safety field. The method comprises the following steps: carrying out image processing and face detection on an image after fusion of collected infrared visible light images, and extracting face features; determining main variables influencing fatigue according to the face features and an ordered multi-classification discrete selection model; determining a fatigue influence factor corresponding to the face feature according to the main variable and each preset fatigue grade critical value; determining a fatigue coefficient and a fatigue score corresponding to each fatigue sensitive action according to each fatigue sensitive action; establishing a controller fatigue evaluation mechanism so that a teacher / controller can quantitatively evaluate the control ability of the controller; and if the control capability of the non-ticket override is quantitatively evaluated, calculating a fatigue integral according to the fatigue coefficient and the fatigue score, determining a fatigue grade, and executing corresponding reminding. Therefore, the controller is prevented from being interfered during image data acquisition, the fatigue state is quantified, and the fatigue state recognition precision and efficiency are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of air traffic control safety, and in particular to a method for detecting fatigue status of a controller based on infrared and visible light image fusion. Background Art

[0002] Air traffic controllers are important personnel who ensure the safe operation of aircraft. Having a good mental and physical state is the prerequisite for completing this mission. Identifying the fatigue state of air traffic controllers in advance can help adjust the working state of air traffic controllers at the right time, thereby effectively avoiding air traffic safety risks caused by air traffic controller fatigue.

[0003] When using computer vision technology to study the fatigue state of air traffic controllers, usually only visible light images are used as input data for human fatigue state detection. However, the human images obtained by this method are easily disturbed by external light and lose some information, which leads to a decrease in the accuracy of fatigue state recognition. In addition, the existing technology does not effectively establish a scientific connection between the fatigue degree and fatigue level of air traffic controllers, nor does it quantitatively evaluate the fatigue state of air traffic controllers. Therefore, the detection method of air traffic controller fatigue state needs further research. Summary of the invention

[0004] The present disclosure provides a method, an apparatus, a device and a storage medium for detecting fatigue status of a controller based on infrared and visible light image fusion.

[0005] According to a first aspect of the present disclosure, a method for detecting fatigue status of a controller based on infrared and visible light image fusion is provided. The method comprises: Perform image fusion on infrared and visible light images of controllers collected by self-developed image acquisition tools; Performing face detection on the fused image after image processing to determine the face image; extracting face feature information from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; According to the facial feature information, factor analysis is used to determine the main variables that affect controller fatigue through an ordered multi-classification discrete selection model; Determining a fatigue influencing factor corresponding to the facial feature information according to the main variables and the pre-classified critical values ​​of each fatigue level; When the controller performs fatigue-sensitive actions, the facial feature information corresponding to each fatigue-sensitive action is determined according to each fatigue-sensitive action, and the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action are determined according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; A controller fatigue evaluation mechanism based on visual technology is established so that instructors or controllers themselves can quantitatively evaluate the controller's control ability based on the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the controller's control ability in the quantitative evaluation, the fatigue score is calculated based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, the fatigue level is determined based on the fatigue score, and corresponding reminders are executed based on the fatigue level.

[0006] In some implementations of the first aspect, extracting facial feature information from a determined facial image includes: Eye movement feature information is determined based on the eye aspect ratio and the percentage of eye closure. Mouth movement feature information is determined based on the mouth aspect ratio and the percentage of mouth opening. Head movement feature information is determined based on the pitch angle of the head when it is raised up and down and the horizontal rotation angle of the head when it is turned left and right. Distance feature information is determined based on the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging.

[0007] In some implementations of the first aspect, determining eye motion feature information according to an eye aspect ratio and an eye closure percentage includes: Calculate the eye aspect ratio corresponding to each image frame in the continuous image frames of the face, and make an eye state map according to the eye aspect ratio corresponding to each image frame; determining eye posture and blinking frequency according to the eye state map; The number of eye-closing frames in continuous face image frames is determined according to the eye posture, and the eye-closing duration is calculated according to the number of eye-closing frames using the eye-closing percentage.

[0008] In some implementations of the first aspect, determining the mouth movement feature information according to the mouth aspect ratio and the mouth opening percentage includes: Determine the mouth area according to the key point information on the face and calculate the mouth aspect ratio corresponding to each image frame in the continuous image frames of the face; Determine the mouth state corresponding to each image frame according to the mouth aspect ratio corresponding to each image frame; Determine the mouth opening duration using the mouth opening percentage according to the mouth state corresponding to each image frame; Determine yawning status based on the duration of mouth opening.

[0009] In some implementations of the first aspect, determining head motion feature information according to a pitch angle of raising the head up and down and a horizontal rotation angle of turning the head left and right includes: The mapping relationship between the face image and the key point information on the face in three-dimensional space is constructed by feature space transformation method; Calculate the pitch angle of the head up and down and the horizontal rotation angle of the head turning left and right corresponding to each image frame in the continuous image frames of the face according to the mapping relationship and the key point information on the face in three-dimensional space; The duration of lowering or raising the head is determined according to the pitch angle of the head corresponding to each image frame; the duration of turning the head left or right is determined according to the horizontal rotation angle of the head corresponding to each image frame.

[0010] In some possible implementations of the first aspect, the distance feature information is determined according to the focal length of the self-developed image acquisition tool, the human pupil distance, and the pixel distance after pupil imaging, including: According to the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging, the distance between the person corresponding to each image frame in the continuous image frames of the face and the self-developed image acquisition tool is calculated by using the triangulation method; The duration during which the distance between the person and the self-developed image acquisition tool exceeds 90 centimeters is obtained according to the distance between the person and the self-developed image acquisition tool corresponding to each image frame.

[0011] In some implementations of the first aspect, calculating the fatigue score according to the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action includes: The fatigue score corresponding to the fatigue-sensitive action is taken as the product of the fatigue coefficient corresponding to the fatigue-sensitive action and the fatigue score corresponding to the fatigue coefficient corresponding to the fatigue-sensitive action; The accumulated value of the fatigue points corresponding to the fatigue-sensitive actions is used as the fatigue point for determining the fatigue level.

[0012] In some implementations of the first aspect, so that an instructor or a controller can quantitatively evaluate the control ability of a controller according to the controller fatigue evaluation mechanism based on visual technology, the method includes: The instructor or the controller himself monitors the controller's control ability and conducts quantitative evaluation of the monitoring situation based on the controller fatigue evaluation mechanism based on visual technology; If the instructor's quantitative evaluation score is lower than 90 points or the controller proactively proposes to suspend command, it means a veto of the controller's control ability.

[0013] After determining the fatigue level according to the fatigue score and executing corresponding reminders according to the fatigue level, the method includes: If the execution of the corresponding reminder according to the fatigue level fails, the fatigue points are calculated, the fatigue level is determined, and the corresponding reminder is executed according to the fatigue level repeatedly until the execution of the corresponding reminder according to the fatigue level is successful.

[0014] According to a second aspect of the present disclosure, a controller fatigue state detection device based on infrared and visible light image fusion is provided. The device comprises: Infrared and visible light image acquisition module, used to fuse infrared and visible light images of controllers collected by self-developed image acquisition tools; A face feature information extraction module is used to perform face detection after image processing on the fused image to determine the face image; extract face feature information from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; A fatigue main variable acquisition module is used to determine the main variables affecting the fatigue of the controller by using the factor analysis method through an ordered multi-classification discrete selection model according to the facial feature information; A fatigue influence factor determination module, used to determine the fatigue influence factor corresponding to the facial feature information according to the main variables and the critical values ​​of each fatigue level divided in advance; A fatigue coefficient and fatigue score determination module is used to determine the facial feature information corresponding to each fatigue-sensitive action according to each fatigue-sensitive action when the controller performs a fatigue-sensitive action, and to determine the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; The quantitative evaluation module is used to establish a controller fatigue evaluation mechanism based on visual technology, so that instructors or controllers themselves can quantitatively evaluate the controller's control ability based on the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the controller's control ability in the quantitative evaluation, the fatigue score is calculated based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, the fatigue level is determined based on the fatigue score, and corresponding reminders are executed based on the fatigue level.

[0015] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described above.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.

[0017] In the present disclosure, infrared and visible light images of air traffic controllers collected by self-developed image acquisition tools are fused; face detection is performed on the fused images after image processing to determine the face image; face feature information is extracted from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; the main variables affecting the fatigue of air traffic controllers are determined by factor analysis through an ordered multi-classification discrete selection model based on the face feature information; the fatigue influencing factors corresponding to the face feature information are determined based on the main variables and the critical values ​​of each fatigue level divided in advance; when the air traffic controller has fatigue-sensitive actions, Determine the facial feature information corresponding to each fatigue-sensitive action according to each fatigue-sensitive action, and determine the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; establish a fatigue evaluation mechanism for controllers based on visual technology, so that instructors or controllers themselves can quantitatively evaluate the control ability of controllers according to the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the control ability of the controller in the quantitative evaluation, calculate the fatigue score according to the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, determine the fatigue level according to the fatigue score, and perform corresponding reminders according to the fatigue level. In this way, it is possible to avoid interference with the controller when collecting image data, quantify the fatigue state, and improve the accuracy and efficiency of fatigue state recognition.

[0018] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which: Figure 1 A flow chart of a method for detecting fatigue status of a controller based on infrared and visible light image fusion provided by an embodiment of the present disclosure is shown; Figure 2 A structural diagram of a controller fatigue state detection device based on infrared and visible light image fusion provided by an embodiment of the present disclosure is shown; Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0021] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0022] In response to the problems arising from the background technology, the embodiments of the present disclosure provide a method and device for detecting the fatigue status of air traffic controllers based on infrared and visible light image fusion. Specifically, image fusion is performed on the infrared and visible light images of air traffic controllers acquired by a self-developed image acquisition tool; face detection is performed on the fused images after image processing to determine the face image; facial feature information is extracted from the determined face image; wherein the facial feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; based on the facial feature information, the factor analysis method is used to determine the main variables affecting the fatigue of the air traffic controller through an ordered multi-classification discrete selection model; based on the main variables and the critical values ​​of each fatigue level divided in advance, the fatigue influencing factor corresponding to the facial feature information is determined; when the air traffic controller has fatigue-sensitive movements, the fatigue influencing factor is determined based on the facial feature information. Determine the facial feature information corresponding to each fatigue-sensitive action according to each fatigue-sensitive action, and determine the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; establish a fatigue evaluation mechanism for controllers based on visual technology, so that instructors or controllers themselves can quantitatively evaluate the control ability of controllers according to the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the control ability of the controller in the quantitative evaluation, calculate the fatigue score according to the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, determine the fatigue level according to the fatigue score, and perform corresponding reminders according to the fatigue level. In this way, interference to the controller can be avoided when collecting image data, the fatigue state can be quantified, and the fatigue state recognition accuracy and efficiency can be improved.

[0023] The following describes in detail the method and device for detecting fatigue status of a controller based on infrared and visible light image fusion provided by the embodiments of the present disclosure through specific embodiments in conjunction with the accompanying drawings.

[0024] Figure 1A flowchart of a method for detecting fatigue status of a controller based on infrared and visible light image fusion provided by an embodiment of the present disclosure is shown. The method 100 includes the following steps: S110, performs image fusion on the infrared and visible light images of the controller collected by the self-developed image acquisition tool.

[0025] In some embodiments, an infrared camera and a visible light camera are integrated into the self-developed image acquisition tool.

[0026] S120, performing face detection after image processing on the fused image to determine the face image; extracting face feature information from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information.

[0027] In some embodiments, image processing includes image transformation, compression coding, image enhancement and restoration. Image processing can improve image quality and improve features of faces in the image.

[0028] In some embodiments, extracting facial feature information from the determined facial image includes: Eye movement feature information is determined based on the eye aspect ratio and the percentage of eye closure. Mouth movement feature information is determined based on the mouth aspect ratio and the percentage of mouth opening. Head movement feature information is determined based on the pitch angle of the head when it is raised up and down and the horizontal rotation angle of the head when it is turned left and right. Distance feature information is determined based on the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging.

[0029] In some embodiments, determining eye movement feature information based on the eye aspect ratio and the eye closure percentage includes: Calculate the eye aspect ratio corresponding to each image frame in the continuous image frames of the face, and make an eye state map according to the eye aspect ratio corresponding to each image frame; determining eye posture and blinking frequency according to the eye state map; Determine the number of eye-closing frames in the continuous image frames of the face according to the eye posture, and calculate the eye-closing duration according to the number of eye-closing frames using the eye-closing percentage; Furthermore, the eye aspect ratio is the ratio of the eye longitudinal value to the eye transverse value; wherein the eye longitudinal value is the distance between the key points of the upper and lower eyelids, and the eye transverse value is the distance between the key points of the inner and outer corners of the eye; The eye closure percentage is the ratio of the number of eye-closing frames to the total number of continuous face image frames; Eye gestures include eyes open or closed.

[0030] In some embodiments, the eye aspect ratio is 0.419 when the eye is open and 0.079 when the eye is closed.

[0031] In some embodiments, determining mouth movement feature information based on the mouth aspect ratio and the mouth opening percentage includes: Determine the mouth area according to the key point information on the face and calculate the mouth aspect ratio corresponding to each image frame in the continuous image frames of the face; Determine the mouth state corresponding to each image frame according to the mouth aspect ratio corresponding to each image frame; Determine the mouth opening duration using the mouth opening percentage according to the mouth state corresponding to each image frame; Determine yawning status based on the duration of mouth opening; Further, the mouth aspect ratio is the ratio of the mouth longitudinal value to the mouth transverse value; Mouth states include mouth open and closed; The mouth opening percentage is the ratio of the number of image frames with the mouth open to the total number of continuous image frames of the face; When the duration of mouth opening reaches the preset yawning time threshold, it indicates that the controller is in a yawning state.

[0032] In some embodiments, the mouth aspect ratio is 1.219 when the mouth is open and 0.471 when the mouth is closed.

[0033] In some embodiments, the head motion feature information is determined based on the pitch angle of the head raising up and down and the horizontal rotation angle of the head turning left and right, including: The mapping relationship between the face image and the key point information on the face in three-dimensional space is constructed by feature space transformation method; Calculate the pitch angle of the head up and down and the horizontal rotation angle of the head turning left and right corresponding to each image frame in the continuous image frames of the face according to the mapping relationship and the key point information on the face in three-dimensional space; Determine the duration of lowering or raising the head according to the pitch angle of the head up and down corresponding to each image frame; determine the duration of turning the head left or right according to the horizontal rotation angle of the head turning left and right corresponding to each image frame; Furthermore, the rotation angle of the head swinging left and right corresponding to each image frame in the continuous image frames of the face can also be calculated according to the mapping relationship and the key point information on the face in three-dimensional space.

[0034] In some embodiments, a positive pitch angle of the head when raising it up and down indicates lowering the head, and a negative angle indicates raising the head; a positive horizontal rotation angle of the head when turning it left and right indicates turning the head to the left, and a negative angle indicates turning the head to the right; a positive rotation angle of the head when swinging it left and right indicates shaking the head to the left, and a negative angle indicates shaking the head to the right.

[0035] In some embodiments, the distance feature information is determined based on the focal length of the self-developed image acquisition tool, the human pupil distance, and the pixel distance after pupil imaging, including: According to the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging, the distance between the person corresponding to each image frame in the continuous image frames of the face and the self-developed image acquisition tool is calculated by using the triangulation method; The duration during which the distance between the person and the self-developed image acquisition tool exceeds 90 centimeters is obtained according to the distance between the person and the self-developed image acquisition tool corresponding to each image frame.

[0036] S130, determining the main variables that affect the controller's fatigue by using a factor analysis method and an ordered multi-classification discrete selection model according to the facial feature information.

[0037] In some embodiments, the main variables affecting controller fatigue are determined by using factor analysis through an ordered multi-classification discrete selection model based on the facial feature information, including: According to the facial feature information, fatigue latent variables are extracted from pre-established fatigue status evaluation indicators using factor analysis; the fatigue latent variables are input into a pre-established ordered multi-classification discrete selection model, and the main variables affecting controller fatigue are output through the ordered multi-classification discrete selection model.

[0038] In some embodiments, fatigue status assessment indicators include: Closing the left eye easily leads to fatigue state, closing the right eye easily leads to fatigue state, yawning easily leads to fatigue state, lowering the head easily leads to fatigue state, turning the head left and right is related to fatigue state, shaking the head left and right is related to fatigue state, and the change of distance between people and self-developed image acquisition tools is related to fatigue state.

[0039] In some embodiments, fatigue latent variables include: Closing the left and / or right eye, yawning, lowering the head, turning the head left and right, shaking the head left and right, and changes in the distance between the person and the self-developed image acquisition tool.

[0040] In some embodiments, the ordered multi-class discrete choice model is established by the following steps: The existing fatigue latent variables are used as training samples; the main variables that affect the controller's fatigue in the existing fatigue latent variables are used as annotations; Train the preset model using the utility function according to the training samples; Obtain the difference between the main variables influencing the fatigue of the controllers in the training output and the main variables influencing the fatigue of the controllers in the existing fatigue latent variables; Based on the difference, the model parameters are determined using the maximum likelihood estimation method; The model after the model parameters are determined is regarded as an ordered multi-class discrete choice model.

[0041] In some embodiments, the ordered multi-class discrete choice model comprises a logistic model.

[0042] S140, determining a fatigue influencing factor corresponding to the facial feature information according to the main variable and the pre-classified critical values ​​of each fatigue level.

[0043] In some embodiments, the fatigue influencing factor corresponding to the facial feature information is determined based on the main variables and the pre-classified critical values ​​of each fatigue level, including: Perform critical value segmentation on each fatigue level divided in advance; determine the probability of fatigue corresponding to each fatigue level based on the main variables and each critical value obtained by segmentation; determine the fatigue influencing factor corresponding to the facial feature information based on the probability of fatigue corresponding to each fatigue level.

[0044] In some embodiments, fatigue levels include Level A (indicating falling asleep and losing control and command capabilities), Level B (indicating deep fatigue, with a significant decline in control work and communication capabilities), Level C (feeling tired, with a significant decline in control work and communication capabilities), Level D (slightly tired, with control work and communication slightly affected), and Level E (normal and effective control work and communication).

[0045] In some embodiments, the magnitude relationship of the fatigue influencing factors corresponding to the facial feature information is as follows: the fatigue influencing factors corresponding to the eye movement feature information, mouth movement feature information, head movement feature information, and distance feature information gradually decrease, which means that the eye movement feature information has the greatest impact on the fatigue state, and the distance feature information has the least impact on the fatigue state.

[0046] S150, when the controller performs fatigue-sensitive actions, the facial feature information corresponding to each fatigue-sensitive action is determined according to each fatigue-sensitive action, and the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action are determined according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information.

[0047] In some embodiments, the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action are determined according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue impact factor corresponding to the facial feature information, including: The fatigue influence factor corresponding to each fatigue-sensitive action is determined according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action are determined according to the fatigue influence factor corresponding to each fatigue-sensitive action.

[0048] In some embodiments, fatigue-sensitive actions include: Closing eyes, yawning, lowering head, raising head, the distance between the person and the self-developed image acquisition tool is too far, turning head to the left, turning head to the right.

[0049] In some embodiments, determining facial feature information corresponding to each fatigue-sensitive action according to each fatigue-sensitive action includes: The facial feature information corresponding to the fatigue-sensitive action of closing eyes is that both eyes are closed for more than 2 seconds; When the fatigue-sensitive action is yawning, the corresponding facial feature information is that the mouth opens for 3 seconds and then closes; The facial feature information corresponding to the fatigue-sensitive action of lowering the head is that the head lowers by more than 15 degrees and lasts for more than 2 seconds; The fatigue-sensitive action is the facial feature information corresponding to the head raising when the head is raised upward by more than 15 degrees and lasts for more than 2 seconds; The fatigue-sensitive action is when the distance between a person and the self-developed image acquisition tool is too far, and the corresponding facial feature information is when the distance between the person and the self-developed image acquisition tool exceeds 90 cm and lasts for more than 2 seconds; The fatigue-sensitive action is turning the head to the left, and the corresponding facial feature information is that the head rotates to the left by more than 30 degrees and lasts for more than 2 seconds; The fatigue-sensitive action is turning the head to the right, and the corresponding facial feature information is that the head rotates to the right by more than 30 degrees and lasts for more than 2 seconds.

[0050] In some embodiments, the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action include: The fatigue coefficient corresponding to the fatigue-sensitive action of closing eyes is 0.35, and the fatigue score is 2; When the fatigue-sensitive action is yawning, the corresponding fatigue coefficient is 0.25, and the fatigue score is 1.5; When the fatigue-sensitive action is lowering the head, the corresponding fatigue coefficient is 0.1, and the fatigue score is 1; The fatigue coefficient corresponding to the fatigue-sensitive action of raising the head is 0.1, and the fatigue score is 1; The fatigue-sensitive action is when the distance between the person and the self-developed image acquisition tool is too far, the corresponding fatigue coefficient is 0.1, and the fatigue score is 1; The fatigue-sensitive action is turning the head to the left, and the corresponding fatigue coefficient is 0.05, and the fatigue score is 1; The fatigue coefficient corresponding to the fatigue-sensitive action of turning the head to the right is 0.05, and the fatigue score is 1.

[0051] In some embodiments, if the controller does not perform fatigue-sensitive actions, the fatigue status detection is terminated.

[0052] S160, establish a controller fatigue evaluation mechanism based on visual technology, so that instructors or controllers themselves can quantitatively evaluate the controller's control ability based on the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the controller's control ability in the quantitative evaluation, calculate the fatigue score based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, determine the fatigue level based on the fatigue score, and perform corresponding reminders based on the fatigue level.

[0053] In some embodiments, the controller fatigue evaluation mechanism based on vision technology is established on the basis of the point fusion system PMS evaluation mechanism.

[0054] In some embodiments, so that the instructor or the controller can quantitatively evaluate the controller's control ability according to the controller fatigue evaluation mechanism based on visual technology, the method includes: The instructor or the controller himself monitors the controller's control ability and conducts quantitative evaluation of the monitoring situation based on the controller fatigue evaluation mechanism based on visual technology; If the instructor's quantitative evaluation score is lower than 90 points or the controller takes the initiative to suspend command, it means a veto of the controller's control ability; Further, regulatory capabilities include: Communication level, monitoring and repetition level, equipment operation ability, emergency response ability, controller awareness and judgment level, and control method level.

[0055] In some embodiments, if the instructor monitors that the controller causes the aircraft to enter a dangerous state due to personal reasons, issues wrong instructions, has bad control habits, does not have a clear grasp of basic control knowledge or skills, or handles special situations or conflicts improperly, the instructor will give a quantitative evaluation score lower than 90 points; When the controller determines that his physical condition is not good, he will take the initiative to suspend the command.

[0056] In some embodiments, the fatigue score is calculated according to the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, including: The fatigue score corresponding to the fatigue-sensitive action is taken as the product of the fatigue coefficient corresponding to the fatigue-sensitive action and the fatigue score corresponding to the fatigue coefficient corresponding to the fatigue-sensitive action; The accumulated value of the fatigue points corresponding to the fatigue-sensitive actions is used as the fatigue point for determining the fatigue level.

[0057] In some embodiments, determining the fatigue level according to the fatigue score and performing corresponding reminders according to the fatigue level include: When the fatigue score is within the score range of grade A fatigue, the fatigue level is determined to be grade A, and the first level reminder is used for grade A fatigue, such as text reminder, voice reminder, high-frequency warning sound, and pop-up window reminder; When the fatigue score is within the score range of B-level fatigue, the fatigue level is determined to be B-level, and the second-level reminder is used for B-level fatigue, such as text reminder, voice reminder, and pop-up window reminder at the same time; When the fatigue score is within the score range of C-level fatigue, the fatigue level is determined to be C-level, and the third-level reminder is used for C-level fatigue, for example, text reminder, voice reminder, etc. are simultaneously performed; When the fatigue score is within the score range of D-level fatigue, the fatigue level is determined to be D-level, and the fourth-level reminder is adopted for D-level fatigue, such as voice reminder, etc.; When the fatigue score is within the score range included in the E-level fatigue, the fatigue level is determined to be the E-level, and the fifth-level reminder is adopted for the E-level fatigue, for example, a text reminder.

[0058] In some embodiments, after determining the fatigue level according to the fatigue score and executing a corresponding reminder according to the fatigue level, the method includes: If the execution of the corresponding reminder according to the fatigue level fails, the fatigue points are calculated, the fatigue level is determined, and the corresponding reminder is executed according to the fatigue level repeatedly until the execution of the corresponding reminder according to the fatigue level succeeds; Furthermore, if the corresponding reminder is successfully executed according to the fatigue level, the fatigue score is cleared to end the fatigue status detection.

[0059] The above is an introduction to the method embodiment. The following further illustrates the scheme of the present disclosure by adopting a specific embodiment of the method.

[0060] Taking the detection of the fatigue state of controller A as an example, the infrared and visible light images of controller A collected by the self-developed image acquisition tool are fused, and face detection is performed after image processing on the fused images, and facial feature information is extracted from the detected faces; According to the facial feature information combined with the factor analysis method, the fatigue latent variables extracted from the pre-established fatigue status evaluation index include left eye and right eye closure, yawning, and head lowering. These fatigue latent variables are input into the ordered multi-classification discrete selection model. The main variables output by the ordered multi-classification discrete selection model are left eye and right eye closure, and yawning. The fatigue levels divided in advance were segmented by critical values, and the fatigue influencing factors corresponding to the facial feature information were determined based on the above main variables and the critical values ​​obtained by segmentation. It was found that the left and right eye closures had the greatest impact on the fatigue state of Person A, and the duration of eye closure obtained based on the eye closure percentage when extracting facial feature information exceeded 2 seconds, i.e., fatigue-sensitive actions occurred. According to the fatigue influencing factors corresponding to the facial feature information corresponding to the fatigue-sensitive actions, the fatigue coefficient corresponding to the fatigue-sensitive actions was determined to be 0.35, and the corresponding fatigue score was 2. The instructor's quantitative evaluation score for the controller's control ability was 95 points, so there was no veto, and the controller continued to perform control work. According to the The fatigue score calculated from the fatigue coefficient and fatigue score corresponding to the fatigue-sensitive action is 0.7, which is within the score range corresponding to the D-level fatigue state. Therefore, it can be judged that controller A is slightly tired, and the control work and communication are slightly affected. In response to this situation, a voice reminder is given to controller A, which may be "It is detected that you are slightly tired, please pay attention to adjust". If A adjusts his state according to the voice reminder, the fatigue state detection is terminated. If A does not adjust his state according to the voice reminder and is still in a fatigued state, the system will repeatedly calculate the fatigue score, determine the fatigue level, and execute the corresponding reminder until A makes timely adjustments according to the corresponding reminder.

[0061] According to the embodiments of the present disclosure, image fusion is performed on the infrared and visible light images of the controller collected by the self-developed image acquisition tool; face detection is performed on the fused image after image processing to determine the face image; face feature information is extracted from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; based on the face feature information, the factor analysis method is used to determine the main variables affecting the fatigue of the controller through an ordered multi-classification discrete selection model; based on the main variables and the critical values ​​of each fatigue level divided in advance, the fatigue influencing factor corresponding to the face feature information is determined; the controller has fatigue-sensitive movements When determining the facial feature information corresponding to each fatigue-sensitive action according to each fatigue-sensitive action, determine the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; establish a fatigue evaluation mechanism for controllers based on visual technology, so that instructors or controllers themselves can quantitatively evaluate the control ability of controllers according to the fatigue evaluation mechanism for controllers based on visual technology; if there is no veto on the control ability of the controller in the quantitative evaluation, calculate the fatigue score according to the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, determine the fatigue level according to the fatigue score, and perform corresponding reminders according to the fatigue level. In this way, interference to the controller can be avoided when collecting image data, the fatigue state can be quantified, and the fatigue state recognition accuracy and efficiency can be improved.

[0062] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0063] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0064] Figure 2 The structure diagram of a controller fatigue state detection device based on infrared and visible light image fusion provided by an embodiment of the present disclosure is shown. The device 200 includes: The infrared and visible light image acquisition module 210 is used to perform image fusion on the infrared and visible light images of the controller acquired by the self-developed image acquisition tool.

[0065] The facial feature information extraction module 220 is used to perform face detection after image processing on the fused image to determine the facial image; extract facial feature information from the determined facial image; wherein the facial feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information.

[0066] In some embodiments, module 220 is specifically configured to: Extract facial feature information from a determined face image, including: Eye movement feature information is determined based on the eye aspect ratio and the percentage of eye closure. Mouth movement feature information is determined based on the mouth aspect ratio and the percentage of mouth opening. Head movement feature information is determined based on the pitch angle of the head when it is raised up and down and the horizontal rotation angle of the head when it is turned left and right. Distance feature information is determined based on the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging.

[0067] In some embodiments, module 220 is further configured to: Eye movement feature information is determined based on the eye aspect ratio and eye closure percentage, including: Calculate the eye aspect ratio corresponding to each image frame in the continuous image frames of the face, and make an eye state map according to the eye aspect ratio corresponding to each image frame; determining eye posture and blinking frequency according to the eye state map; The number of eye-closing frames in continuous face image frames is determined according to the eye posture, and the eye-closing duration is calculated according to the number of eye-closing frames using the eye-closing percentage.

[0068] In some embodiments, module 220 is further configured to: The mouth movement feature information is determined based on the mouth aspect ratio and the mouth opening percentage, including: Determine the mouth area according to the key point information on the face and calculate the mouth aspect ratio corresponding to each image frame in the continuous image frames of the face; Determine the mouth state corresponding to each image frame according to the mouth aspect ratio corresponding to each image frame; Determine the mouth opening duration using the mouth opening percentage according to the mouth state corresponding to each image frame; Determine yawning status based on the duration of mouth opening.

[0069] In some embodiments, module 220 is further configured to: The head movement feature information is determined based on the pitch angle of the head up and down and the horizontal rotation angle of the head turning left and right, including: The mapping relationship between the face image and the key point information on the face in three-dimensional space is constructed by feature space transformation method; Calculate the pitch angle of the head up and down and the horizontal rotation angle of the head turning left and right corresponding to each image frame in the continuous image frames of the face according to the mapping relationship and the key point information on the face in three-dimensional space; The duration of lowering or raising the head is determined according to the pitch angle of the head corresponding to each image frame; the duration of turning the head left or right is determined according to the horizontal rotation angle of the head corresponding to each image frame.

[0070] In some embodiments, module 220 is further configured to: The distance feature information is determined based on the focal length of the self-developed image acquisition tool, the human pupil distance, and the pixel distance after pupil imaging, including: According to the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging, the distance between the person corresponding to each image frame in the continuous image frames of the face and the self-developed image acquisition tool is calculated by using the triangulation method; The duration during which the distance between the person and the self-developed image acquisition tool exceeds 90 centimeters is obtained according to the distance between the person and the self-developed image acquisition tool corresponding to each image frame.

[0071] The fatigue main variable acquisition module 230 is used to determine the main variables that affect the controller's fatigue by using the factor analysis method through an ordered multi-classification discrete selection model based on the facial feature information.

[0072] The fatigue impact factor determination module 240 is used to determine the fatigue impact factor corresponding to the facial feature information according to the main variables and the critical values ​​of each fatigue level divided in advance.

[0073] The fatigue coefficient and fatigue score determination module 250 is used to determine the facial feature information corresponding to each fatigue-sensitive action when the controller performs a fatigue-sensitive action, and to determine the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action based on the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information.

[0074] The quantitative evaluation module 260 is used to establish a controller fatigue evaluation mechanism based on visual technology, so that instructors or controllers themselves can conduct quantitative evaluation of the controller's control ability based on the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the controller's control ability in the quantitative evaluation, the fatigue score is calculated based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, the fatigue level is determined based on the fatigue score, and corresponding reminders are executed based on the fatigue level.

[0075] In some embodiments, module 260 is specifically configured to: In order to enable instructors or controllers themselves to quantitatively evaluate the controller's control ability according to the controller fatigue evaluation mechanism based on visual technology, including: The instructor or the controller himself monitors the controller's control ability and conducts quantitative evaluation of the monitoring situation based on the controller fatigue evaluation mechanism based on visual technology; If the instructor's quantitative evaluation score is lower than 90 points or the controller proactively proposes to suspend command, it means a veto of the controller's control ability.

[0076] In some embodiments, module 260 is further configured to: The fatigue score is calculated based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, including: The fatigue score corresponding to the fatigue-sensitive action is taken as the product of the fatigue coefficient corresponding to the fatigue-sensitive action and the fatigue score corresponding to the fatigue coefficient corresponding to the fatigue-sensitive action; The accumulated value of the fatigue points corresponding to the fatigue-sensitive actions is used as the fatigue point for determining the fatigue level.

[0077] In some embodiments, module 260 is further configured to: After determining the fatigue level according to the fatigue score and executing corresponding reminders according to the fatigue level, the method includes: If the execution of the corresponding reminder according to the fatigue level fails, the fatigue points are calculated, the fatigue level is determined, and the corresponding reminder is executed according to the fatigue level repeatedly until the execution of the corresponding reminder according to the fatigue level is successful.

[0078] Understandably, Figure 2 Each module / unit in the detection device 200 shown has the function of implementing each step in the detection method 100 provided in the embodiment of the present disclosure and can achieve its corresponding technical effect. For the sake of brevity, it will not be repeated here.

[0079] Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. The electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0080] like Figure 3As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0081] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0082] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0083] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0085] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0086] It should be noted that the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in the embodiment of the present disclosure. For the sake of concise description, they are not repeated here.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0088] The systems and techniques described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with the systems and techniques described herein). The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0089] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0090] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0091] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting controller fatigue status based on infrared and visible light image fusion, characterized in that: include: Perform image fusion on infrared and visible light images of controllers collected by self-developed image acquisition tools; Performing face detection on the fused image after image processing to determine the face image; extracting face feature information from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; According to the facial feature information, factor analysis is used to determine the main variables that affect controller fatigue through an ordered multi-classification discrete selection model; Determining a fatigue influencing factor corresponding to the facial feature information according to the main variables and the pre-classified critical values ​​of each fatigue level; When the controller performs fatigue-sensitive actions, the facial feature information corresponding to each fatigue-sensitive action is determined according to each fatigue-sensitive action, and the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action are determined according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; A controller fatigue evaluation mechanism based on visual technology is established so that instructors or controllers themselves can quantitatively evaluate the controller's control ability based on the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the controller's control ability in the quantitative evaluation, the fatigue score is calculated based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, the fatigue level is determined based on the fatigue score, and corresponding reminders are executed based on the fatigue level.

2. The method according to claim 1, characterized in that The step of extracting facial feature information from the determined facial image includes: Eye movement feature information is determined based on the eye aspect ratio and the percentage of eye closure. Mouth movement feature information is determined based on the mouth aspect ratio and the percentage of mouth opening. Head movement feature information is determined based on the pitch angle of the head when it is raised up and down and the horizontal rotation angle of the head when it is turned left and right. Distance feature information is determined based on the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging.

3. The method according to claim 2, characterized in that The step of determining eye movement feature information according to the eye aspect ratio and the eye closure percentage includes: Calculate the eye aspect ratio corresponding to each image frame in the continuous image frames of the face, and make an eye state map according to the eye aspect ratio corresponding to each image frame; determining eye posture and blinking frequency according to the eye state map; The number of eye-closing frames in the continuous image frames of the face is determined according to the eye posture, and the eye-closing duration is calculated according to the number of eye-closing frames using the eye-closing percentage.

4. The method according to claim 2, characterized in that: The determining of the mouth movement feature information according to the mouth aspect ratio and the mouth opening percentage includes: Determine the mouth area according to the key point information on the face and calculate the mouth aspect ratio corresponding to each image frame in the continuous image frames of the face; Determine the mouth state corresponding to each image frame according to the mouth aspect ratio corresponding to each image frame; Determine the mouth opening duration using the mouth opening percentage according to the mouth state corresponding to each image frame; Determine yawning status based on the duration of mouth opening.

5. The method according to claim 2, characterized in that: The determining of head motion feature information according to the pitch angle of the head raising up and down and the horizontal rotation angle of the head turning left and right includes: The mapping relationship between the face image and the key point information on the face in three-dimensional space is constructed by feature space transformation method; Calculate the pitch angle of the head up and down and the horizontal rotation angle of the head turning left and right corresponding to each image frame in the continuous image frames of the face according to the mapping relationship and the key point information on the face in three-dimensional space; The duration of lowering or raising the head is determined according to the pitch angle of the head corresponding to each image frame; the duration of turning the head left or right is determined according to the horizontal rotation angle of the head corresponding to each image frame.

6. The method according to claim 2, characterized in that The distance feature information is determined according to the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging, including: According to the focal length of the self-developed image acquisition tool, the human pupil distance and the pixel distance after pupil imaging, the distance between the person corresponding to each image frame in the continuous image frames of the face and the self-developed image acquisition tool is calculated by using the triangulation method; The duration during which the distance between the person and the self-developed image acquisition tool exceeds 90 centimeters is obtained according to the distance between the person and the self-developed image acquisition tool corresponding to each image frame.

7. The method according to claim 1, characterized in that The step of calculating the fatigue score according to the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action includes: The fatigue score corresponding to the fatigue-sensitive action is taken as the product of the fatigue coefficient corresponding to the fatigue-sensitive action and the fatigue score corresponding to the fatigue coefficient corresponding to the fatigue-sensitive action; The accumulated value of the fatigue points corresponding to the fatigue-sensitive actions is used as the fatigue point for determining the fatigue level.

8. The method according to claim 1, characterized in that The method allows the instructor or the controller to quantitatively evaluate the controller's control ability according to the controller fatigue evaluation mechanism based on visual technology, including: The instructor or the controller himself monitors the controller's control ability and conducts quantitative evaluation of the monitoring situation based on the controller fatigue evaluation mechanism based on visual technology; If the instructor's quantitative evaluation score is lower than 90 points or the controller proactively proposes to suspend command, it means a veto of the controller's control ability.

9. The method according to claim 1, characterized in that: After determining the fatigue level according to the fatigue score and executing corresponding reminder according to the fatigue level, the method includes: If the execution of the corresponding reminder according to the fatigue level fails, the fatigue points are calculated, the fatigue level is determined, and the corresponding reminder is executed according to the fatigue level repeatedly until the execution of the corresponding reminder according to the fatigue level is successful.

10. A controller fatigue state detection device based on infrared and visible light image fusion, characterized in that: include: Infrared and visible light image acquisition module, used to fuse infrared and visible light images of controllers collected by self-developed image acquisition tools; A face feature information extraction module is used to perform face detection after image processing on the fused image to determine the face image; extract face feature information from the determined face image; wherein the face feature information includes eye movement feature information, mouth movement feature information, head movement feature information and distance feature information; A fatigue main variable acquisition module is used to determine the main variables affecting the fatigue of the controller by using the factor analysis method through an ordered multi-classification discrete selection model according to the facial feature information; A fatigue influence factor determination module, used to determine the fatigue influence factor corresponding to the facial feature information according to the main variables and the critical values ​​of each fatigue level divided in advance; A fatigue coefficient and fatigue score determination module is used to determine the facial feature information corresponding to each fatigue-sensitive action according to each fatigue-sensitive action when the controller performs a fatigue-sensitive action, and to determine the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action according to the facial feature information corresponding to each fatigue-sensitive action and the fatigue influence factor corresponding to the facial feature information; The quantitative evaluation module is used to establish a controller fatigue evaluation mechanism based on visual technology, so that instructors or controllers themselves can quantitatively evaluate the controller's control ability based on the controller fatigue evaluation mechanism based on visual technology; if there is no veto on the controller's control ability in the quantitative evaluation, the fatigue score is calculated based on the fatigue coefficient and fatigue score corresponding to each fatigue-sensitive action, the fatigue level is determined based on the fatigue score, and corresponding reminders are executed based on the fatigue level.

Images