A pilot fatigue monitoring method and system using visual technology

Abstract

The present invention relates to the field of image processing technology and proposes a pilot fatigue monitoring method and system using visual technology. The method comprises: extracting continuous frame images during the pilot's work process to obtain facial feature visual areas and facial feature sequences; determining the occlusion probability, occlusion cycle, and normal cycle; marking the target facial features and target occlusion cycle, respectively determining the degree of abnormality of the target facial features in each normal cycle, and correcting the areas of the facial feature visual areas of the facial features in the target occlusion cycle based on the periodic interval between the normal cycle before the target occlusion cycle and the target occlusion cycle, as well as the area of the facial feature visual area corresponding to the target occlusion cycle; and obtaining pilot fatigue monitoring results using visual technology based on the area correction results and the continuous frame images extracted during the pilot's work process. The present invention can improve the accuracy of pilot fatigue monitoring based on vision.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a pilot fatigue monitoring method and system using visual technology. Background Art

[0002] Pilot fatigue can lead to decreased reaction speed, lack of concentration, weakened decision-making ability and increased operational errors, which may directly threaten flight safety and even cause serious aviation accidents. Therefore, pilot fatigue monitoring is an extremely important part of aviation safety management.

[0003] Existing technology can monitor pilot fatigue through visual monitoring based on facial features. However, when encountering strong light during flight, pilots often wear special sunglasses to protect their eyes. These sunglasses block portions of the pilot's face, leading to inaccurate recognition of key facial features and affecting fatigue assessment. Furthermore, the complex flight environment can affect cockpit lighting, which can vary, causing reflections and excessive darkness. This can lead to inaccurate recognition of key facial features, affecting fatigue assessment. Summary of the Invention

[0004] The present invention provides a pilot fatigue monitoring method and system using visual technology to solve the problem that the pilot's face is obscured, resulting in inaccurate recognition of key facial features and affecting fatigue status judgment. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for monitoring pilot fatigue using visual technology, the method comprising the following steps:

[0006] Extracting continuous frame images during the pilot's work process, identifying the pilot's facial features from the continuous frame images, obtaining the facial feature visual area, and determining the facial feature sequence corresponding to the same facial features in the same time period;

[0007] Any one of the five sense organs is recorded as a target five sense organ, and the occlusion possibility of the target five sense organ in the same time period is determined based on the difference between all data contained in the facial feature sequence of the five sense organs in the same time period, and the occlusion period and the normal period are determined based on the occlusion possibility;

[0008] According to the difference between the areas of the facial feature visual areas corresponding to all normal cycles of the target facial features, the abnormality degree of the target facial features in each normal cycle is determined respectively, and any occlusion cycle is recorded as the target occlusion cycle. Combining the period interval between the normal cycle before the target occlusion cycle and the target occlusion cycle, as well as the area of the facial feature visual area corresponding to the target occlusion cycle, the area of the facial feature visual area of the facial features in the target occlusion cycle is corrected respectively;

[0009] Pilot fatigue monitoring results using visual technology are obtained based on the area correction results and continuous frame images extracted during the pilot's work process.

[0010] Furthermore, the method for obtaining the facial feature sequence is:

[0011] Arrange the areas of the facial feature visual regions corresponding to the same facial features in consecutive frame images within the same time period according to the order of acquisition moments corresponding to the images where the facial feature visual regions are located, to obtain a facial feature sequence of the facial features within the same time period;

[0012] The area of the facial feature visual area is the number of pixels contained in the facial feature visual area.

[0013] Furthermore, the method for determining the occlusion possibility of the target facial features in the same time period is as follows:

[0014] The inverse of the variance of all data contained in the facial feature sequence of the target facial features in the target time period is recorded as the occlusion possibility of the target facial features in the target time period.

[0015] Furthermore, the specific method of determining the occlusion period and the normal period according to the occlusion possibility is as follows:

[0016] The occlusion probability is greater than The time period is recorded as the suspected occlusion period. For the continuous time period of the target facial features, it will be greater than or equal to Each suspected occlusion period of consecutive suspected occlusion periods is recorded as an occlusion period, where represents the first preset threshold, represents a second preset threshold;

[0017] All time periods that are not occlusion periods are recorded as normal periods.

[0018] Furthermore, the method of determining the abnormality degree of the target facial features in each normal period according to the difference between the areas of the facial feature visual areas corresponding to all normal periods of the target facial features includes the following specific methods:

[0019] Any normal cycle is recorded as the target normal cycle; the average of the DTW distances between the facial feature sequences of every two normal cycles of the target facial features is recorded as the abnormality of the normal cycle of the target facial features;

[0020] The average value of the areas of the facial feature visual areas corresponding to all normal cycles of the target facial features is recorded as the average area of the target facial features, and the absolute value of the difference between the average value of the areas of the facial feature visual areas corresponding to the target normal cycle and the average area of the target facial features is recorded as the characteristic visual area area difference of the target facial features in the target normal cycle;

[0021] The abnormality degree of the target facial features in the target normal period is determined according to the abnormality degree of the target facial features in the normal period of the target normal period and the difference in the characteristic visual area of the target facial features.

[0022] Furthermore, the method of determining the abnormality degree of the target facial features in the target normal period according to the abnormality degree of the target facial features in the normal period of the target normal period and the difference in the characteristic visual area of the target facial features includes the following specific methods:

[0023] The normalized value of the product of the abnormality of the target facial features in the normal period of the target normal period and the area difference of the characteristic visual areas of the target facial features is recorded as the abnormality degree of the target facial features in the target normal period.

[0024] Furthermore, the periodic interval between the normal period before the target occlusion period and the target occlusion period, and the area of the facial feature visual area corresponding to the target occlusion period, are combined to respectively correct the area of the facial feature visual area of the facial features in the target occlusion period, including the specific method of:

[0025] The normal cycle closest in time to the target occlusion cycle is recorded as the adjacent normal cycle of the target occlusion cycle, the reciprocal of the sum of the number of cycles between the target occlusion cycle and the adjacent normal cycle and the number 1 is recorded as the first weight of the target occlusion cycle, and the difference between the number 1 and the first weight of the target occlusion cycle is recorded as the second weight of the target occlusion cycle;

[0026] The average of the abnormality levels of all organs different from the target facial features in the adjacent normal cycles of the target occlusion cycle is recorded as the comparative facial features mean of the target facial features, and the product of the comparative facial features mean of the target facial features and the second weight of the target occlusion cycle is recorded as the second product of the target facial features in the target occlusion cycle;

[0027] The product of the first weight of the target occlusion period and the abnormality degree of the target facial features in the adjacent normal period of the target occlusion period is recorded as the first product of the target facial features in the target occlusion period;

[0028] The sum of the first product and the second product of the target facial features in the target occlusion period is recorded as the adjustment coefficient of the target facial features in the target occlusion period;

[0029] According to the adjustment coefficient of the target facial features in the target occlusion period and the area of the facial feature visual area corresponding to the target occlusion period, the area of the facial feature visual area of the target facial features in the target occlusion period is corrected.

[0030] Furthermore, the correction of the area of the facial feature visual area of the target facial features in the target occlusion period according to the adjustment coefficient of the target facial features in the target occlusion period and the area of the facial feature visual area corresponding to the target occlusion period is achieved, including the specific method of:

[0031] The product of the adjustment coefficient of the target facial features in the target occlusion period and the third preset threshold is recorded as the adjusted area of the target facial features in the target occlusion period. The rounded value of the difference between the mean of the area of the facial feature visual area corresponding to the target facial features in the target occlusion period and the adjusted area is recorded as the area correction value of the target facial features in the target occlusion period. The area of the facial feature visual area of the target facial features in the target occlusion period is assigned as the area correction value of the target facial features in the target occlusion period to achieve area correction.

[0032] Furthermore, the pilot fatigue monitoring result using visual technology is obtained based on the area correction result and the continuous frame images extracted during the pilot's work process, including the specific method of:

[0033] The area of the facial feature visual area of each organ in all occlusion cycles and normal cycles, as well as the continuous frame images extracted during the pilot's work process, are input into the pilot fatigue monitoring neural network model to obtain the pilot's driving status evaluation value;

[0034] When the pilot's driving state evaluation value is greater than or equal to a fourth preset threshold, determining that the pilot is in a fatigue driving state;

[0035] When the pilot's driving state evaluation value is less than a fourth preset threshold, it is determined that the pilot is in a non-fatigue driving state.

[0036] In a second aspect, an embodiment of the present invention further provides a pilot fatigue monitoring system utilizing visual technology, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above-described methods when executing the computer program.

[0037] The beneficial effects of the present invention are:

[0038] The present application first extracts continuous frame images during the pilot's work process, identifies the pilot's facial features from the continuous frame images, obtains the facial feature visual area, and evaluates the degree to which the organ is occluded in the same time period according to the change trend of the area of the facial feature visual area corresponding to the same organ in the facial features, determines the occlusion possibility, and determines the occlusion period and the normal period according to the occlusion possibility; further, since the facial feature visual area in the image cannot be accurately divided when the pilot's face is occluded, and since the facial feature visual area of the pilot is consistent with the features of the facial feature visual area of the same facial features that are not occluded in the previous adjacent continuous frames, the facial feature visual area before and after the face is occluded is obtained. The facial feature visual area when the face is occluded is corrected. Specifically, the possibility of the pilot working fatigued in the time period corresponding to all normal cycles is evaluated to obtain the abnormality degree of the normal cycle. Then, according to the abnormality degree of the normal cycle, the area of the facial feature visual area of the facial features in the occlusion cycle is corrected respectively; finally, according to the area correction result and the continuous frame images extracted during the pilot's work process, the pilot fatigue monitoring result using visual technology is obtained, which solves the problem that the pilot's face is occluded, resulting in inaccurate recognition of some key feature points of the face, affecting the judgment of the fatigue state, and improving the accuracy of vision-based pilot fatigue detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a pilot fatigue monitoring method using visual technology provided by one embodiment of the present invention;

[0041] Figure 2 A flowchart for obtaining the shielding period and the normal period provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] See also Figure 1, which shows a flow chart of a pilot fatigue monitoring method using visual technology provided by one embodiment of the present invention, the method comprising the following steps:

[0044] Step S001: extract continuous frame images during the pilot's work process, identify the pilot's facial features from the continuous frame images, obtain the facial feature visual area, and determine the facial feature sequence corresponding to the same facial features in the same time period.

[0045] Continuous frames of pilot work were extracted from the ground control center's data storage. For each of these frames, OpenCV was used to identify the pilot's facial features and record them as visual regions of facial features.

[0046] Among them, using OpenCV to identify the facial features of the pilot in the image is a well-known technology and will not be described in detail. It can be understood that the area of the facial feature visual area is the number of pixels contained in the facial feature visual area.

[0047] The areas of the facial feature visual areas corresponding to the same facial features in the continuous frame images within the same time period are arranged in the order of the acquisition moments corresponding to the images where the facial feature visual areas are located to obtain the facial feature sequence of the same time period.

[0048] Preferably, in one embodiment of the present application, one minute is used as the time length corresponding to the time period. In actual application, as other implementation methods, the implementer can decide the value of the time length corresponding to the time period according to actual conditions, and this application does not impose any special restrictions.

[0049] It is understandable that each of the five sense organs, namely the eyebrows, eyes, ears, nose, and mouth, has a facial feature sequence in each time period. For example, in the same time period, the left ear and the right ear each correspond to a facial feature sequence.

[0050] At this point, a sequence of facial features of the pilot's five sense organs at the same time period during his work process is obtained.

[0051] Step S002: record any one of the five senses as the target five senses, determine the occlusion possibility of the target five senses in the same time period based on the difference between all data contained in the facial feature sequence of the five senses in the same time period, and determine the occlusion period and the normal period based on the occlusion possibility.

[0052] Any time period is recorded as the target time period, and any organ among the five senses is recorded as the target five senses. The occlusion possibility of the target five senses in the target time period is determined based on the difference between all data contained in the facial feature sequence of the target five senses in the target time period.

[0053] Preferably, as an embodiment of the present application, the inverse of the variance of all data contained in the facial feature sequence of the target facial features in the target time period is recorded as the occlusion possibility of the target facial features in the target time period.

[0054] In the process of calculating the inverse, it is necessary to use the variance of all data contained in the facial feature sequence of the target facial features in the target time period as the denominator, and calculate the ratio of 1 to the denominator. In the process of calculating the ratio, in order to avoid the situation where the denominator is zero, it is necessary to add a preset value to the denominator. The embodiment of the preset value is 1.

[0055] When the difference between all data contained in the facial feature sequence of the target facial features in the target time period is greater, the possibility that the target facial features will show normal human movements in the target time period is greater, the possibility that the target facial features will be obscured in the target time period is smaller, and the possibility that the pilot is working normally is greater. At this time, the possibility of the target facial features being obscured in the target time period is smaller.

[0056] For example, when the target facial feature is the left eye, the left eye will definitely blink within a time period, the differences between all data contained in the facial feature sequence are large, and the target facial feature is less likely to be occluded during the target time period; when the target facial feature is occluded during the target time period, the area of the facial feature visual area remains unchanged, the differences between all data contained in the facial feature sequence are small, and the target facial feature is more likely to be occluded during the target time period.

[0057] The same method can be used to obtain the occlusion possibility of any organ among the five senses in any time period. That is to say, for each organ among the five senses, there is a corresponding occlusion possibility in each time period.

[0058] The occlusion probability is greater than The time period is recorded as the suspected occlusion period, and the continuous time period of the target facial features is greater than or equal to Each suspected occlusion period in the suspected occlusion period is recorded as an occlusion period. All time periods that are not occlusion periods are recorded as normal periods. The flowchart for obtaining occlusion periods and normal periods is as follows: Figure 2 shown.

[0059] in, represents the first preset threshold, represents the second preset threshold. The first preset threshold and the second preset threshold are both preset constants. In this embodiment, the value of the first preset threshold is 0.7, and the value of the second preset threshold is 3.

[0060] It can be understood that the occlusion period corresponding to the target facial features is the time period when the target facial features are blocked. For the target facial features, multiple occlusion periods may be obtained, or there may be no corresponding occlusion period. When the target facial features have no corresponding occlusion period, that is, the target facial features are not blocked during the pilot's work process, at this time, all time periods of the target facial features are normal periods.

[0061] The same method can be used to obtain all occlusion periods and normal periods of any organ among the five sense organs.

[0062] At this point, all occlusion cycles and normal cycles of all organs in the five senses are obtained.

[0063] Step S003, based on the difference between the areas of the facial feature visual areas corresponding to all normal periods of the target facial features, determine the abnormality level of the target facial features in each normal period respectively, record any occlusion period as the target occlusion period, combine the period interval between the normal period before the target occlusion period and the target occlusion period, and the area of the facial feature visual area corresponding to the target occlusion period, and realize the correction of the area of the facial feature visual area of the facial features in the target occlusion period respectively.

[0064] When a pilot's face is obscured, it's impossible to accurately delineate the facial feature visual area in the image. The identified facial feature visual area is identical in area to the unobstructed facial feature visual area of the same facial features in the previous adjacent frames. Therefore, the extracted facial feature visual area needs to be corrected. Because the facial feature visual area of a pilot's face when obscured cannot be directly obtained, this embodiment corrects the facial feature visual area when the face is obscured based on the facial feature visual area before and during the unobstructed state.

[0065] According to the difference between the areas of the facial feature visual regions corresponding to all normal cycles of the target facial features, the abnormality degree of the target facial features in any normal cycle is determined respectively.

[0066] Any normal cycle is recorded as the target normal cycle; the average of the DTW distances between the facial feature sequences of every two normal cycles of the target facial features in all normal cycles is recorded as the abnormality of the normal cycle of the target facial features; the average of the areas of the facial feature visual areas corresponding to all normal cycles of the target facial features is recorded as the average area of the target facial features; the absolute value of the difference between the average of the areas of the facial feature visual areas corresponding to the target normal cycle and the average area of the target facial features is recorded as the area difference of the characteristic visual areas of the target facial features in the target normal cycle; the normalized value of the product of the abnormality of the target facial features in the normal cycle of the target normal cycle and the area difference of the characteristic visual areas of the target facial features is recorded as the abnormality degree of the target facial features in the target normal cycle.

[0067] The same method can be used to obtain the abnormality level of the target facial features in any normal period.

[0068] As will be appreciated, calculating the DTW distance between sequences is a well-known technique and will not be further described. It should be noted that this embodiment uses the Z-Score standard normalization method to calculate the normalized value. In actual applications, implementers may use other existing methods such as maximum-minimum normalization or the sigmoid function to calculate the normalized value, without limitation herein.

[0069] The closer the facial feature sequences of the target facial features across all normal cycles are, the more similar the pilot's facial features across different normal cycles are. In this case, the smaller the abnormality of the target facial features across the normal cycles, the smaller the abnormality of the target facial features across the target normal cycles. For example, when the target facial feature is the left eye, the closer the blinking frequency of the pilot's left eye across different occlusion cycles, the greater the likelihood that the pilot will function normally during the time periods corresponding to all normal cycles. The greater the difference between the mean area of the facial feature visual areas corresponding to the target normal cycles and the average area of the target facial features, the greater the difference in the area of the target facial features visual areas across the target normal cycles, the greater the abnormality of the target facial features across the target normal cycles, and the greater the likelihood that the pilot will be fatigued during the time periods corresponding to all normal cycles.

[0070] Furthermore, since pilots' facial states are often similar over short periods of time, for example, fatigue can lead to faster blinking and changes in facial features due to movements like yawning. These similar facial features remain similar when the pilot's face is both unobstructed and obstructed. Therefore, the visual area of facial features during occlusion is corrected based on the visual area of the facial features before and during the unobstructed state.

[0071] Any occlusion period is recorded as the target occlusion period. According to the period interval between the normal period before the target occlusion period and the target occlusion period, as well as the abnormality degree of the normal period, the adjustment coefficient of each organ in the five sense organs during the target occlusion period is determined.

[0072] The normal cycle closest in time to the target occlusion cycle is recorded as the adjacent normal cycle of the target occlusion cycle, the reciprocal of the sum of the number of cycles between the target occlusion cycle and the adjacent normal cycle and the number 1 is recorded as the first weight of the target occlusion cycle, and the difference between the number 1 and the first weight of the target occlusion cycle is recorded as the second weight of the target occlusion cycle; the average of the abnormality levels of all organs different from the target facial features in the adjacent normal cycles of the target occlusion cycle is recorded as the comparative facial features average of the target facial features, the product of the comparative facial features average of the target facial features and the second weight of the target occlusion cycle is recorded as the second product of the target facial features in the target occlusion cycle; the product of the first weight of the target occlusion cycle and the abnormality levels of the target facial features in the adjacent normal cycles of the target occlusion cycle is recorded as the first product of the target facial features in the target occlusion cycle; the sum of the first product and the second product of the target facial features in the target occlusion cycle is recorded as the adjustment coefficient of the target facial features in the target occlusion cycle.

[0073] It should be noted that when there is no normal cycle before the target occlusion cycle, the average of the abnormality levels of all normal cycles is taken as the abnormality level of the adjacent normal cycles of the target occlusion cycle, and the first weight and second weight of the target occlusion cycle are both assigned as .

[0074] The same method can be used to obtain the adjustment coefficient of each organ in the five sense organs during the target occlusion period.

[0075] The product of the adjustment coefficient of the target facial features in the target occlusion period and the third preset threshold is recorded as the adjusted area of the target facial features in the target occlusion period, the rounded value of the difference between the mean of the areas of the facial feature visual areas corresponding to the target facial features in the target occlusion period and the adjusted area is recorded as the area correction value of the target facial features in the target occlusion period, and the area of the facial feature visual area of the target facial features in the target occlusion period is assigned as the area correction value of the target facial features in the target occlusion period, so as to achieve the correction of the area of the facial feature visual area of the target facial features in the target occlusion period.

[0076] The third preset threshold is a preset constant. In this embodiment, the value of the third preset threshold is 50.

[0077] The same method can be used to correct the area of the facial feature visual area of each organ in the five senses during the target occlusion period, and to correct the area of the facial feature visual area of each organ in the five senses during each occlusion period.

[0078] At this point, the area correction of the facial feature visual area of each organ in each occlusion cycle is achieved.

[0079] Step S004: Acquire pilot fatigue monitoring results using visual technology based on the area correction result and the continuous frame images extracted during the pilot's work process.

[0080] The area of the facial feature visual area of each organ in all occlusion cycles and normal cycles, and the continuous frame images extracted during the pilot's work process are input into the pilot fatigue monitoring neural network model to obtain the pilot's driving status evaluation value. The pilot's driving status evaluation value ranges from greater than or equal to 0 to less than or equal to 1.

[0081] When the pilot's driving state evaluation value is greater than or equal to a fourth preset threshold, the pilot is determined to be in a fatigued driving state; when the pilot's driving state evaluation value is less than the fourth preset threshold, the pilot is determined to be in a non-fatigued driving state. The fourth preset threshold is a preset constant. In this embodiment, the fourth preset threshold is set to 0.5.

[0082] Among them, the input of the pilot fatigue monitoring neural network model is the area of the facial feature visual area of each organ in all occlusion cycles and normal cycles and the continuous frame images extracted during the pilot's work process, and the output is the pilot's driving status evaluation value. The pilot fatigue monitoring neural network model is a pre-trained DNN network, the network task is prediction, the data set is continuous frame images of the pilot's work process under different postures and expressions, the training set is the area of the facial feature visual area identified by the continuous frame images extracted during the pilot's work process, and the loss function used is the cross-entropy damage function; training the DNN network is a well-known technology and will not be repeated here.

[0083] At this point, the pilot fatigue monitoring results using visual technology are obtained.

[0084] Based on the same inventive concept as the above-mentioned method, an embodiment of the present invention also provides a pilot fatigue monitoring system using visual technology, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned pilot fatigue monitoring methods using visual technology are implemented.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pilot fatigue monitoring method using visual technology, characterized in that: The method comprises the following steps: Extracting continuous frame images during the pilot's work process, identifying the pilot's facial features from the continuous frame images, obtaining the facial feature visual area, and determining the facial feature sequence corresponding to the same facial features in the same time period; Any one of the five sense organs is recorded as a target five sense organ, and the occlusion possibility of the target five sense organ in the same time period is determined based on the difference between all data contained in the facial feature sequence of the five sense organs in the same time period, and the occlusion period and the normal period are determined based on the occlusion possibility; According to the difference between the areas of the facial feature visual areas corresponding to all normal cycles of the target facial features, the abnormality degree of the target facial features in each normal cycle is determined respectively, and any occlusion cycle is recorded as the target occlusion cycle. Combined with the period interval between the normal cycle before the target occlusion cycle and the target occlusion cycle, and the area of the facial feature visual area corresponding to the target occlusion cycle, the area of the facial feature visual area of the facial features in the target occlusion cycle is corrected respectively, specifically including: recording the normal cycle closest in time to the target occlusion cycle as the adjacent normal cycle of the target occlusion cycle, recording the inverse of the sum of the number of cycles between the target occlusion cycle and the adjacent normal cycle and the number 1 as the first weight of the target occlusion cycle, and recording the difference between the number 1 and the first weight of the target occlusion cycle as the first weight of the target occlusion cycle. second weight; the average of the abnormality levels of all organs different from the target facial features in the adjacent normal cycles of the target occlusion cycle is recorded as the comparative facial features mean of the target facial features, and the product of the comparative facial features mean of the target facial features and the second weight of the target occlusion cycle is recorded as the second product of the target facial features in the target occlusion cycle; the product of the first weight of the target occlusion cycle and the abnormality levels of the target facial features in the adjacent normal cycles of the target occlusion cycle is recorded as the first product of the target facial features in the target occlusion cycle; the sum of the first product and the second product of the target facial features in the target occlusion cycle is recorded as the adjustment coefficient of the target facial features in the target occlusion cycle; according to the adjustment coefficient of the target facial features in the target occlusion cycle and the area of the facial feature visual area corresponding to the target occlusion cycle, the correction of the area of the facial feature visual area of the target facial features in the target occlusion cycle is achieved; Pilot fatigue monitoring results using visual technology are obtained based on the area correction results and continuous frame images extracted during the pilot's work process.

2. The pilot fatigue monitoring method using visual technology according to claim 1, characterized in that: The method for obtaining the facial feature sequence of the five sense organs is as follows: Arrange the areas of the facial feature visual regions corresponding to the same facial features in consecutive frame images within the same time period according to the order of acquisition moments corresponding to the images where the facial feature visual regions are located, to obtain a facial feature sequence of the facial features within the same time period; The area of the facial feature visual area is the number of pixels contained in the facial feature visual area.

3. The pilot fatigue monitoring method using visual technology according to claim 1, characterized in that: The method for determining the occlusion probability of the target facial features in the same time period is as follows: The inverse of the variance of all data contained in the facial feature sequence of the target facial features in the target time period is recorded as the occlusion possibility of the target facial features in the target time period.

4. The pilot fatigue monitoring method using visual technology according to claim 1, characterized in that: The specific method of determining the occlusion period and the normal period according to the occlusion possibility is as follows: The occlusion probability is greater than The time period is recorded as the suspected occlusion period. For the continuous time period of the target facial features, it will be greater than or equal to Each suspected occlusion period of consecutive suspected occlusion periods is recorded as an occlusion period, where represents the first preset threshold, represents a second preset threshold; All time periods that are not occlusion periods are recorded as normal periods.

5. The pilot fatigue monitoring method using visual technology according to claim 1, characterized in that: The specific method of determining the abnormality degree of the target facial features in each normal cycle according to the difference between the areas of the facial feature visual areas corresponding to all normal cycles of the target facial features is as follows: Any normal cycle is recorded as the target normal cycle; the average of the DTW distances between the facial feature sequences of every two normal cycles of the target facial features is recorded as the abnormality of the normal cycle of the target facial features; The average value of the areas of the facial feature visual areas corresponding to all normal cycles of the target facial features is recorded as the average area of the target facial features, and the absolute value of the difference between the average value of the areas of the facial feature visual areas corresponding to the target normal cycle and the average area of the target facial features is recorded as the characteristic visual area area difference of the target facial features in the target normal cycle; The abnormality degree of the target facial features in the target normal period is determined according to the abnormality degree of the target facial features in the normal period of the target normal period and the difference in the characteristic visual area of the target facial features.

6. The pilot fatigue monitoring method using visual technology according to claim 5, characterized in that: The specific method of determining the abnormality degree of the target facial features in the target normal period according to the abnormality degree of the target facial features in the normal period of the target normal period and the difference in the characteristic visual area of the target facial features is as follows: The normalized value of the product of the abnormality of the target facial features in the normal period of the target normal period and the area difference of the characteristic visual areas of the target facial features is recorded as the abnormality degree of the target facial features in the target normal period.

7. The pilot fatigue monitoring method using visual technology according to claim 1, characterized in that: The method of correcting the area of the facial feature visual area of the target facial features in the target occlusion period according to the adjustment coefficient of the target facial features in the target occlusion period and the area of the facial feature visual area corresponding to the target occlusion period includes the following specific methods: The product of the adjustment coefficient of the target facial features in the target occlusion period and the third preset threshold is recorded as the adjusted area of the target facial features in the target occlusion period. The rounded value of the difference between the mean of the area of the facial feature visual area corresponding to the target facial features in the target occlusion period and the adjusted area is recorded as the area correction value of the target facial features in the target occlusion period. The area of the facial feature visual area of the target facial features in the target occlusion period is assigned as the area correction value of the target facial features in the target occlusion period to achieve area correction.

8. The pilot fatigue monitoring method using visual technology according to claim 1, characterized in that: The method of obtaining the pilot fatigue monitoring result using visual technology based on the area correction result and the continuous frame images extracted during the pilot's work process includes the following specific methods: The area of the facial feature visual area of each organ in all occlusion cycles and normal cycles, as well as the continuous frame images extracted during the pilot's work process, are input into the pilot fatigue monitoring neural network model to obtain the pilot's driving status evaluation value; When the pilot's driving state evaluation value is greater than or equal to a fourth preset threshold, determining that the pilot is in a fatigue driving state; When the pilot's driving state evaluation value is less than a fourth preset threshold, it is determined that the pilot is in a non-fatigue driving state.

9. A pilot fatigue monitoring system using visual technology, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images