Pilot fatigue monitoring method and system using visual technology

Abstract

The invention relates to the technical field of image processing, and provides a pilot fatigue monitoring method and system using a visual technology, and the method comprises the steps: extracting continuous frame images in a working process of a pilot, and obtaining a facial feature visual region and a facial feature sequence of five sense organs; determining a shielding possibility, a shielding period and a normal period; marking the target five sense organs and a target shielding period, respectively determining the abnormal degree of the target five sense organs in each normal period, and combining the period interval between the normal period before the target shielding period and the target shielding period and the area of the facial feature visual region corresponding to the target shielding period to obtain the facial feature visual region. Respectively correcting the area of the facial feature visual region of the five sense organs in the target shielding period; and according to the area correction result and the continuous frame images extracted in the working process of the pilot, acquiring a pilot fatigue monitoring result by using a visual technology. According to the invention, the accuracy of pilot fatigue monitoring based on vision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method and system for monitoring pilot fatigue using vision technology. Background Art

[0002] Pilot fatigue can lead to a decrease in reaction speed, inattentiveness, weakened decision-making ability, and an increase in operational errors, which may directly threaten flight safety and even cause serious aviation accidents. Therefore, pilot fatigue monitoring is an extremely important part of aviation safety management.

[0003] Existing technologies can monitor pilot fatigue through vision monitoring technology based on facial features. However, when encountering strong light during flight, pilots will wear special sunglasses to avoid the impact of strong light on the eyes, and the sunglasses will block part of the pilot's face, resulting in inaccurate recognition of some key facial feature points and affecting the judgment of the fatigue state. At the same time, the flight environment is complex, and the light in the cockpit will change due to the influence of the flight environment, resulting in situations such as reflection and excessive darkness, which cause inaccurate recognition of some key facial feature points of the pilot and affect the judgment of the fatigue state. Summary of the Invention

[0004] The present invention provides a method and system for monitoring pilot fatigue using vision technology to solve the problem that the pilot's face is blocked, resulting in inaccurate recognition of some key facial feature points and affecting the judgment of the fatigue state. The specific technical solutions adopted are as follows: In a first aspect, an embodiment of the present invention provides a method for monitoring pilot fatigue using vision technology, the method comprising the following steps: Extract consecutive frame images during the pilot's work process, identify the facial feature regions of the pilot from the consecutive frame images, obtain the facial feature visual regions, and determine the facial feature sequences of the facial features of the same five sense organs in the same time period corresponding to the same five sense organs; Denote any one of the five sense organs as the target five sense organ, determine the occlusion possibility of the target five sense organ in the same time period according to the differences between all the data included in the facial feature sequence of the five sense organs of the target five sense organ in the same time period, and determine the occlusion period and the normal period according to the occlusion possibility; According to the differences between the areas of the facial feature visual regions corresponding to all the normal periods of the target five sense organ, respectively determine the abnormality degree of the target five sense organ in each normal period, denote any one occlusion period as the target occlusion period, and combine the interval between the normal period before the target occlusion period and the target occlusion period, and the area of the facial feature visual region corresponding to the target occlusion period to respectively correct the area of the facial feature visual region of the five sense organs in the target occlusion period; Based on the area correction result and the consecutive frame images extracted during the pilot's work, obtain the pilot fatigue monitoring result using vision technology.

[0005] Further, the method for obtaining the facial feature sequence of the five sense organs is as follows: Arrange the areas of the facial feature visual regions corresponding to the same five sense organs in consecutive frame images within the same time period in the order of the acquisition times of the images where the facial feature visual regions are located, and obtain the facial feature sequence of the five sense organs for the same time period; The area of the facial feature visual region is the number of pixel points contained in the facial feature visual region.

[0006] Further, the method for determining the occlusion possibility of the target five sense organs in the same time period is as follows: Denote the reciprocal of the variance of all data contained in the facial feature sequence of the target five sense organs in the target time period as the occlusion possibility of the target five sense organs in the target time period.

[0007] Further, the method for determining the occlusion period and the normal period according to the occlusion possibility includes the following specific methods: Denote the time period with an occlusion possibility greater than as a suspected occlusion period. For consecutive time periods of the target five sense organs, denote each suspected occlusion period that is greater than or equal to consecutive suspected occlusion periods as an occlusion period, where represents the first preset threshold, and represents the second preset threshold; Denote all time periods that are not occlusion periods as normal periods.

[0008] Further, the method for respectively determining the abnormality degree of the target five sense organs in each normal period according to the differences between the areas of the facial feature visual regions corresponding to all normal periods of the target five sense organs includes the following specific methods: Denote any one normal period as the target normal period; denote the mean of the DTW distances between the facial feature sequences of every two normal periods among all normal periods of the target five sense organs as the abnormality degree of the normal periods of the target five sense organs; Denote the mean of the areas of the facial feature visual regions corresponding to all normal periods of the target five sense organs as the average area of the target five sense organs, and denote the absolute value of the difference between the mean of the area of the facial feature visual region corresponding to the target normal period and the average area of the target five sense organs as the area difference of the feature visual region of the target five sense organs in the target normal period; Determine the abnormality degree of the target five sense organs in the target normal period according to the abnormality degree of the normal periods of the target five sense organs and the area difference of the feature visual region of the target five sense organs in the target normal period.

[0009] Further, the method for determining the abnormality degree of the target facial feature in the target normal period based on the abnormality degree of the target facial feature in the normal period of the target normal period and the area difference of the characteristic visual region of the target facial feature includes the following specific steps: Denote the normalized value of the product of the abnormality degree of the target facial feature in the normal period of the target normal period and the area difference of the characteristic visual region of the target facial feature as the abnormality degree of the target facial feature in the target normal period.

[0010] Further, the method for respectively correcting the area of the facial feature visual region of the facial features in the target occlusion period by combining the period interval between the normal period before the target occlusion period and the target occlusion period, and the area of the facial feature visual region corresponding to the target occlusion period includes the following specific steps: Denote the normal period closest to the target occlusion period in time before the target occlusion period as the adjacent normal period of the target occlusion period, denote the reciprocal of the sum of the number of periods between the target occlusion period and the adjacent normal period and the number 1 as the first weight of the target occlusion period, and denote the difference between the number 1 and the first weight of the target occlusion period as the second weight of the target occlusion period; Denote the average value of the abnormality degrees of all organs different from the target facial feature in the adjacent normal period of the target occlusion period as the comparison facial feature average value of the target facial feature, and denote the product of the comparison facial feature average value of the target facial feature and the second weight of the target occlusion period as the second product of the target facial feature in the target occlusion period; Denote the product of the first weight of the target occlusion period and the abnormality degree of the target facial feature in the adjacent normal period of the target occlusion period as the first product of the target facial feature in the target occlusion period; Denote the sum of the first product and the second product of the target facial feature in the target occlusion period as the adjustment coefficient of the target facial feature in the target occlusion period; Based on the adjustment coefficient of the target facial feature in the target occlusion period and the area of the facial feature visual region corresponding to the target occlusion period, correct the area of the facial feature visual region of the target facial feature in the target occlusion period.

[0011] Further, the method for correcting the area of the facial feature visual region of the target facial feature in the target occlusion period based on the adjustment coefficient of the target facial feature in the target occlusion period and the area of the facial feature visual region corresponding to the target occlusion period includes the following specific steps: Multiply the adjustment coefficient of the target facial feature in the target occlusion period by the third preset threshold, and denote it as the adjusted area of the target facial feature in the target occlusion period. Round the difference between the average value of the area of the facial feature visual region corresponding to the target facial feature in the target occlusion period and the adjusted area, and denote it as the area correction value of the target facial feature in the target occlusion period. Assign the area of the facial feature visual region of the target facial feature in the target occlusion period to the area correction value of the target facial feature in the target occlusion period to achieve area correction.

[0012] Further, the specific method for obtaining the pilot fatigue monitoring result using vision technology according to the area correction result and the continuous frame images extracted during the pilot's work includes: Input the areas of the facial feature visual regions of each organ in all occlusion periods and normal periods among the facial features, as well as the continuous frame images extracted during the pilot's work, into the pilot fatigue monitoring neural network model to obtain the driving state evaluation value of the pilot. When the driving state evaluation value of the pilot is greater than or equal to the fourth preset threshold, it is determined that the pilot is in a fatigue driving state. When the driving state evaluation value of the pilot is less than the fourth preset threshold, it is determined that the pilot is in a non-fatigue driving state.

[0013] In a second aspect, an embodiment of the present invention further provides a pilot fatigue monitoring system using vision technology, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the method described in any one of the above are implemented.

[0014] The beneficial effects of the present invention are: This application first extracts consecutive frame images during the pilot's work process, identifies the facial feature regions of the pilot from the consecutive frame images, obtains the facial feature visual regions, evaluates the degree of occlusion of an organ within the same time period based on the change trend of the area of the facial feature visual regions corresponding to the same organ in the facial features, determines the occlusion possibility, and determines the occlusion period and normal period according to the occlusion possibility; further, since when the pilot's face is occluded, the facial feature visual regions in the image cannot be accurately divided, and since the features of the facial feature visual regions of the same facial features that are not occluded in the consecutive frames adjacent to the previous one of the pilot's face are coherent, therefore, according to the facial feature visual regions when the face is not occluded before being occluded, the facial feature visual regions when the face is occluded are corrected. Specifically, first, the possibility of fatigue work during the time periods corresponding to all normal periods of the pilot is evaluated to obtain the abnormality degree of the normal periods, and then, according to the abnormality degree of the normal periods, the areas of the facial feature visual regions of the facial features during the occlusion period are corrected respectively; finally, according to the area correction results and the consecutive frame images extracted during the pilot's work process, the pilot fatigue monitoring results using vision technology are obtained, solving the problem that when the pilot's face is occluded, some key feature points on the face are inaccurately recognized, affecting the judgment of the fatigue state, and improving the accuracy of pilot fatigue detection based on vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0016] Figure 1 It is a flowchart of a method for monitoring pilot fatigue using vision technology provided by an embodiment of the present invention; Figure 2 It is a flowchart for obtaining the occlusion period and the normal period provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0018] Please refer to Figure 1, which shows a flowchart of a pilot fatigue monitoring method using vision technology provided by an embodiment of the present invention. The method includes the following steps: Step S001, extract consecutive frame images during the pilot's work process, identify the facial feature regions of the pilot from the consecutive frame images, obtain the facial feature visual regions, and determine the facial feature sequences of the facial features of the facial features corresponding to the same facial features in the same time period.

[0019] Extract consecutive frame images of the pilot's work process from the data storage center of the ground control center. For each image in the extracted consecutive frame images, use opencv to identify the facial feature regions of the pilot in the image, and record the facial feature regions as facial feature visual regions.

[0020] Among them, using opencv to identify the facial feature regions of the pilot in the image is a well-known technology and will not be elaborated. It can be understood that the area of the facial feature visual region is the number of pixel points included in the facial feature visual region.

[0021] Arrange the areas of the facial feature visual regions corresponding to the same facial features in the consecutive frame images within the same time period in the order of the acquisition times corresponding to the images where the facial feature visual regions are located, and obtain the facial feature sequences of the facial features in the same time period.

[0022] Preferably, in an embodiment of the present application, one minute is used as the time length corresponding to the time period. In actual application, as other implementation manners, the implementer can determine the value of the time length corresponding to the time period according to the actual situation, and the present application does not make special restrictions.

[0023] It can be understood that each of the two eyebrows, two eyes, two ears, nose, and mouth in the facial features has a facial feature sequence in each time period. For example, in the same time period, the left ear and the right ear both correspond to a facial feature sequence.

[0024] So far, the facial feature sequences of the facial features in the same time period during the pilot's work process are obtained.

[0025] Step S002, record any one of the facial features as the target facial feature, determine the occlusion possibility of the target facial feature in the same time period according to the differences between all the data included in the facial feature sequence of the target facial feature in the same time period, and determine the occlusion period and the normal period according to the occlusion possibility.

[0026] Record any one time period as the target time period, record any one of the facial features as the target facial feature, and determine the occlusion possibility of the target facial feature in the target time period according to the differences between all the data included in the facial feature sequence of the target facial feature in the target time period.

[0027] Preferably, as an embodiment of the present application, the reciprocal of the variance of all data included in the facial feature sequence of the target facial feature in the target time period is denoted as the occlusion possibility of the target facial feature in the target time period.

[0028] Wherein, in the process of calculating the reciprocal, the variance of all data included in the facial feature sequence of the target facial feature in the target time period is used as the denominator, and the ratio of 1 to the denominator is calculated. In the process of calculating the ratio, in order to avoid the case where the denominator is zero, a preset value needs to be added to the denominator. In an embodiment of the preset value, the value is 1.

[0029] When the difference between all data included in the facial feature sequence of the target facial feature in the target time period is greater, the possibility of the target facial feature showing normal human movements in the target time period is greater, the possibility of the target facial feature being occluded in the target time period is smaller, and the possibility of the pilot working normally is greater. At this time, the occlusion possibility of the target facial feature in the target time period is smaller.

[0030] For example, when the target facial feature is the left eye, the left eye will definitely blink within a time period, and the difference between all data included in the facial feature sequence is relatively large, and the occlusion possibility of the target facial feature in the target time period is relatively small; when the target facial feature is occluded in the target time period, the area of the facial feature visual region remains unchanged, and the difference between all data included in the facial feature sequence is relatively small, and the occlusion possibility of the target facial feature in the target time period is relatively large.

[0031] The occlusion possibility of any organ in any time period of the facial features can be obtained in the same way. That is to say, for each organ of the facial features, there is a corresponding occlusion possibility in each time period.

[0032] The time period with an occlusion possibility greater than is denoted as a suspected occlusion period. In the continuous time periods of the target facial feature, each suspected occlusion period that is continuous and greater than or equal to is denoted as an occlusion period. All time periods that are not occlusion periods are denoted as normal periods. The flowchart for obtaining the occlusion period and the normal period is as shown in Figure 2 shown.

[0033] Wherein, represents the first preset threshold, represents the second preset threshold. The first preset threshold and the second preset threshold are both preset constants. In this embodiment, the value of the first preset threshold is 0.7, and the value of the second preset threshold is 3.

[0034] It can be understood that the occlusion period corresponding to the target facial feature is the time period during which the target facial feature is occluded. For the target facial feature, multiple occlusion periods may be obtained, or there may be no corresponding occlusion period. When there is no corresponding occlusion period for the target facial feature, that is, the target facial feature does not appear to be occluded during the pilot's work. At this time, all time periods of the target facial feature are normal periods.

[0035] All occlusion periods and normal periods of any one of the facial features can be obtained in the same way.

[0036] So far, all occlusion periods and normal periods of all facial features have been obtained.

[0037] Step S003: According to the differences between the areas of the facial feature visual regions corresponding to all normal periods of the target facial feature, determine the abnormality degree of the target facial feature in each normal period respectively. Denote any one occlusion period as the target occlusion period, and combine the interval between the normal period before the target occlusion period and the target occlusion period, as well as the area of the facial feature visual region corresponding to the target occlusion period, to respectively correct the area of the facial feature visual region of the facial feature at the target occlusion period.

[0038] When the pilot's face is occluded, it is impossible to accurately divide the facial feature visual region in the image. At this time, the area of the recognized facial feature visual region is the same as that of the facial feature visual region of the same unoccluded facial feature in the previous adjacent consecutive frames. Therefore, the extracted facial feature visual region needs to be corrected. Since the facial feature visual region when the pilot's face is occluded cannot be directly obtained, in this embodiment, the facial feature visual region when the face is occluded is corrected according to the facial feature visual region when the face is not occluded before being occluded.

[0039] According to the differences between the areas of the facial feature visual regions corresponding to all normal periods of the target facial feature, determine the abnormality degree of the target facial feature in any one normal period respectively.

[0040] Denote any one normal period as the target normal period; denote the average value of the DTW distances between the facial feature sequences of the facial feature in every two normal periods among all normal periods of the target facial feature as the abnormality degree of the normal period of the target facial feature; denote the average value of the areas of the facial feature visual regions corresponding to all normal periods of the target facial feature as the average area of the target facial feature, and denote the absolute value of the difference between the average value of the area of the facial feature visual region corresponding to the target normal period and the average area of the target facial feature as the area difference of the feature visual region of the target facial feature in the target normal period; denote the normalized value of the product of the abnormality degree of the normal period of the target facial feature and the area difference of the feature visual region of the target facial feature in the target normal period as the abnormality degree of the target facial feature in the target normal period.

[0041] In the same way, the abnormality degree of the target facial feature in any normal period can be obtained.

[0042] It can be understood that calculating the DTW distance between sequences is a well-known technology and will not be elaborated here. It should be noted that in this embodiment, the Z-Score standard normalization method is used to calculate the normalization value. In the actual application process, the implementer can adopt other methods of existing technologies, such as the maximum-minimum normalization method, sigmoid function, etc., to calculate the normalization value, which is not limited here.

[0043] When the facial feature sequences of all normal periods of the target facial feature are closer, the facial features of the pilot in different normal periods are more similar. At this time, the abnormality degree of the normal period of the target facial feature is smaller, and the abnormality degree of the target facial feature in the target normal period is smaller. For example, when the target facial feature is the left eye, the blinking frequencies of the left eye of the pilot in different occlusion periods are closer, and the possibility of the pilot working normally in the corresponding time periods of all normal periods is greater. When the difference between the mean value of the area of the corresponding facial feature visual area in the target normal period and the average area of the target facial feature is greater, the area difference of the feature visual area of the target facial feature in the target normal period is greater, and the abnormality degree of the target facial feature in the target normal period is greater. At this time, the possibility of the pilot working fatigued in the corresponding time periods of all normal periods is greater.

[0044] Furthermore, since the facial states of the pilot are relatively similar in a short period of time. For example, when the pilot is fatigued, the blinking of the eyes will become faster, and due to actions such as yawning, the facial features will change. Whether the pilot's face is unoccluded or occluded, these facial features that are similar in a short period of time are still similar. Therefore, the facial feature visual area when the face is occluded is corrected according to the facial feature visual area when the face is unoccluded before being occluded.

[0045] Denote any occlusion period as the target occlusion period, and determine the adjustment coefficient of each organ in the five sense organs in the target occlusion period respectively according to the interval between the normal period before the target occlusion period and the target occlusion period, and the abnormality degree of the normal period.

[0046] Denote the normal period closest to the time before the target occlusion period as the adjacent normal period of the target occlusion period. Denote the reciprocal of the sum of the number of periods between the target occlusion period and the adjacent normal period and the number 1 as the first weight of the target occlusion period. Denote the difference between the number 1 and the first weight of the target occlusion period as the second weight of the target occlusion period. Denote the average abnormal degree of all organs different from the target facial features in the adjacent normal period of the target occlusion period as the comparison facial features average of the target facial features. Denote the product of the comparison facial features average of the target facial features and the second weight of the target occlusion period as the second product of the target facial features in the target occlusion period. Denote the product of the first weight of the target occlusion period and the abnormal degree of the target facial features in the adjacent normal period of the target occlusion period as the first product of the target facial features in the target occlusion period. Denote the sum of the first product and the second product of the target facial features in the target occlusion period as the adjustment coefficient of the target facial features in the target occlusion period.

[0047] It should be noted that when there is no normal period before the target occlusion period, take the average abnormal degree of all normal periods as the abnormal degree of the adjacent normal period of the target occlusion period, and assign the first weight and the second weight of the target occlusion period to .

[0048] The adjustment coefficient of each organ in the five facial features in the target occlusion period can be obtained in the same way.

[0049] Denote the product of the adjustment coefficient of the target facial features in the target occlusion period and the third preset threshold as the adjustment area of the target facial features in the target occlusion period. Denote the rounded value of the difference between the average area of the facial feature visual region corresponding to the target facial features in the target occlusion period and the adjustment area as the area correction value of the target facial features in the target occlusion period. Assign the area of the facial feature visual region of the target facial features in the target occlusion period to the area correction value of the target facial features in the target occlusion period to realize the correction of the area of the facial feature visual region of the target facial features in the target occlusion period.

[0050] Among them, the third preset threshold is a preset constant, and the value of the third preset threshold in this embodiment is 50.

[0051] The area of the facial feature visual region of each organ in the five facial features in the target occlusion period can be corrected in the same way, and the area of the facial feature visual region of each organ in each occlusion period can be corrected.

[0052] Thus, the area of the facial feature visual region of each organ in each occlusion period is corrected.

[0053] Step S004: Obtain the pilot fatigue monitoring result using vision technology based on the area correction result and the consecutive frame images extracted during the pilot's work process.

[0054] Input the areas of the facial feature visual regions of each organ in the five sense organs during all occlusion cycles and normal cycles, as well as the consecutive frame images extracted during the pilot's work process, into the pilot fatigue monitoring neural network model to obtain the pilot's driving state evaluation value. The value range of the pilot's driving state evaluation value is greater than or equal to 0 and less than or equal to 1.

[0055] When the pilot's driving state evaluation value is greater than or equal to the fourth preset threshold, it is determined that the pilot is in a fatigued driving state; when the pilot's driving state evaluation value is less than the fourth preset threshold, it is determined that the pilot is in a non-fatigued driving state. Among them, the fourth preset threshold is a preset constant, and in this embodiment, the value of the fourth preset threshold is 0.5.

[0056] Among them, the input of the pilot fatigue monitoring neural network model is the areas of the facial feature visual regions of each organ in the five sense organs during all occlusion cycles and normal cycles and the consecutive frame images extracted during the pilot's work process, and the output is the pilot's driving state evaluation value. The pilot fatigue monitoring neural network model is a pre-trained DNN network, the network task is prediction, the data set is the consecutive frame images during the pilot's work process under different postures and expression conditions, the training set is the areas of the facial feature visual regions identified from the consecutive frame images extracted during the pilot's work process, and the loss function used is the cross-entropy loss function; Training the DNN network is a well-known technology and will not be elaborated here.

[0057] Thus far, the pilot fatigue monitoring result using vision technology is obtained.

[0058] Based on the same inventive concept as the above method, an embodiment of the present invention further provides a pilot fatigue monitoring system using vision technology, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above methods of a pilot fatigue monitoring method using vision technology.

[0059] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for monitoring pilot fatigue using vision technology, characterized in that, The method includes the following steps: During the pilot's work process, consecutive frame images are extracted. The facial feature visual regions of the pilot's facial features are identified from the consecutive frame images, and the facial feature visual regions corresponding to the same facial features in the same time period are obtained, and the facial feature sequences of the facial features in the same time period corresponding to the same facial features are determined; Any one of the facial features is denoted as the target facial feature. According to the differences between all the data included in the facial feature sequence of the target facial feature in the same time period, the occlusion possibility of the target facial feature in the same time period is determined. According to the occlusion possibility, the occlusion period and the normal period are determined; According to the differences between the areas of the facial feature visual regions corresponding to all the normal periods of the target facial feature, the abnormality degree of the target facial feature in each normal period is determined respectively. Any one of the occlusion periods is denoted as the target occlusion period. Combining the interval between the normal period before the target occlusion period and the target occlusion period, and the area of the facial feature visual region corresponding to the target occlusion period, the area correction of the facial feature visual region of the facial feature in the target occlusion period is realized respectively; According to the area correction result and the consecutive frame images extracted during the pilot's work process, the pilot fatigue monitoring result using visual technology is obtained.

2. The pilot fatigue monitoring method using vision technology according to claim 1, characterized in that, The method for obtaining the facial feature sequence is as follows: The areas of the facial feature visual regions corresponding to the same facial features in the consecutive frame images in the same time period are arranged in the order of the acquisition times corresponding to the images where the facial feature visual regions are located, and the facial feature sequence of the facial features in the same time period is obtained; The area of the facial feature visual region is the number of pixel points included in the facial feature visual region.

3. A pilot fatigue monitoring method using vision technology according to claim 1, characterized in that The method for determining the occlusion possibility of the target facial feature in the same time period is as follows: The reciprocal of the variance of all the data included in the facial feature sequence of the target facial feature in the target time period is denoted as the occlusion possibility of the target facial feature in the target time period.

4. A pilot fatigue monitoring method using vision technology according to claim 1, characterized in that, The specific method for determining the occlusion period and the normal period according to the occlusion possibility includes: The time period with an occlusion possibility greater than is recorded as a suspected occlusion period. For the continuous time period of the target facial features, each suspected occlusion period that is greater than or equal to consecutive suspected occlusion periods is recorded as an occlusion period, where represents the first preset threshold, and represents the second preset threshold. All time periods that are not occlusion periods are denoted as normal periods.

5. A pilot fatigue monitoring method using vision technology according to claim 1, characterized in that, The specific method for determining the abnormality degree of the target facial feature in each normal period according to the differences between the areas of the facial feature visual regions corresponding to all the normal periods of the target facial feature includes: Any one of the normal periods is denoted as the target normal period; the average value of the DTW distances between the facial feature sequences of every two normal periods among all the normal periods of the target facial feature is denoted as the abnormality degree of the normal period of the target facial feature; the average value of the areas of the facial feature visual regions corresponding to all the normal periods of the target facial feature is denoted as the average area of the target facial feature, and the absolute value of the difference between the average value of the area of the facial feature visual region corresponding to the target normal period and the average area of the target facial feature is denoted as the area difference of the feature visual region of the target facial feature in the target normal period; According to the abnormality degree of the normal period of the target facial feature in the target normal period and the area difference of the feature visual region of the target facial feature, the abnormality degree of the target facial feature in the target normal period is determined.

6. The pilot fatigue monitoring method using vision technology according to claim 5, wherein, Determining the abnormality degree of the target facial features in the target normal period according to the abnormality degree of the target facial features in the target normal period and the area difference of the characteristic visual regions of the target facial features, the specific method included is as follows: Denote the normalized value of the product of the abnormality degree of the target facial features in the target normal period and the area difference of the characteristic visual regions of the target facial features as the abnormality degree of the target facial features in the target normal period.

7. A method for monitoring pilot fatigue using vision technology according to claim 1, characterized in that, Combining the cycle interval between the normal period before the target occlusion period and the target occlusion period, and the area of the facial feature visual region corresponding to the target occlusion period, respectively realizing the correction of the area of the facial feature visual region of the facial features in the target occlusion period, the specific method included is as follows: Denote the normal period closest to the target occlusion period in terms of time before the target occlusion period as the adjacent normal period of the target occlusion period, denote the reciprocal of the sum of the number of intervals between the target occlusion period and the adjacent normal period and the number 1 as the first weight of the target occlusion period, and denote the difference between the number 1 and the first weight of the target occlusion period as the second weight of the target occlusion period; Denote the average value of the abnormality degrees of all organs different from the target facial features in the adjacent normal period of the target occlusion period as the comparison facial feature average value of the target facial features, and denote the product of the comparison facial feature average value of the target facial features and the second weight of the target occlusion period as the second product of the target facial features in the target occlusion period; Denote the product of the first weight of the target occlusion period and the abnormality degree of the target facial features in the adjacent normal period of the target occlusion period as the first product of the target facial features in the target occlusion period; Denote the sum of the first product and the second product of the target facial features in the target occlusion period as the adjustment coefficient of the target facial features in the target occlusion period; According to the adjustment coefficient of the target facial features in the target occlusion period and the area of the facial feature visual region corresponding to the target occlusion period, realize the correction of the area of the facial feature visual region of the target facial features in the target occlusion period.

8. A method for monitoring pilot fatigue using vision technology according to claim 7, characterized in that, According to the adjustment coefficient of the target facial features in the target occlusion period and the area of the facial feature visual region corresponding to the target occlusion period, realizing the correction of the area of the facial feature visual region of the target facial features in the target occlusion period, the specific method included is as follows: Denote the product of the adjustment coefficient of the target facial features in the target occlusion period and the third preset threshold as the adjusted area of the target facial features in the target occlusion period, denote the rounded value of the difference between the average value of the area of the facial feature visual region corresponding to the target facial features in the target occlusion period and the adjusted area as the area correction value of the target facial features in the target occlusion period, and assign the area of the facial feature visual region of the target facial features in the target occlusion period to the area correction value of the target facial features in the target occlusion period to realize the area correction.

9. A method for monitoring pilot fatigue using vision technology according to claim 1, characterized in that, According to the area correction result and the continuous frame images extracted during the pilot's work, obtaining the pilot fatigue monitoring result using visual technology, the specific method included is as follows: Input the areas of the facial feature visual regions of each organ in the facial features in all occlusion periods and normal periods, and the continuous frame images extracted during the pilot's work into the pilot fatigue monitoring neural network model to obtain the driving state evaluation value of the pilot; When the driving state evaluation value of the pilot is greater than or equal to the fourth preset threshold, it is determined that the pilot is in a fatigued driving state; When the driving state evaluation value of the pilot is less than the fourth preset threshold, it is determined that the pilot is in a non-fatigued driving state.

10. A pilot fatigue monitoring system using vision technology, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-9.

Images