Non-wearable pilot fatigue state intelligent detection method and device

Abstract

The invention discloses a non-wearable pilot fatigue state intelligent detection method and device, and the method comprises the steps: installing an array camera in front of a pilot seat, and collecting face images of a pilot at different angles; the face image is preprocessed; a CNN-BiMGRU-MHAM network model is built, and the CNN-BiMGRU- And optimizing the network model by using a sparrow search algorithm based on a sine and cosine strategy and a Gaussian-Cauchy variation strategy to obtain an optimal network model, and inputting the preprocessed face image into the optimal network model to carry out intelligent detection on the fatigue state of the pilot. According to the invention, a machine vision technology, an intelligent optimization technology and a deep learning technology are combined, so that accurate detection of the fatigue state of the pilot is realized; on one hand, inconvenience caused by a traditional wearable device to pilot operation is avoided, on the other hand, the accuracy of fatigue state recognition is effectively improved, normal flight safety cannot be affected, and reliable guarantee is provided for flight safety.

Description

Technical Field

[0001] The present invention relates to the field of aircraft driving and artificial intelligence technology, and in particular to a non-wearable intelligent detection method and device for pilot fatigue status. Background Art

[0002] In recent years, with the continuous development and growth of the aviation industry and the increasing demand for civil aviation flights, the workload of pilots has become increasingly heavy. Especially for pilots who often perform flight missions at night, the closed space and boring environment can easily make pilots fatigued, and pilot fatigue flight will bring great safety hazards. Therefore, pilot fatigue status detection and early warning are of great significance to eliminate potential flight hazards, ensure flight safety and reduce the occurrence of aviation accidents.

[0003] At present, pilot fatigue detection mainly includes two methods: subjective evaluation and objective evaluation. Subjective evaluation determines whether the subject is tired by asking the subject to fill in a fatigue scale. This method is greatly affected by personal subjective feelings and can only be used as an auxiliary means. The objective evaluation method mainly uses various instruments to collect multiple physiological signals such as blood pressure, electrocardiogram, skin electricity, myoelectricity, brain waves, etc. of the pilot to determine the fatigue state and degree. This method is relatively objective and accurate, but it requires the pilot to wear a variety of physiological detection equipment, which not only increases the detection cost, but also is inconvenient for flight driving, and may even increase the psychological burden of the pilot. Therefore, it is difficult to be widely used in actual operating environments. In addition to physiological signals that can reflect the fatigue state of a person, facial information such as facial features and pupil reactions can also be used to identify the fatigue state. Compared with the difficulty of obtaining physiological signals, the collection of facial information can be achieved through a camera without physical contact. In addition, with the development of computer vision technology, it is easy to obtain a high-definition face image set even in extreme environments such as facial occlusion and dim light in the cockpit. Therefore, the method of fatigue detection through facial information has good applicability and broad application prospects.

[0004] In this regard, the present invention considers combining computer vision technology and artificial intelligence technology to develop a non-wearable intelligent detection method and device for pilot fatigue status, which can be used to quickly and accurately diagnose the pilot's fatigue status, with low detection cost and no interference to the pilot. Summary of the invention

[0005] In view of this, in order to solve the problems of low accuracy, high cost and impact on normal flight operations of pilots in existing detection methods, the present invention provides a non-wearable intelligent detection method and device for pilot fatigue status. The use of the present invention can accurately and quickly understand the pilot's fatigue status; at the same time, the detection cost is low and will not interfere with the pilot.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, an embodiment of the present invention provides a non-wearable intelligent detection method for pilot fatigue status, the method comprising the following steps:

[0008] S1. Install an array camera in front of the pilot seat;

[0009] S2, using the array camera to collect facial images of the pilot at different angles;

[0010] S3, preprocessing the face image;

[0011] S4. Build the CNN-BiMGRU-MHAM network model;

[0012] S5. Use the sparrow search algorithm based on the sine-cosine strategy and Gauss-Cauchy mutation strategy to optimize the CNN-BiMGRU-MHAM network model to obtain the best network model, and input the preprocessed face image into the best network model for intelligent detection of pilot fatigue status.

[0013] Optionally, the method further includes:

[0014] S6. Visually display the test results on the terminal platform.

[0015] Optionally, in step S1, the array camera includes three wide-angle cameras, which are respectively installed in front of and on both sides of the pilot's seat.

[0016] Optionally, in step S2, the array camera acquires three face images at the same time, and uses an XGBoost detection algorithm based on Haar features to identify key face areas from the images respectively, and generates new face images to form a face image set.

[0017] Optionally, in step S3, all newly generated face images are subjected to grayscale conversion and histogram equalization processing one by one, and the processed images are resized to obtain a set of face grayscale images with uniform size.

[0018] Optionally, in step S4, a CNN-BiMGRU-MHAM network model is constructed, specifically:

[0019] First, a CNN network consisting of convolutional layers, pooling layers, and fully connected layers is built to extract the spatial features of the grayscale face image.

[0020] Then, the Mogrifier data coupling module is introduced to form an MGRU network together with the GRU lightweight model to replace the long short-term memory network to form a BiMGRU network. Through the interaction of contexts, the information features of time correlation in the data are extracted; the model expression of the BiMGRU network is:

[0021]

[0022] In the formula, is the forward hidden state output by the forward MGRU network at time t; is the reverse hidden state output by the reverse MGRU network at time t; For input x t The hidden state output of the network; ω t1 and ω t2 are the weights for forward and backward propagation respectively; b t is bias;

[0023] Finally, the multi-head attention mechanism is introduced to extract key core features from the extracted features and remove redundant information. The whole process is:

[0024] Q=XW Q

[0025] K=XW K

[0026] V=XW V

[0027]

[0028] Multihead(Q,K,V)=concat(head1,L,head h )W o

[0029] Where X is the output of BiMGRU; W Q , W K , W V They are three trainable parameter matrices respectively; d k is the dimension; W o is the output weight matrix; Softmax() means normalizing the scores.

[0030] Optionally, in step S5, the CNN-BiMGRU-MHAM network model is optimized using a sparrow search algorithm based on a sine-cosine strategy and a Gauss-Cauchy mutation strategy, and the specific process includes:

[0031] In the sparrow search algorithm, the sine and cosine strategies are used to continuously update the discoverer position, so as to maintain the individual diversity of the discoverers and improve the global optimization ability of the sparrow search algorithm. The specific strategies are:

[0032]

[0033] In the formula, is the position of the i-th sparrow in the j-th dimension in the t-th iteration; is the updated position; is the current global optimal position; λ is the adaptive balance factor; β is the search factor; r1 determines the moving distance of the sparrow; r2 controls the influence of the optimal individual on the next position of the sparrow; α is the adjustment factor; R2 represents the warning value; ST represents the safety value;

[0034] At the same time, in order to avoid the algorithm falling into the local optimum, the Gauss-Cauchy mutation strategy is used to update the position of the follower, and the new follower position is updated as follows:

[0035]

[0036] In the formula, δ 2 is the standard deviation; Iter max is the maximum number of iterations;

[0037] To ensure safety during foraging, some sparrows are used for alerting. The initial positions of these sparrows are randomly generated, and the position update is performed as follows:

[0038]

[0039] Where μ is the step size control parameter; K is a random number between -1 and 1; f i is the fitness value of the current sparrow; f g is the current global optimal fitness value; f ω is the current global worst fitness value; ε is a constant.

[0040] In a second aspect, the present invention also provides a non-wearable intelligent detection device for pilot fatigue status, comprising: a data acquisition device, a memory and a processor, wherein:

[0041] The data acquisition device is used to collect facial images of pilots at different angles;

[0042] The memory stores computer instructions;

[0043] The processor is data-connected to the data acquisition device and the memory. The processor executes the computer instructions to perform the above-mentioned non-wearable pilot fatigue state detection method and intelligently detect the pilot fatigue state.

[0044] In a third aspect, the present invention also provides a storage medium having stored thereon one or more programs readable by a computing device, wherein the one or more programs include instructions, which, when executed by the computing device, enable the computing device to execute a non-wearable intelligent detection method for pilot fatigue status as described above.

[0045] Compared with the prior art, the present invention has at least the following beneficial effects:

[0046] 1. Based on the CNN-BiMGRU-MHAM model, the present invention provides a non-wearable intelligent detection method for pilot fatigue status. The present invention combines computer vision technology and deep learning to detect pilot fatigue status, thereby improving the detection accuracy. Compared with the method using physiological signals, the present method does not require contact with the pilot, does not interfere with the pilot's normal operation, and also reduces the detection cost.

[0047] 2. The non-wearable intelligent detection method for pilot fatigue status of the present invention can simultaneously capture three facial images of the pilot at different angles at the same time, which can avoid the problem of face occlusion compared with single image acquisition; at the same time, the XGBoost detection algorithm based on Haar features is used to identify key facial areas from the image, and clear facial features can be obtained even in dim conditions, which is beneficial to improving the quality of image samples.

[0048] 3. The non-wearable intelligent detection method for pilot fatigue status of the present invention, the designed deep learning network can not only mine spatial features from samples, but also extract temporal correlation features, and can more accurately reveal the occurrence and evolution process and laws of fatigue expressions at the previous and next moments. The extracted feature information is richer, which helps to improve the detection accuracy; at the same time, it introduces a unique context interaction mechanism and a multi-head attention mechanism, which improves the training speed of the network model.

[0049] 4. In a non-wearable intelligent detection method for pilot fatigue status of the present invention, a sparrow search algorithm based on sine-cosine strategy and Gauss-Cauchy variation strategy (SCGCSSA) is used to optimize the network structure, hyperparameters and attention weights, so that the network model with the best detection performance can be obtained in a relatively short time, thereby improving the training speed of the model.

[0050] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0054] Figure 1 A schematic diagram of the principle of a non-wearable intelligent detection method for pilot fatigue status provided by an embodiment of the present invention.

[0055] Figure 2 A flowchart of a non-wearable intelligent detection method for pilot fatigue status provided by an embodiment of the present invention.

[0056] Figure 3 A schematic diagram of the model training and testing process provided by an embodiment of the present invention.

[0057] Figure 4 A schematic diagram of the changing trends of accuracy and loss function during the model training process provided in an embodiment of the present invention.

[0058] Figure 5 A schematic diagram of a confusion matrix for comparing the results of an embodiment of the present invention with other intelligent swarm optimization algorithms.

[0059] Figure 6 Schematic diagram of the detection result visualization interface developed based on Python provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0061] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0062] Therefore, the detailed description of the embodiments of the present invention provided below is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Embodiment 1:

[0064] See also Figure 1 and Figure 2 As shown, the embodiment of the present invention provides a non-wearable intelligent detection method for pilot fatigue status, which uses an array camera to collect the pilot's front, left and right facial images respectively, and after pre-processing, sends them to a trained deep learning network model to realize fatigue status detection, which is convenient and fast to operate. The general process includes the following steps:

[0065] S1, using multiple wide-angle cameras to form an array camera, evenly placed in front of the pilot seat;

[0066] S2, using an array camera composed of multiple cameras to collect facial image sets from different angles of the pilot in different fatigue states (awake state, mild fatigue state, and severe fatigue state);

[0067] S3, performing grayscale conversion and histogram equalization processing operations on the acquired face image set in sequence, and resizing the processed face grayscale image set to obtain a data set of uniform size, and dividing it into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0068] S4. Build the CNN-BiMGRU-MHAM network model and initialize the network structure and parameters;

[0069] S5. Use the sparrow search algorithm based on the sine-cosine strategy and Gauss-Cauchy mutation strategy (SCGCSSA) to optimize the network architecture, hyperparameters and attention weights simultaneously to obtain the best network model, and test the optimal network model with the test set;

[0070] S6. Develop a terminal user interface to visualize the results of the above test analysis on the terminal platform.

[0071] The working principle and specific implementation methods of the method of the present invention are described in detail below:

[0072] In this embodiment, first, an array camera is installed. The array camera includes three wide-angle cameras, which are installed in front of the pilot and on both sides respectively.

[0073] Furthermore, for the three face images acquired by the array camera at the same time, the XGBoost detection algorithm based on Haar features is used to identify the key face areas from the images respectively, and generate new face images.

[0074] Furthermore, the face images are preprocessed, specifically including: grayscale and histogram equalization processing are performed on all newly generated face images one by one, and the size of the processed grayscale images is uniformly adjusted to 200×200, and then divided into training set, validation set and test set according to the ratio of 7:2:1.

[0075] Furthermore, a CNN-BiMGRU-MHAM network model is built, which specifically includes:

[0076] First, a CNN network consisting of 2 convolutional layers, 2 pooling layers, and 1 fully connected layer is built to extract the spatial features of the face grayscale image. Then, the Mogrifier data coupling module is introduced to form an MGRU network together with the GRU lightweight model to replace the long short-term memory network (LSTM) to form a BiMGRU network. Through the interaction of contexts, the information features of time correlation in the data can be quickly extracted. The model expression of the BiMGRU network is shown as follows:

[0077]

[0078] In the formula, is the forward hidden state output by the forward MGRU network at time t; is the reverse hidden state output by the reverse MGRU network at time t; For input x t The hidden state output of the network; ω t1 and ω t2 are the weights for forward and backward propagation respectively; b t For bias.

[0079] Finally, the multi-head attention mechanism (MHAM) is introduced to extract key core features from the extracted features and remove redundant information, which is conducive to speeding up network training. The whole process is shown in the following formula:

[0080] Q=XW Q

[0081] K=XW K

[0082] V=XW V

[0083]

[0084] Multihead(Q,K,V)=concat(head1,L,head h )W o

[0085] Where X is the output of BiMGRU; W Q , W K , W V They are three trainable parameter matrices respectively; d k is the dimension; W o is the output weight matrix; Softmax() means normalizing the scores.

[0086] Furthermore, the sparrow search algorithm based on the sine-cosine strategy and Gauss-Cauchy mutation strategy is used to optimize the CNN-BiMGRU-MHAM network model to obtain the best network model, and the preprocessed face image is input into the best network model for intelligent detection of pilot fatigue status, including:

[0087] First, the sine-cosine strategy (SC) is used in the sparrow search algorithm (SSA) to continuously update the discoverer position, so as to maintain the diversity of discoverers and improve the global optimization ability of SSA. The whole strategy is shown in the following formula:

[0088]

[0089] In the formula, is the position of the i-th sparrow in the j-th dimension in the t-th iteration; is the updated position; is the current global optimal position; λ is the adaptive balance factor; β is the search factor; r1 determines the moving distance of the sparrow; r2 controls the influence of the optimal individual on the next position of the sparrow; α is the adjustment factor; R2 represents the warning value; ST represents the safety value.

[0090] At the same time, in order to avoid the algorithm falling into the local optimum, the Gauss-Cauchy mutation strategy (GC) is used to update the position of the follower and improve the global optimization ability. The new follower position update is shown in the following formula:

[0091]

[0092] In the formula, is the new position after adopting the Gauss-Cauchy mutation strategy; δ 2 is the standard deviation; Iter max is the maximum number of iterations.

[0093] In order to ensure safety during foraging, some sparrows are used for alerting. The initial positions of these sparrows are randomly generated, and the position update is performed according to the following formula:

[0094]

[0095] In the formula, is the current global optimal position; μ is the step size control parameter; K is a random number in [-1, 1]; f i is the fitness value of the current sparrow; f g is the current global optimal fitness value; f ω is the current global worst fitness value; ε is a very small constant.

[0096] In this embodiment, the parameters that need to be set for the SCGCSSA algorithm are: the number of sparrow populations N, the number of iterations M, the safety threshold ST, the proportion of discoverers PD, the proportion of scouts SD, the warning threshold R, the optimization dimension D, and the fitness function F. The present invention selects the cross entropy loss function as the fitness function. The algorithm flow is:

[0097] (1) Initialize the population and set the initial values ​​of parameters such as N, M, ST, PD, SD, R, and D;

[0098] (2) Calculate the fitness value of each sparrow and sort them, and determine the individuals with the best and worst fitness at the moment;

[0099] (3) Update the finder's location;

[0100] (4) Update the joiner's location;

[0101] (5) Update the position of the alerter;

[0102] (6) Recalculate the fitness value of the population and update the position of the new sparrow individual;

[0103] (7) Determine whether the current number of iterations meets the termination condition. If so, end the process; otherwise, jump to (2).

[0104] The proposed SCGCSSA algorithm is used to optimize the CNN network structure, hyperparameters, and attention weights to obtain the optimal network structure; then the test set is sent to the trained network model for testing.

[0105] Furthermore, a terminal user interface is developed to visualize the results of the above test analysis on the terminal platform: specifically, first, a terminal user platform embedded with a trained deep learning network model is developed based on Python, and it is saved on a storage unit with computing functions; then, the computing storage unit is connected to the array camera; finally, the user can use the operation interface to visualize the pilot's fatigue status in real time on the terminal.

[0106] Figure 3The figure is a flow chart of the model training and testing provided by the embodiment of the present invention. The deep network consists of a CNN network, a BiMGRU network with context information interaction, and an MHAM model. The initial structure of the CNN includes a layer consisting of 2 convolutional layers, 2 pooling layers, and 1 fully connected layer. The convolutional layer search range is 2 to 6, and the learning rate search range is set to 10. -4 ~10 -2 , Epoach=250, the batch size search range is 10-20, the MGRU neuron number search range is set to less than 256, the MHAM model head number search range is 4-8, and the dimension of each head is 64.

[0107] The SCGCSSA algorithm is used to optimize the relevant parameters in the CNN-BiMGRU-MHAM network training process. After obtaining the optimal network structure, the test set is used for testing and verification. The hyperparameters of the learning network after optimization are shown in Table 1.

[0108] Table 1 Results after hyperparameter optimization

[0109] parameter Number of CNN convolutional layers Learning Rate Batch size MGRU Neuron MHAM Head Count Value 4 0.00152 10 68 6

[0110] Figure 4 It is the changing trend of the accuracy curve and loss function curve of the model provided by the embodiment of the present invention during the training process. Figure 4 It can be seen that during the training process, when Epoch = 100, the accuracy of the network model tends to stabilize at around 99%, the verification accuracy is around 98%, and the loss function also gradually shows a convergence trend.

[0111] Figure 5 This is the comparison test result of the SCGCSSA intelligent optimization algorithm adopted by the present invention and other intelligent optimization algorithms. Figure 5 Part (a) is a schematic diagram of the confusion matrix of the detection results after the network is optimized using the immune optimization algorithm (IA). Figure 5 Part (b) is a schematic diagram of the confusion matrix of the detection results after the network is optimized using the particle swarm algorithm (PSO). Figure 5 Part (c) is a schematic diagram of the confusion matrix of the detection results after the network is optimized using the ant colony algorithm (ACO). Figure 5 Part (d) is a schematic diagram of the confusion matrix of the detection results after the network is optimized using the Grey Wolf Algorithm (GWO). Figure 5 Part (e) is a schematic diagram of the confusion matrix of the detection results after the network is optimized using the standard sparrow algorithm (SSA). Figure 5 Part (f) is a schematic diagram of the confusion matrix of the detection results after the SCGCSSA optimization network of the present invention is used. It can be seen that:

[0112] In terms of accuracy, the accuracy rates achieved by using IA, PSO, ACO, GWO, and SSA to optimize the network are 91.00%, 91.00%, 87.00%, 78.33%, and 93.33%, respectively. The accuracy rate achieved by using SCGCSSA to optimize the network is 99.00%, which are improved by 8.00%, 8.00%, 12.00%, 20.67%, and 5.67%, respectively.

[0113] In terms of F1-score, the F1-scores obtained by using the IA, PSO, ACO, GWO, and SSA optimization networks are 91.04%, 91.05%, 87.03%, 78.40%, and 93.33%, respectively, and the F1-score obtained by using the SCGCSSA optimization network is 99.00%, which are increased by 7.96%, 7.95%, 11.97%, 20.60%, and 5.67%, respectively. Therefore, the detection performance obtained by using the SCGCSSA optimization network proposed by the present invention is the best.

[0114] Figure 6 It is a test result visualization interface developed based on Python. The user interface platform here is developed based on Python language. The platform has functions such as result analysis and visualization display, data storage, etc. Through the development of this user interface platform, the pilot fatigue status test results are intuitively presented in the form of data and images.

[0115] From the description of the above embodiments, those skilled in the art can know that the present invention provides a non-wearable intelligent detection method for pilot fatigue status. The present invention combines machine vision technology, intelligent optimization technology and deep learning technology. On the one hand, it avoids the inconvenience caused by traditional wearable devices to pilot operation, and on the other hand, it effectively improves the accuracy of fatigue status recognition and provides reliable protection for flight safety. The non-wearable pilot fatigue status detection device of the present invention is simple in layout and easy to operate, and will not affect normal flight safety. .

[0116] Embodiment 2:

[0117] The present invention also provides a non-wearable intelligent detection device for pilot fatigue status, comprising: a data acquisition device, a memory and a processor, wherein:

[0118] The data acquisition device is used to collect facial images of pilots at different angles;

[0119] The memory stores computer instructions;

[0120] The processor is data-connected to the data acquisition device and the memory. The processor executes the computer instructions to perform the above-mentioned non-wearable pilot fatigue state detection method and intelligently detect the pilot fatigue state.

[0121] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0122] Example 3

[0123] An embodiment of the present invention also provides a storage medium on which one or more programs readable by a computing device are stored. The one or more programs include instructions. When the instructions are executed by the computing device, the computing device executes a non-wearable intelligent detection method for pilot fatigue status in Example 1.

[0124] Examples of computer-readable storage media include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on networked computer devices, so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0125] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0127] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0128] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A non-wearable intelligent detection method for pilot fatigue status, characterized in that: The method comprises the following steps: S1. Install an array camera in front of the pilot seat; S2, using the array camera to collect facial images of the pilot at different angles; S3, preprocessing the face image; S4. Build the CNN-BiMGRU-MHAM network model; S5. Use the sparrow search algorithm based on the sine-cosine strategy and Gauss-Cauchy mutation strategy to optimize the CNN-BiMGRU-MHAM network model to obtain the best network model, and input the preprocessed face image into the best network model for intelligent detection of pilot fatigue status.

2. A non-wearable intelligent detection method for pilot fatigue status according to claim 1, characterized in that: The method further includes: S6. Visually display the test results on the terminal platform.

3. The non-wearable intelligent detection method for pilot fatigue state according to claim 1 is characterized in that: In step S1, the array camera includes three wide-angle cameras, which are respectively installed in front of and on both sides of the pilot's seat.

4. The non-wearable intelligent detection method for pilot fatigue state according to claim 3 is characterized in that: In step S2, the array camera acquires three face images at the same time, and uses the XGBoost detection algorithm based on Haar features to identify key face areas from the images respectively, and generates new face images to form a face image set.

5. The non-wearable intelligent detection method for pilot fatigue state according to claim 4 is characterized in that: In step S3, all newly generated face images are subjected to grayscale conversion and histogram equalization processing one by one, and the processed images are resized to obtain a set of face grayscale images with uniform size.

6. A non-wearable intelligent detection method for pilot fatigue status according to claim 5, characterized in that: In step S4, a CNN-BiMGRU-MHAM network model is constructed, specifically: First, a CNN network consisting of convolutional layers, pooling layers, and fully connected layers is built to extract the spatial features of the grayscale face image. Then, the Mogrifier data coupling module is introduced to form an MGRU network together with the GRU lightweight model to replace the long short-term memory network to form a BiMGRU network. Through the interaction of contexts, the information features of time correlation in the data are extracted; the model expression of the BiMGRU network is: In the formula, is the forward hidden state output by the forward MGRU network at time t; is the reverse hidden state output by the reverse MGRU network at time t; For input x t The hidden state output of the network; ω t1 and ω t2 are the weights for forward and backward propagation respectively; b t is bias; Finally, the multi-head attention mechanism is introduced to extract key core features from the extracted features and remove redundant information. The whole process is: Q=XW Q K=XW K V=XW V Multihead(Q,K,V)=concat(head1,L,head h )W o Where X is the output of BiMGRU; W Q , W K , W V They are three trainable parameter matrices respectively; d k is the dimension; W o is the output weight matrix; Softmax() means normalizing the scores.

7. The non-wearable intelligent detection method for pilot fatigue state according to claim 6 is characterized in that: In step S5, the CNN-BiMGRU-MHAM network model is optimized using a sparrow search algorithm based on the sine-cosine strategy and the Gauss-Cauchy mutation strategy. The specific process includes: In the sparrow search algorithm, the sine and cosine strategies are used to continuously update the discoverer position, so as to maintain the individual diversity of the discoverers and improve the global optimization ability of the sparrow search algorithm. The specific strategies are: In the formula, is the position of the i-th sparrow in the j-th dimension in the t-th iteration; is the updated position; is the current global optimal position; λ is the adaptive balance factor; β is the search factor; r1 determines the moving distance of the sparrow; r2 controls the influence of the optimal individual on the next position of the sparrow; α is the adjustment factor; R2 represents the warning value; ST represents the safety value. At the same time, in order to avoid the algorithm falling into the local optimum, the Gauss-Cauchy mutation strategy is used to update the position of the follower, and the new follower position is updated as follows: In the formula, δ 2 is the standard deviation; Iter max is the maximum number of iterations; To ensure safety during foraging, some sparrows are used for alerting. The initial positions of these sparrows are randomly generated, and the position update is performed as follows: Where μ is the step size control parameter; K is a random number between -1 and 1; f i is the fitness value of the current sparrow; f g is the current global optimal fitness value; f ω is the current global worst fitness value; ε is a constant.

8. A non-wearable intelligent detection device for pilot fatigue status, characterized in that: include: A data acquisition device, a memory and a processor, wherein: The data acquisition device is used to collect facial images of pilots at different angles; The memory stores computer instructions; The processor is data-connected to the data acquisition device and the memory. The processor executes the non-wearable pilot fatigue state detection method according to any one of claims 1 to 7 by executing the computer instructions, thereby performing intelligent detection of the pilot fatigue state.

9. A storage medium having stored thereon one or more programs readable by a computing device, characterized in that: The one or more programs include instructions, which, when executed by a computing device, enable the computing device to execute a non-wearable intelligent detection method for a pilot's fatigue state as described in any one of claims 1 to 7.

Images