A method, device and system for monitoring submarine crew alertness

Abstract

The present invention discloses a method, device and system for monitoring the alertness of submarine crew members. The method comprises the following steps: step 1, collecting eye movement behavior, facial expression, and heart rate signal feature data from multiple perspectives during the submarine crew members' mission execution; step 2, pre-processing the feature data collected in step 1, and extracting feature indicators sensitive to alertness; step 3, respectively calculating eye movement behavior alertness fatigue scores, facial expression alertness fatigue scores, and heart rate signal alertness fatigue scores according to the alertness sensitive feature data obtained in step 2; step 4, calculating the alertness fatigue comprehensive monitoring score according to the calculation result of step 3. The present invention uses eye movement behavior, facial expression, and heart rate signals to perform non-interference monitoring of the crew members and the tasks they are performing, thereby realizing long-term non-interference monitoring, and improving the accuracy of alertness monitoring through the fusion and complementation of multi-source indicators of multi-source eye movement, multi-source facial expression, and heart rate signals.

Description

Technical Field

[0001] The invention belongs to the field of human-computer interaction equipment, and in particular relates to a method, equipment and system for monitoring the alertness of submarine crew members. Background Art

[0002] Alertness refers to the ability of operators to maintain attention for a long time to monitor whether there are abnormal targets or abnormal situations while performing typical tasks, and to make timely judgments on the nature and urgency of abnormal targets or abnormal situations.

[0003] During the long voyage of submarines, submarine crews need to be on duty for a long time. The alertness of the crews tends to decline with the sailing time, and the alertness of the crews will also decrease with the duty time during a single duty, which will have an adverse effect on the crew's continuous work efficiency. During the mission, monitoring the alertness of the crew in a non-interfering way and giving prompts according to the alertness state of the crew are of great significance for guiding the crew's alertness intervention and regulation and maintaining the submarine's human-machine ergonomics. The existing alertness intention measurement paradigms mainly include cognitive testing, subjective evaluation, behavioral monitoring (eye behavior, facial expression), and neurobiological signal monitoring (electrooculography, electroencephalography, electrocardiography, skin conduction, etc.). Among them, cognitive testing and subjective evaluation require interrupting the current tasks of the subjects, which are not suitable for monitoring the alertness of personnel during the actual mission execution; the measurement of electroencephalography, electrocardiography, and skin conduction signals requires wearing special data acquisition equipment for a long time, which will cause physiological discomfort to the crew. Behavior monitoring signals based on eye behavior, facial expression, and operation behavior have been used to monitor the alertness of drivers and pilots, but there are certain differences in the mission environment and mission characteristics between submarine crew members and drivers and pilots. However, the layout of the upper and lower screens on the submarine display console makes the head movement and eye behavior characteristics of the crew members different from the eye behavior and head movement noise of the drivers and pilots. In addition, the existing commercial and scientific eye trackers are mainly used for horizontal viewing angles and cannot be applied to monitoring scenarios with large vertical wide angles such as vertical screens or upper and lower screens, resulting in the inability to use existing eye movement and facial expression monitoring models and tools to monitor the alertness of the crew members.

[0004] The alertness monitoring method based on eye movement characteristics usually uses eye-type, desktop-type, and strip-type eye trackers to collect eye movement characteristic data of operators. However, eye-type eye trackers have certain interference to operators, and the vertical wide angle that can be monitored by desktop and strip-type eye trackers cannot meet the vertical wide angle monitoring requirements of the upper and lower screens. The alertness monitoring method based on EEG monitors the alertness and fatigue level of operators by monitoring the EEG signals of operators, but there are problems such as strong interference to operators and tasks, the signal is easily interfered by the outside world, high environmental requirements, and unsuitable for long-term continuous monitoring.

[0005] In addition, the above methods all belong to single type of indicator monitoring and their accuracy is low. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method, device and system for monitoring the alertness of submarine crew members. Based on multi-source physiological characteristics monitoring, it can collect the physiological characteristics data of the crew members from multiple screens and multiple wide angles from top to bottom, and realize effective monitoring of the crew members' alertness by fusing sensitivity indicators.

[0007] The object of the present invention is achieved through the following technical scheme: a method for monitoring the alertness of submarine crew members, comprising the following steps:

[0008] Step 1: Collect eye movement behavior, facial expression, and heart rate signal feature data from multiple perspectives during the submarine crew's mission;

[0009] Step 2, preprocessing the feature data collected in step 1 to extract feature indicators that are sensitive to alertness;

[0010] Step 3, according to the alertness sensitive feature data obtained in step 2, respectively calculating the eye movement behavior alertness fatigue score, the facial expression alertness fatigue score and the heart rate signal alertness fatigue score;

[0011] Step 4: Calculate the alertness fatigue comprehensive monitoring score based on the calculation results of step 3.

[0012] Preferably, in step 2, extracting alertness sensitive feature indicators corresponding to eye movement behavior and facial expression includes at least the following steps:

[0013] Step 2.1: Through 3D simulation modeling, integrate the collected original images of eye movement behavior and facial expressions to obtain an integrated image:

[0014] Step 2.1.1, pre-processing the collected eye movement behavior data and facial expression data;

[0015] Step 2.1.2, time synchronization, through timestamp calibration or synchronization signal application, make all eye movement behavior collection data consistent in time, and make all facial expression collection data consistent in time;

[0016] Step 2.1.3: spatial calibration, aligning the eye movement behavior and facial expression image data collected from different perspectives into a unified coordinate system;

[0017] Step 2.1.4, based on the machine learning algorithm, perform data fusion on the eye movement behavior and facial expression image data, build a three-dimensional simulation model, and obtain the integrated eye behavior integrated image and facial expression integrated image;

[0018] Step 2.2: Extract eye movement behavior alertness sensitivity features and facial expression alertness sensitivity features from the eye movement behavior integrated image and facial expression integrated image acquired in real time.

[0019] Preferably, in step 2, the extraction of the alertness sensitive feature of the heart rate signal comprises at least the following steps:

[0020] Heart rate data collection is achieved by collecting PPG signals in real time. The maximum non-overlapping wavelet decomposition method is used to decompose the PPG signals collected in real time to remove body motion artifacts. Bandpass filtering and median filtering are used to complete signal preprocessing, and the periodic characteristics of the heart rate signals are extracted. The heart rate alertness sensitivity features are extracted based on the periodic characteristics of the heart rate signals.

[0021] Preferably, in step 3, the eye movement behavior alertness fatigue score AS i Calculated by the following formula:

[0022] AS i =A-Aw 1i AV i +Aw 2i AD i

[0023] In the formula, A is a constant, Aw 1i , Aw 2i is the weight of eye movement speed and average pupil diameter; AV i is the eye movement speed, AD i is the average pupil diameter.

[0024] Preferably, in step 3, the facial expression alertness fatigue score BS i The calculation is done using the following formula:

[0025] BS i =B+Bw 1i B1 i +Bw 2i B2 i ...+Bw Ni BN i

[0026] In the formula, B is a constant, Bw 1i ...Bw Ni is the weight of the facial expression sensitivity feature, each weight value can be positive or negative, and N represents the number of facial expression sensitivity feature values.

[0027] Preferably, in step 3, the heart rate signal alertness fatigue score CS i Calculated by the following formula:

[0028] CS i =C+Cw1i C1 i +Cw 2i C2 i …+Cw ni Cn i

[0029] In the formula, C is a constant, Cw 1i ……Cw ni is the weight of the heart rate signal sensitivity feature, each weight value can be positive or negative, and n represents the number of heart rate signal sensitivity feature values.

[0030] Preferably, in step 4, the alertness fatigue comprehensive monitoring score SS i Calculated by the following formula:

[0031]

[0032] In the formula, WA i is the eye movement behavior alertness fatigue score weight, WB i is the facial expression behavior alertness fatigue score weight, WC i The weight of the heart rate signal alertness fatigue score.

[0033] In addition to providing a method for monitoring the alertness of submarine crew members, the present invention also provides a device for implementing the above-mentioned alertness monitoring method, the device comprising:

[0034] Multiple eye movement behavior acquisition devices are set at different positions in the eye acquisition area.

[0035] A plurality of facial expression acquisition devices are arranged at different positions in the facial expression acquisition area.

[0036] Heart rate signal collection terminal, collects crew heart rate data in real time.

[0037] The data processing terminal is connected to the eye movement behavior acquisition device, the facial expression acquisition device and the heart rate signal acquisition terminal by wire or wirelessly, receives the raw data collected by the above devices, extracts and fuses the features of the raw data according to a prefabricated program, and calculates the comprehensive monitoring score of alertness fatigue.

[0038] The display terminal is arranged inside the cabin and displays in real time the comprehensive monitoring score of alertness fatigue obtained by the data processing terminal.

[0039] Preferably, the positions at which the eye movement behavior acquisition device and the facial expression acquisition device are set in the corresponding acquisition area include at least: upper left, upper right, middle left, middle right, lower left and lower right.

[0040] The present invention also provides a submarine crew alertness monitoring system for implementing the above-mentioned submarine crew alertness monitoring method, the system comprising:

[0041] The alertness physiological behavior feature acquisition module obtains the eye movement behavior, facial expression, and heart rate signal feature data collected by the alertness physiological behavior acquisition device, and outputs it to the alertness sensitivity feature extraction and fusion module.

[0042] The alertness sensitivity feature extraction and fusion module receives input eye movement behavior, facial expression, and heart rate signal feature data, performs feature extraction and fusion, extracts features that are sensitive to alertness, and outputs them to the alertness status prompt module.

[0043] The alertness status prompt module calculates the alertness fatigue comprehensive monitoring score based on the input alertness sensitivity characteristics, outputs it to the display terminal, and gives corresponding prompts based on the monitoring score.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] The present invention provides a method, device and system for monitoring the alertness of submarine crew members. The eye movement behavior data of submarine crew members using vertical screens and upper and lower screens are collected during the execution of tasks through a multi-source eye movement behavior terminal and a facial expression behavior terminal, and integrated eye movement behavior and facial expression images are obtained through three-dimensional square modeling. Alertness sensitivity features are extracted based on the integrated eye movement behavior and facial expression images, and the collection of crew physiological characteristic data from multiple screens and multiple wide angles is realized, thereby improving the monitoring effectiveness of crew alertness. The present invention uses eye movement behavior, facial expression and heart rate signal lights to perform non-interference monitoring of crew members and the tasks they are performing, realizing long-term non-interference monitoring, and improving the accuracy of alertness monitoring through the fusion and complementation of multi-source indicators of multi-source eye movement, multi-source facial expression and heart rate signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the method for monitoring the alertness of submarine crew members in the present invention;

[0047] Figure 2 Schematic diagram of three-dimensional simulation modeling of a six-source facial expression acquisition terminal in an embodiment of the present invention;

[0048] Figure 3 4 is a diagram showing the architecture of a submarine crew alertness monitoring system in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0050] like Figure 1As shown, the technical solution of the present invention provides a method for monitoring the alertness of submarine crew members, comprising the following steps:

[0051] Step 1: Collect eye movement behavior, facial expression, and heart rate signal feature data from multiple perspectives during the submarine crew's mission;

[0052] Step 2, preprocessing the feature data collected in step 1 to extract feature indicators that are sensitive to alertness;

[0053] Step 3, according to the alertness sensitive feature data obtained in step 2, respectively calculating the eye movement behavior alertness fatigue score, the facial expression alertness fatigue score and the heart rate signal alertness fatigue score;

[0054] Step 4: Calculate the alertness fatigue comprehensive monitoring score based on the calculation results of step 3.

[0055] In one embodiment of the present invention, a method for extracting eye movement behavior alertness sensitive feature index and calculating alertness fatigue score for step 2 and step 3 is provided, which extracts sensitive features from the original eye movement behavior data and calculates the eye movement behavior alertness fatigue score AS. i ; At least the following steps:

[0056] Step 2.1, through three-dimensional simulation modeling, integrate the original images collected by six sources of eye movement behavior to obtain an integrated image of eye movement behavior;

[0057] Step 2.1.1. Preprocess the eye images captured by each eye tracker in detail. This includes key steps such as image correction, denoising, and enhancement. Image correction ensures that the eye features in the image will not be distorted due to the physical position or angle deviation of the camera. Denoising removes irrelevant information in the image, and enhancement further highlights the details of the eye features, providing clearer visual data for eye behavior feature recognition.

[0058] Step 2.1.2: Time synchronization: Ensure that the video streams captured by all eye trackers are temporally consistent through precise timestamp calibration or application of synchronization signals.

[0059] Step 2.1.3, spatial calibration. Accurately calibrate the position of each eye tracker's viewpoint, determine the relative position and direction of the eye tracker in space, align the image data from different viewpoints into a unified coordinate system, and provide a unified reference framework for three-view simulation modeling and feature recognition of eye behavior.

[0060] Step 2.1.4: Data fusion based on machine learning algorithm: Input the image features from different eye trackers into the classifier for preliminary eye feature recognition, and then fuse them based on the eye behavior feature recognition results to build a three-dimensional simulation model and obtain the integrated eye behavior integrated image.

[0061] Step 2.2: Extract eye movement behavior alertness sensitivity features from the real-time integrated eye movement behavior images

[0062] Step 2.2.1, extract eight sensitive features including the X-axis coordinate of the left eye fixation point, the Y-axis coordinate of the left eye fixation point, the duration of each fixation point of the left eye, the pupil diameter of the left eye, the X-axis coordinate of the right eye fixation point, the Y-axis coordinate of the right eye fixation point, the duration of each fixation point of the right eye, and the pupil diameter of the right eye;

[0063] Step 2.2.2, take the average X-axis coordinate, Y-axis coordinate, and gaze time of the i-th left and right eye gaze points as the i-th average gaze point X-axis coordinate, Y-axis coordinate, and gaze time, and define them as AX i AY i , AT i ;

[0064] Step 2.2.3, using the X-axis coordinate, Y-axis coordinate and fixation time of the ith average fixation point as input, calculate the eye movement speed AV according to the following formula: i :

[0065]

[0066] Step 2.2.4: Take the average of the i-th left eye pupil diameter and the right eye pupil diameter as the average AD of the i-th pupil diameter. i .

[0067] Step 3: Eye movement speed AV i and the average pupil diameter AD i As input, the eye movement behavior alertness fatigue score AS is calculated according to the following formula i :

[0068] AS i =A-Aw 1i AV i +Aw 2i AD i

[0069] In the formula, A is a constant, Aw 1i , Aw 2i are the weights for eye movement speed and average pupil diameter.

[0070] In one embodiment of the present invention, a method for extracting facial expression alertness sensitivity feature index and calculating alertness fatigue score for step 2 and step 3 is provided, which extracts alertness sensitivity feature from the original facial expression image data acquired in real time, and calculates the facial expression alertness fatigue score BS. i , including at least the following steps:

[0071] Step 2.1: Figure 2 As shown, through three-dimensional simulation modeling, the original images collected by six sources of facial expressions are integrated to obtain the facial expression integrated image;

[0072] Step 2.1.1: Perform detailed preprocessing on the images captured by each camera, including key links such as image correction, denoising, and enhancement. Image correction ensures that the facial features in the image will not be distorted due to the physical position or angle deviation of the camera, denoising removes irrelevant information in the image, and enhancement further highlights the details of facial expressions, providing clearer visual data for facial expression recognition.

[0073] Step 2.1.2: Time synchronization: Ensure that the video streams captured by all cameras are consistent in time through precise timestamp calibration or application of synchronization signals.

[0074] Step 2.1.3, spatial calibration. Accurately calibrate the position of each camera perspective, determine the relative position and direction of the camera in space, align the image data from different perspectives into a unified coordinate system, and provide a unified reference framework for facial expression three-dimensional simulation modeling and feature recognition.

[0075] Step 2.1.4: Data fusion based on machine learning algorithm: Input the image features from different cameras into the classifier for preliminary facial expression recognition, and then fuse them based on the facial expression feature recognition results to build a three-dimensional simulation model and obtain the integrated facial expression integrated image.

[0076] Step 2.2, extract facial expression alertness sensitivity features from the facial expression integrated image acquired in real time. In this embodiment, facial expression feature indices of facial pitch angle, right inner eyebrow lifting amplitude, left inner eyebrow lifting amplitude, right eyelid tightening amplitude, left eyelid tightening amplitude, left outer eyebrow lifting amplitude, right outer eyebrow lifting amplitude, lip tightening amplitude, lip stretching amplitude, lip pressing amplitude, and lip opening amplitude are extracted and recorded as BA respectively. i BB i , B.C. i , BD i , BE i , BF i , BG i , BH i , BI i , BJ i , BK i ;

[0077] Step 3: Using the above eleven sensitive facial expression feature indicators as input, calculate the facial expression alertness fatigue score BS according to the following formula: i

[0078] BS i =B+Bw 1i BA i +Bw 2i BB i +Bw 3i BC i +Bw 4i BD i +Bw 5i BE i -Bw 6i BF i +Bw 7i BG i +Bw 8i BH i -Bw 9i BI i -Bw 10i BJ i -Bw 11 BK i

[0079] In the formula, B is a constant, Bw 1i ...Bw 11i are the weights of the eleven sensitivity features.

[0080] In one embodiment of the present invention, a method for extracting the alertness sensitivity characteristic index of the heart rate signal and calculating the alertness fatigue score for step 2 and step 3 is provided, wherein the heart rate alertness sensitivity characteristic is extracted from the original characteristic data of the heart rate signal acquired in real time, and the heart rate alertness fatigue score CS is calculated. i , the specific steps are as follows:

[0081] Step 2.1, decompose the PPG signal collected during real-time monitoring using the maximum non-overlapping wavelet decomposition method to remove body motion artifacts, use bandpass filtering and median filtering to complete signal preprocessing, and extract the periodic characteristics of the heart rate signal;

[0082] Step 2.2, extract the heart rate alertness sensitivity feature of the heart rate cycle characteristic data, and extract eleven sensitive heart rate characteristic indicators including breathingrate, p_total, sdsd, ibi, pnn50, sdl / sd2, sd2, lf_perc, s, sdl, and sdnn;

[0083] Step 3: Using the above eleven sensitive facial expression feature indicators as input, calculate the facial expression alertness fatigue score CS according to the following formula: i

[0084] CS i =C-Cw 1i CA i -Cw 2i CB i -Cw 3i CC i -Cw 4i CD i -Cw 5i CE i -Cw 6i CF i -Cw 7i CG i +Cw 8i CH i -Cw 9i CI i +Cw 10i CJ i +Cw 11i CK i

[0085] In the formula, C is a constant, Cw 1i ……Cw 11i are the weights of eleven facial expression features.

[0086] In one embodiment of the present invention, a comprehensive monitoring score SS of alertness fatigue is provided for step 4. i The calculation method uses the eye movement, face, and ECG alertness fatigue prediction scores as input, and calculates the alertness fatigue comprehensive monitoring score according to the following formula:

[0087]

[0088] In the formula, WA i is the eye movement behavior alertness fatigue score weight, WS i is the facial expression behavior alertness fatigue score weight, WC i The weight of the heart rate signal alertness fatigue score.

[0089] It should be noted that in the above embodiments, the original data of eye movement behavior, facial expression and heart rate signal are obtained through the six-source acquisition terminal of eye movement behavior (taking six sources as an example, but not limited to six sources), the six-source acquisition terminal of facial expression (taking six sources as an example, but not limited to six sources) and the heart rate signal acquisition terminal, wherein the six-source devices of eye movement behavior and facial expression are the six-source acquisition terminals of upper left, upper right, middle left, middle right, lower left and lower right respectively.

[0090] In addition to providing a method for monitoring the alertness of submarine crew members, the technical solution of the present invention also provides a device for implementing the above-mentioned alertness monitoring method, the device comprising:

[0091] Multiple eye movement behavior acquisition devices are set at different positions in the eye acquisition area.

[0092] A plurality of facial expression acquisition devices are arranged at different positions in the facial expression acquisition area.

[0093] Heart rate signal collection terminal, collects crew heart rate data in real time.

[0094] The data processing terminal is connected to the eye movement behavior acquisition device, the facial expression acquisition device and the heart rate signal acquisition terminal by wire or wirelessly, receives the raw data collected by the above devices, extracts and fuses the features of the raw data according to a prefabricated program, and calculates the comprehensive monitoring score of alertness fatigue.

[0095] The display terminal is arranged inside the cabin and displays in real time the comprehensive monitoring score of alertness fatigue obtained by the data processing terminal.

[0096] In one embodiment of the present invention, the positions of the eye movement behavior acquisition device and the facial expression acquisition device in the corresponding acquisition area include at least: upper left, upper right, middle left, middle right, lower left and lower right.

[0097] like Figure 3 As shown, the technical solution of the present invention also provides a submarine crew alertness monitoring system for implementing the above-mentioned submarine crew alertness monitoring method, the system comprising:

[0098] The alertness physiological behavior feature acquisition module obtains the eye movement behavior, facial expression, and heart rate signal feature data collected by the alertness physiological behavior acquisition device, and outputs it to the alertness sensitivity feature extraction and fusion module.

[0099] The alertness sensitivity feature extraction and fusion module receives input eye movement behavior, facial expression, and heart rate signal feature data, performs feature extraction and fusion, extracts features of sensitivity to alertness-fatigue-distraction states, and outputs them to the alertness state prompt module.

[0100] The alertness status prompt module calculates the alertness-fatigue-distraction score (i.e., the comprehensive monitoring score) based on the input alertness sensitive features, outputs it to the display terminal, and gives corresponding prompts based on the monitoring score.

[0101] In this embodiment, the calculated alertness fatigue score is displayed on the system interface, ranging from 0 to 100. The higher the score, the higher the alertness and the lower the fatigue level. The lower the score, the worse the alertness and the higher the fatigue level. When the alertness is lower than 60 points, it indicates mild fatigue, lower than 70 points, it indicates moderate fatigue, lower than 80 points, it indicates severe fatigue, and lower than 90 points, it indicates extreme fatigue.

[0102] In the above-mentioned embodiments of the present invention, a method, device and system for monitoring the alertness of submarine crew members oriented to the needs of ergonomics are provided, which collects eye movement behavior, facial expression and heart rate signal feature data of submarine crew members using vertical screens and upper and lower screens during the execution of tasks through six-source eye movement, six-source facial expression and heart rate signal acquisition terminals. The eye movement behavior images acquired in real time are subjected to three-dimensional simulation modeling and image data integration, characteristic indicators sensitive to alertness are extracted, and eye movement behavior and facial expression alertness fatigue scores are calculated; the facial expression images acquired in real time are subjected to three-dimensional simulation modeling and image data integration, characteristic indicators sensitive to alertness are extracted, and eye movement behavior and facial expression alertness fatigue scores are calculated; the PPG signals collected during the real-time monitoring process are decomposed using the maximum non-overlapping wavelet decomposition method to remove body motion artifacts, and signal preprocessing is completed using bandpass filtering and median filtering to extract the periodic characteristics of the heart rate signal, extract characteristic indicators sensitive to alertness, and calculate the alertness fatigue score of the heart rate signal; the submarine crew monitoring model formula of the face ergonomics requirements constructed by the present invention is input to obtain a comprehensive alertness score for evaluating the alertness state of the crew, and prompts are given according to the decrease in alertness. By using eye movement behavior, facial expressions and heart rate signal lights to perform non-interference monitoring of the crew and the tasks they are performing, long-term non-interference monitoring is achieved, and the accuracy of alertness monitoring is improved by the fusion and complementation of multi-source indicators of multi-source eye movement, multi-source facial expression fusion and heart rate signals.

[0103] The above are preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for monitoring submarine crew alertness, characterized in that: The monitoring method comprises the following steps: Step 1: Collect eye movement behavior, facial expression, and heart rate signal feature data from multiple perspectives during the submarine crew's mission; Step 2, preprocessing the characteristic data collected in step 1 to extract characteristic indicators sensitive to alertness; Step 3, according to the alertness sensitive feature data obtained in step 2, respectively calculating the eye movement behavior alertness fatigue score, the facial expression alertness fatigue score and the heart rate signal alertness fatigue score; Step 4: Calculate the alertness fatigue comprehensive monitoring score based on the calculation result of step 3.

2. A method for monitoring submarine crew alertness as claimed in claim 1, characterized in that: In step 2, the extraction of alertness sensitive feature indicators corresponding to eye movement behavior and facial expression includes at least the following steps: Step 2.1: Through 3D simulation modeling, integrate the collected original images of eye movement behavior and facial expressions to obtain an integrated image: Step 2.1.1, pre-processing the collected eye movement behavior data and facial expression data; Step 2.1.2, time synchronization, through timestamp calibration or synchronization signal application, make all eye movement behavior collection data consistent in time, and make all facial expression collection data consistent in time; Step 2.1.3: spatial calibration, aligning the eye movement behavior and facial expression image data collected from different perspectives into a unified coordinate system; Step 2.1.4, based on the machine learning algorithm, perform data fusion on the eye movement behavior and facial expression image data, build a three-dimensional simulation model, and obtain the integrated eye behavior integrated image and facial expression integrated image; Step 2.2: Extract eye movement behavior alertness sensitivity features and facial expression alertness sensitivity features from the eye movement behavior integrated image and facial expression integrated image acquired in real time.

3. A method for monitoring submarine crew alertness as claimed in claim 1, characterized in that: In step 2, the extraction of the alertness sensitive feature of the heart rate signal includes at least the following steps: Heart rate data collection is achieved by collecting PPG signals in real time. The maximum non-overlapping wavelet decomposition method is used to decompose the PPG signals collected in real time to remove body motion artifacts. Bandpass filtering and median filtering are used to complete signal preprocessing, and the periodic characteristics of the heart rate signals are extracted. The heart rate alertness sensitivity features are extracted based on the periodic characteristics of the heart rate signals.

4. A method for monitoring submarine crew alertness as claimed in claim 1, characterized in that: In step 3, the eye movement behavior alertness fatigue score AS i Calculated by the following formula: <h2 style=";text-align:left;direction:ltr">AS<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (A-Aw)<h2 style=";text-align:left;direction:ltr"> 1i <h2 style=";text-align:left;direction:ltr"> AV<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +Aw<h2 style=";text-align:left;direction:ltr"> 2i <h2 style=";text-align:left;direction:ltr"> AD<h2 style=";text-align:left;direction:ltr"> i In the formula, A is a constant, Aw 1i , Aw 2i is the weight of eye movement speed and average pupil diameter; AV i is the eye movement speed, AD i is the average pupil diameter.

5. A method for monitoring submarine crew alertness as claimed in claim 4, characterized in that: In step 3, the facial expression alertness fatigue score BS i The calculation is done using the following formula: <h2 style=";text-align:left;direction:ltr">BS<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (B+Bw)<h2 style=";text-align:left;direction:ltr"> 1i <h2 style=";text-align:left;direction:ltr"> B1<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +Bw<h2 style=";text-align:left;direction:ltr"> 2i <h2 style=";text-align:left;direction:ltr"> B2<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> …+Bw<h2 style=";text-align:left;direction:ltr"> Ni <h2 style=";text-align:left;direction:ltr"> BN<h2 style=";text-align:left;direction:ltr"> i In the formula, B is a constant, Bw 1i ...Bw Ni is the weight of the facial expression sensitivity feature, each weight value can be positive or negative, and N represents the number of facial expression sensitivity feature values.

6. A method for monitoring submarine crew alertness as claimed in claim 5, characterized in that: In step 3, the heart rate signal alertness fatigue score CS i Calculated by the following formula: CS i =C+Cw 1i C1 i +Cw 2i C2 i …+Cw ni Cn i In the formula, C is a constant, Cw 1i ……Cw ni is the weight of the heart rate signal sensitivity feature, each weight value can be positive or negative, and n represents the number of heart rate signal sensitivity feature values.

7. A method for monitoring submarine crew alertness as claimed in claim 6, characterized in that: In step 4, the alertness fatigue comprehensive monitoring score SS i Calculated by the following formula: In the formula, WA i is the eye movement behavior alertness fatigue score weight, WB i is the facial expression behavior alertness fatigue score weight, WC i The weight of the heart rate signal alertness fatigue score.

8. A submarine crew alertness monitoring device, characterized in that: The device is used to implement the alertness monitoring method according to any one of claims 1 to 7, and the device comprises: Multiple eye movement behavior acquisition devices are set at different positions in the eye acquisition area; A plurality of facial expression acquisition devices are arranged at different positions in the facial expression acquisition area; Heart rate signal collection terminal, collects crew heart rate data in real time; The data processing terminal is connected to the eye movement behavior acquisition device, the facial expression acquisition device and the heart rate signal acquisition terminal by wire or wirelessly, receives the raw data collected by the above devices, extracts and fuses the features of the raw data according to a pre-made program, and calculates the comprehensive monitoring score of alertness fatigue; The display terminal is arranged inside the cabin and displays in real time the comprehensive monitoring score of alertness fatigue obtained by the data processing terminal.

9. A submarine crew alertness monitoring device as claimed in claim 8, characterized in that: The positions where the eye movement behavior acquisition device and the facial expression acquisition device are set in the corresponding acquisition area include at least: upper left, upper right, middle left, middle right, lower left and lower right.

10. A submarine crew alertness monitoring system, characterized in that: The system is used to implement the method for monitoring submarine crew alertness according to any one of claims 1 to 7, and the system comprises: The alertness physiological behavior feature acquisition module obtains the eye movement behavior, facial expression, and heart rate signal feature data collected by the alertness physiological behavior acquisition device, and outputs it to the alertness sensitivity feature extraction and fusion module; The alertness sensitivity feature extraction and fusion module receives input eye movement behavior, facial expression, and heart rate signal feature data, performs feature extraction and fusion, extracts features that are sensitive to alertness, and outputs them to the alertness status prompt module; The alertness status prompt module calculates the alertness fatigue comprehensive monitoring score based on the input alertness sensitivity characteristics, outputs it to the display terminal, and gives corresponding prompts based on the monitoring score.

Images