Fatigue detection method and device based on multi-modal characteristics, equipment and medium

Abstract

The invention provides a fatigue detection method and device based on multi-modal features, equipment and a medium, is used for the technical field of fatigue detection of control tower personnel, and can solve the problem of low efficiency and accuracy of fatigue detection of the control tower personnel in a remote control tower scene. Comprising the following steps: identifying initial facial features to obtain facial coordinates and depth coordinates corresponding to preset identification points, and calculating facial fatigue feature vectors; determining a signal sequence according to a signal peak value of the pulse signal, and determining a physiological fatigue feature vector according to a difference value of adjacent sequence points in the signal sequence and signal power; according to an operation corresponding to the remote control tower, calculating an execution fatigue feature vector, and fusing the facial fatigue feature vector, the physiological fatigue feature vector and the execution fatigue feature vector to obtain a target fatigue feature vector; according to a fatigue feature value corresponding to the target fatigue feature vector, determining a fatigue detection result of the control tower personnel; therefore, the fatigue detection efficiency and precision of the control tower personnel are improved.

Description

Technical Field

[0001] This application relates to the field of fatigue detection technology for tower personnel, and in particular to a fatigue detection method, device, equipment and medium based on multimodal characteristics. Background Technology

[0002] Remote control towers refer to the use of high-definition cameras, radar, and sensor arrays installed at airports to transmit real-time images and data from airports to a centralized control center hundreds or even thousands of kilometers away. This allows tower personnel to monitor the real-time dynamics of one or more airports and issue commands via electronic screens. However, because tower personnel need to stare at multiple high-resolution two-dimensional screens for extended periods, this planar visual information processing requires a high degree of concentration, which can easily lead to screen-induced visual fatigue, resulting in command errors and affecting normal dispatching.

[0003] Current methods for detecting fatigue among control tower personnel generally involve recording feelings of fatigue through timed questionnaires or fatigue self-assessment scales during shifts, thereby determining fatigue levels; or monitoring brain electrical activity by wearing an electrode cap, where alpha and theta wave activities in the electroencephalogram (EEG) will significantly increase when fatigued.

[0004] However, existing methods for detecting fatigue among control tower personnel suffer from low efficiency and accuracy. Summary of the Invention

[0005] This application provides a fatigue detection method, apparatus, equipment, and medium based on multimodal features to solve the problems of low detection efficiency and accuracy in existing fatigue detection methods for tower personnel.

[0006] Firstly, this application provides a fatigue detection method based on multimodal features, the method comprising: Based on the preset facial recognition model, the initial facial features of the tower personnel are identified to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point. Based on the facial coordinates and depth coordinates, multiple fatigue index values ​​are calculated to obtain the facial fatigue feature vector. Based on multiple signal peaks of the pulse signal, the corresponding signal sequence is determined, and the physiological fatigue feature vector is determined based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence. Based on the operation corresponding to the remote control tower, the abnormal operation entropy value and operation delay value are calculated to obtain the execution fatigue feature vector. The facial fatigue feature vector, physiological fatigue feature vector and execution fatigue feature vector are then fused to obtain the target fatigue feature vector. Based on the fatigue feature value corresponding to the target fatigue feature vector and the preset fatigue threshold, the fatigue detection result for the tower personnel is determined.

[0007] In some embodiments of this application, based on a preset facial recognition model, feature recognition is performed on the initial facial features of tower personnel to obtain facial coordinates and depth coordinates corresponding to each preset recognition point, including: Based on the preset image acquisition equipment, the visible light continuous image and the infrared continuous image of the tower personnel's face within the target time are determined, and the visible light continuous image and the infrared continuous image of the face are input into the dual-channel CNN model to obtain the initial facial features; Based on a preset facial recognition model, feature recognition is performed on the initial facial features to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point.

[0008] In some embodiments of this application, based on a preset facial recognition model, feature recognition is performed on initial facial features to obtain facial coordinates and depth coordinates corresponding to each preset recognition point, including: Based on the preset facial recognition model, the feature heatmap corresponding to the initial facial features is determined, as well as the initial coordinates and coordinate offsets corresponding to each preset recognition point in the feature heatmap. Based on the sum of the initial coordinates and coordinate offsets, the facial coordinates corresponding to the preset recognition points are determined. For multiple preset image acquisition devices, determine the facial coordinates corresponding to the preset recognition points, as well as the coordinate differences between different facial coordinates. Based on the preset device focal length, baseline distance, and coordinate differences, calculate the depth coordinates corresponding to the preset recognition points.

[0009] In some embodiments of this application, multiple fatigue index values ​​are calculated based on facial coordinates and depth coordinates to obtain a facial fatigue feature vector, including: Based on multiple preset recognition points, the corresponding recognition points for each fatigue index are determined, as well as the facial coordinates and depth coordinates corresponding to the recognition points. The fatigue indices include eye opening and closing, average blinking frequency and duration, yawn detection, head posture angle, and gaze deviation. For all facial coordinates of the same indicator recognition point within the target time, calculate the coordinate difference between adjacent facial coordinates, and compare the coordinate difference with the position coordinate threshold, as well as the depth coordinate with the depth coordinate threshold, to obtain the corresponding comparison results; If the comparison result shows that the coordinate difference is greater than the position coordinate threshold and the depth coordinate is less than the depth coordinate threshold, then it is determined that the tower personnel's facial state has changed, and the target facial state of the tower personnel after the change is determined based on multiple preset facial states. Based on the duration of the target facial state, calculate the fatigue index value corresponding to the index recognition point, and encapsulate multiple fatigue index values ​​to obtain a facial fatigue feature vector. If the comparison result shows that the coordinate difference is not greater than the position coordinate threshold or the depth coordinate is not less than the depth coordinate threshold, then it is determined that the facial condition of the tower personnel has not changed.

[0010] In some embodiments of this application, a physiological fatigue feature vector is determined based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence, including: The pulse of the tower personnel was collected to obtain the pulse signal and its corresponding multiple signal peaks. The difference between adjacent signal peaks was calculated to obtain multiple sequence points in the signal sequence. The difference between adjacent sequence points is calculated to obtain multiple sequence differences. The root mean square of the sequence differences within the target time is then calculated to obtain the time-domain physiological fatigue characteristics. Perform a Fourier transform on the signal sequence to obtain the corresponding frequency domain signal and the signal power corresponding to the frequency domain signal, and determine the frequency domain signals corresponding to the preset high-frequency signal band and low-frequency signal band respectively. Power integration is performed on the frequency domain signals corresponding to the high-frequency signal band and the low-frequency signal band respectively to obtain the corresponding high-frequency signal power and low-frequency signal power. The ratio of high-frequency signal power to low-frequency signal power is then calculated to obtain the frequency domain physiological fatigue characteristics. The physiological fatigue features in the time domain and frequency domain are encapsulated to determine the physiological fatigue feature vector.

[0011] In some embodiments of this application, based on the operation corresponding to the remote control tower, the abnormal operation entropy value and operation delay value are calculated to obtain an execution fatigue feature vector, including: Determine the effective operation time and tower prompt time for each operation, as well as any abnormal operations during the operation. Based on the time difference between the effective operation time and the remote tower prompt time, calculate the average of all time differences to obtain the operation delay value. The number of abnormal operations within the target time period, as well as the corresponding operation movement curvature and tower scene calibration coefficient, are determined. The abnormal operation entropy value is calculated, and the operation delay value and abnormal operation entropy value are encapsulated to obtain the execution fatigue feature vector.

[0012] In some embodiments of this application, the fatigue detection result for tower personnel is determined based on the fatigue feature value corresponding to the target fatigue feature vector and a preset fatigue threshold, including: The target fatigue feature vector is mapped to a preset feature value range to obtain fatigue feature values. The fatigue feature values ​​are then compared with the preset fatigue threshold values ​​to obtain the corresponding comparison results. If the comparison result shows that the fatigue characteristic value is greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are currently fatigued, and the corresponding fatigue prompt information is output. If the comparison result shows that the fatigue characteristic value is not greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are not currently fatigued.

[0013] Secondly, this application provides a fatigue detection device based on multimodal features, the device comprising: The recognition module is used to perform feature recognition on the initial facial features of tower personnel based on a preset facial recognition model, obtain the facial coordinates and depth coordinates corresponding to each preset recognition point, and calculate multiple fatigue index values ​​based on the facial coordinates and depth coordinates to obtain a facial fatigue feature vector. The vector determination module is used to determine the corresponding signal sequence based on multiple signal peaks of the pulse signal, and to determine the physiological fatigue feature vector based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence. The calculation module is used to calculate the abnormal operation entropy value and operation delay value according to the operation corresponding to the remote tower, obtain the execution fatigue feature vector, and fuse the facial fatigue feature vector, physiological fatigue feature vector and execution fatigue feature vector to obtain the target fatigue feature vector. The result determination module is used to determine the fatigue detection result of the tower personnel based on the fatigue feature value corresponding to the target fatigue feature vector and the preset fatigue threshold.

[0014] Thirdly, this application provides a computer device, including: a processor, and a memory communicatively connected to the processor; The memory stores the instructions that the computer executes; The processor executes computer execution instructions stored in memory to implement the method of this application.

[0015] Fourthly, this application provides a computer-readable storage medium storing program code, which, when executed by a processor, is used to implement the method of this application.

[0016] This application provides a fatigue detection method, device, equipment, and medium based on multimodal features. It identifies the initial facial features of control tower personnel using a preset facial recognition model, obtaining facial coordinates and depth coordinates corresponding to each preset recognition point. Based on these coordinates, multiple fatigue index values ​​are calculated to obtain a facial fatigue feature vector. A corresponding signal sequence is determined based on multiple peak values ​​of the pulse signal, and a physiological fatigue feature vector is determined based on the difference between adjacent sequence points and the corresponding signal power. An abnormal operation entropy value and operation delay value are calculated based on the operation corresponding to the remote control tower to obtain an execution fatigue feature vector. The facial fatigue feature vector, physiological fatigue feature vector, and execution fatigue feature vector are then fused to obtain a target fatigue feature vector. Finally, the fatigue detection result for the control tower personnel is determined based on the fatigue feature value corresponding to the target fatigue feature vector and a preset fatigue threshold.

[0017] Thus, by recognizing initial facial features using a pre-set facial recognition model, the facial coordinates and depth coordinates of each pre-set recognition point are determined. This allows for the calculation of multiple fatigue indicators for tower personnel, such as eye opening / closing, yawning frequency, head posture angle, and gaze deviation, thereby obtaining a facial fatigue feature vector. This approach is adaptable to the complex environment of remote towers, including low light and screen glare, enabling accurate capture of visual fatigue signals from controllers and solving the problem of high false alarm rates in traditional single-visual detection. Furthermore, by determining the signal sequence through pulse signal peaks and calculating temporal features such as RMSSD using the inter-interval point difference (IBI), and combining this with statistical features derived from signal power, such as power mean and standard deviation, a physiological fatigue feature vector is determined. This accurately reflects changes in the controller's autonomic nervous system function, allowing for fatigue assessment from a physiological perspective and avoiding the pitfalls of traditional physiological detection methods. The limitations of invasive equipment are addressed by calculating abnormal operation entropy and delay values ​​based on remote tower operations, forming an execution fatigue feature vector. This allows for timely detection of issues such as decreased operational accuracy and sluggish response caused by tower personnel fatigue, based on the interaction between controllers and the remote tower system. By fusing facial, physiological, and execution fatigue feature vectors to obtain a target fatigue feature vector, multimodal data complementarity is achieved, overcoming the incomplete characterization of fatigue states by single-modal features and improving detection accuracy and precision. Furthermore, based on the fatigue feature values ​​corresponding to the target fatigue feature vector, such as the RDTC-CFI index, and a preset fatigue threshold, fatigue detection results are determined, enabling graded assessment and precise early warning of fatigue states. This allows for timely output of alerts, significantly improving the safety and sustainability of remote air traffic control operations. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 A flowchart illustrating a fatigue detection method based on multimodal features provided in this application embodiment; Figure 2 A flowchart illustrating another fatigue detection method based on multimodal features provided in this application embodiment; Figure 3 A schematic diagram of a fatigue detection device based on multimodal features provided in an embodiment of this application; Figure 4 This is a structural block diagram of an apparatus for performing a fatigue detection method based on multimodal features according to an embodiment of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0021] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a fatigue detection method based on multimodal features provided in an embodiment of this application. Figure 1 As shown, this fatigue detection method based on multimodal features may include the following steps: S110. Based on the preset facial recognition model, the initial facial features of the tower personnel are identified to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point. Based on the facial coordinates and depth coordinates, multiple fatigue index values ​​are calculated to obtain the facial fatigue feature vector.

[0023] The pre-set facial recognition model refers to a deep learning network that has been pre-trained using a large amount of labeled data, so as to receive images and output the positions of key facial points; for example, it can be a pre-trained 68-point facial model. The 68-point facial model is a pre-trained deep learning model used to standardize the localization of key structures of the human face, so as to divide the face into 68 fixed feature points through algorithms. Each feature point corresponds to a specific physiological structure of the face, such as the eyelid edge, corner of the mouth, and jaw angle, forming a unified facial feature localization template.

[0024] Air traffic controllers are professionals responsible for directing aircraft takeoffs, landings, taxiing, and parking space allocation within the airport control tower. Remote control towers, on the other hand, transmit real-time images and data from airports to centralized control centers hundreds or even thousands of kilometers away via high-definition cameras, radar, and sensor arrays. This allows tower controllers to monitor the real-time dynamics of one or more airports and issue commands through electronic screens. However, because tower controllers need to stare at multiple high-resolution two-dimensional screens for extended periods, this planar visual information processing requires a high degree of concentration, which can easily lead to screen-induced visual fatigue, resulting in command errors and disrupting normal operations.

[0025] Initial facial features refer to the original image feature map obtained after the image acquisition equipment has acquired and undergone preliminary preprocessing. This map is used for subsequent facial feature recognition, thereby enabling the identification of features in the acquired tower personnel's facial images, determining the corresponding facial features and recognition points, and judging whether the tower personnel are fatigued based on the facial features.

[0026] The preset recognition points refer to 68 standardized feature points that are predefined and cover key fatigue-related areas of the face. These can include 12 points in the eye area, such as the upper and lower eyelid edges and pupil periphery; 20 points in the mouth area, such as the upper and lower lip apex and corner of the mouth; and 36 points in the head contour area, such as the mandibular angle and forehead apex. By locating these key points, the dynamic changes of key facial areas can be accurately quantified, such as eyelid opening and closing, mouth opening and closing, and head posture deviation.

[0027] Facial coordinates refer to the position coordinates of preset recognition points on the two-dimensional plane of the image, in pixels, and represented by a two-dimensional vector (x, y), where the x-axis is the horizontal direction of the image and the y-axis is the vertical direction of the image. The coordinate values ​​are generated by the coordinate regression network of the preset facial recognition model, and thus serve as the core parameters for calculating facial dynamic fatigue indicators. For example, the eyelid opening and closing degree is calculated by the change of the y-coordinate of the eye recognition point, and the y-coordinate span of the mouth recognition point is used to determine whether a yawning event has occurred, thereby determining whether the tower personnel are fatigued.

[0028] Depth coordinates refer to the vertical distance coordinates of a preset recognition point relative to the camera in three-dimensional space, represented by the z-value: z = f is the focal length, and b is the binocular baseline distance. The depth coordinate represents the pixel difference between feature points in the left and right views. In practical applications, depth coordinates can achieve three-dimensional quantization of facial features, thereby further determining changes in facial state. For example, by changing the z-coordinate of the head recognition point, slight head nodding can be accurately detected, which is a positional fluctuation in the depth direction. This avoids the two-dimensional facial coordinates from mistakenly moving the head back and forth, and instead determines it as a stable posture.

[0029] Fatigue index values ​​refer to numerical indicators calculated based on the facial and depth coordinates of preset recognition points, used to quantify the visual fatigue state of tower controllers. Fatigue indicators may include eye opening and closing, average blinking frequency and duration, yawn detection, head posture angle, and gaze deviation. The fatigue index value is the specific quantitative value corresponding to each fatigue indicator for tower personnel, thereby determining the specific degree of fatigue of tower personnel in that indicator.

[0030] Facial fatigue feature vector refers to a high-dimensional numerical vector formed by standardizing fatigue index values ​​and splicing them in a fixed-dimensional order. This integrates scattered fatigue index values ​​into a unified vector, facilitating subsequent judgment based on the fatigue detection results of tower personnel.

[0031] Based on this, the processed initial facial features are identified using a pre-set facial recognition model to determine multiple pre-set recognition points on the tower personnel's face, as well as the two-dimensional facial coordinates and three-dimensional depth coordinates corresponding to each pre-set recognition point. The facial coordinates and depth coordinates can reflect the changes in the front and back poses of key facial points, which can be used to determine changes in the tower personnel's facial state. The quantitative index values ​​corresponding to various fatigue indicators are calculated to obtain the corresponding facial fatigue feature vector, so that the tower personnel can be further determined to see if they are currently fatigued based on the facial fatigue feature vector.

[0032] S120. Based on the multiple signal peaks of the pulse signal, determine the corresponding signal sequence, and based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence, determine the physiological fatigue feature vector.

[0033] Among them, the pulse signal refers to the physiological signal collected that reflects the heartbeat and blood circulation status of the remote tower controller. It can be collected by a non-contact wristband photoplethysmography (PPG) sensor, and the sensor sampling rate is higher than the Nyquist frequency required for heart rate variability (HRV) analysis. In practical applications, the non-contact wristband acquisition design can achieve 24-hour continuous monitoring without interfering with the controller's operation, solving the problems of high invasiveness and poor applicability of traditional physiological detection equipment. The heart rate variability (HRV) information contained in the signal is a key physiological basis for judging fatigue status. Fatigue will lead to autonomic nervous system imbalance, which will cause changes in the periodicity and amplitude stability of the pulse signal. Therefore, the pulse signal can be used to determine whether the tower personnel are fatigued.

[0034] The peak value of a pulse signal refers to the highest amplitude point of the pulse signal within each cardiac cycle, identified by a peak detection algorithm after bandpass filtering and denoising of the original pulse signal. It corresponds to the signal peak when blood is rapidly perfused into the blood vessels during the systolic phase of the heart, and the occurrence of each peak value is synchronized with a heartbeat.

[0035] A signal sequence refers to a sequence of peak timestamps formed by arranging the acquisition timestamps corresponding to the signal peaks in chronological order; while adjacent sequence points are two consecutive timestamp points, and their time difference (IBI) can be used to calculate the time-domain index of heart rate variability (HRV), and IBI is the interval between heartbeats.

[0036] Signal power is a quantitative indicator that reflects the energy intensity or fluctuation stability of a pulse signal, calculated based on the peak amplitude of the pulse signal or the corresponding IBI sequence.

[0037] Physiological fatigue feature vector refers to a high-dimensional numerical vector formed by standardizing the difference between adjacent sequence points and the HRV frequency domain feature based on signal power, and then splicing them together in a fixed-dimensional order; thereby, based on physiological data, it can be determined whether tower personnel are experiencing fatigue.

[0038] Based on this, since fatigue can lead to autonomic nervous system imbalance, which in turn causes changes in the periodicity and amplitude stability of pulse signals, by collecting pulse signals from tower personnel, the peak value and corresponding signal power in the pulse signals can be further analyzed and calculated. Based on the changes in the pulse before and after, the corresponding physiological fatigue feature vector can be obtained, so as to realize the fatigue detection of tower personnel from a physiological level.

[0039] S130. Based on the operation corresponding to the remote control tower, calculate the abnormal operation entropy value and operation delay value to obtain the execution fatigue feature vector. Then, fuse the facial fatigue feature vector, physiological fatigue feature vector, and execution fatigue feature vector to obtain the target fatigue feature vector.

[0040] Among them, the abnormal operation entropy value is a statistical indicator that quantifies the complexity, randomness, and standardization of remote tower controllers' operational behavior. In practical applications, fatigue can lead to controllers' distraction and memory bias, which in turn can cause operational logic confusion, such as repeatedly executing invalid operations or forgetting operational steps. Furthermore, fatigue can lead to a decline in muscle control, causing tower personnel to become stiff when using the mouse. The abnormal operation entropy value is a specific quantitative value obtained by quantifying such abnormal operations through probability entropy terms, thus avoiding misjudgments caused by relying solely on the frequency of abnormal operations.

[0041] Operation delay value refers to the average time difference between the remote tower system issuing an operation prompt and the controller completing the first effective operation within the target time, thus reflecting the actual reaction time of tower personnel to the operation to be performed.

[0042] The execution fatigue feature vector is a high-dimensional numerical vector formed by concatenating abnormal operation entropy values ​​and average response delays in a fixed-dimensional order after standardization. Different components in the vector correspond to different operational fatigue behaviors, such as reaction speed and operational standardization, thus characterizing the degree of fatigue of tower personnel in specific operations.

[0043] The target fatigue feature vector refers to a unified high-dimensional vector containing three-dimensional fatigue information of vision, physiology and behavior, generated by dynamically weighting and fusing three types of heterogeneous features: facial fatigue feature vector, physiological fatigue feature vector and executive fatigue feature vector.

[0044] Based on this, by determining the degree of execution delay and the quantitative index value corresponding to abnormal operations of tower personnel in specific operations, a specific feature vector that can characterize the user's execution behavior can be obtained, namely the execution fatigue feature vector. Then, by integrating the three heterogeneous features of facial fatigue feature vector, physiological fatigue feature vector and execution fatigue feature vector, multimodal feature fusion is achieved to obtain the target fatigue feature vector.

[0045] S140. Based on the fatigue feature value corresponding to the target fatigue feature vector and the preset fatigue threshold, determine the fatigue detection result corresponding to the tower personnel.

[0046] Among them, fatigue feature value refers to the standardized and quantifiable cognitive fatigue quantification index generated after nonlinear mapping of the target fatigue feature vector. It transforms the high-dimensional and heterogeneous fatigue information contained in the target fatigue feature vector into a single and comparable scalar value, that is, converting the abstract and high-dimensional feature vector into a specific and subsequently comparable actual quantification value.

[0047] The preset fatigue threshold refers to a pre-determined threshold used to determine whether tower personnel are currently experiencing fatigue, thereby achieving fatigue determination through a clearly defined numerical critical point.

[0048] Fatigue detection results refer to the comprehensive detection results generated by comparing real-time calculated fatigue characteristic values ​​with preset fatigue thresholds, which can reflect whether tower personnel are fatigued and output corresponding fatigue warning information.

[0049] Based on this, by quantifying the target fatigue feature vector into specific fatigue feature values, and then using the fatigue feature values ​​and preset fatigue thresholds, it is determined whether the current tower personnel are experiencing fatigue, thereby generating corresponding fatigue detection results. This allows for timely alerts to fatigued tower personnel, preventing execution errors.

[0050] Based on the feasible implementation of S110 described above, this application further provides a method for feature recognition of the initial facial features of tower personnel based on a preset facial recognition model, obtaining facial coordinates and depth coordinates corresponding to each preset recognition point, including: Based on the preset image acquisition equipment, the visible light continuous image and the infrared continuous image of the tower personnel's face within the target time are determined, and the visible light continuous image and the infrared continuous image of the face are input into the dual-channel CNN model to obtain the initial facial features; Based on a preset facial recognition model, feature recognition is performed on the initial facial features to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point.

[0051] The preset image acquisition equipment refers to a dedicated imaging device with dual-mode infrared and visible light acquisition capabilities that is pre-deployed at the remote tower controller's workstation. In practical applications, it can be a multi-modal camera array embedded in the center of the top of the remote tower's panoramic screen, thereby enabling real-time facial video acquisition of tower personnel. Dual-mode acquisition solves the shortcomings of single-mode acquisition, thus outputting high-quality facial images.

[0052] The target time refers to a pre-set time window for acquiring continuous facial images and extracting corresponding features, so as to detect and determine fatigue based on the various characteristic indicators exhibited by tower personnel within the target time.

[0053] Facial visible light continuous images are image sequences continuously captured by a visible light lens, containing the facial area of ​​a remote tower controller; facial infrared continuous images are thermal imaging image sequences continuously captured by an infrared lens, containing the facial area of ​​a remote tower controller.

[0054] A dual-channel CNN model is a deep learning model architecture that includes two parallel convolutional neural network (CNN) branches and a feature-level fusion layer. It can process two heterogeneous inputs: continuous visible light images and continuous infrared images of the face. By integrating the detail advantages of visible light images with the anti-interference advantages of infrared images, it outputs clear facial features.

[0055] Based on this, a dual-channel CNN model is used to integrate the visible light and infrared images acquired within the target time period to determine the initial facial features. This allows for subsequent facial detection and fatigue index calculation based on the initial facial features, thereby obtaining the corresponding facial fatigue feature vector.

[0056] Based on the feasible implementation of S110 described above, this application further provides a method for feature recognition of initial facial features based on a preset facial recognition model, obtaining facial coordinates and depth coordinates corresponding to each preset recognition point, including: Based on the preset facial recognition model, the feature heatmap corresponding to the initial facial features is determined, as well as the initial coordinates and coordinate offsets corresponding to each preset recognition point in the feature heatmap. Based on the sum of the initial coordinates and coordinate offsets, the facial coordinates corresponding to the preset recognition points are determined. For multiple preset image acquisition devices, determine the facial coordinates corresponding to the preset recognition points, as well as the coordinate differences between different facial coordinates. Based on the preset device focal length, baseline distance, and coordinate differences, calculate the depth coordinates corresponding to the preset recognition points.

[0057] Among them, the feature heatmap refers to the visual feature map output by the preset facial recognition model after processing the initial facial features. It represents the position of the preset facial recognition point in the form of pixel probability distribution, thus representing the probability heat distribution of each preset recognition point on the image plane. That is, the value of each pixel in the heatmap represents the confidence that the pixel position is the target preset recognition point. The higher the value (the brighter the color), the greater the probability that the position is the corresponding recognition point, thereby determining multiple preset recognition points.

[0058] Initial coordinates refer to the reference position coordinates of each preset recognition point on the two-dimensional plane of the image, output by the preset facial recognition model based on the feature heatmap. It can be understood as the center coordinates of the high confidence area in the feature heatmap.

[0059] Coordinate offset refers to the position compensation value of each preset recognition point on the two-dimensional plane output by the preset facial recognition model to correct the initial coordinate deviation. In practical applications, remote tower controllers have scenario characteristics such as slight head tilt and wearing headphones that cover part of their face. The initial coordinates deviate from the actual position. The coordinate offset can compensate for these deviations in a targeted manner, so that the final facial coordinates can accurately match the actual facial shape of the controller.

[0060] Coordinate difference refers to the difference in pixel values ​​on the horizontal x-axis among the facial coordinates output by multiple preset image acquisition devices for the same preset recognition point. In practical applications, coordinate difference can be caused by the difference in the viewing angle of the binocular cameras, thus reflecting the left and right position of the recognition point relative to the camera baseline. The coordinate difference of the same recognition point should be within a reasonable range. For example, if the distance between the remote tower controller and the camera is fixed, the difference will fluctuate little. If the difference is too large or abrupt, it indicates that there may be a recognition point matching error in the binocular image.

[0061] The preset device focal length refers to the pre-calibrated focal length of the optical system of each preset image acquisition device.

[0062] Baseline distance refers to the straight-line distance between the optical centers of the two cameras in a binocular camera system, as pre-calibrated.

[0063] Based on this, in practical applications, the preset facial recognition model can output a corresponding heatmap based on the input initial facial features. The heatmap can represent the position of the preset facial recognition points in the form of pixel probability distribution, thereby determining multiple preset recognition points, as well as the initial coordinates and coordinate offsets of each preset recognition point. In order to obtain the facial coordinates corresponding to each preset recognition point based on the sum of the initial coordinates and coordinate offsets, the corresponding facial coordinates can be calculated by determining the coordinate difference between the two cameras for the same facial coordinates, as well as the focal length and baseline distance of the cameras.

[0064] Based on the feasible implementation of S110 described above, this application further provides a method for calculating multiple fatigue index values ​​based on facial coordinates and depth coordinates to obtain a facial fatigue feature vector, including: Based on multiple preset recognition points, the corresponding recognition points for each fatigue index are determined, as well as the facial coordinates and depth coordinates corresponding to the recognition points. The fatigue indices include eye opening and closing, average blinking frequency and duration, yawn detection, head posture angle, and gaze deviation. For all facial coordinates of the same indicator recognition point within the target time, calculate the coordinate difference between adjacent facial coordinates, and compare the coordinate difference with the position coordinate threshold, as well as the depth coordinate with the depth coordinate threshold, to obtain the corresponding comparison results; If the comparison result shows that the coordinate difference is greater than the position coordinate threshold and the depth coordinate is less than the depth coordinate threshold, then it is determined that the tower personnel's facial state has changed, and the target facial state of the tower personnel after the change is determined based on multiple preset facial states. Based on the duration of the target facial state, calculate the fatigue index value corresponding to the index recognition point, and encapsulate multiple fatigue index values ​​to obtain a facial fatigue feature vector. If the comparison result shows that the coordinate difference is not greater than the position coordinate threshold or the depth coordinate is not less than the depth coordinate threshold, then it is determined that the facial condition of the tower personnel has not changed.

[0065] Among them, fatigue indexes refer to pre-determined specific measurement indicators used to quantify specific fatigue signs. These indicators can include multiple dimensions such as eye opening and closing, average blinking frequency and duration, yawn detection, head posture angle, and gaze deviation. The five types of indicators comprehensively depict different manifestations of visual fatigue of controllers from multiple levels such as eye state, mouth behavior, head posture, and gaze focus. After standardization, the calculation results of each type of indicator can be spliced ​​into a facial fatigue feature vector, which becomes the visual dimension input for multimodal fusion, avoiding the one-sidedness of a single indicator in depicting the fatigue state.

[0066] The indicator recognition point refers to a specific facial key point selected from 68 preset facial recognition points, which is specifically used to calculate a certain type of fatigue indicator. Each fatigue indicator corresponds to a set of exclusive indicator recognition points, and there is a fixed mapping relationship between the two. For example, the eye opening and closing degree or the average blinking frequency and duration correspond to the eye area recognition points, such as the upper and lower eyelid edges and the pupil periphery points, totaling 12. By focusing on the recognition points that are strongly correlated with the indicator, the interference of irrelevant recognition point data is avoided, thereby improving the efficiency and accuracy of indicator calculation.

[0067] The target time is a pre-determined time window used to calculate the corresponding fatigue index.

[0068] Coordinate difference refers to the difference between the facial coordinates of the same indicator recognition point in two consecutive frames within the target time. It reflects the magnitude of the position change of the recognition point in adjacent frames. By calculating the coordinate difference, the position fluctuation of the indicator recognition point can be quantified. For example, a large y-coordinate difference of the eyelid recognition point indicates a change in the opening and closing state of the eyelid, thereby determining whether the controller has facial movements related to fatigue.

[0069] The position coordinate threshold is a predetermined pixel difference threshold used to determine whether changes in the position of the indicator identification point have fatigue significance. When the coordinate difference is greater than the position coordinate threshold, it is determined to be a facial movement actively performed by the controller that is related to fatigue; otherwise, it is determined to be noise or slight shaking, so as to avoid invalid data interfering with the calculation of fatigue indicators.

[0070] The depth coordinate threshold is a pre-defined critical depth value used to determine whether the facial area corresponding to the indicator recognition point is within the effective monitoring distance. This threshold is set based on the fixed distance between the remote tower controller's workstation and the image acquisition equipment, and is usually the maximum value of the effective monitoring distance.

[0071] Preset facial states refer to predefined standard facial states corresponding to various fatigue indicators, used to match specific state types after facial state changes. For example, for the eyes, it includes eyelid closure and blinking states, and for the mouth, it includes yawning and normal mouth states.

[0072] The target facial state refers to the latest facial state of the tower personnel after a change in facial condition; if the y-coordinate difference of the eye recognition point suddenly increases while the depth coordinate is within the valid range, and after comparing with the preset fatigue state, the target facial state is determined to be the blinking state. State duration refers to the total duration of the target facial state. In practical applications, most fatigue indicators need to be derived from state duration. The longer the state duration, the more severe the fatigue level. For example, the longer the eyelid closure state lasts, the more obvious the tendency for microsleep. The longer the gaze deviates from the state, the more severe the cognitive drift. This provides a quantitative basis for judging the fatigue level. For example, the average blink frequency and duration can be the number of blinks per unit time and the average duration of each blink, reflecting the compensatory blink changes caused by dryness of the ocular surface in a low-light environment. Yawn detection can be determined by analyzing the opening and closing degree and duration of key points of the mouth to judge the frequency of yawning events. A yawn is counted once if the vertical span of the mouth continuously exceeds the empirical threshold and conforms to the respiratory cycle.

[0073] Based on this, by determining the indicator identification points corresponding to each fatigue indicator, the coordinate transformation of the indicator identification points within the target time, i.e., the coordinate difference, is determined. By comparing the coordinate difference with the position coordinate threshold and the depth coordinate with the depth coordinate threshold, the comparison results determine whether the state of the tower personnel's face changes after the coordinates of the indicator identification points change. If so, the current target facial state is determined, and the duration of the state is calculated. Based on the duration of the state, the corresponding indicator fatigue value is calculated and encapsulated to obtain a facial fatigue feature vector.

[0074] Based on the feasible implementation of S120 described above, this application further provides a method for determining a physiological fatigue feature vector based on the difference between adjacent sequence points in a signal sequence and the signal power corresponding to the signal sequence, including: The pulse of the tower personnel was collected to obtain the pulse signal and its corresponding multiple signal peaks. The difference between adjacent signal peaks was calculated to obtain multiple sequence points in the signal sequence. The difference between adjacent sequence points is calculated to obtain multiple sequence differences. The root mean square of the sequence differences within the target time is then calculated to obtain the time-domain physiological fatigue characteristics. Perform a Fourier transform on the signal sequence to obtain the corresponding frequency domain signal and the signal power corresponding to the frequency domain signal, and determine the frequency domain signals corresponding to the preset high-frequency signal band and low-frequency signal band respectively. Power integration is performed on the frequency domain signals corresponding to the high-frequency signal band and the low-frequency signal band respectively to obtain the corresponding high-frequency signal power and low-frequency signal power. The ratio of high-frequency signal power to low-frequency signal power is then calculated to obtain the frequency domain physiological fatigue characteristics. The physiological fatigue features in the time domain and frequency domain are encapsulated to determine the physiological fatigue feature vector.

[0075] The root mean square (RMS) calculation refers to the mathematical operation of first squaring the sequence differences within a target time period, then taking the average, and finally taking the square root. This can be done using... RMSSD refers to the calculated time-domain physiological fatigue characteristics, where IBIi is the i-th heartbeat interval and N is the total number of heartbeat intervals within the window.

[0076] Temporal physiological fatigue characteristics refer to physiological indicators that quantify the short-term regulatory function of the cardiac autonomic nervous system, obtained by calculating the temporal dimension characteristics of pulse signal sequences; they can reflect the short-term physiological fluctuations corresponding to pulse signals.

[0077] Fourier transform refers to the mathematical transformation that converts a signal sequence from the time domain to the frequency domain; frequency domain signal refers to the signal form obtained after the signal sequence is transformed by Fourier transform (FFT), characterized by frequency and power spectral density.

[0078] The preset high-frequency and low-frequency signal bands refer to predefined physiologically effective frequency ranges related to cardiac autonomic nerve function. The low-frequency signal band (LF band) has a frequency range of 0.04-0.15Hz. The signal in this band mainly reflects the combined effect of the sympathetic and parasympathetic nervous systems and is affected by factors such as blood pressure regulation and emotional fluctuations. The high-frequency signal band (HF band) has a frequency range of 0.15Hz-0.4Hz. The signal in this band mainly reflects parasympathetic nerve activity and is highly correlated with respiratory rate.

[0079] Power integration refers to the definite integration of the power spectral density curve of a frequency domain signal within a preset high-frequency or low-frequency signal band.

[0080] High-frequency signal power refers to the signal energy value obtained by integrating the power of a frequency domain signal within a preset high-frequency signal band (0.15Hz-0.4Hz); low-frequency signal power refers to the signal energy value obtained by integrating the power of a frequency domain signal within a preset low-frequency signal band (0.04Hz-0.15Hz).

[0081] Frequency domain physiological fatigue characteristics refer to physiological indicators that quantify the balance of the cardiac autonomic nervous system, obtained by calculating the frequency dimension characteristics of pulse signal sequences.

[0082] Based on this, after acquiring pulse signals and determining the corresponding signal sequence according to the signal peak, multiple sequence differences are obtained by calculating the differences between adjacent sequence points. These sequence differences are the IBI differences. Further root mean square calculation of the differences yields the time-domain physiological fatigue characteristics of the pulse signal, reflecting the short-term regulatory capacity of the parasympathetic nervous system in the autonomic nervous system of the heart of the controller. By performing frequency domain transformation on the signal sequence, power integration is performed on the physiologically effective frequency range related to the function of the autonomic nervous system of the heart to obtain the high-frequency signal power corresponding to the high-frequency band and the low-frequency signal power corresponding to the low-frequency band. This allows for the calculation of the ratio of high-frequency signal power to low-frequency signal power, thus obtaining the frequency-domain physiological fatigue characteristics. By encapsulating the time-domain and frequency-domain physiological fatigue characteristics, a physiological fatigue feature vector is obtained, which can comprehensively characterize the physiological state of the controller and avoid the one-sidedness of a single dimension.

[0083] Based on the feasible implementation of S130 described above, this application further provides a method for calculating abnormal operation entropy values ​​and operation delay values ​​according to the operation corresponding to the remote control tower, to obtain an execution fatigue feature vector, including: Determine the effective operation time and tower prompt time for each operation, as well as any abnormal operations during the operation. Based on the time difference between the effective operation time and the remote tower prompt time, calculate the average of all time differences to obtain the operation delay value. The number of abnormal operations within the target time period, as well as the corresponding operation movement curvature and tower scene calibration coefficient, are determined. The abnormal operation entropy value is calculated, and the operation delay value and abnormal operation entropy value are encapsulated to obtain the execution fatigue feature vector.

[0084] Among them, the effective operation time refers to the operation execution timestamp recorded by the system when the remote tower controller performs an operation that conforms to the standard operating procedures and can trigger an effective system response in response to the system prompts.

[0085] Tower alert time refers to the timestamp recorded by the system when a remote tower system sends an alert message to the controller that requires an interactive response.

[0086] Abnormal operation refers to the behavior of a remote tower controller that deviates from the standard operating template of the remote tower during interaction with the system.

[0087] Operational movement curvature refers to the geometric parameters that quantify the smoothness and regularity of an operator's movement on equipment such as a mouse. Specifically, it is the second derivative of the curvature of the mouse movement trajectory. This can be achieved by collecting the continuous position coordinates of the mouse within a target time, generating a movement trajectory curve, and calculating the first and second derivatives of the trajectory curve. By taking the absolute value of the second derivative, the operational movement curvature is obtained. The larger the value, the more rigid the mouse trajectory and the more frequent the directional changes, such as irregular movements when fatigued; the smaller the value, the smoother the trajectory and the more fluid the operation.

[0088] The tower scenario calibration coefficient refers to a dimensionless coefficient that is pre-calibrated based on the operation data of different task scenarios of a remote tower, and is used to balance the weights of the abnormal operation probability entropy term and the operation movement curvature correction term. By dynamically adjusting the weights, the operation entropy can accurately reflect the normal operation fluctuations and anomalies caused by fatigue under different task scenarios, avoiding the problem of false alarms in high-complexity scenarios and false alarms in low-complexity scenarios caused by the general coefficient.

[0089] Therefore, the operation latency value can be represented by ARL. t_prompt is the system prompt time, t_action is the controller's first valid operation time, and M is the total number of valid events within the window; while the operation entropy can be used... express, , This is an abnormal operation. (e.g., the probability of erroneous instruction correction) occurring within a time window. Total number of abnormal operation types (such as erroneous instructions, duplicate corrections, etc.). It is the second derivative of the curvature of the mouse movement trajectory (quantifying the smoothness of operation). It is the calibration coefficient for remote tower scenarios.

[0090] Based on the feasible implementation of S140 described above, this application further provides a method for determining the fatigue detection result of tower personnel based on the fatigue feature value corresponding to the target fatigue feature vector and a preset fatigue threshold, including: The target fatigue feature vector is mapped to a preset feature value range to obtain fatigue feature values. The fatigue feature values ​​are then compared with the preset fatigue threshold values ​​to obtain the corresponding comparison results. If the comparison result shows that the fatigue characteristic value is greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are currently fatigued, and the corresponding fatigue prompt information is output. If the comparison result shows that the fatigue characteristic value is not greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are not currently fatigued.

[0091] The preset characteristic value range refers to the standardized interval set in advance to regulate the numerical distribution of fatigue characteristic values ​​(RDTC-CFI), with the core being the closed interval [0,1].

[0092] Fatigue alert information refers to various forms of notification content automatically generated and output by the system when the fatigue characteristic value exceeds the preset fatigue threshold, used to remind controllers or administrators of the current fatigue risk.

[0093] Based on this, in practical applications, RDTC-CFI(t) represents the fatigue characteristic value at time point t, where σ represents the Sigmoid function, and W and b are trainable parameters. C(t) is the remote tower scene correction factor, and C(t) is the context vector generated by the attention mechanism at time t. That is, the target fatigue feature vector is compressed by a fully connected layer and Sigmoid, and the output is a scalar between 0 and 1 to obtain the corresponding fatigue feature value.

[0094] Please refer to Figure 2 , Figure 2 A flowchart illustrating another fatigue detection method based on multimodal features provided in this application embodiment; as shown. Figure 2 As shown, by monitoring fatigue characteristic values ​​in real time, once the value exceeds one or more preset thresholds, a graded early warning and intervention sub-process is immediately triggered; specifically, three thresholds can be set: attention threshold T1=0.70, warning threshold T2=0.85, and intervention threshold T3=0.95. Level 1 Alarm (Attention Level): Triggered when 0.70 < Fatigue Characteristic Value ≤ 0.85; Performs mild, primarily alerting actions, including displaying a status indicator light that changes from green to yellow at the edge of the controller's screen interface; and can be linked to the environmental controller to fine-tune the screen color temperature to reduce visual stimulation.

[0095] Level 2 Alarm (Warning Level): Triggered when 0.85 < Fatigue Characteristic Value ≤ 0.95; Executes enhanced actions and notifies the manager, including turning the screen status indicator light orange and flashing at a low frequency; Linking the vibration motor in the controller's seat or wristband to provide a short tactile reminder; Simultaneously, Sending a silent attention request message to the remote tower operation supervisor's management terminal.

[0096] Level 3 Alarm (Intervention Level): Triggered when fatigue characteristic value > 0.95; executes the highest priority action requiring immediate manual intervention, including the screen status indicator light turning red and flashing at high frequency; playing a preset voice prompt through the controller's headset; simultaneously, pushing a high-priority pop-up alarm to the remote tower operation supervisor's management terminal and suggesting intervention measures such as shift change; the system automatically starts enhanced data recording for that position.

[0097] Furthermore, on the remote tower operations supervisor's management terminal, a visual dashboard is used to display the fatigue characteristic value change trend of each controller in the area of ​​responsibility in the form of a real-time curve graph; all collected and calculated data are anonymized and stored for long-term fatigue trend analysis, quantitative evaluation of scheduling strategies, and aviation safety audits, thus forming a complete data-driven safety management closed loop.

[0098] In some embodiments of this application, based on a preset facial recognition model, feature recognition is performed on initial facial features to obtain facial coordinates and depth coordinates corresponding to each preset recognition point. Multiple fatigue index values ​​are calculated based on the facial coordinates and depth coordinates to obtain a facial fatigue feature vector. Based on multiple signal peaks of the pulse signal, a corresponding signal sequence is determined. Based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence, a physiological fatigue feature vector is determined. Based on the operation corresponding to the remote control tower, abnormal operation entropy and operation delay values ​​are calculated to obtain an execution fatigue feature vector. The facial fatigue feature vector, physiological fatigue feature vector, and execution fatigue feature vector are fused to obtain a target fatigue feature vector. Based on the fatigue feature value corresponding to the target fatigue feature vector and a preset fatigue threshold, the fatigue detection result corresponding to the control tower personnel is determined.

[0099] Thus, by recognizing initial facial features using a pre-set facial recognition model, the facial coordinates and depth coordinates of each pre-set recognition point are determined. This allows for the calculation of multiple fatigue indicators for tower personnel, such as eye opening / closing, yawning frequency, head posture angle, and gaze deviation, thereby obtaining a facial fatigue feature vector. This approach is adaptable to the complex environment of remote towers, including low light and screen glare, enabling accurate capture of visual fatigue signals from controllers and solving the problem of high false alarm rates in traditional single-visual detection. Furthermore, by determining the signal sequence through pulse signal peaks and calculating temporal features such as RMSSD using the inter-interval point difference (IBI), and combining this with statistical features derived from signal power, such as power mean and standard deviation, a physiological fatigue feature vector is determined. This accurately reflects changes in the controller's autonomic nervous system function, allowing for fatigue assessment from a physiological perspective and avoiding the pitfalls of traditional physiological detection methods. The limitations of invasive equipment are addressed by calculating abnormal operation entropy and delay values ​​based on remote tower operations, forming an execution fatigue feature vector. This allows for timely detection of issues such as decreased operational accuracy and sluggish response caused by tower personnel fatigue, based on the interaction between controllers and the remote tower system. By fusing facial, physiological, and execution fatigue feature vectors to obtain a target fatigue feature vector, multimodal data complementarity is achieved, overcoming the incomplete characterization of fatigue states by single-modal features and improving detection accuracy and precision. Furthermore, based on the fatigue feature values ​​corresponding to the target fatigue feature vector, such as the RDTC-CFI index, and a preset fatigue threshold, fatigue detection results are determined, enabling graded assessment and precise early warning of fatigue states. This allows for timely output of alerts, significantly improving the safety and sustainability of remote air traffic control operations.

[0100] Figure 3 This is a schematic diagram of a fatigue detection device 300 based on multimodal features, provided as an embodiment of this application. Figure 3 As shown, the fatigue detection device 300 based on multimodal features includes: an identification module 310, a vector determination module 320, a calculation module 330, and a result determination module 340; wherein: The recognition module 310 is used to perform feature recognition on the initial facial features of tower personnel based on a preset facial recognition model, obtain the facial coordinates and depth coordinates corresponding to each preset recognition point, and calculate multiple fatigue index values ​​based on the facial coordinates and depth coordinates to obtain a facial fatigue feature vector. The vector determination module 320 is used to determine the corresponding signal sequence based on multiple signal peaks of the pulse signal, and to determine the physiological fatigue feature vector based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence. The calculation module 330 is used to calculate the abnormal operation entropy value and operation delay value according to the operation corresponding to the remote tower, obtain the execution fatigue feature vector, and fuse the facial fatigue feature vector, physiological fatigue feature vector and execution fatigue feature vector to obtain the target fatigue feature vector. The result determination module 340 is used to determine the fatigue detection result of the tower personnel based on the fatigue feature value corresponding to the target fatigue feature vector and the preset fatigue threshold.

[0101] In this embodiment of the application, the identification module 310 can also be specifically used for: Based on the preset image acquisition equipment, the visible light continuous image and the infrared continuous image of the tower personnel's face within the target time are determined, and the visible light continuous image and the infrared continuous image of the face are input into the dual-channel CNN model to obtain the initial facial features; Based on a preset facial recognition model, feature recognition is performed on the initial facial features to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point.

[0102] In this embodiment of the application, the identification module 310 can also be specifically used for: Based on the preset facial recognition model, the feature heatmap corresponding to the initial facial features is determined, as well as the initial coordinates and coordinate offsets corresponding to each preset recognition point in the feature heatmap. Based on the sum of the initial coordinates and coordinate offsets, the facial coordinates corresponding to the preset recognition points are determined. For multiple preset image acquisition devices, determine the facial coordinates corresponding to the preset recognition points, as well as the coordinate differences between different facial coordinates. Based on the preset device focal length, baseline distance, and coordinate differences, calculate the depth coordinates corresponding to the preset recognition points.

[0103] In this embodiment of the application, the identification module 310 can also be specifically used for: Based on multiple preset recognition points, the corresponding recognition points for each fatigue index are determined, as well as the facial coordinates and depth coordinates corresponding to the recognition points. The fatigue indices include eye opening and closing, average blinking frequency and duration, yawn detection, head posture angle, and gaze deviation. For all facial coordinates of the same indicator recognition point within the target time, calculate the coordinate difference between adjacent facial coordinates, and compare the coordinate difference with the position coordinate threshold, as well as the depth coordinate with the depth coordinate threshold, to obtain the corresponding comparison results; If the comparison result shows that the coordinate difference is greater than the position coordinate threshold and the depth coordinate is less than the depth coordinate threshold, then it is determined that the tower personnel's facial state has changed, and the target facial state of the tower personnel after the change is determined based on multiple preset facial states. Based on the duration of the target facial state, calculate the fatigue index value corresponding to the index recognition point, and encapsulate multiple fatigue index values ​​to obtain a facial fatigue feature vector. If the comparison result shows that the coordinate difference is not greater than the position coordinate threshold or the depth coordinate is not less than the depth coordinate threshold, then it is determined that the facial condition of the tower personnel has not changed.

[0104] In this embodiment of the application, the vector determination module 320 can also be specifically used for: The pulse of the tower personnel was collected to obtain the pulse signal and its corresponding multiple signal peaks. The difference between adjacent signal peaks was calculated to obtain multiple sequence points in the signal sequence. The difference between adjacent sequence points is calculated to obtain multiple sequence differences. The root mean square of the sequence differences within the target time is then calculated to obtain the time-domain physiological fatigue characteristics. Perform a Fourier transform on the signal sequence to obtain the corresponding frequency domain signal and the signal power corresponding to the frequency domain signal, and determine the frequency domain signals corresponding to the preset high-frequency signal band and low-frequency signal band respectively. Power integration is performed on the frequency domain signals corresponding to the high-frequency signal band and the low-frequency signal band respectively to obtain the corresponding high-frequency signal power and low-frequency signal power. The ratio of high-frequency signal power to low-frequency signal power is then calculated to obtain the frequency domain physiological fatigue characteristics. The physiological fatigue features in the time domain and frequency domain are encapsulated to determine the physiological fatigue feature vector.

[0105] In this embodiment of the application, the calculation module 330 can also be specifically used for: Determine the effective operation time and tower prompt time for each operation, as well as any abnormal operations during the operation. Based on the time difference between the effective operation time and the remote tower prompt time, calculate the average of all time differences to obtain the operation delay value. The number of abnormal operations within the target time period, as well as the corresponding operation movement curvature and tower scene calibration coefficient, are determined. The abnormal operation entropy value is calculated, and the operation delay value and abnormal operation entropy value are encapsulated to obtain the execution fatigue feature vector.

[0106] In this embodiment of the application, the result determination module 340 can also be specifically used for: The target fatigue feature vector is mapped to a preset feature value range to obtain fatigue feature values. The fatigue feature values ​​are then compared with the preset fatigue threshold values ​​to obtain the corresponding comparison results. If the comparison result shows that the fatigue characteristic value is greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are currently fatigued, and the corresponding fatigue prompt information is output. If the comparison result shows that the fatigue characteristic value is not greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are not currently fatigued.

[0107] Figure 4 This is a schematic diagram of the structure of an apparatus for performing a fatigue detection method based on multimodal features according to an embodiment of this application. Figure 4 As shown, the device 400 includes: The device 400 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0108] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the fatigue detection method based on multimodal features as described above.

[0109] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0110] Furthermore, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0111] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0112] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0113] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described fatigue detection methods based on multimodal features.

[0114] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0115] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware, and these instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0116] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of program codes, which can be loaded by a processor to execute the steps in any of the fatigue detection methods based on multimodal features provided in embodiments of this application.

[0117] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0118] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0119] Since the instructions stored in the storage medium can execute the steps of any of the fatigue detection methods based on multimodal features provided in the embodiments of this application, the beneficial effects that any of the fatigue detection methods based on multimodal features provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0120] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

[0121] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A fatigue detection method based on multimodal features, characterized in that, The method includes: Based on a preset facial recognition model, the initial facial features of the tower personnel are identified to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point. Based on the facial coordinates and depth coordinates, multiple fatigue index values ​​are calculated to obtain a facial fatigue feature vector. Based on multiple signal peaks of the pulse signal, the corresponding signal sequence is determined, and based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence, the physiological fatigue feature vector is determined. Based on the operation corresponding to the remote control tower, the abnormal operation entropy value and operation delay value are calculated to obtain the execution fatigue feature vector. The facial fatigue feature vector, the physiological fatigue feature vector, and the execution fatigue feature vector are then fused to obtain the target fatigue feature vector. The fatigue detection result of the tower personnel is determined based on the fatigue feature value corresponding to the target fatigue feature vector and the preset fatigue threshold.

2. The method according to claim 1, characterized in that, The preset facial recognition model performs feature recognition on the initial facial features of the tower personnel to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point, including: Based on a preset image acquisition device, a continuous visible light image and a continuous infrared image of the face of the tower personnel within a target time period are determined, and the continuous visible light image and the continuous infrared image are input into a dual-channel CNN model to obtain the initial facial features; Based on the preset facial recognition model, the initial facial features of the tower personnel are identified to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point.

3. The method according to claim 2, characterized in that, The method, based on the preset facial recognition model, performs feature recognition on the initial facial features of the tower personnel to obtain the facial coordinates and depth coordinates corresponding to each preset recognition point, including: Based on the preset facial recognition model, a feature heatmap corresponding to the initial facial features is determined, as well as the initial coordinates and coordinate offsets corresponding to each preset recognition point in the feature heatmap. The facial coordinates corresponding to the preset recognition point are determined according to the sum of the initial coordinates and the coordinate offsets. For multiple preset image acquisition devices, the facial coordinates corresponding to the preset recognition points and the coordinate differences between different facial coordinates are determined. Based on the preset device focal length, baseline distance, and coordinate differences, the depth coordinates corresponding to the preset recognition points are calculated.

4. The method according to claim 1, characterized in that, The step of calculating multiple fatigue index values ​​based on the facial coordinates and the depth coordinates to obtain a facial fatigue feature vector includes: Based on multiple preset recognition points, the corresponding indicator recognition points for each fatigue indicator, as well as the facial coordinates and depth coordinates corresponding to the indicator recognition points, are determined; the fatigue indicators include eye opening and closing, average blinking frequency and duration, yawn detection, head posture angle, and gaze deviation. For all facial coordinates of the same indicator recognition point within the target time, calculate the coordinate difference between adjacent facial coordinates, and compare the coordinate difference with the position coordinate threshold, and the depth coordinate with the depth coordinate threshold, to obtain the corresponding comparison result; If the comparison result is that the coordinate difference is greater than the position coordinate threshold and the depth coordinate is less than the depth coordinate threshold, then it is determined that the facial state of the tower personnel has changed, and the target facial state of the tower personnel after the change is determined according to multiple preset facial states. Based on the duration of the target facial state, the fatigue index value corresponding to the index identification point is calculated, and multiple fatigue index values ​​are encapsulated to obtain the facial fatigue feature vector. If the comparison result is that the coordinate difference is not greater than the position coordinate threshold, or the depth coordinate is not less than the depth coordinate threshold, then it is determined that the facial state of the tower personnel has not changed.

5. The method according to claim 1, characterized in that, The step of determining the physiological fatigue feature vector based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence includes: The pulse of the tower personnel is collected to obtain the pulse signal and its corresponding multiple signal peaks, and the difference between adjacent signal peaks is calculated to obtain multiple sequence points in the signal sequence; The difference between adjacent sequence points is calculated to obtain multiple sequence differences, and the root mean square of the sequence differences within the target time is calculated to obtain the time-domain physiological fatigue characteristics. Perform a Fourier transform on the signal sequence to obtain the corresponding frequency domain signal and the signal power corresponding to the frequency domain signal, and determine the frequency domain signals corresponding to the preset high-frequency signal band and low-frequency signal band respectively; Power integration is performed on the frequency domain signals corresponding to the high-frequency signal band and the low-frequency signal band respectively to obtain the corresponding high-frequency signal power and low-frequency signal power, and the ratio of the high-frequency signal power to the low-frequency signal power is calculated to obtain the frequency domain physiological fatigue characteristics. The time-domain physiological fatigue features and the frequency-domain physiological fatigue features are encapsulated to determine the physiological fatigue feature vector.

6. The method according to claim 1, characterized in that, The step of calculating the abnormal operation entropy value and operation delay value based on the operation corresponding to the remote control tower to obtain the execution fatigue feature vector includes: Determine the effective operation time and tower prompt time corresponding to each operation, as well as the abnormal operation in the operation, and calculate the average of all the time differences based on the time difference between the effective operation time and the remote tower prompt time to obtain the operation delay value; The number of abnormal operations within the target time period, as well as the corresponding operation movement curvature and tower scene calibration coefficient, are determined. The entropy value of the abnormal operation is calculated, and the operation delay value and the entropy value of the abnormal operation are encapsulated to obtain the execution fatigue feature vector.

7. The method according to claim 1, characterized in that, The step of determining the fatigue detection result of the tower personnel based on the fatigue feature value corresponding to the target fatigue feature vector and a preset fatigue threshold includes: The target fatigue feature vector is mapped to a preset feature value range to obtain the fatigue feature value, and the fatigue feature value is compared with the preset fatigue threshold to obtain the corresponding comparison result. If the comparison result shows that the fatigue feature value is greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are currently fatigued, and the corresponding fatigue prompt information is output. If the comparison result indicates that the fatigue characteristic value is not greater than the preset fatigue threshold, then the fatigue detection result is determined to be that the tower personnel are not currently fatigued.

8. A fatigue detection device based on multimodal features, characterized in that, The device includes: The recognition module is used to perform feature recognition on the initial facial features of tower personnel based on a preset facial recognition model, obtain the facial coordinates and depth coordinates corresponding to each preset recognition point, and calculate multiple fatigue index values ​​based on the facial coordinates and the depth coordinates to obtain a facial fatigue feature vector. The vector determination module is used to determine the corresponding signal sequence based on multiple signal peaks of the pulse signal, and to determine the physiological fatigue feature vector based on the difference between adjacent sequence points in the signal sequence and the signal power corresponding to the signal sequence. The calculation module is used to calculate the abnormal operation entropy value and operation delay value according to the operation corresponding to the remote tower, obtain the execution fatigue feature vector, and fuse the facial fatigue feature vector, the physiological fatigue feature vector and the execution fatigue feature vector to obtain the target fatigue feature vector. The result determination module is used to determine the fatigue detection result of the tower personnel based on the fatigue feature value corresponding to the target fatigue feature vector and the preset fatigue threshold.

9. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be called by a processor to perform the method as described in any one of claims 1 to 7.

Images