Method for detecting fatigue of reviewer in real time based on multi-feature fusion video analysis

Abstract

The invention relates to the technical field of specific post personnel state video analysis and early warning, and discloses a reviewer fatigue real-time detection method based on multi-feature fusion video analysis, and the method comprises the steps: obtaining a reviewer face detection data stream, marking as an interruption starting moment when a face is not detected in multiple continuous frames, and carrying out the early warning of the state of a specific post; when the human face is detected again, marking as an interruption ending moment, and generating an interruption event record; segmenting the recovery period according to time, and generating a recovery period interference sensitivity time sequence; extracting a decay time constant as a cognitive elastic recovery time constant; comparing the current cognitive elastic recovery time constant with a historical reference value of the reviewer, and calculating a recovery time extension rate as a cognitive elastic fatigue index; when the cognitive elastic fatigue index exceeds a set threshold value, a fatigue alarm signal is output; the method overcomes the limitation that a traditional fatigue detection method only pays attention to the steady state expression and neglects the dynamic recovery capability, and solves the technical problem that an early warning cannot be given out at the early stage of fatigue.

Description

Technical Field

[0001] This invention relates to the field of video analysis and early warning technology for personnel in specific positions, and more specifically, it relates to a real-time fatigue detection method for reviewers based on multi-feature fusion video analysis. Background Technology

[0002] In the field of video analytics, reviewers need to stare at the screen for extended periods to identify individuals in the video who resemble the target. Review work is frequently interrupted due to phone calls, short breaks, shift handovers, and other reasons. Under normal circumstances, reviewers can quickly rebuild their ability to distinguish similar interfering targets after an interruption; however, under fatigue, the recovery of memory resistance to interference after an interruption is slower, manifested as a slower rate of decrease in the rate of mis-sighting of highly similar interfering targets and a longer recovery time.

[0003] Traditional fatigue detection methods suffer from the following technical problems: First, they only focus on physiological characteristics and cognitive performance during steady-state work, failing to utilize the naturally occurring scenario of task interruption to assess cognitive function. Second, existing memory interference detection methods only assess interference sensitivity during continuous work, without analyzing the recovery process of interference resistance after interruption. Third, interruption recovery assessment methods only focus on overall alertness recovery, failing to capture the recovery trajectory of fine cognitive functions such as memory discrimination. Therefore, when an examiner's cognitive resilience is depleted due to fatigue, existing methods struggle to issue early warnings before a significant decline in steady-state performance. Summary of the Invention

[0004] This invention provides a real-time fatigue detection method for reviewers based on multi-feature fusion video analysis, which solves the limitation of traditional fatigue monitoring methods in related technologies that only focus on steady-state performance and ignore dynamic recovery capabilities, and solves the technical problem of not being able to issue early warnings in the early stages of fatigue.

[0005] This invention provides a real-time method for detecting viewer fatigue based on multi-feature fusion video analysis, comprising: Acquire the reviewer's face detection data stream, mark the start time of the interruption when no face is detected for several consecutive frames, and mark the end time of the interruption when a face is detected again, and generate an interruption event record; Obtain a reference image of the target person and extract the target feature vector. Calculate the similarity between the face detected in the video stream to be reviewed and the target feature vector. Mark faces that have a similarity exceeding the threshold but are not the target person as interference targets. The recovery period time window is defined with the end of the interruption as the starting point. Eye-tracking data within this window is obtained, the viewing time of the interfering target is counted, and the mis-viewing rate of the interfering target is calculated. The recovery period is divided into time segments, and the average misfixation rate within each segment is calculated to generate a time series of interference sensitivity during the recovery period. The time series of disturbance sensitivity during the recovery period was fitted with an exponential decay function, and the decay time constant was extracted as the cognitive elasticity recovery time constant. The recovery time constant of the current cognitive resilience is compared with the reviewer's historical benchmark value, and the recovery time extension rate is calculated as the cognitive resilience fatigue index. When the cognitive elasticity fatigue index exceeds the set threshold, a fatigue alarm signal is output. Among them, the exponential decay function characterizes the change law of the disturbance sensitivity from the initial high value to the steady state value as the recovery time increases, and the decay time constant characterizes the characteristic time required for the sensitivity to decay to the steady state.

[0006] Furthermore, the mis-fixation rate of interfering targets is calculated as the ratio of the sum of fixation durations on interfering targets to the total fixation duration during the recovery period.

[0007] Furthermore, the historical benchmark value is obtained by reading the cognitive elasticity recovery time constant records of similar interruption events within the preset time range of the reviewer, and calculating their average value as the historical benchmark value.

[0008] Furthermore, the recovery time extension rate is calculated as the ratio of the current cognitive elasticity recovery time constant to the historical baseline value minus 1.

[0009] Furthermore, it also includes the following steps: Arrange the cognitive resilience fatigue index of multiple consecutive interruption events that occur on the same day in chronological order, apply a linear regression algorithm to calculate the growth slope of the fatigue index, and generate a cognitive resilience reserve depletion rate index. When the exhaustion rate indicator is positive and exceeds the warning threshold, a cognitive resilience reserve exhaustion warning signal is output.

[0010] Furthermore, the similarity is calculated by extracting the feature vector of the face in the video stream and calculating the cosine similarity between the feature vector and the target feature vector.

[0011] Furthermore, the interruption event is determined by the following condition: the number of consecutive frames in which no face is detected exceeds a preset frame rate threshold, which is set according to the frame rate of the camera device.

[0012] Furthermore, the fatigue warning signal also includes a suggested rest duration, which is determined by looking up a table based on the numerical range of the cognitive resilience fatigue index.

[0013] This invention provides a real-time fatigue detection system for video reviewers based on multi-feature fusion video analysis, characterized in that it includes: The interruption event detection module is used to acquire the reviewer's facial detection data stream and identify task interruption events; The interference target identification module is used to extract target feature vectors and identify interference targets in the video stream; The recovery period behavior statistics module is used to collect statistics on the gaze behavior of reviewers towards interfering targets during the recovery period and calculate the misgaze rate. The sensitivity sequence generation module is used to segment the recovery period over time and generate a time series of interference sensitivity. The recovery time constant extraction module is used to perform exponential decay fitting on the interference sensitivity time series and extract the decay time constant. The fatigue index calculation module is used to compare the recovery time constant with historical benchmark values ​​and calculate the cognitive elasticity fatigue index. The alarm output module is used to output fatigue alarm signals based on the fatigue index determination results.

[0014] The beneficial effects of this invention are as follows: This invention introduces a dynamic tracking method for memory interference resistance recovery after task interruption and a method for quantifying the cognitive elasticity recovery time constant, using naturally occurring task interruptions as trigger points for cognitive function testing.

[0015] Since the recovery rate of memory interference resistance directly reflects the elastic reserve level of prefrontal executive control function, and prefrontal executive control function is highly sensitive to fatigue accumulation, the extension of the recovery time constant can reveal the latent fatigue of the cognitive system.

[0016] Because this invention uses similarity interference targets contained in the video content itself as test probes for memory function, changes in the viewer's memory discrimination ability can be evaluated without imposing additional test tasks.

[0017] Because this invention applies an exponential decay function to fit the time series of interference sensitivity during the recovery period and extracts the decay time constant, it can transform the subjectively imperceptible differences in recovery speed into quantifiable and comparable numerical indicators.

[0018] Because this invention performs trend analysis on the cognitive resilience fatigue index of multiple interruption events, it can capture the continuous depletion process of cognitive resilience reserves and issue an early warning before the reviewer's steady-state performance declines significantly.

[0019] Therefore, this invention overcomes the limitation of traditional fatigue detection methods that only focus on steady-state performance and ignore dynamic recovery capabilities, and solves the technical problem of not being able to issue early warnings in the early stages of fatigue. Attached Figure Description

[0020] Figure 1 This is a flowchart of a real-time method for detecting reviewer fatigue based on multi-feature fusion video analysis according to the present invention; Figure 2 This is a line graph showing the dynamic change of the cumulative misfocus rate during the recovery period according to the present invention. Figure 3 This is a hybrid graph of the interference sensitivity time series and the exponential decay fitting bar graph of the present invention; Figure 4 This is a scatter plot and line graph of the fatigue index trend analysis of multiple interruptions on the same day, as described in this invention. Figure 5 This is a bar chart showing the comparison and analysis of the similarity of interference targets according to the present invention; Figure 6 This is a box plot of the recovery time constant distribution under different fatigue states of the present invention; Figure 7 This is a heatmap of fatigue index changes over a week's workday, based on the present invention. Detailed Implementation

[0021] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0022] At least one embodiment of the present invention discloses a real-time method for detecting viewer fatigue based on multi-feature fusion video analysis, such as... Figure 1 As shown, it includes the following steps: Step 100: Obtain the reviewer's face detection data stream, identify task interruption events based on the face detection results, and generate an interruption event timestamp and interruption duration record table.

[0023] Specifically, the computing device acquires a sequence of video frames of the reviewer's facial area from the camera device, and applies a face detection network to each frame to determine whether a valid face exists. When no face is detected for several consecutive frames, the time corresponding to the first frame where no face is detected is marked as the interruption start time; when a face is detected again, the time corresponding to that frame is marked as the interruption end time. The difference between the interruption end time and the interruption start time is calculated to obtain the interruption duration. The interruption start time, interruption end time, and interruption duration are stored as a record in the interruption event timestamp and interruption duration record table.

[0024] The aforementioned face detection network comprises an input layer, a feature extraction layer, a region proposal layer, and an output layer. The input layer receives video frame images, the feature extraction layer extracts multi-scale features from the images, the region proposal layer generates candidate face regions, and the output layer outputs the face bounding box coordinates and confidence scores. The face detection network is trained using supervised learning, taking images containing face annotations as input and face bounding box coordinates and classification labels as targets. Network parameters are updated using a gradient descent algorithm, and the loss function includes a bounding box regression loss function and a classification loss function.

[0025] Furthermore, the aforementioned method for determining whether a valid face exists is as follows: when the confidence score output by the face detection network is greater than 0.9, the frame is determined to contain a valid face; when the confidence score is less than or equal to 0.9, the frame is determined not to contain a valid face. The setting of the confidence threshold ensures that the detected faces have sufficiently high reliability, avoiding misclassification of non-face regions or blurred faces as valid faces.

[0026] It should be noted that the frame rate threshold for "continuous multiple frames" mentioned above can be set according to the frame rate of the camera device. For example, when the frame rate is 30fps, 60 consecutive frames (corresponding to 2 seconds) without detecting a face can be used as the condition for determining interruption, so as to filter out the momentary detection failure caused by the reviewer briefly looking down or turning their head.

[0027] On the morning of December 19, 2024, a video reviewer in a certain city was performing a video review task with the goal of identifying targets in the surveillance video of a shopping mall. The camera captured facial images of the reviewer at a frame rate of 30fps, and a face detection network detected each frame. At 10:23:15 AM, the reviewer answered a work call and left their seat. The face detection network failed to detect a valid face for 60 consecutive frames, and the system marked 10:23:17 AM as the start of the interruption. At 10:25:42 AM, the reviewer returned to their seat, and the face detection network re-detected a valid face with a confidence level of 0.94. The system marked this moment as the end of the interruption, calculating the interruption duration to be 145 seconds. This record is stored in the interruption event timestamp and interruption duration record table, as shown in Table 1.

[0028] Table 1: Interrupt Event Timestamps and Interrupt Duration Records Step 200: Obtain the reference image of the target person and the video stream to be reviewed, calculate the target feature vector using the face feature extraction algorithm, perform similarity matching on the faces in the video stream, and generate a list of interference targets with location coordinates and similarity values.

[0029] Specifically, a reference image of the target person specified in the current review task is obtained. A facial feature extraction algorithm is used to extract features from this reference image, generating a feature vector of the target person as a baseline. Face detection is performed on each frame of the video stream to be reviewed, and feature vectors are extracted from each detected face region. L2 normalization is applied to both the extracted feature vectors and the target feature baseline to ensure the Euclidean norm of the vectors equals 1. A cosine similarity algorithm is applied to calculate the similarity between the normalized feature vectors and the target feature baseline. Faces with similarity exceeding a set threshold but confirmed not to be the target person are marked as interference targets. Their position coordinates in the frame are extracted and stored along with their similarity values ​​in the interference target list.

[0030] Furthermore, the aforementioned threshold is used to filter out highly similar interfering targets, with a value ranging from 0.5 to 0.8. Setting the threshold too low will introduce too many low-similarity targets, leading to inaccurate interference assessment; setting it too high will miss similar targets that actually interfere with the reviewer. The threshold can be adjusted based on the quality of the reference image of the target person and the scene complexity of the video to be reviewed. For cases with high-quality reference images and densely populated scenes in the video, the threshold can be set to 0.6.

[0031] Furthermore, the aforementioned method for identifying non-target individuals is as follows: For faces with a similarity exceeding a set threshold, their feature vectors are matched against a database of previously identified target individuals' feature vectors. The cosine similarity with the feature vectors of each identified target individual in the database is calculated. When all similarities are below 0.85, the face is determined to be a non-target individual; when there is a match with a similarity greater than or equal to 0.85, the face is determined to be the target individual. For highly similar faces encountered for the first time, the system automatically marks them as pending confirmation. After the reviewer confirms their identity through keyboard or mouse operations, the face feature vector is added to the corresponding feature vector database.

[0032] The aforementioned cosine similarity algorithm takes two feature vectors to be compared as input and outputs a similarity value between -1 and 1. The cosine similarity algorithm calculates the dot product of the two feature vectors and then divides it by the product of the magnitudes of the two vectors to obtain the cosine similarity value. The closer the value is to 1, the more similar the two feature vectors are.

[0033] It should be noted that the above face feature extraction algorithm can use a pre-trained deep convolutional neural network. The deep convolutional neural network takes the face image region as input and outputs a feature vector of fixed dimensions.

[0034] In this review task, the target's ID photo was obtained as a reference image. A facial feature extraction algorithm extracted a 512-dimensional feature vector from this image, which served as the target feature baseline. The mall surveillance video stream to be reviewed was processed. During the recovery period from 10:25:42 to 10:28:42, face detection detected 47 faces. Feature vectors were extracted from each face and cosine similarity was calculated with the target feature baseline. Three faces had similarities exceeding the set threshold of 0.6. After matching with a confirmed target feature vector database, the highest similarities between these three faces and the database were 0.62, 0.58, and 0.71, respectively, all below 0.85, and were therefore determined to be non-target individuals. The bounding box coordinates of these three interfering targets in the image were extracted and stored along with their similarity values ​​in an interfering target list, as shown in Table 2.

[0035] Table 2 List of Interference Targets: The aforementioned deep convolutional neural network includes an input layer, multiple convolutional layers, pooling layers, fully connected layers, and an output layer. The input layer receives normalized face image regions, the convolutional layers extract local facial features, the pooling layers reduce feature dimensionality, the fully connected layers integrate global features, and the output layer outputs a feature vector. The deep convolutional neural network is obtained through pre-training, and its training employs supervised learning. It uses face images from a large-scale face dataset as input and corresponding identity labels as targets. The network parameters are updated using backpropagation and Adam optimization algorithms. The loss function uses either cross-entropy loss or triplet loss, ensuring that feature vectors of faces with the same identity are close together in the feature space, while feature vectors of faces with different identities are far apart.

[0036] Figure 5 The facial similarity values ​​of three highly similar interference targets detected during the recovery period (data in Table 2) are shown.

[0037] Step 300: Based on the interruption event timestamp, interruption duration record table and eye-tracking data, analyze the examiner's gaze behavior towards the interfering target during the recovery period, calculate the mis-gaze rate of the interfering target, and generate a feature sequence of interfering gaze behavior during the recovery period.

[0038] Specifically, the end time of each interruption is read from the interruption event timestamp and interruption duration record table, and the recovery period time window is defined starting from this time. The time sequence of gaze coordinates within this window is obtained from the eye-tracking device. The spatial region matching judgment method is applied to match the gaze coordinates with the position coordinates of each interference target in the interference target list. When the gaze falls into the region of a certain interference target, the gaze is recorded as a gaze at that interference target. The number of gazes at each interference target and the cumulative gaze duration during the recovery period are counted, and the interference target mis-gazing rate is calculated, which is defined as the ratio of the sum of gaze durations at interference targets to the total gaze duration during the recovery period. The interference target mis-gazing rates are arranged in chronological order to generate a sequence of interference gaze behavior characteristics during the recovery period.

[0039] Furthermore, the specific method for arranging the data chronologically is as follows: the recovery period time window is divided into several time points according to the sampling time interval of the eye-tracking device. For each time point, the cumulative mis-fixation rate of the interfering target before the current moment is recorded, forming time series data with time as the horizontal axis and the cumulative mis-fixation rate as the vertical axis. The time series data starts from the end of the interruption and records the dynamic change process of the mis-fixation rate during the recovery period with the sampling frequency of the eye-tracking device (usually 60Hz or 120Hz) as the time resolution.

[0040] The aforementioned spatial region matching method takes the coordinates of the gaze point and the bounding box coordinates of the interfering target as input, and outputs the result of whether the gaze point falls within the region of the interfering target. The spatial region matching method determines whether the x-coordinate and y-coordinate of the gaze point are simultaneously within the horizontal and vertical ranges of the bounding box of the interfering target. If both conditions are met, the gaze point is determined to have fallen within the region of the interfering target.

[0041] It should be noted that the length of the recovery period window mentioned above can be set to 3 minutes after recovery. The length of the recovery period window is sufficient to observe the recovery process of interference resistance from low to high.

[0042] The interruption end time of 10:25:42 was read from the interruption event timestamp and interruption duration record table, and a 180-second recovery period window was defined starting from this time. The eye-tracking device recorded the gaze coordinates within this window at a sampling frequency of 60Hz. When the gaze coordinates fell within the bounding box of the interfering target in Table 2, it was recorded as a gaze at that interfering target. Statistical results showed that the cumulative gaze duration for interfering target D1 during the recovery period was 8.2 seconds, for D2 it was 12.5 seconds, and for D3 it was 6.8 seconds, totaling 27.5 seconds. The total gaze duration during the recovery period was 162 seconds. The mis-gaze rate for the interfering target was calculated. The mis-fixation rate was recorded in chronological order to generate a sequence of interference fixation behavior characteristics during the recovery period. Some data are shown in Table 3.

[0043] Table 3. Characteristic sequences of interference gaze behavior during the recovery period (partial data): Figure 2 The text describes the changes in the cumulative misfocus rate of reviewers within 180 seconds after a task is resumed following an interruption.

[0044] Step 400: Perform time-segmented statistical analysis on the feature sequence of interference gaze behavior during the recovery period to generate a time series of interference sensitivity during the recovery period.

[0045] Specifically, the recovery period interference gaze behavior feature sequence is segmented according to the time after recovery, with each segment having a length of 30 seconds. The average mis-gaze rate of the interference target within each segment is calculated as the interference sensitivity value for that time period. The interference sensitivity values ​​of each segment are arranged in chronological order to generate a recovery period interference sensitivity time series.

[0046] Furthermore, the aforementioned method for calculating the average mis-fixation rate of interfering targets within each segment is as follows: For the Each time segment (counted from 0) is used to extract the timestamps from the feature sequence of distracting gaze behavior during the recovery period. For all data points within a second interval, sum the corresponding misjudgment rate values ​​of the interference targets and divide by the number of data points to obtain the interference sensitivity value for each time segment. When there are no valid data points in a certain time segment, the interference sensitivity value for that time segment is set to the interference sensitivity value of the previous time segment to maintain the continuity of the time series.

[0047] The recovery period interference gaze behavior feature sequence was segmented into 30-second segments, resulting in 6 time segments. For the 0th time segment (0 to 30 seconds), the cumulative misgaze rate data of 1800 sampling points within this interval was extracted, and the average value was calculated to be 0.352; for the 1st time segment (30 to 60 seconds), the average value was calculated to be 0.306; and so on, to obtain the interference sensitivity value for each segment. The interference sensitivity values ​​of the 6 segments were arranged in chronological order to generate the recovery period interference sensitivity time series, as shown in Table 4.

[0048] Table 4 Time series of interference sensitivity during the recovery period: Step 500: Apply an exponential decay function to fit the time series of disturbance sensitivity during the recovery period, extract the decay time constant, and generate the cognitive elasticity recovery time constant for the current interruption event.

[0049] Specifically, the time series of disturbance sensitivity during the recovery period is fitted using the least squares method with an exponential decay function. The sequence number of each time period is converted to the actual time, i.e., the [number of time periods]. The time corresponding to each time period is Seconds, where 30 seconds is the length of a single time interval. Starting from 0, the 0th time interval corresponds to the time window of 0 to 30 seconds after recovery. The expression for the exponential decay function is: in, Indicates the time after recovery (Unit: seconds) Interference sensitivity value for the corresponding time period; This represents the offset of the initial disturbance sensitivity relative to the steady-state value; This represents the decay time constant (in seconds), which is the characteristic time required for the sensitivity to decay from an initial high value to near a steady-state value. This represents the steady-state disturbance sensitivity value. The parameters are estimated using the least squares method. , and Extract the decay time constant This serves as the cognitive resilience recovery time constant for the current interruption event.

[0050] Furthermore, the aforementioned time variable The range of values ​​is ,in The recovery period time window length is set to 180 seconds. Data point index. The range of values ​​is ,in The total number of time segments within the recovery period is calculated using the following formula: That is, the recovery period time window length divided by the segment length and rounded up. For a recovery period of 180 seconds, .

[0051] The aforementioned least squares method takes the recovery period disturbance sensitivity time series and the exponential decay function model as input, and outputs the fitted model parameters. The least squares method iteratively solves for the parameter values ​​that minimize the sum of squares of the differences between the observed values ​​and the model predictions.

[0052] Furthermore, the objective function of the aforementioned least squares method is: in, Let be the parameter vector to be optimized. Indicates transpose. For the first Each time segment corresponds to a specific moment (in seconds). For at any time Observed interference sensitivity values, This represents the total number of time segments. The objective function, by summing the squared residuals at each time point within the recovery period, reflects the accumulation of fitting error over time, ensuring that the fitted curve deviates from the observed data to a minimum throughout the entire recovery period.

[0053] Furthermore, the aforementioned iterative solution process is implemented using the Levenberg-Marquardt algorithm, which combines the advantages of gradient descent and Gauss-Newton methods.

[0054] Specifically, the parameter vector to be fitted is set as In the In this iteration, the parameter update formula is: , The number of iterations The range of values ​​is , The maximum number of iterations is preset (set to 1000). for The Jacobian matrix of dimension , for A residual vector of dimension 1 The damping factor, for A dimensional identity matrix. The th dimension of the residual vector. Each element is defined as ,in , For the model at time The predicted value, For the observed values. The Jacobian matrix of the th... Line 1 Column elements are ,in Corresponding parameters , , The specific calculation is as follows: , , Damping factor The initial value is set to 0.01, and when the sum of squared residuals decreases, it will... Reduced to 0.1 times the original value, when the sum of squared residuals increases, It has increased to 10 times its original size.

[0055] Furthermore, the initial value selection method for the aforementioned parameters is as follows: the interference sensitivity value of the first time period is used as... The initial value estimation uses the interference sensitivity value of the last time period as... The initial value estimate is obtained, thus yielding The initial value is the difference between the two; The initial value is set to one-third of the recovery time window length, i.e., 60 seconds. The initial value selection method is based on the physical meaning of the exponential decay function, ensuring that the initial value is close to the true parameter value and accelerating iterative convergence.

[0056] Furthermore, the termination condition for the aforementioned iterative solution process is that the relative value of the parameter change between two adjacent iterations is less than a set convergence threshold, or the number of iterations reaches a preset maximum number of iterations. The mathematical expression for the convergence criterion is: in, The convergence threshold is set to 0.001, with the superscript... This indicates the number of iterations. When the above conditions are met or the number of iterations is reached... When the maximum number of iterations (1000) is reached, the iteration terminates and the current parameter values ​​are used as the fitting result. During the fitting process, the following constraints are applied to the parameters: Among them, the decay time constant The constraint range covers typical timescales from rapid recovery to slow recovery under severe fatigue; steady-state disturbance sensitivity value The constraint range is based on the proportion of its representation to the total fixation time; initial offset The constraint range ensures that the output value of the fitted function is within the entire recovery period. Keep within a physically reasonable range Inside.

[0057] It should be noted that the cognitive elasticity recovery time constant This reflects the speed at which reviewers rebuild their resistance to memory interference after an interruption. The higher the value, the slower the recovery and the less cognitive resilience reserves there are.

[0058] Apply the exponential decay function to the recovery period interference sensitivity time series in Table 4. Perform least-squares fitting. Initial parameter values ​​are chosen as follows: (Sensitivity value for the first time period) (The sensitivity value of the last time period), thus obtaining , Seconds. The Levenberg-Marquardt algorithm was applied for iterative optimization, with the initial value of the damping factor set to 0.01.

[0059] After 23 iterations, the parameter changes between two consecutive iterations satisfy the convergence condition: The iteration terminates. The optimal parameters obtained from the fitting are... , Second, Extract the decay time constant. The second is used as the cognitive elasticity recovery time constant for the current interruption event, and the fitting results are shown in Table 5.

[0060] Table 5. Fitting results of the exponential decay function: Figure 3 The graph displays the interference sensitivity values ​​(bar chart) and the fitted curve of the exponential decay function (line graph) during the recovery period, which are statistically analyzed in 30-second segments.

[0061] Step 600: Compare the current cognitive elasticity recovery time constant with the historical benchmark value, calculate the recovery time extension rate, and generate a single cognitive elasticity fatigue index.

[0062] Specifically, the cognitive resilience recovery time constant records of similar interruption events for the reviewer within the past 30 days are retrieved from the historical database, and their average value is calculated as a historical benchmark. Calculate the cognitive elasticity recovery time constant for the current event. Compared with historical benchmarks The ratio of the two values ​​is then subtracted by 1 to obtain the recovery time extension rate. This recovery time extension rate is then used as the output of the single-experiment cognitive elasticity fatigue index.

[0063] Furthermore, the formula for calculating the aforementioned single-experiment cognitive elasticity fatigue index is as follows: in, The single-experiment cognitive elastic fatigue index has a range of values. , The cognitive resilience recovery time constant (in seconds) for the current interruption event, with a value range of [value range missing]. , This is a historical baseline value (unit: seconds), with a range of values ​​of [missing value]. .when This indicates that the recovery rate is slower than the historical average. This indicates that the recovery rate is faster than the historical average. This indicates that the recovery speed is consistent with the historical average.

[0064] Furthermore, the aforementioned similar interruption events refer to historical interruption events whose duration differs from the current interruption duration by no more than 1 minute. The classification criteria are based on research in cognitive psychology regarding the differences in the impact of interruptions of different durations on working memory. Short interruptions (1 to 3 minutes) and long interruptions (more than 5 minutes) have different mechanisms of impact on cognitive recovery, thus requiring differentiation and comparison.

[0065] Furthermore, the aforementioned 30-day time window length is set to capture the individual baseline cognitive resilience level of the reviewer, while avoiding the influence of long-term changes in the reviewer's skill improvement or physical condition due to an excessively long window. When there are fewer than 10 valid interruption events in the historical record, the time window is extended to 60 days to obtain sufficient statistical samples; when the standard deviation of the interruption duration of the same type of interruption event in the historical record exceeds 1.5 times the standard deviation of the interruption duration of all interruption events of the reviewer, it is determined that the interruption type classification is unstable. In this case, only the historical records in which the interruption duration differs from the current interruption duration by no more than 30 seconds are used to calculate the baseline value.

[0066] Furthermore, the aforementioned historical benchmark values The calculation method is as follows: Sort the filtered historical cognitive elasticity recovery time constant records by numerical value, remove one maximum and one minimum value, and calculate the arithmetic mean of the remaining data as the historical baseline value. The specific calculation formula is: in, This represents the total number of historical records after filtering. For the sorted number The cognitive resilience recovery time constant of each historical record, summed within a certain range. This indicates removing the minimum (first) and maximum (second) values. Intermediate data after (number) records. When the filtered historical records are less than 3 (i.e., ... When a valid historical baseline cannot be calculated, the single cognitive resilience fatigue index of the current interruption event is used. Setting it to 0 will not trigger fatigue alarms, and the cognitive elasticity recovery time constant for that instance will be directly stored in the historical database for subsequent baseline calculations. This approach avoids baseline bias caused by insufficient samples.

[0067] A search of the historical database was conducted for interruption events within the past 30 days that occurred between 85 and 205 seconds (the current interruption was 145 ± 60 seconds). A total of 12 records were retrieved. The cognitive resilience recovery time constants of these 12 records were extracted and sorted by value as follows: 48.3, 52.7, 55.1, 57.9, 59.2, 61.4, 62.8, 64.5, 66.3, 68.7, 72.1, and 89.5 seconds. After removing the minimum value of 48.3 seconds and the maximum value of 89.5 seconds, the arithmetic mean of the middle 10 values ​​was calculated. Seconds. Calculate the ratio of the current cognitive resilience recovery time constant to the historical baseline: Subtracting 1 yields a recovery time extension rate of 0.362. This recovery time extension rate is output as the single-experiment cognitive elastic fatigue index. .

[0068] Step 700: When the cognitive elasticity fatigue index exceeds the set threshold in a single instance, output a cognitive elasticity fatigue alarm signal.

[0069] Specifically, the single cognitive elasticity fatigue index is compared with a preset fatigue alarm threshold. When the single cognitive elasticity fatigue index exceeds the fatigue alarm threshold, a cognitive elasticity fatigue alarm signal is generated, and the recommended rest duration is determined by looking up a table based on the numerical range of the fatigue index. The alarm signal and the recommended rest duration are then output to the display terminal or management system.

[0070] Furthermore, the aforementioned fatigue alarm threshold is set to 0.3, meaning an alarm is triggered when the recovery time constant is extended by more than 30% compared to the historical baseline. This fatigue alarm threshold is set based on typical amplitudes of executive function recovery delays under fatigue conditions observed in cognitive psychology research. The aforementioned lookup table mapping for recommended rest duration is as follows: when a single cognitive resilience fatigue index is between 0.3 and 0.5, a 10-minute rest is recommended; when the index is between 0.5 and 0.8, a 20-minute rest is recommended; and when the index exceeds 0.8, a 30-minute rest is recommended. This mapping is determined based on the shortest effective rest duration required for cognitive function recovery under different levels of fatigue.

[0071] Single cognitive elastic fatigue index Compare with the fatigue alarm threshold of 0.3 to determine The system triggers a cognitive fatigue alarm. Based on a fatigue index value falling between 0.3 and 0.5, a recommended rest period of 10 minutes is determined from a table. The system generates an alarm signal, displaying a message on the terminal: "Cognitive fatigue detected; recovery time increased by 36.2%. It is recommended to rest for 10 minutes before continuing to review the task." Simultaneously, the alarm record is uploaded to the management system.

[0072] In this embodiment of the application, in order to capture the continuous depletion process of cognitive resilience reserves, the following steps are also included: Step 800: Perform trend analysis on the cognitive resilience fatigue index of multiple consecutive interruption events that occurred on the same day, and generate a cognitive resilience reserve depletion rate index.

[0073] Specifically, the interruption events that occurred on the same day were arranged in chronological order, and the single-event cognitive resilience fatigue index corresponding to each interruption event was extracted. Using the interruption event number as the independent variable and the single-event cognitive resilience fatigue index as the dependent variable, a linear regression model was applied to calculate the regression slope. The regression slope characterizes the growth rate of the single-event cognitive resilience fatigue index with the number of interruptions, and is output as an indicator of the rate of depletion of cognitive resilience reserves.

[0074] The aforementioned linear regression model takes the sequence number of the interruption event as input and outputs the predicted single-event cognitive resilience fatigue index. The expression for the linear regression model is: ,in This is the sequence number of the interrupt event, with a value range of [value range missing]. , This represents the total number of interruption events that occurred that day. The predicted single-instance cognitive elasticity fatigue index has a value range of [value range missing]. , The regression slope, This is the intercept.

[0075] Furthermore, the aforementioned regression slope and intercept The calculation method is as follows: Let the total number of occurrences on that day be... Secondary interruption event ( ), No. The sequence number of the interrupt event is ( From 1 to The corresponding single cognitive elasticity fatigue index is The formula for calculating the regression slope is: The formula for calculating the intercept is: Among them, the regression slope The range of values ​​is ,when The time indicates that the fatigue index increases with the number of interruptions. The time indicates that the fatigue index decreases with the number of interruptions. The time indicates that the fatigue index remains stable. The calculation method is the least-squares analytical solution of univariate linear regression, by minimizing the objective function. The optimal parameters are obtained. The aforementioned cognitive elasticity reserve depletion rate index is the regression slope. .

[0076] The reviewer experienced a total of 5 interruption events on that day. The single cognitive resilience fatigue index corresponding to each interruption event is arranged in chronological order as shown in Table 6.

[0077] Table 6: Fatigue Index Record of Interruption Events on the Same Day: Using the interrupt event number as the independent variable The fatigue index is the dependent variable. A linear regression model was applied for fitting the data. Intermediate variables were calculated. , , , Substitute into the regression slope formula to calculate: The intercept is calculated as follows: Extracting the regression slope As an indicator of the rate of depletion of cognitive elasticity reserves.

[0078] Figure 4 The graph displays the interference sensitivity values ​​(bar chart) and the fitted curve of the exponential decay function (line graph) during the recovery period, which are statistically analyzed in 30-second segments.

[0079] Step 900: When the cognitive elasticity reserve depletion rate indicator is positive and exceeds the warning threshold, output a cognitive elasticity reserve depletion warning signal.

[0080] Specifically, the sign and value of the cognitive resilience reserve depletion rate index are determined. When the cognitive resilience reserve depletion rate index is positive and exceeds the preset warning threshold, it indicates that the reviewer's cognitive resilience recovery ability is showing a continuous downward trend, and a cognitive resilience reserve depletion warning signal is generated and output.

[0081] Furthermore, the aforementioned warning threshold is set at 0.05, meaning that an alert is triggered when the average growth rate of the cognitive resilience fatigue index with the number of interruptions exceeds 0.05 per interruption. The warning threshold is set based on the following constraint: when a reviewer experiences 8 interruptions in a workday, if the exhaustion rate is 0.05, the fatigue index at the 8th interruption will reach 0.4, which is close to the critical state requiring mandatory rest. Therefore, the warning threshold provides an early warning window before cognitive reserves reach a dangerous level.

[0082] Indicators for assessing the rate of depletion of cognitive elasticity reserves The symbols and values, confirm and The system meets the early warning trigger conditions. It generates an early warning signal for the depletion of cognitive elasticity reserves and outputs the following on the display terminal: "Alarm: Cognitive elasticity reserves are continuously depleted. The fatigue index growth rate is 0.207 times / time, which exceeds the safety threshold by 4.1 times. It is recommended to arrange a longer rest period or replace the review personnel immediately." The early warning signal is also uploaded to the management system, where managers decide whether to adjust the work arrangements.

[0083] Figure 6 The simulation test data is presented to compare the statistical distribution of the cognitive elasticity recovery time constant τ of the reviewer under four states: normal state, mild fatigue, moderate fatigue, and severe fatigue.

[0084] This implementation method introduces a dynamic tracking method for memory interference resistance recovery after task interruption and a method for quantifying the cognitive elasticity recovery time constant, using naturally occurring task interruptions as the trigger point for cognitive function testing.

[0085] Since the recovery rate of memory interference resistance directly reflects the elastic reserve level of prefrontal executive control function, and prefrontal executive control function is highly sensitive to fatigue accumulation, the extension of the recovery time constant can reveal the latent fatigue of the cognitive system.

[0086] Because this method uses similarity interference targets contained in the video content itself as test probes for memory function, changes in the viewer's memory discrimination ability can be assessed without imposing additional test tasks.

[0087] Because this method applies an exponential decay function to fit the time series of disturbance sensitivity during the recovery period and extracts the decay time constant, it can transform the subjectively imperceptible differences in recovery speed into quantifiable and comparable numerical indicators.

[0088] Because this method performs trend analysis on the cognitive resilience fatigue index of multiple interruption events, it can capture the continuous depletion process of cognitive resilience reserves and issue an early warning before the reviewer's steady-state performance declines significantly.

[0089] Therefore, this method overcomes the limitation of traditional fatigue detection methods that only focus on steady-state performance and ignore dynamic recovery ability, and solves the technical problem of not being able to issue early warnings in the early stages of fatigue.

[0090] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for real-time detection of viewer fatigue based on multi-feature fusion video analysis, characterized in that, Includes the following steps: Acquire the reviewer's face detection data stream, mark the start time of the interruption when no face is detected for several consecutive frames, and mark the end time of the interruption when a face is detected again, and generate an interruption event record; Obtain a reference image of the target person and extract the target feature vector. Calculate the similarity between the face detected in the video stream to be reviewed and the target feature vector. Mark faces that have a similarity exceeding the threshold but are not the target person as interference targets. The recovery period time window is defined with the end of the interruption as the starting point. Eye-tracking data within this window is obtained, the viewing time of the interfering target is counted, and the mis-viewing rate of the interfering target is calculated. The recovery period is divided into time segments, and the average misfixation rate within each segment is calculated to generate a time series of interference sensitivity during the recovery period. The time series of disturbance sensitivity during the recovery period was fitted with an exponential decay function, and the decay time constant was extracted as the cognitive elasticity recovery time constant. The recovery time constant of the current cognitive resilience is compared with the reviewer's historical benchmark value, and the recovery time extension rate is calculated as the cognitive resilience fatigue index. When the cognitive elasticity fatigue index exceeds the set threshold, a fatigue alarm signal is output. Among them, the exponential decay function characterizes the change law of the disturbance sensitivity gradually decaying from the initial high value to the steady state value with the recovery time, and the decay time constant characterizes the characteristic time required for the sensitivity to decay to the steady state.

2. The real-time fatigue detection method for reviewers based on multi-feature fusion video analysis according to claim 1, characterized in that, The mis-fixation rate of the interfering target is calculated as the ratio of the sum of fixation durations on the interfering target to the total fixation duration during the recovery period.

3. The real-time fatigue detection method for reviewers based on multi-feature fusion video analysis according to claim 1, characterized in that, The historical benchmark value is obtained by reading the cognitive elastic recovery time constant records of similar interruption events within a preset time range of the reviewer, and calculating their average value as the historical benchmark value.

4. The real-time fatigue detection method for reviewers based on multi-feature fusion video analysis according to claim 1, characterized in that, The recovery time extension rate is calculated as follows: the ratio of the current cognitive elasticity recovery time constant to the historical baseline value minus 1.

5. The real-time fatigue detection method for reviewers based on multi-feature fusion video analysis according to claim 1, characterized in that, It also includes the following steps: The cognitive resilience fatigue index of multiple consecutive interruption events that occurred on the same day was arranged in chronological order, and the growth slope of the fatigue index was calculated by applying a linear regression algorithm to generate a cognitive resilience reserve depletion rate index. When the exhaustion rate indicator is positive and exceeds the warning threshold, a cognitive resilience reserve exhaustion warning signal is output.

6. The method for real-time detection of reviewer fatigue based on multi-feature fusion video analysis according to claim 1, characterized in that, The similarity is calculated by extracting the feature vector of the face in the video stream and calculating the cosine similarity between the feature vector and the target feature vector.

7. The real-time fatigue detection method for reviewers based on multi-feature fusion video analysis according to claim 1, characterized in that, The interruption event is determined when the number of consecutive frames in which no face is detected exceeds a preset frame rate threshold, which is set according to the frame rate of the camera device.

8. The real-time fatigue detection method for reviewers based on multi-feature fusion video analysis according to claim 1, characterized in that, The fatigue alarm signal also includes a suggested rest duration, which is determined by looking up a table based on the numerical range of the cognitive resilience fatigue index.

9. A real-time fatigue detection system for reviewers based on multi-feature fusion video analysis, used to execute the real-time fatigue detection method for reviewers based on multi-feature fusion video analysis as described in any one of claims 1-8, characterized in that, include: The interruption event detection module is used to acquire the reviewer's facial detection data stream and identify task interruption events; The interference target identification module is used to extract target feature vectors and identify interference targets in the video stream; The recovery period behavior statistics module is used to collect statistics on the gaze behavior of reviewers towards interfering targets during the recovery period and calculate the misgaze rate. The sensitivity sequence generation module is used to segment the recovery period over time and generate a time series of interference sensitivity. The recovery time constant extraction module is used to perform exponential decay fitting on the interference sensitivity time series and extract the decay time constant. The fatigue index calculation module is used to compare the recovery time constant with historical benchmark values ​​and calculate the cognitive elasticity fatigue index. The alarm output module is used to output fatigue alarm signals based on the fatigue index determination results.

Images