A multimodal data video inspection concentration analysis and early warning method and system

Abstract

The present application relates to the technical field of image detection, in particular to a multi-modal data video inspection concentration analysis and early warning method and system, the method comprising collecting facial video data, extracting eye micro-motion features, eyebrow shape features and forehead muscle television features in facial features; collecting heart rate variability data and skin electricity reaction data in physiological signals; performing individualized standardization processing, converting into Z-score feature vectors of deviation degree relative to user's own baseline level value; based on the Z-score feature vectors, constructing time sequence feature vectors and inputting into a multi-modal time sequence fusion network for time sequence collaborative mode analysis, outputting current concentration probability value and concentration prediction value at a specified time point in the future; triggering a hierarchical early warning mechanism based on the current concentration probability value and the concentration prediction value. The present application utilizes information complementation in multi-modal data fusion, thereby improving the robustness and accuracy of state recognition.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method and system for multimodal data video inspection focus analysis and early warning. Background Technology

[0002] Existing methods for monitoring the status of video inspection personnel mainly rely on simple facial recognition technology or manual supervision by managers, which generally have the following limitations: they can only passively determine whether employees are on duty, but cannot effectively quantify their work focus; they lack early warning mechanisms, and can only deal with problems after they occur; the evaluation criteria are singular and rigid, ignoring individual differences between different employees; at the same time, they fail to incorporate multimodal physiological data such as heart rate variability, resulting in incomplete evaluation dimensions and inaccurate results. Summary of the Invention

[0003] In view of this, the purpose of this invention is to propose a multimodal data video inspection focus analysis and early warning method and system to solve the problem that it is only possible to passively determine whether an employee is on duty, but cannot effectively quantify their work focus.

[0004] To achieve the above objectives, this invention provides a method for multimodal data video inspection focus analysis and early warning, comprising the following steps:

[0005] Collect facial video data and extract eye micro-movement features, eyebrow morphology features, and forehead electromyographic visual features from facial features;

[0006] Collect heart rate variability data and skin conductance response data from physiological signals;

[0007] The eye micro-movement features, eyebrow morphology features, forehead electromyography visual features, heart rate variability data, and skin conductance response data are personalized and standardized, and transformed into Z-score feature vectors that represent the degree of deviation from the user's own baseline level.

[0008] Based on the Z-score feature vector, a temporal feature vector is constructed and input into a multimodal temporal fusion network to perform temporal collaborative pattern analysis, and output the current focus probability value and the focus prediction value at a specified future time point;

[0009] A tiered early warning mechanism is triggered based on the current focus probability value and the focus prediction value.

[0010] Optionally, the extraction of eye micro-motion features specifically includes:

[0011] Capture continuous video frames of the user's eye region, use the Farneback algorithm to calculate dense optical flow, and obtain the horizontal optical flow Flow_x(t) and the vertical optical flow Flow_y(t);

[0012] Calculate the optical flow amplitude: M(t) = sqrt(Flow_x(t)² + Flow_y(t)²);

[0013] in:

[0014] -M(t) represents the amplitude of the light flow at time t;

[0015] -Flow_x(t) represents the horizontal component of the light flow at time t;

[0016] -Flow_y(t) represents the vertical component of the light flow at time t;

[0017] Short-time Fourier transform (STFT) is performed on the M(t) sequence within a 3-second time window, and the energy integral of the 2Hz-5Hz frequency band is extracted as the micro-saccade energy f_e(t) in the Z-score feature vector.

[0018] Where the micro-eye saccade energy f_e(t) = |STFT{M(t)}(ω)|²dω;

[0019] in:

[0020] -f_e(t) represents the micro-eye saccade energy at time t;

[0021] -STFT{ M ( t )}( oh ) represents the short-time Fourier transform of the M(t) sequence at frequency oh The value at that location, oh The range is from 2Hz to 5Hz.

[0022] Optionally, the extraction of eyebrow morphological features and forehead electromyographic visual features includes:

[0023] Eyebrow morphology feature extraction: Extract depth texture feature vectors from the eyebrow region image. _brow(t) calculates the cosine distance between the current frame feature and the baseline level value of the user's eyebrow shape feature to obtain the frowning index h_b(t) in the Z-score feature vector;

[0024] Where h_b(t)=1-( brow(t)·μ brow) / (|| brow(t)||×||μ brow||),

[0025] in:

[0026] -h_b(t) represents the frown index at time t;

[0027] - brow( t ) represents the eyebrow texture feature vector at time t;

[0028] -· represents the dot product of vectors;

[0029] - m brow represents the average vector of eyebrow texture features during the baseline period;

[0030] -||·|| denotes the Euclidean norm of a vector;

[0031] Forehead electromyography visual feature extraction: Capture image sequences of the forehead region, generate motion magnification images based on phase motion magnification algorithm, and calculate the variance of the inter-frame difference image of the magnified sequence as the micro-strain energy e_f(t) in the Z-score feature vector;

[0032] D(t)=R_fore_amplified_t-R_fore_amplified_{t-1}; e_f(t)=Var(D(t));

[0033] in:

[0034] -D(t) represents the inter-frame difference image at time t, which is the difference between the current frame and the motion-magnified forehead region image of the previous frame;

[0035] -R_fore_amplified_t represents the motion-amplified image of the forehead region at time t;

[0036] -R_fore_amplified_{t-1} represents the motion-amplified image of the forehead region at time t-1.

[0037] Optionally, the personalized standardization process, which transforms the data into a Z-score feature vector representing the degree of deviation from the user's baseline level, specifically includes:

[0038] A baseline period is established for each user, and facial feature and physiological signal data are collected when the user is in a calm and natural state.

[0039] Calculate the mean μ and standard deviation σ for each feature, where μ = (1 / N) × ∑A;

[0040] σ=sqrt((1 / N)×∑(A-μ)²);

[0041] in:

[0042] A represents f_e(t), h_b(t), e_f(t), hrv(t), or gsr(t), where hrv(t) represents heart rate variability data and gsr(t) represents skin conductance response data.

[0043] Z-score standardization was performed on the real-time feature values ​​using the formula z(t)=(A-μ) / σ to obtain the feature vectors of the eyes, eyebrows, and forehead [z_fe(t),z_hb(t),z_ef(t)], and the feature vectors of heart rate variability and skin conductance response [z_hrv(t),z_gsr(t)].

[0044] -z(t) represents the standardized eigenvalue at time t;

[0045] -μ and σ represent the mean and standard deviation of this feature during the baseline period, respectively;

[0046] The baseline data collection shall be no less than 180 frames.

[0047] Optionally, constructing the time-series feature vector based on the Z-score feature vector includes:

[0048] For the current time t, set the time window length T, collect feature values ​​from t-T+1 to t, and obtain the facial temporal feature vector: X_face=[z_fe(t-T+1),z_hb(t-T+1),z_ef(t-T+1);...;z_fe(t),z_hb(t),z_ef(t)], with shape Tx3; physiological temporal feature vector: X_physio=[z_hrv(t-T+1),z_gsr(t-T+1);...;z_hrv(t),z_gsr(t)], with shape Tx2.

[0049] Optionally, the multimodal temporal fusion network performs temporal collaborative pattern analysis and outputs the current focus probability value and the focus prediction value at a specified future time point, including:

[0050] A dual-branch TCN structure is adopted to process the facial temporal feature vector and the physiological temporal feature vector respectively, and output the processed facial feature vector F_face(t)∈R^d, where d is the feature dimension, and the physiological feature vector F_physio(t)∈R^d, where d is the feature dimension.

[0051] The fused feature X_extended(t) is obtained by fusing F_face(t) and F_physio(t) through a cross-attention mechanism:

[0052] The fused feature X_extended(t) is input into the joint TCN encoder, and the TCN encoder outputs the current focus probability and the future focus prediction value.

[0053] Wherein, the current focus probability is: s_raw(t) = M_TCN(X_extended(t));

[0054] Future focus prediction: _raw(t+Δt)=M_Predictor(X_extended(t)), where Δt=30 / 60 / 90 / 120 seconds;

[0055] -M_TCN indicates the TCN encoder model, a convolutional neural network used to process time-series data, which is widely used in sequence classification, regression and probabilistic prediction tasks;

[0056] -M_Predictor indicates the predictor model.

[0057] Optionally, the step of fusing F_face(t) and F_physio(t) through a cross-attention mechanism to obtain the fused feature X_extended(t) includes:

[0058] Define the query matrix, key matrix, and value matrix:

[0059] Query matrix W_q: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors into the query space of the cross-attention mechanism to generate a query vector Q, which is used to calculate the similarity with the key vector.

[0060] Key matrix W_k: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors onto the key space in the cross-attention mechanism to generate key vector K, which is used by the query vector to calculate similarity.

[0061] Value matrix W_v: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors into the value space of the cross-attention mechanism to generate a value vector V, which is used to generate the attention-weighted output.

[0062] d_k: The dimensions of the query vector Q and the key vector K; using facial features as the query and physiological features as the keys and values: Q_face = W_q F_face(t); K_physio=W_k F_physio(t); V_physio=W_v F_physio(t);

[0063] Calculate the attention score: S = Q_face K_physio^T / sqrt(d_k); Attention weights: A_s = softmax(S); Output: F_attention_face = A_s V_physio;

[0064] Query the facial branch from the physiological branch: Q_physio=W_q F_physio(t); K_face=W_k F_face(t); V_face=W_v F_face(t);F_attention_physio=softmax(Q_physio K_face^T / sqrt(d_k)) V_face;

[0065] Finally, the fused features are calculated: X_extended(t) = [F_face(t), F_physio(t), F_attention_face, F_attention_physio].

[0066] Optionally, the tiered early warning mechanism includes four levels:

[0067] Attention-level alert: When predicting focus level for the next 30 seconds When (t+30s)<0.7, a visual cue is triggered;

[0068] Alert level warning: When predicting focus level in the next 60 seconds When (t+60s)<0.6 and the trend is downward, trigger visual and auditory cues;

[0069] Warning level alert: When predicting focus level for the next 90 seconds When (t+90s)<0.5 and at least two physiological indicators are abnormal, a device vibration warning is triggered.

[0070] Emergency alert: When predicting focus level for the next 120 seconds When (t+120s)<0.4 and continues to decrease, a forced rest prompt is triggered.

[0071] Optionally, when the system predicts future focus levels When (t+Δt) falls below a preset threshold, the root cause analysis module is triggered to perform correlation analysis on factors that may lead to a decrease in focus.

[0072] The root cause analysis includes: environmental factor correlation analysis, which matches the predicted period of decreased focus with noise, light, and temperature data collected by environmental sensors to identify environmental interference factors; task type correlation analysis, which correlates the predicted period of decreased focus with the type of inspection task being processed to identify task difficulty suitability issues; and time rhythm correlation analysis, which combines historical work logs to analyze the user's focus fluctuation patterns in specific time periods.

[0073] Based on the root cause analysis results, targeted personalized intervention suggestions are generated, including: when environmental noise is identified as the main interfering factor, it is recommended to "wear noise-canceling headphones"; when the task type is identified as not matching the user's current state, it is recommended to "adjust the task order and move complex tasks to the peak period of concentration"; when fatigue from long-term work is identified, it is recommended to "take a 5-minute short break and adjust the ambient lighting".

[0074] The intervention recommendations are implemented in conjunction with the early warning level, and specific improvement measures are pushed out at the same time as the early warning is issued.

[0075] When the system predicts a decline in focus, it not only issues an alert but also simultaneously initiates root cause analysis. This analysis uses data correlation technology to match the predicted low point in focus with concurrent environmental data (noise, lighting), task attributes (type, difficulty), and personal historical patterns (biological clock) to identify the most likely influencing factors. Then, based on the identified root causes, it generates highly targeted and actionable solutions, which are pushed to the user along with the alert. This upgrades the system from a simple "monitor" to an "intelligent work coach." It solves the dilemma of "knowing there's a problem but not knowing what to do," providing users with clear action guidelines. This integrated "alert-diagnosis-suggestion" closed loop helps users fundamentally understand and improve their focus issues, thereby achieving long-term work efficiency improvements, demonstrating the system's advanced intelligence and practical value.

[0076] Based on the same invention, this invention also provides a system for multimodal data video inspection focus analysis and early warning method, comprising:

[0077] Acquisition and extraction module: Acquires facial video data and extracts facial features such as eye micro-movement features, eyebrow morphology features, and forehead electromyography visual features; acquires physiological signals such as heart rate variability data and skin conductance response data.

[0078] Standardization processing module: Performs personalized standardization processing on the eye micro-movement features, eyebrow morphology features, forehead electromyography visual features, heart rate variability data and skin conductance response data, and converts them into Z-score feature vectors that represent the degree of deviation from the user's own baseline level value;

[0079] Prediction module: Based on Z-score feature vectors, a temporal feature vector is constructed and input into a multimodal temporal fusion network to perform temporal collaborative pattern analysis, and output the current focus probability value and the focus prediction value at a specified future time point;

[0080] Early warning module: Triggers a tiered early warning mechanism based on the current focus probability value and focus prediction value.

[0081] This method uses two independent channels—a camera and a smart wearable device—to collect facial micro-motion information and physiological signal data from users, respectively. Facial features primarily reflect micro-expressions and muscle movements triggered by cognitive activities, while physiological signals reveal changes in the state of the autonomic nervous system. By personalizing and standardizing these two types of heterogeneous data, interference from individual physiological differences is eliminated. The temporal fusion network identifies stable states associated with high attention by analyzing the collaborative change patterns of multimodal feature vectors within past T frames, rather than isolated instantaneous values. Based on the persistence of these patterns, it predicts future attention trends, thus achieving a leap from current state assessment to future trend prediction. This method realizes a transformation in assessing work attention from a "static snapshot" to a "dynamic trend." Multimodal data fusion utilizes complementary information to improve the robustness and accuracy of state recognition. The predictive capability based on temporal patterns enables the system to provide early warnings, offering a valuable time window for proactive intervention and fundamentally changing the passive management model of post-event recording. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0085] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. Figure 1 As shown, a multimodal data video inspection focus analysis and early warning method includes the following steps:

[0086] S1: Collect facial video data, which can be collected through a camera and extracted from facial features such as eye micro-movement features, eyebrow shape features, and forehead electromyographic visual features.

[0087] S2: Collects heart rate variability (HRV) and skin conductance response (GSR) data from physiological signals, which can be acquired via smart wearable devices.

[0088] S3: Perform personalized standardization processing on the eye micro-movement features, eyebrow morphology features, forehead electromyography visual features, heart rate variability data and skin conductance response data, and convert them into Z-score feature vectors that represent the degree of deviation from the user's own baseline level.

[0089] S4: Based on the Z-score feature vector, construct a temporal feature vector and input it into a multimodal temporal fusion network to perform temporal collaborative pattern analysis, and output the current focus probability value and the focus prediction value at a specified future time point;

[0090] S5: A tiered early warning mechanism is triggered based on the current focus probability value and the predicted focus value. This method collects facial micro-motion information and physiological signal data from two independent channels: a camera and a smart wearable device. Facial features mainly reflect micro-expressions and muscle movements triggered by cognitive activities, while physiological signals reveal changes in the state of the autonomic nervous system. By personalizing and standardizing these two types of heterogeneous data, interference from individual basic physiological differences is eliminated. The temporal fusion network identifies stable states associated with high focus by analyzing the collaborative change patterns of multimodal feature vectors within the past T frames, rather than isolated instantaneous values. Based on the persistence of this pattern, it predicts future focus trends, thus achieving a leap from current state assessment to future trend prediction. This method realizes the transformation of work focus assessment from a "static snapshot" to a "dynamic trend." Multimodal data fusion utilizes complementary information to improve the robustness and accuracy of state recognition. The predictive capability based on temporal patterns enables the system to issue early warnings, providing a valuable time window for proactive intervention and fundamentally changing the passive management model of post-event recording.

[0091] In some embodiments, the extraction of eye micro-motion features specifically includes:

[0092] Capture continuous video frames of the user's eye region, use the Farneback algorithm to calculate dense optical flow, and obtain the horizontal optical flow Flow_x(t) and the vertical optical flow Flow_y(t);

[0093] Calculate the optical flow amplitude: M(t) = sqrt(Flow_x(t)² + Flow_y(t)²);

[0094] in:

[0095] -M(t) represents the amplitude of the light flow at time t;

[0096] -Flow_x(t) represents the horizontal component of the light flow at time t;

[0097] -Flow_y(t) represents the vertical component of the light flow at time t;

[0098] Short-time Fourier transform (STFT) is performed on the M(t) sequence within a 3-second time window, and the energy integral of the 2Hz-5Hz frequency band is extracted as the micro-saccade energy f_e(t) in the Z-score feature vector.

[0099] Where the micro-eye saccade energy f_e(t) = |STFT{M(t)}(ω)|²dω;

[0100] in:

[0101] -f_e(t) represents the micro-eye saccade energy at time t;

[0102] -STFT{ M ( t )}( oh ) represents the short-time Fourier transform of the M(t) sequence at frequency oh The value at that location, oh The range is from 2Hz to 5Hz.

[0103] When the human eye is highly focused, the brain suppresses microsaccades to stabilize visual input. This claim captures minute eye movements by calculating the optical flow field between video frames in the eye region. The optical flow amplitude sequence M(t) reflects the intensity of these microsaccades. By converting the time-domain signal to the frequency domain using STFT, the energy of the frequency band (2-5Hz) specific to microsaccades can be accurately separated, thereby quantifying the energy reduction phenomenon of microsaccades caused by cognitive inhibition. A non-contact, high-precision microsaccade detection method is provided. Compared with traditional electrode measurements, this method is non-invasive and easy to deploy. Frequency domain analysis can effectively filter out interference actions such as large-scale head movements and blinking, making the extracted features more specific and more directly and reliably correlated with focus.

[0104] In some embodiments, the extraction of eyebrow morphological features and forehead electromyographic visual features includes:

[0105] Eyebrow morphology feature extraction: Extract depth texture feature vectors from the eyebrow region image. _brow(t) calculates the cosine distance between the current frame feature and the baseline level value of the user's eyebrow shape feature to obtain the frowning index h_b(t) in the Z-score feature vector;

[0106] Where h_b(t)=1-( brow(t)·μ brow) / (|| brow(t)||×||μ brow||),

[0107] in:

[0108] -h_b(t) represents the frown index at time t;

[0109] - brow( t ) represents the eyebrow texture feature vector at time t;

[0110] -· represents the dot product of vectors;

[0111] - m brow represents the average vector of eyebrow texture features during the baseline period;

[0112] -||·|| denotes the Euclidean norm of a vector;

[0113] Forehead electromyography visual feature extraction: Capture image sequences of the forehead region, generate motion magnification images based on phase motion magnification algorithm, and calculate the variance of the inter-frame difference image of the magnified sequence as the micro-strain energy e_f(t) in the Z-score feature vector;

[0114] D(t) = R_fore_amplified_t - R_fore_amplified_{t-1}; e_f(t) = Var(D(t)); (Calculate the variance of the differences between all pixels within the image block).

[0115] in:

[0116] -D(t) represents the inter-frame difference image at time t, which is the difference between the current frame and the motion-magnified forehead region image of the previous frame;

[0117] -R_fore_amplified_t represents the motion-amplified image of the forehead region at time t;

[0118] -R_fore_amplified_{t-1} represents the magnified image of the forehead region at time t-1. Frowning and frontalis muscle tension are facial muscle coordination responses accompanying cognitive effort. This claim quantifies the degree of downward pressure and contraction of the eyebrow shape by analyzing changes in the texture features of the eyebrow region (using LBP or CNN). For the subtle tension of the frontalis muscle, phase motion magnification technology is used to amplify the micro-strain of the skin that is invisible to the naked eye, and then the energy of this subtle motion is quantified by calculating the inter-frame variance of the amplified sequence, thereby transforming the physiological activity of the muscle into calculable visual features. This achieves the objective quantification of weak, involuntary facial muscle activity. Phase magnification technology allows for the capture of subtle tension of the frontalis muscle without special hardware, reducing system costs. It transforms the abstract concepts of "frowning" and "tension" into concrete numerical indicators, providing a stable and repeatable objective basis for attention analysis.

[0119] In some embodiments, the personalized standardization process, which transforms the deviation from the user's own baseline level value into a Z-score feature vector, specifically includes:

[0120] A baseline period is established for each user, and facial feature and physiological signal data are collected when the user is in a calm and natural state.

[0121] Calculate the mean μ and standard deviation σ for each feature, where μ = (1 / N) × ∑A;

[0122] σ=sqrt((1 / N)×∑(A-μ)²);

[0123] in:

[0124] A represents f_e(t), h_b(t), e_f(t), hrv(t), or gsr(t), where hrv(t) represents heart rate variability data and gsr(t) represents skin conductance response data.

[0125] Z-score standardization was performed on the real-time feature values ​​using the formula z(t)=(A-μ) / σ to obtain the feature vectors of the eyes, eyebrows, and forehead [z_fe(t),z_hb(t),z_ef(t)], and the feature vectors of heart rate variability and skin conductance response [z_hrv(t),z_gsr(t)].

[0126] -z(t) represents the standardized eigenvalue at time t;

[0127] -μ and σ represent the mean and standard deviation of this feature during the baseline period, respectively;

[0128] The baseline data collection shall be no less than 180 frames.

[0129] Significant differences exist in the baseline physiological state and facial habits of individuals (e.g., some habitually frown, while others have a high baseline frequency of twitching). This method, upon initialization for each user, first establishes a personal baseline (mean μ and standard deviation σ) in their calm state. All subsequent real-time data is standardized using Z-scores, transforming it into a numerical value representing the degree of deviation from their baseline ("deviation by several standard deviations"). This effectively eliminates the influence of individual differences on attention assessment, achieving truly personalized evaluation. This allows the same set of judgment criteria to be used across different users, greatly improving the system's universality and fairness. Simultaneously, it enables the system to focus on the "relative changes" in user states rather than "absolute values," which is crucial for detecting subtle fluctuations in attention.

[0130] In some embodiments, constructing a time-series feature vector based on the Z-score feature vector includes:

[0131] For the current time t, set the time window length T, collect feature values ​​from t-T+1 to t, and obtain the facial temporal feature vector: X_face=[z_fe(t-T+1),z_hb(t-T+1),z_ef(t-T+1);...;z_fe(t),z_hb(t),z_ef(t)], with shape Tx3; physiological temporal feature vector: X_physio=[z_hrv(t-T+1),z_gsr(t-T+1);...;z_hrv(t),z_gsr(t)], with shape Tx2;

[0132] In some embodiments, the multimodal temporal fusion network performs temporal cooperative pattern analysis including:

[0133] The multimodal temporal fusion network performs temporal collaborative pattern analysis and outputs the current focus probability value and the focus prediction value at a specified future time point, including:

[0134] A dual-branch TCN structure is adopted to process the facial temporal feature vector and the physiological temporal feature vector respectively, and output the processed facial feature vector F_face(t)∈R^d, where d is the feature dimension, and the physiological feature vector F_physio(t)∈R^d, where d is the feature dimension.

[0135] The fused feature X_extended(t) is obtained by fusing F_face(t) and F_physio(t) through a cross-attention mechanism:

[0136] The fused feature X_extended(t) is input into the joint TCN encoder, and the TCN encoder outputs the current focus probability and the future focus prediction value.

[0137] Wherein, the current focus probability is: s_raw(t) = M_TCN(X_extended(t));

[0138] Future focus prediction: _raw(t+Δt)=M_Predictor(X_extended(t)), where Δt=30 / 60 / 90 / 120 seconds;

[0139] -M_TCN indicates the TCN encoder model, a convolutional neural network used to process time-series data, which is widely used in sequence classification, regression and probabilistic prediction tasks;

[0140] -M_Predictor represents the predictor model, and its internal processing flow is as follows:

[0141] Input module: Receives an extended feature vector X_extended(t) containing temporal context, generated by a multimodal temporal fusion network. This vector is a high-level feature representation that has been fully fused by the preceding dual-branch TCN and cross-attention mechanism.

[0142] Temporal feature extraction module: Consists of one or more stacked temporal convolutional layers. This module is responsible for:

[0143] It receives the extended feature vector X_extended(t) from the input module and performs deep temporal dependency learning. It operates in the time dimension through dilated causal convolution, efficiently capturing long-range patterns and trends in the input sequence, providing a solid temporal foundation for future predictions.

[0144] Feature mapping and regression module: This typically consists of a fully connected regression layer. This module is responsible for:

[0145] It receives the compressed and refined high-level features after processing by the temporal feature extraction module; performs a non-linear mapping from high-dimensional features to a single predicted value; and uses the Sigmoid activation function to limit the final output value to between 0 and 1.

[0146] Output module: Outputs a scalar value, namely the predicted focus level at a specified future time point t+Δt. `_raw(t+Δt)`, where `Δt` is a configurable prediction time interval, such as 30, 60, 90, or 120 seconds. This network employs a dual-branch structure to process facial features (reflecting external behavior) and physiological signals (reflecting internal states) separately, respecting the heterogeneity of the data source. Through a cross-attention mechanism, the facial branch "attention" to the contextual information provided by the physiological branch, and vice versa, thereby achieving deep feature complementarity and synergy. The final joint encoder comprehensively utilizes all this information, not only determining the current state but also predicting future focus probabilities and trend classifications through sequence prediction. This achieves multi-level, deep-dimensional fusion of information. The cross-attention mechanism can uncover strong cross-modal correlation patterns, such as "when skin conductance increases, the frown index also increases," which is more discriminative than simply concatenating features. Outputting future predictions and trend classifications provides direct, quantitative decision-making basis for tiered early warning systems, making them more intelligent and forward-looking.

[0147] In some embodiments, the step of fusing F_face(t) and F_physio(t) through a cross-attention mechanism to obtain the fused feature X_extended(t) includes:

[0148] Define the query matrix, key matrix, and value matrix:

[0149] Query matrix W_q: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors into the query space of the cross-attention mechanism to generate a query vector Q, which is used to calculate the similarity with the key vector.

[0150] Key matrix W_k: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors onto the key space in the cross-attention mechanism to generate key vector K, which is used by the query vector to calculate similarity.

[0151] Value matrix W_v: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors into the value space of the cross-attention mechanism to generate a value vector V, which is used to generate the attention-weighted output.

[0152] d_k: The dimensions of the query vector Q and the key vector K; using facial features as the query and physiological features as the keys and values: Q_face = W_q F_face(t); K_physio=W_k F_physio(t); V_physio=W_v F_physio(t);

[0153] Calculate the attention score: S = Q_face K_physio^T / sqrt(d_k); Attention weights: A_s = softmax(S); Output: F_attention_face = A_s V_physio;

[0154] Query the facial branch from the physiological branch: Q_physio=W_q F_physio(t); K_face=W_k F_face(t); V_face=W_v F_face(t);F_attention_physio=softmax(Q_physio K_face^T / sqrt(d_k)) V_face;

[0155] Finally, the fused features are calculated: X_extended(t) = [F_face(t), F_physio(t), F_attention_face, F_attention_physio].

[0156] In some embodiments, the tiered early warning mechanism includes four levels:

[0157] Attention-level alert: When predicting focus level for the next 30 seconds When (t+30s)<0.7, a visual cue is triggered;

[0158] Alert level warning: When predicting focus level in the next 60 seconds When (t+60s)<0.6 and the trend is downward, trigger visual and auditory cues;

[0159] Warning level alert: When predicting focus level for the next 90 seconds When (t+90s)<0.5 and at least two physiological indicators are abnormal, a device vibration warning is triggered.

[0160] Emergency alert: When predicting focus level for the next 120 seconds When (t+120s) < 0.4 and continues to decline, a mandatory rest prompt is triggered. This mechanism sets up a tiered intervention hierarchy based on the predicted severity and urgency of the attention deficit. It ranges from gentle visual cues (via computer), to multi-sensory (visual and auditory) alerts, to strong device vibration alerts (via smart wearable devices that collect physiological signals), and finally to mandatory rest suggestions. The triggering conditions comprehensively consider the synergistic verification of predicted values, the downward trend, and physiological indicators to ensure the accuracy of the warning. This avoids the user aversion and resistance that might result from a single, harsh alarm method. The tiered mechanism makes the intervention more humane and precise, while also allowing for decisive measures to ensure work safety when necessary (emergency level). The multi-indicator fusion triggering conditions effectively reduce the system's false alarm rate, improving user experience and the effectiveness of the alarms.

[0161] In some embodiments, when the system predicts future focus levels When (t+Δt) falls below a preset threshold, the root cause analysis module is triggered to perform correlation analysis on factors that may lead to a decrease in focus.

[0162] The root cause analysis includes: environmental factor correlation analysis, which matches the predicted period of decreased focus with noise, light, and temperature data collected by environmental sensors to identify environmental interference factors; task type correlation analysis, which correlates the predicted period of decreased focus with the type of inspection task being processed to identify task difficulty suitability issues; and time rhythm correlation analysis, which combines historical work logs to analyze the user's focus fluctuation patterns in specific time periods.

[0163] Based on the root cause analysis results, targeted personalized intervention suggestions are generated, including: when environmental noise is identified as the main interfering factor, it is recommended to "wear noise-canceling headphones"; when the task type is identified as not matching the user's current state, it is recommended to "adjust the task order and move complex tasks to the peak period of concentration"; when fatigue from long-term work is identified, it is recommended to "take a 5-minute short break and adjust the ambient lighting".

[0164] The intervention recommendations are executed in conjunction with the early warning level, pushing specific improvement measures along with the warning alert. When the system predicts a decline in focus, it not only issues an alarm but also simultaneously initiates root cause analysis. This analysis uses data association technology to match the predicted low focus point with environmental data (noise, light), task attributes (type, difficulty), and personal historical patterns (biological clock) at the same time to identify the most likely influencing factors. Then, based on the identified root causes, it generates highly targeted and actionable solutions, which are pushed to the user along with the warning information. This upgrades the system from a simple "monitor" to an "intelligent work coach." It solves the dilemma of "knowing there is a problem but not knowing what to do," providing users with clear action guidelines. This integrated "early warning-diagnosis-recommendation" closed loop helps users fundamentally understand and improve their focus problems, thereby achieving long-term work efficiency improvements, demonstrating the system's advanced intelligence and practical value.

[0165] To further implement this invention, the present invention also provides a system for multimodal data video inspection focus analysis and early warning method, comprising:

[0166] Acquisition and extraction module: Acquires facial video data and extracts facial features such as eye micro-movement features, eyebrow morphology features, and forehead electromyography visual features; acquires physiological signals such as heart rate variability data and skin conductance response data.

[0167] Standardization processing module: Performs personalized standardization processing on the eye micro-movement features, eyebrow morphology features, forehead electromyography visual features, heart rate variability data and skin conductance response data, and converts them into Z-score feature vectors that represent the degree of deviation from the user's own baseline level value;

[0168] Prediction module: Based on Z-score feature vectors, a temporal feature vector is constructed and input into a multimodal temporal fusion network to perform temporal collaborative pattern analysis, and output the current focus probability value and the focus prediction value at a specified future time point;

[0169] Early warning module: Triggers a tiered early warning mechanism based on the current focus probability value and focus prediction value.

[0170] The present invention will be further described in detail below with reference to specific embodiments.

[0171] Example 1: Application of multimodal early warning system in work scenarios.

[0172] For example, a video inspection worker in front of a computer would have a system configuration that includes a camera and a smart bracelet.

[0173] Multimodal data synchronization and alignment:

[0174] Camera data (30fps) and wristband data (1Hz) are synchronized using a unified timestamp. The physiological data is resampled to 30Hz using an interpolation method to match the video frame rate.

[0175] Construct a 5-dimensional feature vector: Z_extended(t) = [z_fe(t), z_hb(t), z_ef(t), z_hrv(t), z_gsr(t)].

[0176] Continuous attention monitoring:

[0177] A focus score (0-100) is output every 5 seconds, and the focus change trend (slope of the past 30 seconds) is calculated in real time to predict the focus change in the next 60 seconds.

[0178] Example of an alert being triggered:

[0179] Scenario description: After working continuously for 45 minutes, workers begin to show signs of decreased concentration.

[0180] Data performance:

[0181] Current focus level: s_raw(t) = 0.62 (62 points);

[0182] Historical trend: The score dropped from 78 to 62 in the past 2 minutes, with a slope of -0.13 / second;

[0183] Physiological indicators: z_hrv = -2.8, z_gsr = 3.2, z_hr = 1.5;

[0184] Prediction results: (t+60s) = 0.48;

[0185] Early warning decision-making:

[0186] Meets the warning level criteria: (t+60s)<0.5 AND z_hrv<-2.0 AND z_gsr>2.0;

[0187] Triggering warning-level intervention: Watch vibrates + screen flashes red + "Decreased focus detected, short break recommended" message.

[0188] Intervention effect:

[0189] Workers paused their tasks and performed one minute of deep breathing exercises; after the intervention, their concentration gradually recovered to 75 points.

[0190] The system records the effectiveness of this intervention and optimizes the early warning parameters.

[0191] Example 2: Personalized baseline establishment and trend analysis.

[0192] Using worker inspection videos as an example, this paper illustrates the process of establishing multimodal baselines and predicting trends.

[0193] Step 1: Calculate the baseline mean and standard deviation of the multimodal workers;

[0194] Facial feature baseline:

[0195] Microsaccade energy: μ_fe = 1.05, σ_fe = 0.15;

[0196] Frowning index: μ_hb = 0.70, σ_hb = 0.08;

[0197] Frontalis muscle tension: μ_ef = 0.55, σ_ef = 0.07;

[0198] Physiological signal baseline:

[0199] Heart rate variability: μ_hrv = 45, σ_hrv = 6.1;

[0200] Skin conductance response: μ_gsr = 2.1, σ_gsr = 0.3;

[0201] Heart rate: μ_hr = 72, σ_hr = 4.2;

[0202] Step 2: Real-time monitoring and trend prediction.

[0203] Attention deficit warning pattern recognition:

[0204] Time series analysis (past 30 seconds):

[0205] Facial features: z_fe: -3.5 → -2.1 → -0.8 (inhibition released);

[0206] z_hb: 2.2 → 1.5 → 0.7 (muscle relaxation);

[0207] z_ef: 2.8 → 1.8 → 0.9 (tension relief);

[0208] Physiological signal: z_hrv: -1.8 → -2.5 → -3.2 (continuously decreasing);

[0209] z_gsr: 2.1 → 2.8 → 3.5 (continuously increasing);

[0210] Trend indicators:

[0211] - Attention slope: -0.15 / second;

[0212] - Predicted focus level (after 60 seconds): 0.45;

[0213] - Pattern matching accuracy: 87%;

[0214] Warning triggered:

[0215] The system detected a pattern of continuously declining focus;

[0216] Predicts that focus will fall below the threshold (0.5) in the next 60 seconds;

[0217] Based on abnormal physiological indicators, a warning-level alert is triggered.

[0218] Example 3: Optimization of closed-loop early warning feedback.

[0219] The system continuously learns and optimizes its early warning capabilities.

[0220] Feedback data collection:

[0221] 1. Warning triggering time, level, and method;

[0222] 2. User response behavior (whether to rest, rest duration);

[0223] 3. Focus level change curve after intervention;

[0224] 4. User subjective feedback (effectiveness rating).

[0225] Parameter adaptive adjustment:

[0226] Initial threshold: (t+60s)<0.6 triggers an early warning.

[0227] Optimization process:

[0228] - If focus recovers by more than 15 points after the warning: Maintain the threshold;

[0229] - If the change in focus after the warning is less than 5 points: Increase the threshold to 0.55;

[0230] - If users frequently ignore alerts: Increase the alert strength;

[0231] - If users report "excessive interference": reduce the frequency of warnings.

[0232] Personalized early warning strategies:

[0233] 1. For highly sensitive users: Use mild warnings (visual cues only);

[0234] 2. For low-sensitivity users: Employ powerful early warning systems (multi-channel combined);

[0235] 3. The warning threshold is automatically adjusted according to working hours, and is lowered in the afternoon.

[0236] Through the above mechanisms, the system can achieve accurate monitoring, trend prediction, and intelligent intervention of focus, providing effective support for workers to maintain their focus.

[0237] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in the details for the sake of brevity.

[0238] This invention is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for multimodal data video inspection focus analysis and early warning, characterized in that, Includes the following steps: Collect facial video data and extract eye micro-movement features, eyebrow morphology features, and forehead electromyographic visual features from facial features; Collect heart rate variability data and skin conductance response data from physiological signals; The eye micro-movement features, eyebrow morphology features, forehead electromyography visual features, heart rate variability data, and skin conductance response data are personalized and standardized, and transformed into Z-score feature vectors that represent the degree of deviation from the user's own baseline level. Based on the Z-score feature vector, a temporal feature vector is constructed and input into a multimodal temporal fusion network to perform temporal collaborative pattern analysis, and output the current focus probability value and the focus prediction value at a specified future time point; A tiered early warning mechanism is triggered based on the current focus probability value and the focus prediction value; The extraction of micro-movement features of the eyes specifically includes: Capture continuous video frames of the user's eye region, use the Farneback algorithm to calculate dense optical flow, and obtain the horizontal optical flow Flow_x(t) and the vertical optical flow Flow_y(t); Calculate the optical flow amplitude: M(t) = sqrt(Flow_x(t)² + Flow_y(t)²); in: -M(t) represents the amplitude of the light flow at time t; -Flow_x(t) represents the horizontal component of the light flow at time t; -Flow_y(t) represents the vertical component of the light flow at time t; Short-time Fourier transform (STFT) is performed on the M(t) sequence within a 3-second time window, and the energy integral of the 2Hz-5Hz frequency band is extracted as the micro-saccade energy f_e(t) in the Z-score feature vector. Where the micro-eye saccade energy f_e(t) = |STFT{M(t)}(ω)|²dω; in: -f_e(t) represents the micro-eye saccade energy at time t; -STFT{M(t)}(ω) represents the value of the short-time Fourier transform of the M(t) sequence at frequency ω, where ω ranges from 2Hz to 5Hz; The extraction of eyebrow morphological features and forehead electromyographic visual features includes: Eyebrow morphology feature extraction: Extract depth texture feature vectors from the eyebrow region image. _brow(t) calculates the cosine distance between the current frame feature and the baseline level value of the user's eyebrow shape feature to obtain the frowning index h_b(t) in the Z-score feature vector; where, h_b(t)=1 - ( brow(t)·μ brow) / (|| brow(t)||×||μ brow||), in: -h_b(t) represents the frown index at time t; - brow( t ) represents the eyebrow texture feature vector at time t; -· represents the dot product of vectors; - μ brow represents the average vector of eyebrow texture features during the baseline period; -||·|| denotes the Euclidean norm of a vector; Forehead electromyography visual feature extraction: Capture image sequences of the forehead region, generate motion magnification images based on phase motion magnification algorithm, and calculate the variance of the inter-frame difference image of the magnified sequence as the micro-strain energy e_f(t) in the Z-score feature vector; D(t)=R_fore_amplified_t-R_fore_amplified_{t-1}; e_f(t)=Var(D(t)); in: -D(t) represents the inter-frame difference image at time t, which is the difference between the current frame and the motion-magnified forehead region image of the previous frame; -R_fore_amplified_t represents the motion-amplified image of the forehead region at time t; -R_fore_amplified_{t-1} represents the motion-amplified image of the forehead region at time t-1; The personalized standardization process, which transforms the deviation from the user's baseline level into a Z-score feature vector, specifically includes: A baseline period is established for each user, and facial feature and physiological signal data are collected when the user is in a calm and natural state. Calculate the mean μ and standard deviation σ for each feature, where μ = (1 / N) × ∑A; σ=sqrt((1 / N)×∑(A-μ)²); in: A represents f_e(t), h_b(t), e_f(t), hrv(t), or gsr(t), where hrv(t) represents heart rate variability data and gsr(t) represents skin conductance response data. Z-score standardization was performed on the real-time feature values ​​using the formula z(t)=(A-μ) / σ to obtain the feature vectors of the eyes, eyebrows, and forehead [z_fe(t),z_hb(t),z_ef(t)], and the feature vectors of heart rate variability and skin conductance response [z_hrv(t),z_gsr(t)]. -z(t) represents the standardized eigenvalue at time t; -μ and σ represent the mean and standard deviation of this feature during the baseline period, respectively; The baseline data acquisition shall be no less than 180 frames; The construction of time-series feature vectors based on Z-score feature vectors includes: For the current time t, set the time window length T, collect feature values ​​from t-T+1 to t, and obtain the facial temporal feature vector: X_face=[z_fe(t-T+1),z_hb(t-T+1),z_ef(t-T+1);...;z_fe(t),z_hb(t),z_ef(t)], with shape Tx3; physiological temporal feature vector: X_physio=[z_hrv(t-T+1),z_gsr(t-T+1);...;z_hrv(t),z_gsr(t)], with shape Tx2; The multimodal temporal fusion network performs temporal collaborative pattern analysis and outputs the current focus probability value and the focus prediction value at a specified future time point, including: A dual-branch TCN structure is adopted to process the facial temporal feature vector and the physiological temporal feature vector respectively, and output the processed facial feature vector F_face(t)∈R^d, where d is the feature dimension, and the physiological feature vector F_physio(t)∈R^d, where d is the feature dimension. The fused feature X_extended(t) is obtained by fusing F_face(t) and F_physio(t) through a cross-attention mechanism: The fused feature X_extended(t) is input into the joint TCN encoder, and the TCN encoder outputs the current focus probability and the future focus prediction value. Wherein, the current focus probability is: s_raw(t) = M_TCN(X_extended(t)); Future focus prediction: _raw(t+Δt)=M_Predictor(X_extended(t)), where Δt=30 / 60 / 90 / 120 seconds; -M_TCN indicates the TCN encoder model; -M_Predictor indicates the predictor model; The tiered early warning mechanism includes four levels: Attention-level alert: When predicting focus level for the next 30 seconds When (t+30s)<0.7, a visual cue is triggered; Alert level warning: When predicting focus level in the next 60 seconds When (t+60s)<0.6 and the trend is downward, trigger visual and auditory cues; Warning level alert: When predicting focus level for the next 90 seconds When (t+90s)<0.5 and at least two physiological indicators are abnormal, a device vibration warning is triggered. Emergency alert: When predicting focus level for the next 120 seconds When (t+120s) < 0.4 and continues to decrease, a forced rest prompt is triggered; When the system predicts future focus When (t+Δt) falls below a preset threshold, the root cause analysis module is triggered to perform correlation analysis on factors that may lead to a decrease in focus. The root cause analysis includes: environmental factor correlation analysis, which matches the predicted period of decreased focus with noise, light, and temperature data collected by environmental sensors to identify environmental interference factors; task type correlation analysis, which correlates the predicted period of decreased focus with the type of inspection task being processed to identify task difficulty suitability issues; and time rhythm correlation analysis, which combines historical work logs to analyze the user's focus fluctuation patterns in specific time periods. Based on the root cause analysis results, targeted personalized intervention suggestions are generated, including: when environmental noise is identified as the main interfering factor, it is recommended to "wear noise-canceling headphones"; when the task type is identified as not matching the user's current state, it is recommended to "adjust the task order and move complex tasks to the peak period of concentration"; when fatigue from long-term work is identified, it is recommended to "take a 5-minute short break and adjust the ambient lighting". The intervention recommendations are implemented in conjunction with the early warning level, and specific improvement measures are pushed out at the same time as the early warning is issued.

2. The method for multimodal data video inspection focus analysis and early warning according to claim 1, characterized in that, The process of fusing F_face(t) and F_physio(t) through a cross-attention mechanism to obtain the fused feature X_extended(t) includes: Define the query matrix, key matrix, and value matrix: Query matrix W_q: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors into the query space of the cross-attention mechanism to generate a query vector Q, which is used to calculate the similarity with the key vector. Key matrix W_k: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors onto the key space in the cross-attention mechanism to generate key vector K, which is used by the query vector to calculate similarity. Value matrix W_v: A shared, learnable weight matrix used to linearly project facial temporal feature vectors or physiological temporal feature vectors into the value space of the cross-attention mechanism to generate a value vector V, which is used to generate the attention-weighted output. d_k: The dimensions of the query vector Q and the key vector K; using facial features as the query and physiological features as the keys and values: Q_face = W_q F_face(t); K_physio=W_k F_physio(t); V_physio=W_v F_physio(t); Calculate the attention score: S = Q_face K_physio^T / sqrt(d_k); Attention weights: A_s = softmax(S); Output: F_attention_face = A_s V_physio; Query the facial branch from the physiological branch: Q_physio=W_q F_physio(t); K_face=W_k F_face(t); V_face=W_v F_face(t);F_attention_physio=softmax(Q_physio K_face^T / sqrt(d_k)) V_face; Finally, the fused features are calculated: X_extended(t) = [F_face(t), F_physio(t), F_attention_face, F_attention_physio].

3. A system for implementing a multimodal data video inspection focus analysis and early warning method according to any one of claims 1-2, characterized in that, include: Acquisition and extraction module: Acquires facial video data and extracts facial features such as eye micro-movement features, eyebrow morphology features, and forehead electromyography visual features; Collect heart rate variability data and skin conductance response data from physiological signals; Standardization processing module: Performs personalized standardization processing on the eye micro-movement features, eyebrow morphology features, forehead electromyography visual features, heart rate variability data and skin conductance response data, and converts them into Z-score feature vectors that represent the degree of deviation from the user's own baseline level value; Prediction module: Based on Z-score feature vectors, a temporal feature vector is constructed and input into a multimodal temporal fusion network to perform temporal collaborative pattern analysis, and output the current focus probability value and the focus prediction value at a specified future time point; Early warning module: Triggers a tiered early warning mechanism based on the current focus probability value and focus prediction value.

Images