A Personnel Status Perception Method and System Based on Time Delay Awareness Multimodal Feature Fusion

By employing cross-modal time-delay perception alignment and dynamic weighted fusion techniques, the problem of inaccurate personnel status perception in existing technologies has been solved, enabling accurate representation of multimodal features and reliable identification of operational risks.

CN123069288APending Publication Date: 2026-11-10ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER COMPANY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611539180.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-30
Publication Date
2026-11-10

Smart Images

  • Figure CN123069288A_ABST
    Figure CN123069288A_ABST
Patent Text Reader

Abstract

This invention relates to the field of data processing technology and proposes a method and system for personnel state perception based on time-delay-aware multimodal feature fusion. By mapping features of endogenous modalities, behavioral modalities, and exogenous modalities to a unified task semantic space, the comparability between features of different modalities is improved. Furthermore, based on cross-modal cosine similarity, the time offset between different modalities is determined and feature-level time alignment is implemented to reduce feature misalignment caused by the time delay of state responses in different modalities. Simultaneously, the fusion weights are dynamically determined by combining the cross-modal semantic consistency after time alignment and the data quality of each modality, reducing the impact of abnormal, missing, or low-quality modalities on the fusion results. This invention can improve the accuracy and stability of multimodal features in representing the actual state of personnel, thereby improving the reliability of personnel state perception and operational risk identification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for perceiving human status based on time-delay-aware multimodal feature fusion. Background Technology

[0002] With the increasing demands for personnel safety management in complex work scenarios such as power maintenance, equipment inspection, high-altitude operations, and live-line work, relying solely on manual observation or single sensor information is no longer sufficient to accurately identify changes in personnel status and potential operational risks. The risk status of workers is typically influenced by multiple factors, including physiological and cognitive states, behavioral actions, and external environmental conditions. Single-modal information can only reflect a portion of these states and is easily affected by motion artifacts, occlusion, environmental interference, and individual differences.

[0003] Existing methods for perceiving human state typically first synchronize different sensor data using a unified timestamp, resampling, interpolation, or a fixed sliding window, and then fuse multimodal features through feature concatenation, fixed weighting, or attention networks. However, due to differences in sampling frequency, processing cycle, and state response speed among different modalities, the representation of the same risk state in EEG, behavioral, and environmental data is not entirely consistent. Synchronizing solely based on absolute timestamps or fixed time windows can easily lead to the direct fusion of features corresponding to different state stages, resulting in cross-modal feature misalignment. Furthermore, different modalities differ in data dimension, numerical scale, and physical semantics. Without a unified feature representation space, directly calculating the similarity between different modalities cannot accurately reflect their consistency in representing the same human state. In addition, existing fixed weights or fusion weights entirely dependent on model training cannot simultaneously reflect the consistency between modalities at the current moment as well as the changes in data quality within each modality itself. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a personnel status perception method and system based on time-delay-aware multimodal feature fusion. By employing cross-modal time-delay alignment and dynamic weighted fusion, the accuracy and stability of personnel status perception and operational risk identification are improved.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: One or more embodiments provide a method for personnel state perception based on time-delay-aware multimodal feature fusion, including the following steps: For the personnel to be tested, time-series data of endogenous modalities, behavioral modalities, and exogenous modalities used to identify the personnel's state are acquired and preprocessed. Feature extraction was performed on the preprocessed temporal data of endogenous modalities, behavioral modalities, and exogenous modalities, and the extracted modal features were mapped to a unified task semantic space to obtain endogenous modal semantic features, behavioral modal semantic features, and exogenous modal semantic features. Based on cross-modal cosine similarity, the temporal offset of behavioral modalities and exogenous modalities relative to endogenous modalities is determined, and feature-level temporal alignment of modal semantic features of different modalities is performed based on the temporal offset. Based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality, the dynamic fusion weights of semantic features of each modality are calculated. The time-aligned semantic features of each modality are weighted and fused according to the dynamic fusion weight to obtain multimodal fusion features, and the status or operational risks of the personnel to be detected are identified based on the multimodal fusion features.

[0006] A further technical solution involves extracting features from the preprocessed temporal data of endogenous modalities, behavioral modalities, and exogenous modalities, respectively, and mapping the extracted modal features to a unified task semantic space to obtain endogenous modal semantic features, behavioral modal semantic features, and exogenous modal semantic features. This includes the following steps: The time series data of endogenous modalities, behavioral modalities and exogenous modalities are respectively processed by the corresponding feature encoders to extract the corresponding modal features; Semantic mapping networks are set up for endogenous modalities, behavioral modalities and exogenous modalities respectively, and the features of each modality are mapped to a unified task semantic space to obtain modal semantic features of a unified dimension; L2 normalization is performed on the semantic features of each modality after semantic mapping to obtain normalized semantic features of each modality, including endogenous modality features, behavioral modality features and exogenous semantic features.

[0007] A further technical solution involves jointly optimizing the feature encoder and semantic mapping network through task loss and cross-modal semantic alignment loss.

[0008] A further technical solution involves determining the temporal offset of behavioral modalities and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity, and performing feature-level temporal alignment of modal semantic features of different modalities according to the temporal offset, including the following steps: Using the endogenous modality as the time reference, candidate time offset ranges are set for both behavioral and exogenous modalities. Calculate the average cosine similarity between the corresponding mode and the endogenous mode within the local time range under different candidate time offset ranges; Based on the average cosine similarity corresponding to different candidate time offsets, the candidate time offset corresponding to the one with the highest average cosine similarity is determined as the target time offset. The feature sequences of behavioral and external modalities are adjusted in time based on the target time offset to obtain the time-aligned semantic features of each modality.

[0009] A further technical solution involves calculating the dynamic fusion weights of the semantic features of each modality based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality. This includes the following steps: Calculate the cosine similarity between the semantic features of each modality after time alignment, map it to a set interval, and calculate the cross-modal consistency parameter of each modality; Calculate the corresponding data quality parameters based on the integrity, continuity, and stability of each modal data. The cross-modal consistency parameter and data quality parameter are weighted to calculate the credibility score of each modality, and the fusion weight of each modality is calculated based on the credibility score.

[0010] A further technical solution involves calculating the pairwise cosine similarity between the time-aligned endogenous modality, behavioral modality, and exogenous modality for the k-th time window, and mapping the cosine similarity to the interval [0,1]. Based on the similarity between each mode and the other two modes, the cross-modal consistency parameter of the corresponding mode is calculated. The calculation formula is as follows: ; ; ; in, This represents the cross-modal consistency parameter of the endogenous modes within the k-th time window; The cross-modal consistency parameter represents the behavioral modality within the k-th time window; This represents the cross-modal consistency parameter between internal and external modes within the k-th time window; This represents the normalized cosine similarity between endogenous modalities and behavioral modalities. This represents the normalized cosine similarity between endogenous and exogenous modes. It represents the normalized cosine similarity between the behavioral modality and the external modality.

[0011] Further technical solutions involve preprocessing the acquired time-series data for each modality, including: For the acquired modal time series data, the timestamps of different data sources are corrected based on a preset unified clock reference; For the timestamp-corrected data, resample the data at different sampling frequencies according to the preset target sampling frequency; The resampled modal time series data are divided into sliding time windows according to the preset time window length and sliding step size, so that the same time window contains endogenous state data, behavioral state data and external working condition data respectively. For data from different modalities within each time window, standardization is performed separately within each modality.

[0012] One or more embodiments provide a personnel state perception system based on time-delay-aware multimodal feature fusion, including: The multimodal time series data acquisition module is configured to acquire time series data of endogenous modalities, behavioral modalities, and exogenous modalities for identifying the state of the person to be detected, and to perform preprocessing. The feature extraction and mapping module is configured to extract features from the preprocessed endogenous modality, behavioral modality and exogenous modality time series data respectively, and map the extracted different modality features to a unified task semantic space to obtain endogenous modality semantic features, behavioral modality semantic features and exogenous modality semantic features; The time alignment module is configured to determine the time offset of behavioral modalities and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity, and to perform feature-level time alignment of modal semantic features of different modalities according to the time offset. The weight calculation module is configured to calculate the dynamic fusion weights of each modality's semantic features based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality's data. The fusion module is configured to perform weighted fusion of the time-aligned semantic features of each modality according to the dynamic fusion weight to obtain multimodal fusion features, and to identify the status or operational risks of the personnel to be detected based on the multimodal fusion features.

[0013] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein when executed by the processor, the computer instructions perform the steps in the aforementioned method for human state perception based on time-delay-aware multimodal feature fusion.

[0014] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the aforementioned method for personnel state perception based on time-delay-aware multimodal feature fusion.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention improves the comparability of features between different modalities by mapping features of endogenous, behavioral, and exogenous modalities to a unified task semantic space. Furthermore, it determines the time offset between different modalities based on cross-modal cosine similarity and implements feature-level time alignment to reduce feature misalignment caused by response delays in different modalities. Simultaneously, it dynamically determines the fusion weights by combining the cross-modal semantic consistency after time alignment with the data quality of each modality, reducing the impact of abnormal, missing, or low-quality modalities on the fusion results. This invention can improve the accuracy and stability of multimodal features in representing the actual state of personnel, thereby enhancing the reliability of personnel state perception and operational risk identification.

[0016] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0018] Figure 1 This is the overall flowchart of the personnel state perception method based on time-delay-aware multimodal feature fusion according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the unified task semantic mapping of endogenous modalities, behavioral modalities, and exogenous modalities in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the joint dynamic weight calculation of cross-modal consistency and data quality in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the personnel status recognition result in an example of Embodiment 2 of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0022] Example 1 In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 3 As shown, a method for personnel state perception based on time-delay-aware multimodal feature fusion includes the following steps: Step 1: For the person to be detected, acquire time-series data of endogenous modalities, behavioral modalities, and exogenous modalities used to identify the person's state, and perform preprocessing; Step 2: Extract features from the preprocessed endogenous modality, behavioral modality, and exogenous modality time-series data respectively, and map the extracted modality features to a unified task semantic space to obtain endogenous modality semantic features, behavioral modality semantic features, and exogenous modality semantic features; Step 3: Determine the temporal offset of behavioral modalities and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity, and perform feature-level temporal alignment of modal semantic features of different modalities according to the temporal offset; Step 4: Calculate the dynamic fusion weights of the semantic features of each modality based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality. Step 5: Perform weighted fusion of the time-aligned semantic features according to the dynamic fusion weight to obtain multimodal fusion features, and identify the status or operational risks of the personnel to be detected based on the multimodal fusion features; This embodiment addresses the problems of inconsistencies in feature representation spaces, temporal shifts in state responses, and inaccurate fusion results caused by changes in the validity of data across different modalities over time. It maps features of endogenous, behavioral, and exogenous modalities to a unified task semantic space, enabling comparison of modal information with different physical meanings and feature forms at a unified semantic level. Furthermore, it determines the temporal shift of behavioral and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity and performs feature-level time alignment accordingly to reduce feature misalignment caused by differences in acquisition and processing processes and state response speeds. Simultaneously, it dynamically determines fusion weights by combining the consistency of cross-modal semantic features after time alignment with the data quality of each modality. This ensures that modalities with higher consistency in state representation and better data quality contribute more to the fusion, while reducing the impact of abnormal, missing, or interfered modalities on the fusion results. This improves the accuracy and stability of multimodal fusion features in representing the actual state of the personnel being tested, thereby enhancing the reliability of personnel state perception and operational risk identification.

[0023] In step 1, time-series data of the intrinsic modalities, behavioral modalities, and extrinsic modalities of the person to be tested are obtained; Endogenous modal time series data are endogenous state data used to characterize the internal physiological and cognitive states of the person being tested. They can be acquired by physiological signal acquisition devices set in the corresponding parts of the person being tested's body, and may include electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, electromyogram (EMG) signals, and electrooculogram (EOG) signals.

[0024] It is feasible to acquire the EEG signals of the person being tested through EEG acquisition equipment and the ECG signals through ECG acquisition equipment to reflect changes in the person's neural activity, physiological load, or cognitive state.

[0025] Behavioral modal time-series data refers to behavioral state data used to characterize the overt actions and operational behaviors of the person being tested. Behavioral modal time-series data can be acquired through image acquisition devices and / or inertial measurement devices, and may include human posture data, limb movement data, motion trajectory data, movement speed data, joint position data, and inertial measurement data, etc.

[0026] The feasible technical solution is that the image acquisition device can be a camera or a depth camera, used to continuously acquire images of the person being inspected, and obtain behavior-related time-series data through human key point detection, posture estimation or action recognition; the inertial measurement device can be set on the body of the person being inspected or the work equipment, used to acquire acceleration, angular velocity and posture change data.

[0027] External modal time series data refers to external working condition data used to characterize the working environment and equipment operating conditions of the personnel being tested. External working condition data can be obtained through environmental sensors and equipment status acquisition devices installed at the work site and on related equipment. It may include ambient temperature, ambient humidity, noise intensity, electromagnetic environment parameters, illuminance, equipment operating parameters, and equipment abnormal status information, and is used to characterize the external working conditions and potential risk environment of the personnel being tested.

[0028] Further technical solutions involve preprocessing the acquired modal time series data, including timestamp correction, resampling, sliding time window division, and standardization, as follows; Step 11: Correct the timestamps of different data sources for the acquired modal timing data based on a preset unified clock reference; After obtaining the timing data of each of the different modalities, this step sets the acquisition timestamps for each data record and corrects the timestamps of different data sources according to a preset unified clock reference, which can eliminate the time reference differences caused by clock deviations between different acquisition devices.

[0029] Step 12: Resample the data at different sampling frequencies according to the preset target sampling frequency after the timestamp correction. After completing the timestamp correction, data with different sampling frequencies are resampled according to the preset target sampling frequency. Specifically, for data with a sampling frequency higher than the target sampling frequency, downsampling can be used to obtain the target sampling point; for data with a sampling frequency lower than the target sampling frequency, interpolation can be used to supplement the data value corresponding to the target sampling time, so that different data sources form corresponding data sequences on a unified time axis.

[0030] Step 13: Divide the resampled modal time series data into sliding time windows according to the preset time window length and sliding step size, so that the same time window contains endogenous state data, behavioral state data and external working condition data respectively. Let the three types of modes be denoted as follows: The corresponding original time series data is: ; A unified timeline is established based on the timestamps carried by each modality's data, with preset time intervals. Complete resampling or interpolation, and adjust according to the window length. and window step size Forming the first A sliding time window .

[0031] Step 14: Standardize the data of different modes within each time window, and then standardize the data within each mode. This step standardizes the data from different data sources within each time window to reduce the impact of differences in units, numerical ranges, and acquisition devices between different data sources.

[0032] Optionally, standardization can be performed using zero-mean unit variance standardization, min-max normalization, or other methods that can unify the data scale.

[0033] To reduce the numerical scale differences between different channels within the same mode, a standardization process is performed within each mode, using the following formula: ; in, and They represent the first The mean and standard deviation of the modality within a preset time window, To prevent the default positive number with a denominator of zero. This represents the data of the m-th mode after standardization at the n-th sampling point; This represents the original data of the m-th mode at the n-th sampling point; Timestamp correction and resampling are used to complete the initial synchronization of the physical time base, while the sliding time window is used to construct the continuous temporal feature sequence required for subsequent feature-level time alignment. After the above timestamp correction, resampling, sliding time window division, and standardization processes, the endogenous state data, behavioral state data, and external operating condition data with different original sampling frequencies and acquisition periods are transformed into three types of modal data based on the same time axis, with corresponding time windows and uniform data scale processing, providing input for subsequent modal feature extraction and cross-modal time delay alignment.

[0034] In step 2, features are extracted from the preprocessed temporal data of endogenous modalities, behavioral modalities, and exogenous modalities, respectively. The extracted modal features are then mapped to a unified task semantic space to obtain semantic features of endogenous modalities, behavioral modalities, and exogenous modalities. This includes the following steps: Step 21: Extract features from the time-series data of endogenous modalities, behavioral modalities, and exogenous modalities respectively using the corresponding feature encoders to obtain the corresponding modal features; Specifically, corresponding feature encoders are constructed for endogenous modalities, behavioral modalities, and exogenous modalities to extract features. Feature encoders are set separately for endogenous modalities, behavioral modalities, and exogenous modalities. , , Then the first The modal features corresponding to each time window are represented as follows: ; in, This represents the time series data of the m-th modality within the k-th time window. This represents the feature encoder corresponding to the m-th modality. This represents the mode-specific features extracted by the feature encoder. , , These represent endogenous modalities, behavioral modalities, and exogenous modalities, respectively.

[0035] It is feasible to use one-dimensional convolutional networks, recurrent neural networks, temporal Transformers, or coding networks based on time-frequency features to extract features from physiological time-series signals such as EEG, ECG, EMG, or EEG for endogenous modalities. For behavioral modalities, convolutional neural networks, pose coding networks, temporal motion coding networks, or inertial data coding networks can be used to extract features from behavioral data such as human posture, motion trajectory, joint position, acceleration, or angular velocity. For exogenous modalities, multilayer perceptrons, temporal coding networks, or other feature extraction networks can be used to extract features from environmental parameters and equipment operating parameters.

[0036] Different modalities can be configured with feature encoders of different structures according to their data format and temporal characteristics, so as to preserve the information expression ability of each modality.

[0037] Step 22: Set up corresponding semantic mapping networks for endogenous modalities, behavioral modalities and exogenous modalities respectively, and map the features of each modality to a unified task semantic space to obtain modal semantic features of a unified dimension.

[0038] Since endogenous modalities, behavioral modalities, and exogenous modalities have different data sources and physical meanings, the modal features extracted by the corresponding feature encoders differ in feature dimensions and semantic expressions, and cannot be directly used for cross-modal similarity comparisons.

[0039] Specifically, corresponding semantic mapping networks are set up for endogenous modalities, behavioral modalities, and exogenous modalities. , , The features of different modalities are mapped to a task semantic space of a predefined uniform dimension d, as expressed by the formula: ; in, This represents the semantic mapping network corresponding to the m-th modality. This represents the semantic features of the m-th modality corresponding to the k-th time window, including endogenous modality features. Behavioral modal characteristics and exogenous modal features ;d represents the preset feature dimension of the unified task semantic space.

[0040] A unified task semantic space is used to represent common high-level semantics related to the status of the personnel being tested or the risks of the operation. This allows different modal data to be transformed into feature representations with unified dimensions and comparable semantics while retaining their respective information expression capabilities, rather than directly comparing the original physical quantities corresponding to different modalities.

[0041] It is feasible; semantic mapping networks can employ fully connected networks, multilayer perceptrons, projection layers, or other network structures capable of feature dimension transformation and semantic projection.

[0042] Step 23: Perform L2 normalization on the semantic features of each modality after semantic mapping to obtain the normalized semantic features of each modality, including endogenous modality features, behavioral modality features and exogenous semantic features; To reduce the impact of differences in the magnitudes of semantic feature vectors from different modalities on subsequent cosine similarity calculations, L2 normalization is applied to the endogenous modal semantic features, behavioral modal semantic features, and exogenous modal semantic features obtained in step 22, as expressed by the following formula: ; in, This represents the m-th normalized modal semantic feature corresponding to the k-th time window; Representing modal semantic features The L2 norm; It is a preset positive number to avoid the denominator being zero.

[0043] After L2 normalization, the vector lengths of each modality semantic feature are constrained to be basically consistent, so that when calculating the cosine similarity between different modality semantic features, the cosine similarity mainly reflects the consistency between the directions of different modality features, thereby reducing the impact of feature amplitude differences.

[0044] Steps 21 to 23 enable the mapping of modal features with different data formats, feature dimensions, and physical meanings to a unified task semantic space while preserving the feature representation capabilities of endogenous, behavioral, and exogenous modalities. L2 normalization reduces the impact of differences in feature amplitudes between different modalities on similarity calculation, thereby improving the comparability between features of different modalities. This allows subsequent cross-modal cosine similarity to more accurately reflect the consistency of different modalities in representing the same person's state or operational risk, and provides a reliable feature basis for cross-modal time offset search and fine time alignment.

[0045] Furthermore, during the model training phase, personnel status labels or job risk labels can be used to jointly train the feature encoder and semantic mapping network, so that the semantic features of different modalities corresponding to the same personnel status or risk status have similar feature representations, thereby enhancing the comparability of different modalities in a unified task semantic space. During training, the feature encoder and semantic mapping network can be jointly optimized using task loss and cross-modal semantic alignment loss. The loss function is expressed as: ; ; in, To pre-determine the monitoring losses of downstream tasks, the downstream tasks can be personnel status identification or operational risk identification; The semantic alignment loss for different modalities under the same task label. is the balancing coefficient, a preset balancing coefficient used to adjust the relative weights of task loss and cross-modal semantic alignment loss. and This represents the semantic features corresponding to different modalities.

[0046] Through the joint training described above, the semantic features of different modalities under the same personnel state or risk state have more similar feature directions in the unified task semantic space, thereby improving the comparability between semantic features of different modalities and providing a feature basis for subsequent cross-modal cosine similarity calculation and time delay alignment.

[0047] Step 3: Determine the temporal offset of behavioral modalities and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity, and perform feature-level temporal alignment of modal semantic features of different modalities according to the temporal offset, including the following steps: Step 31: Using the endogenous mode as the time reference, set candidate time offset ranges for both the behavioral mode and the exogenous mode; Step 32: Calculate the average cosine similarity between the corresponding mode and the endogenous mode within the local time range under different candidate time offset ranges; Step 33: Based on the mean cosine similarity corresponding to different candidate time offsets, determine the candidate time offset corresponding to the one with the largest mean cosine similarity as the target time offset. Step 34: Adjust the temporal position of the feature sequences of the behavioral modality and the external modality according to the target time offset to obtain the temporally aligned semantic features of each modality.

[0048] In step 31, candidate time offset sets are set for both behavioral and exogenous modes. Using the endogenous mode as the time reference, candidate time offset sets are set for the behavioral mode. Set a candidate time offset set for the external mode. .

[0049] The candidate time offset set can be predetermined based on the sampling period, data processing latency, and response latency relative to changes in personnel state for different modalities. Different candidate time offset ranges can be set for behavioral modalities and external modalities to accommodate the latency differences in the acquisition and processing and state response processes of different modalities.

[0050] In step 32, for each candidate time offset range, the average cosine similarity between the corresponding mode and the endogenous mode within the candidate time offset range is calculated; for the th A time window and any candidate time offset In the first Within a local time range centered on a time window, calculate the mean cosine similarity between endogenous modes and behavioral or exogenous modes; Specifically, for the first Time windows and candidate time offsets In the local area centered Intrinsic computation of endogenous modes and the first Mean cosine similarity of class modes: ; in, This indicates that in the k-th time window, the candidate time offset is... At that time, the local mean cosine similarity between the endogenous mode and the m-th type mode; I represents the endogenous mode; m represents the target mode that is time-aligned with the endogenous mode; B represents the behavioral modality; E represents the external mode; k represents the sequence number of the current sliding time window; This represents the candidate time offset currently being calculated; This indicates the half-window length of the preset local search window; This represents the local window offset index centered at the k-th time window, where r∈[-H,H]; Indicates the first Normalized endogenous modal semantic features corresponding to each time window; Indicates the time offset of the candidate Then, the normalized modal semantic features corresponding to the m-th modality; This represents the vector dot product operation.

[0051] Since step 23 has already performed L2 normalization on the semantic features of each modality, therefore, and The vector dot product is used to characterize the cosine similarity between corresponding semantic features of two modalities.

[0052] In step 33, the target time offset is determined based on the mean cosine similarity corresponding to different candidate time offsets.

[0053] For both behavioral and exogenous modalities, the mean cosine similarity of each candidate time offset in the corresponding candidate time offset set is compared. The candidate time offset with the highest mean cosine similarity is determined as the target time offset for the current time window. The formula is as follows: ; in, This represents the target time offset of the m-th mode relative to the endogenous mode within the k-th time window. This represents the set of candidate time offsets between the endogenous mode and the m-th mode; when m = B or E; In step 34, the feature sequences of the behavioral modality and the exogenous modality are time-aligned according to the target time offset, while keeping the time position of the endogenous modality features, which serve as the time reference, unchanged. Then, the semantic feature sequences of the behavioral modality and the exogenous modality are shifted according to their respective target time offsets to obtain the time-aligned modal semantic features. , and ,in: ; ; ; in, This represents the endogenous modal semantic features after time alignment for the k-th time window; This represents the behavioral modal semantic features after time alignment for the k-th time window; This represents the semantic features of the external modality after time alignment in the k-th time window; Represents the endogenous modal semantic features of the k-th time window; Sequence of semantic features representing behavioral modalities; Sequence of semantic features representing exogenous modalities; This represents the target time offset of the sub-mode relative to the endogenous mode in the k-th time window; This represents the target time offset of the exogenous mode relative to the endogenous mode in the k-th time window; This indicates an alignment operation that retrieves aligned features from the corresponding modal feature sequence according to the target time offset.

[0054] Furthermore, the feature sequences of the behavioral modality and the exogenous modality are shifted according to the target time offset. When the target time offset corresponds to a non-integer sampling position, linear interpolation or higher-order interpolation is used to obtain the alignment features of the corresponding position, thereby obtaining the behavioral modality semantic features and exogenous modality semantic features that correspond to the endogenous modality in time.

[0055] Through the above processing, based on the initial time synchronization of different modal data achieved by timestamp correction and resampling in step 1, the response delay of behavioral modalities and exogenous modalities relative to endogenous modalities is further corrected according to the semantic consistency of different modalities with the same person's state. This reduces the time misalignment caused by acquisition and processing delays and state response delays of different modal features, and makes the endogenous modalities, behavioral modalities and exogenous modal features participating in subsequent fusion have a more accurate time correspondence.

[0056] In step 4, the pairwise cosine similarity between the time-aligned endogenous modalities, behavioral modalities, and exogenous modalities is first calculated, and the corresponding cross-modal consistency parameters are obtained based on the average similarity between each modality and the other modalities. Furthermore, data quality parameters are determined based on the integrity, continuity, and stability of the data of each modality. The cross-modal consistency parameters and data quality parameters are combined to obtain the credibility score of each modality, and the credibility score of each modality is converted into dynamic fusion weights through the Softmax function.

[0057] In step 4, based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality, the dynamic fusion weights of the semantic features of each modality are calculated, including the following steps: Step 41: Calculate the cosine similarity between the semantic features of each modality after time alignment, map it to a set interval, and calculate the cross-modal consistency parameter of each modality; The calculation process for the cross-modal consistency parameter involves, specifically, calculating the pairwise cosine similarity between the time-aligned endogenous modality, behavioral modality, and exogenous modality for the k-th time window, and mapping the cosine similarity from the interval [-1,1] to the interval [0,1]. The formula is as follows: ; in, This represents the normalized cross-modal cosine similarity between the m-th modality and the n-th modality within the k-th time window; m and n represent the category identifiers of different modalities. This represents the temporally aligned semantic features of the m-th modality within the k-th time window obtained in step 3. Represents the temporally aligned semantic features of the nth modality; Furthermore, based on the similarity between each mode and the other two modes, the cross-modal consistency parameter of the corresponding mode is calculated, and the calculation formula is as follows: ; ; ; in, This represents the cross-modal consistency parameter of the endogenous modes within the k-th time window; The cross-modal consistency parameter represents the behavioral modality within the k-th time window; This represents the cross-modal consistency parameter between internal and external modes within the k-th time window; This represents the normalized cosine similarity between endogenous modalities and behavioral modalities. This represents the normalized cosine similarity between endogenous and exogenous modes. It represents the normalized cosine similarity between the behavioral modality and the external modality.

[0058] Step 42: Calculate the corresponding data quality parameters based on the integrity, continuity, and stability of each modal data; For each modal of data within the k-th time window, determine the data integrity index, data continuity index, and data stability index, and calculate the corresponding data quality parameters for each modality: ; in, This represents the data quality parameter for the m-th modality within the k-th time window; , , The coefficients are preset to be nonnegative and their sum is 1; The data integrity metric representing the m-th modality: ; in, This represents the number of actually valid data locations within the current window. The number of data locations that should be present in the current window according to the modal sampling or update rule, and which is greater than zero.

[0059] The data continuity index representing the m-th modality is calculated based on the length of the continuous valid data segment: ; in, This represents the number of consecutive valid data segments within the current window. The sequence number of the consecutive valid data segment; For the first The number of consecutive valid data locations contained in a segment; when all data is missing, this indicator is 0.

[0060] The data stability index representing the m-th modality divides the current window into several sub-time periods. First, it calculates the quality score for each period, then simultaneously considers the average quality and quality fluctuation. ; in, Indicates the first Data quality score for each sub-time period; This represents the average quality score for each sub-time period; The standard deviation of the quality score for each sub-time period is represented by the overall standard deviation. The basic quality scores for different modes can be determined as follows: Endogenous modes are calculated by subtracting the proportion of artifact contamination from 1, based on pre-determined artifact detection rules; behavioral modes are calculated by subtracting the proportion of abnormal observations from 1, based on detection rules such as sensor saturation and data corruption; and exogenous modes are calculated by subtracting the proportion of invalid observations from 1, based on sensor fault signs and invalid readings.

[0061] Step 43: Weight the cross-modal consistency parameters and data quality parameters, calculate the credibility score for each modality, and calculate the fusion weight for each modality based on the credibility score; Specifically, the cross-modal consistency parameters and data quality parameters of the m-th modality within the k-th time window are weighted and combined to obtain the credibility score of the corresponding modality: ; in, .

[0062] This represents the credibility score of the m-th modality within the k-th time window; This represents the cross-modal consistency parameter for the m-th mode; This represents the data quality parameter for the m-th modality; This represents the preset weighting coefficients corresponding to the cross-modal consistency parameters; This represents the preset weighting coefficients corresponding to the data quality parameters; Furthermore, the credibility scores of the three modalities are normalized using the Softmax function to obtain the dynamic fusion weights corresponding to the k-th time window: ; in, This represents the dynamic fusion weight of the m-th modality within the k-th time window; This represents the credibility score of the m-th modality; This represents the confidence score of the j-th mode during the summation process; j represents the modality index during the Softmax normalization process; T represents the preset Softmax temperature parameter; The dynamic fusion weights satisfy: ; Therefore, when a modality has a high degree of semantic consistency with other modalities and its own data quality is good, the modality can obtain a relatively large dynamic fusion weight; when a modality is affected by missing data, abnormal interference, or inconsistent state representation, its dynamic fusion weight will decrease accordingly.

[0063] A further technical solution, to reduce weight abrupt changes caused by accidental fluctuations in adjacent time windows, is to perform exponential smoothing on the dynamic fusion weights obtained in step 43 to obtain the final fusion weights. : ; in, The preset fusion coefficient, .

[0064] Through the above processing, the dynamic fusion weights are no longer determined solely by fixed parameters or the attention within a single model. Instead, they simultaneously reflect the consistency of state representations among different modalities within the current time window, as well as the data validity of each modality itself. This reduces the impact of abnormal, missing, or low-quality modalities on the fusion results, enabling the fusion weights to be dynamically adjusted according to changes in personnel status, working environment, and sensor data quality. This provides a more reliable weight basis for subsequent multimodal feature fusion and personnel risk identification.

[0065] Step 5: Perform weighted fusion of the time-aligned semantic features according to the dynamic fusion weight to obtain multimodal fusion features, and identify the status or operational risks of the personnel to be detected based on the multimodal fusion features; Specifically, based on the dynamic fusion weights of each modality obtained in step 4, the time-aligned endogenous modality semantic features, behavioral modality semantic features, and exogenous modality semantic features obtained in step 3 are weighted and summed to obtain the multimodal fusion features corresponding to the k-th time window: ; in, This represents the multimodal fusion feature corresponding to the k-th time window; This represents the dynamic fusion weight of endogenous modes within the k-th time window; This represents the dynamic fusion weights of behavioral modalities within the k-th time window; This represents the dynamic fusion weights of internal and external modalities in the k-th time window; This represents the time-aligned endogenous modality semantic features corresponding to the k-th time window; This represents the time-aligned behavioral modal semantic features corresponding to the k-th time window; This represents the time-aligned external modal semantic features corresponding to the k-th time window.

[0066] Optionally, when step 4 uses exponential smoothing to smooth the dynamic fusion weights over time, the smoothed dynamic fusion weights are used for feature fusion. Furthermore, the obtained multimodal fusion features Input a preset personnel status recognition model or operation risk recognition model to determine the personnel status or operation risk status corresponding to the current time window.

[0067] It is feasible that the personnel status or operational risk may include normal status, fatigue status, abnormal cognitive load status, abnormal action status, and / or high-risk operational status, and the specific identification category can be set according to the actual application task.

[0068] Through the above processing, the effective contributions of different modalities to the current personnel status are dynamically fused according to their cross-modal consistency and their own data quality. This allows modalities with higher data quality and greater consistency with other modal status representations to make a higher contribution in the fusion result, thereby reducing the impact of abnormal, missing, or low-quality modalities on personnel status representation and improving the accuracy and stability of personnel status perception and operational risk identification in complex work environments.

[0069] Therefore, the above-mentioned scheme in this embodiment can establish comparable relationships between different modalities within a unified semantic space, and reduce feature misalignment caused by asynchronous cross-modal state responses through time-delay-aware alignment. At the same time, it dynamically adjusts the contribution of each modality to the fusion result by combining cross-modal consistency and the data quality of the modality itself, thereby reducing the impact of abnormal, missing or low-quality modalities and improving the accuracy and stability of personnel state representation and operational risk identification in complex working environments.

[0070] Example 2 Based on Embodiment 1, this embodiment provides a specific implementation example to illustrate the implementation process and corresponding effects of the method in Embodiment 1; First, the three types of modal data are preprocessed using a unified time base. Then, modality-specific feature extraction and unified task semantic mapping are performed respectively. Based on this, the target time offsets of behavioral modalities and exogenous modalities are searched using the endogenous modality as the time base to achieve fine-grained time alignment at the feature level. Subsequently, the cross-modal cosine consistency and data quality of the three types of modalities are calculated to obtain dynamic fusion weights. Finally, the aligned features of the three types of modalities are weighted and fused.

[0071] Step 1: Preprocess the three types of modal time series data.

[0072] In this embodiment, the original sampling rate of the EEG signal is 500Hz, the sampling rate of the inertial measurement data is 100Hz, the human posture data is generated according to the video frame rate, and the update cycle of the external environment and device status data is 1 second to several seconds.

[0073] First, the system clock deviation is corrected based on the acquisition timestamps carried by each data point, with 0.5s as the uniform characteristic time step. For external data that is updated slowly, a data sequence corresponding to a unified time axis is established using either preserve or linear interpolation methods.

[0074] Then, the sliding time window was divided according to a 2-second window length and a 0.5-second window step. Each type of signal retains its original or pre-processed sampling structure within the same window to avoid directly downsampling high-frequency EEG signals to the sampling rate of low-frequency environmental data.

[0075] Select the appropriate preprocessing method based on the characteristics of the data: The EEG signals are processed within the window, including baseline removal, power frequency notch filtering, and standardization. Behavioral data is normalized to coordinates and missing frames are filled in. External data undergoes outlier truncation and standardization.

[0076] Step 2: Extract modal features and perform unified task semantic mapping.

[0077] Regarding the first Each time window is used to input the EEG window, behavior window, and external source window into the corresponding feature encoder.

[0078] like Figure 2 As shown, in this embodiment, the endogenous modal encoder uses a combination of one-dimensional convolution and Transformer to extract EEG time-frequency and temporal features; the behavioral modal encoder uses a temporal Transformer to encode the inertial measurement sequence and attitude key point sequence; and the exogenous modal encoder uses a multilayer perceptron to encode the environmental and device state sequence.

[0079] The three types of encoders can output different feature dimensions, which are then uniformly mapped to a semantic mapping network through a fully connected layer. Dimensional task semantic space: ; In this embodiment, .

[0080] Step 3: Calculate the target time offset and complete fine time alignment.

[0081] Considering the typically short response delay between changes in internal EEG states and overt behavioral manifestations, this embodiment sets the candidate time offset range of behavioral modalities relative to endogenous modalities to... .

[0082] Considering the potential longer-term impacts between environmental factors, personnel status, and risk characterization, the candidate time offset range of the exogenous mode relative to the endogenous mode is set to... .

[0083] After discretization using a 0.5s characteristic time step, candidate time offset sets are obtained respectively: The above range can be adjusted according to actual sensor latency and business scenarios.

[0084] In the A time window is set to a preset half-window length for the local search window. Construct a local comparison range, that is, use the current window and the windows before and after it. In this embodiment, the mean cosine similarity under the candidate offset is calculated within a time window. .

[0085] For each behavioral modality, calculate individually: time Take the maximum value to obtain the target time offset of the mode relative to the endogenous mode in the k-th time window. ; For external modes, calculate them one by one: time Take the maximum value to obtain the target time offset of the exogenous mode relative to the endogenous mode in the k-th time window. When the target time offset or When corresponding to non-integer physical time locations, the corresponding alignment features are obtained through linear interpolation of adjacent semantic features. After this processing, the three types of features are in the [missing information - likely a specific position or location]. The time windows are no longer based solely on the correspondence of the same absolute timestamp, but rather on establishing fine-grained time relationships based on the maximum state consistency in a unified semantic space.

[0086] Step 4: Calculate cross-modal consistency, data quality, and dynamic fusion weights.

[0087] like Figure 3 As shown, using time alignment , , Calculate the similarity between the modal data separately: ; Based on the similarity between each mode and the other two modes, the cross-modal consistency parameter of the corresponding mode is calculated, resulting in: ; In this embodiment, for each modal data within the k-th time window, data integrity, data continuity, and data stability indices are determined, and corresponding data quality parameters are calculated. Each data quality parameter consists of three normalized indices with coefficients set as follows: ; ; ; Among them, EEG modalities Using the effective channel ratio, Use the effective sampling ratio within the window. Determined based on normalized signal-to-noise ratio and artifact ratio; behavioral modality Employ the proportion of effective attitude key points or effective inertial channels. Use effective frame rate or data continuity rate. The attitude tracking confidence level or inertial data anomaly rate is used for determination; external modalities... Using the effective sensor ratio, Using data update completeness rate, The validity of the sensor status and the proportion of outliers within the window are used to determine this.

[0088] In this embodiment, the cross-modal consistency parameters and data quality parameters of the m-th modality within the k-th time window are weighted and combined to obtain the credibility score of the corresponding modality. The modal credibility calculation uses preset weight coefficients as follows: ; ; Cross-modal consistency is taken as the primary criterion, while the weights are adjusted based on data quality; In this embodiment, the Softmax temperature parameter is: ; When a modality has a high degree of consistency with other modalities in a unified semantic space and its own data quality is good, its Increase the fusion weight to obtain a higher fusion weight; when the data quality of a certain modality is reduced due to noise, packet loss, or tracking instability, its fusion weight will still be affected even if it has some similarity to other modalities. The inhibition of decline.

[0089] For scenarios requiring reduced short-term weight fluctuations, this embodiment uses a preset fusion coefficient. : ; Perform exponential smoothing on the weights.

[0090] Step 5: Based on the dynamic fusion weights obtained in Step 4, adjust the weights for the first... The three types of aligned semantic features within each time window are weighted and summed to form a 128-dimensional multimodal fusion feature. ; The fusion features simultaneously include information on the person's internal physiological state, outward behavioral performance, and external working conditions, and the main feature-level time misalignment has been eliminated through step 3.

[0091] In practical applications, continuous time windows can be... Input state classifiers, risk assessment networks, or other time-series decision models for worker state identification, cognitive load assessment, fatigue analysis, and job risk assessment.

[0092] The specific downstream models mentioned above can be selected according to the application task, without affecting the time alignment and dynamic feature fusion process described in this invention.

[0093] In another embodiment, the endogenous modality may simultaneously include multiple signals from EEG, ECG, EMG, EEG, or TENS, and the fusion of similar physiological information is first completed within the endogenous modality encoder; Behavioral modalities can be derived from one or more of the following: video pose, motion trajectory, and inertial measurement data. External modalities can be one or more of environmental parameters, equipment status, and operational process parameters.

[0094] As long as the three modalities obtain the temporal features in the unified task semantic space through the corresponding encoder and semantic mapping network, the time delay-aware alignment and dynamic weight fusion can be completed according to steps 3 to 5 of this invention.

[0095] This embodiment does not limit the number of network layers, hidden layer dimensions, or training optimizer of the specific feature encoder; the above structure can be adjusted according to the data scale and computing resources.

[0096] In this embodiment, the state includes three behavioral states of the test subject: standing, walking, and climbing stairs. By jointly utilizing endogenous physiological modalities and behavioral modalities, the activity state of the test subject can be identified, and exogenous modalities can be introduced to supplement the characterization of the external working conditions of the area where the test subject is located.

[0097] The endogenous physiological modality refers to information used to characterize the internal physiological activities of the subject. In this embodiment, an electrocardiogram (ECG) signal is used as the input to this modality. The ECG signal reflects the electrical activity of the heart and can provide information for analyzing the physiological responses of the subject under different activity states.

[0098] The behavioral modality refers to information used to characterize the overt actions and body movements of the test subject. In this embodiment, triaxial acceleration signals collected from the chest and ankle positions are used as inputs. Chest acceleration primarily reflects trunk movement and overall body movement characteristics, while ankle acceleration primarily reflects lower limb movement and gait changes. Behavioral information from different acquisition locations complements each other and, together with endogenous physiological information, is used to characterize the subject's state.

[0099] The exogenous modality is information used to characterize the external working conditions of the tested personnel. It can be obtained by collecting the temperature, humidity and illuminance of the working area where the tested personnel are located, and using them as the input of the exogenous modality to provide environmental information for personnel status perception.

[0100] In the data processing, the endogenous physiological modality, behavioral modality, and exogenous modality data are first timestamped. Features are extracted from electrocardiogram signals, triaxial acceleration signals at the chest and ankle positions, and temperature, humidity, and illuminance data of the work area. These modal features are then mapped to a unified task semantic space to establish comparable relationships between features from different sources. Based on this, the temporal offset between modalities is determined through feature similarity search within a local time range, achieving temporal delay alignment of cross-modal features. Furthermore, the consistency of aligned features and the data quality of each modality are combined to calculate fusion weights, and the features of each modality are dynamically weighted and fused to obtain fusion features that comprehensively represent the internal physiological activities, overt behaviors, and external working conditions of the test subjects. These features are used for the identification of three behavioral states: standing, walking, and climbing stairs.

[0101] Figure 4 The image shows the confusion matrix of the three behavioral state recognition results in this embodiment, where the vertical axis represents the true state and the horizontal axis represents the predicted state. Figure 4 As can be seen, both standing and walking states were correctly identified, and only a small number of cases of climbing stairs were misidentified as walking, with an overall recognition accuracy of 98.6%. These results demonstrate that, under the test conditions of this embodiment, the multimodal fusion recognition method employed can effectively extract and fuse state representation information from different sources, accurately distinguishing the actual state of the tested personnel and providing a basis for personnel state perception.

[0102] Example 3 Based on Embodiment 1, this embodiment provides a personnel state perception system based on time-delay-aware multimodal feature fusion, including: The multimodal time series data acquisition module is configured to acquire time series data of endogenous modalities, behavioral modalities, and exogenous modalities for identifying the state of the person to be detected, and to perform preprocessing. The feature extraction and mapping module is configured to extract features from the preprocessed endogenous modality, behavioral modality and exogenous modality time series data respectively, and map the extracted different modality features to a unified task semantic space to obtain endogenous modality semantic features, behavioral modality semantic features and exogenous modality semantic features; The time alignment module is configured to determine the time offset of behavioral modalities and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity, and to perform feature-level time alignment of modal semantic features of different modalities according to the time offset. The weight calculation module is configured to calculate the dynamic fusion weights of each modality's semantic features based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality's data. The fusion module is configured to perform weighted fusion of the time-aligned semantic features of each modality according to the dynamic fusion weight to obtain multimodal fusion features, and to identify the status or operational risks of the personnel to be detected based on the multimodal fusion features.

[0103] In some embodiments, the multimodal time series data acquisition module is configured to acquire time series data of the endogenous modality, behavioral modality and exogenous modality of the person to be detected; Endogenous modal time series data are endogenous state data used to characterize the internal physiological and cognitive states of the person being tested. They can be acquired by physiological signal acquisition devices set in the corresponding parts of the person being tested's body, and may include electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, electromyogram (EMG) signals, and electrooculogram (EOG) signals.

[0104] It is feasible to acquire the EEG signals of the person being tested through EEG acquisition equipment and the ECG signals through ECG acquisition equipment to reflect changes in the person's neural activity, physiological load, or cognitive state.

[0105] Behavioral modal time-series data refers to behavioral state data used to characterize the overt actions and operational behaviors of the person being tested. Behavioral modal time-series data can be acquired through image acquisition devices and / or inertial measurement devices, and may include human posture data, limb movement data, motion trajectory data, movement speed data, joint position data, and inertial measurement data, etc.

[0106] The feasible technical solution is that the image acquisition device can be a camera or a depth camera, used to continuously acquire images of the person being inspected, and obtain behavior-related time-series data through human key point detection, posture estimation or action recognition; the inertial measurement device can be set on the body of the person being inspected or the work equipment, used to acquire acceleration, angular velocity and posture change data.

[0107] External modal time series data refers to external working condition data used to characterize the working environment and equipment operating conditions of the personnel being tested. External working condition data can be obtained through environmental sensors and equipment status acquisition devices installed at the work site and on related equipment. It may include ambient temperature, ambient humidity, noise intensity, electromagnetic environment parameters, illuminance, equipment operating parameters, and equipment abnormal status information, and is used to characterize the external working conditions and potential risk environment of the personnel being tested.

[0108] Further technical solutions involve preprocessing the acquired modal time series data, including timestamp correction, resampling, sliding time window division, and standardization, as follows; Step 11: Correct the timestamps of different data sources for the acquired modal timing data based on a preset unified clock reference; After obtaining the timing data of each of the different modalities, this step sets the acquisition timestamps for each data record and corrects the timestamps of different data sources according to a preset unified clock reference, which can eliminate the time reference differences caused by clock deviations between different acquisition devices.

[0109] Step 12: Resample the data at different sampling frequencies according to the preset target sampling frequency after the timestamp correction. After completing the timestamp correction, data with different sampling frequencies are resampled according to the preset target sampling frequency. Specifically, for data with a sampling frequency higher than the target sampling frequency, downsampling can be used to obtain the target sampling point; for data with a sampling frequency lower than the target sampling frequency, interpolation can be used to supplement the data value corresponding to the target sampling time, so that different data sources form corresponding data sequences on a unified time axis.

[0110] Step 13: Divide the resampled modal time series data into sliding time windows according to the preset time window length and sliding step size, so that the same time window contains endogenous state data, behavioral state data and external working condition data respectively. Let the three types of modes be denoted as follows: The corresponding original time series data is: ; A unified timeline is established based on the timestamps carried by each modality's data, with preset time intervals. Complete resampling or interpolation, and adjust according to the window length. and window step size Forming the first A sliding time window .

[0111] Step 14: Standardize the data of different modes within each time window, and then standardize the data within each mode. This step standardizes the data from different data sources within each time window to reduce the impact of differences in units, numerical ranges, and acquisition devices between different data sources.

[0112] Optionally, standardization can be performed using zero-mean unit variance standardization, min-max normalization, or other methods that can unify the data scale.

[0113] To reduce the numerical scale differences between different channels within the same mode, a standardization process is performed within each mode, using the following formula: ; in, and They represent the first The mean and standard deviation of the modality within a preset time window, To prevent the default positive number with a denominator of zero. This represents the data of the m-th mode after standardization at the n-th sampling point; This represents the original data of the m-th mode at the n-th sampling point; Timestamp correction and resampling are used to complete the initial synchronization of the physical time base, while the sliding time window is used to construct the continuous temporal feature sequence required for subsequent feature-level time alignment. After the above timestamp correction, resampling, sliding time window division, and standardization processes, the endogenous state data, behavioral state data, and external operating condition data with different original sampling frequencies and acquisition periods are transformed into three types of modal data based on the same time axis, with corresponding time windows and uniform data scale processing, providing input for subsequent modal feature extraction and cross-modal time delay alignment.

[0114] In some embodiments, the feature extraction and mapping module is configured to perform the following steps: Step 21: Extract features from the time-series data of endogenous modalities, behavioral modalities, and exogenous modalities respectively using the corresponding feature encoders to obtain the corresponding modal features; Specifically, corresponding feature encoders are constructed for endogenous modalities, behavioral modalities, and exogenous modalities to extract features. Feature encoders are set separately for endogenous modalities, behavioral modalities, and exogenous modalities. , , Then the first The modal features corresponding to each time window are represented as follows: ; in, This represents the time series data of the m-th modality within the k-th time window. This represents the feature encoder corresponding to the m-th modality. This represents the mode-specific features extracted by the feature encoder. , , These represent endogenous modalities, behavioral modalities, and exogenous modalities, respectively.

[0115] Step 22: Set up corresponding semantic mapping networks for endogenous modalities, behavioral modalities and exogenous modalities respectively, and map the features of each modality to a unified task semantic space to obtain modal semantic features of a unified dimension.

[0116] Since endogenous modalities, behavioral modalities, and exogenous modalities have different data sources and physical meanings, the modal features extracted by the corresponding feature encoders differ in feature dimensions and semantic expressions, and cannot be directly used for cross-modal similarity comparisons.

[0117] Specifically, corresponding semantic mapping networks are set up for endogenous modalities, behavioral modalities, and exogenous modalities. , , The features of different modalities are mapped to a task semantic space of a predefined uniform dimension d, as expressed by the formula: ; in, This represents the semantic mapping network corresponding to the m-th modality. This represents the semantic features of the m-th modality corresponding to the k-th time window, including endogenous modality features. Behavioral modal characteristics and exogenous modal features ;d represents the preset feature dimension of the unified task semantic space.

[0118] A unified task semantic space is used to represent common high-level semantics related to the status of the personnel being tested or the risks of the operation. This allows different modal data to be transformed into feature representations with unified dimensions and comparable semantics while retaining their respective information expression capabilities, rather than directly comparing the original physical quantities corresponding to different modalities.

[0119] Step 23: Perform L2 normalization on the semantic features of each modality after semantic mapping to obtain the normalized semantic features of each modality, including endogenous modality features, behavioral modality features and exogenous semantic features; To reduce the impact of differences in the magnitudes of semantic feature vectors from different modalities on subsequent cosine similarity calculations, L2 normalization is applied to the endogenous modal semantic features, behavioral modal semantic features, and exogenous modal semantic features obtained in step 22, as expressed by the following formula: ; in, This represents the m-th normalized modal semantic feature corresponding to the k-th time window; Representing modal semantic features The L2 norm; It is a preset positive number to avoid the denominator being zero.

[0120] After L2 normalization, the vector lengths of each modality semantic feature are constrained to be basically consistent, so that when calculating the cosine similarity between different modality semantic features, the cosine similarity mainly reflects the consistency between the directions of different modality features, thereby reducing the impact of feature amplitude differences.

[0121] Steps 21 to 23 enable the mapping of modal features with different data formats, feature dimensions, and physical meanings to a unified task semantic space while preserving the feature representation capabilities of endogenous, behavioral, and exogenous modalities. L2 normalization reduces the impact of differences in feature amplitudes between different modalities on similarity calculation, thereby improving the comparability between features of different modalities. This allows subsequent cross-modal cosine similarity to more accurately reflect the consistency of different modalities in representing the same person's state or operational risk, and provides a reliable feature basis for cross-modal time offset search and fine time alignment.

[0122] A further technical solution, the time alignment module, is configured to perform the following steps: Step 31: Using the endogenous mode as the time reference, set candidate time offset ranges for both the behavioral mode and the exogenous mode; Step 32: Calculate the average cosine similarity between the corresponding mode and the endogenous mode within the local time range under different candidate time offset ranges; Step 33: Based on the mean cosine similarity corresponding to different candidate time offsets, determine the candidate time offset corresponding to the one with the largest mean cosine similarity as the target time offset. Step 34: Adjust the temporal position of the feature sequences of the behavioral modality and the external modality according to the target time offset to obtain the temporally aligned semantic features of each modality.

[0123] In step 31, candidate time offset sets are set for both behavioral and exogenous modes. Using the endogenous mode as the time reference, candidate time offset sets are set for the behavioral mode. Set a candidate time offset set for the external mode. .

[0124] The candidate time offset set can be predetermined based on the sampling period, data processing latency, and response latency relative to changes in personnel state for different modalities. Different candidate time offset ranges can be set for behavioral modalities and external modalities to accommodate the latency differences in the acquisition and processing and state response processes of different modalities.

[0125] In step 32, for each candidate time offset range, the average cosine similarity between the corresponding mode and the endogenous mode within the candidate time offset range is calculated; for the th A time window and any candidate time offset In the first Within a local time range centered on a time window, calculate the mean cosine similarity between endogenous modes and behavioral or exogenous modes; Specifically, for the first Time windows and candidate time offsets In the local area centered Intrinsic computation of endogenous modes and the first Mean cosine similarity of class modes: ; in, This indicates that in the k-th time window, the candidate time offset is... At that time, the local mean cosine similarity between the endogenous mode and the m-th type mode; I represents the endogenous mode; m represents the target mode that is time-aligned with the endogenous mode; B represents the behavioral modality; E represents the external mode; k represents the sequence number of the current sliding time window; This represents the candidate time offset currently being calculated; This indicates the half-window length of the preset local search window; This represents the local window offset index centered at the k-th time window, where r∈[-H,H]; Indicates the first Normalized endogenous modal semantic features corresponding to each time window; Indicates the time offset of the candidate Then, the normalized modal semantic features corresponding to the m-th modality; This represents the vector dot product operation.

[0126] Since step 23 has already performed L2 normalization on the semantic features of each modality, therefore, and The vector dot product is used to characterize the cosine similarity between corresponding semantic features of two modalities.

[0127] In step 33, the target time offset is determined based on the mean cosine similarity corresponding to different candidate time offsets.

[0128] For both behavioral and exogenous modalities, the mean cosine similarity of each candidate time offset in the corresponding candidate time offset set is compared. The candidate time offset with the highest mean cosine similarity is determined as the target time offset for the current time window. The formula is as follows: ; in, This represents the target time offset of the m-th mode relative to the endogenous mode within the k-th time window. This represents the set of candidate time offsets between the endogenous mode and the m-th mode; when m = B or E; In step 34, the feature sequences of the behavioral modality and the exogenous modality are time-aligned according to the target time offset, while keeping the time position of the endogenous modality features, which serve as the time reference, unchanged. Then, the semantic feature sequences of the behavioral modality and the exogenous modality are shifted according to their respective target time offsets to obtain the time-aligned modal semantic features. , and ,in: ; ; ; in, This represents the endogenous modal semantic features after time alignment for the k-th time window; This represents the behavioral modal semantic features after time alignment for the k-th time window; This represents the semantic features of the external modality after time alignment in the k-th time window; Represents the endogenous modal semantic features of the k-th time window; Sequence of semantic features representing behavioral modalities; Sequence of semantic features representing exogenous modalities; This represents the target time offset of the sub-mode relative to the endogenous mode in the k-th time window; This represents the target time offset of the exogenous mode relative to the endogenous mode in the k-th time window; This indicates an alignment operation that retrieves aligned features from the corresponding modal feature sequence according to the target time offset.

[0129] Furthermore, the feature sequences of the behavioral modality and the exogenous modality are shifted according to the target time offset. When the target time offset corresponds to a non-integer sampling position, linear interpolation or higher-order interpolation is used to obtain the alignment features of the corresponding position, thereby obtaining the behavioral modality semantic features and exogenous modality semantic features that correspond to the endogenous modality in time.

[0130] A further technical solution, the fusion module, is configured to perform the following steps: Step 41: Calculate the cosine similarity between the semantic features of each modality after time alignment, map it to a set interval, and calculate the cross-modal consistency parameter of each modality; The calculation process for the cross-modal consistency parameter involves, specifically, calculating the pairwise cosine similarity between the time-aligned endogenous modality, behavioral modality, and exogenous modality for the k-th time window, and mapping the cosine similarity from the interval [-1,1] to the interval [0,1]. The formula is as follows: ; in, This represents the normalized cross-modal cosine similarity between the m-th modality and the n-th modality within the k-th time window; m and n represent the category identifiers of different modalities. Represents the temporally aligned semantic features of the m-th modality within the k-th time window; Represents the temporally aligned semantic features of the nth modality; Furthermore, based on the similarity between each mode and the other two modes, the cross-modal consistency parameter of the corresponding mode is calculated, and the calculation formula is as follows: ; ; ; in, This represents the cross-modal consistency parameter of the endogenous modes within the k-th time window; The cross-modal consistency parameter represents the behavioral modality within the k-th time window; This represents the cross-modal consistency parameter between internal and external modes within the k-th time window; This represents the normalized cosine similarity between endogenous modalities and behavioral modalities. This represents the normalized cosine similarity between endogenous and exogenous modes. It represents the normalized cosine similarity between the behavioral modality and the external modality.

[0131] Step 42: Calculate the corresponding data quality parameters based on the integrity, continuity, and stability of each modal data; For each modal of data within the k-th time window, determine the data integrity index, data continuity index, and data stability index, and calculate the corresponding data quality parameters for each modality: ; in, This represents the data quality parameter for the m-th modality within the k-th time window; , , The coefficients are preset to be nonnegative and their sum is 1; The data integrity metric representing the m-th modality: ; in, This represents the number of actually valid data locations within the current window. The number of data locations that should be present in the current window according to the modal sampling or update rule, and which is greater than zero.

[0132] The data continuity index representing the m-th modality is calculated based on the length of the continuous valid data segment: ; in, This represents the number of consecutive valid data segments within the current window. The sequence number of the consecutive valid data segment; For the first The number of consecutive valid data locations contained in a segment; when all data is missing, this indicator is 0.

[0133] The data stability index representing the m-th modality divides the current window into several sub-time periods. First, it calculates the quality score for each period, then simultaneously considers the average quality and quality fluctuation. ; in, Indicates the first Data quality score for each sub-time period; This represents the average quality score for each sub-time period; The standard deviation of the quality score for each sub-time period is represented by the overall standard deviation. The basic quality scores for different modes can be determined as follows: Endogenous modes are calculated by subtracting the proportion of artifact contamination from 1, based on pre-determined artifact detection rules; behavioral modes are calculated by subtracting the proportion of abnormal observations from 1, based on detection rules such as sensor saturation and data corruption; and exogenous modes are calculated by subtracting the proportion of invalid observations from 1, based on sensor fault signs and invalid readings.

[0134] Step 43: Weight the cross-modal consistency parameters and data quality parameters, calculate the credibility score for each modality, and calculate the fusion weight for each modality based on the credibility score; Specifically, the cross-modal consistency parameters and data quality parameters of the m-th modality within the k-th time window are weighted and combined to obtain the credibility score of the corresponding modality: ; in, .

[0135] This represents the credibility score of the m-th modality within the k-th time window; This represents the cross-modal consistency parameter for the m-th mode; This represents the data quality parameter for the m-th modality; This represents the preset weighting coefficients corresponding to the cross-modal consistency parameters; This represents the preset weighting coefficients corresponding to the data quality parameters; Furthermore, the credibility scores of the three modalities are normalized using the Softmax function to obtain the dynamic fusion weights corresponding to the k-th time window: ; in, This represents the dynamic fusion weight of the m-th modality within the k-th time window; This represents the credibility score of the m-th modality; This represents the confidence score of the j-th mode during the summation process; j represents the modality index during the Softmax normalization process; T represents the preset Softmax temperature parameter; The dynamic fusion weights satisfy: ; Therefore, when a modality has a high degree of semantic consistency with other modalities and its own data quality is good, the modality can obtain a relatively large dynamic fusion weight; when a modality is affected by missing data, abnormal interference, or inconsistent state representation, its dynamic fusion weight will decrease accordingly.

[0136] A further technical solution, to reduce weight abrupt changes caused by accidental fluctuations between adjacent time windows, is to perform exponential smoothing on the obtained dynamic fusion weights to obtain the final fusion weights. : ; in, The preset fusion coefficient, .

[0137] Through the above processing, the dynamic fusion weights are no longer determined solely by fixed parameters or the attention within a single model. Instead, they simultaneously reflect the consistency of state representations among different modalities within the current time window, as well as the data validity of each modality itself. This reduces the impact of abnormal, missing, or low-quality modalities on the fusion results, enabling the fusion weights to be dynamically adjusted according to changes in personnel status, working environment, and sensor data quality. This provides a more reliable weight basis for subsequent multimodal feature fusion and personnel risk identification.

[0138] A further technical solution involves weighted fusion of the temporally aligned semantic features based on dynamic fusion weights to obtain multimodal fusion features, and then identifying the status or operational risks of the personnel to be detected based on these multimodal fusion features. Specifically, based on the dynamic fusion weights of each modality, the obtained time-aligned endogenous modality semantic features, behavioral modality semantic features, and exogenous modality semantic features are weighted and summed to obtain the multimodal fusion features corresponding to the k-th time window: ; in, This represents the multimodal fusion feature corresponding to the k-th time window; This represents the dynamic fusion weight of endogenous modes within the k-th time window; This represents the dynamic fusion weights of behavioral modalities within the k-th time window; This represents the dynamic fusion weights of internal and external modalities in the k-th time window; This represents the time-aligned endogenous modality semantic features corresponding to the k-th time window; This represents the time-aligned behavioral modal semantic features corresponding to the k-th time window; This represents the time-aligned external modal semantic features corresponding to the k-th time window.

[0139] Optionally, when using exponential smoothing to smooth the dynamic fusion weights over time, the smoothed dynamic fusion weights are used for feature fusion. Furthermore, the obtained multimodal fusion features Input a preset personnel status recognition model or operation risk recognition model to determine the personnel status or operation risk status corresponding to the current time window.

[0140] It is feasible that the personnel status or operational risk may include normal status, fatigue status, abnormal cognitive load status, abnormal action status, and / or high-risk operational status, and the specific identification category can be set according to the actual application task.

[0141] Through the above processing, the effective contributions of different modalities to the current personnel status are dynamically fused according to their cross-modal consistency and their own data quality. This allows modalities with higher data quality and greater consistency with other modal status representations to make a higher contribution in the fusion result, thereby reducing the impact of abnormal, missing, or low-quality modalities on personnel status representation and improving the accuracy and stability of personnel status perception and operational risk identification in complex work environments.

[0142] Therefore, the above-mentioned scheme in this embodiment can establish comparable relationships between different modalities within a unified semantic space, and reduce feature misalignment caused by asynchronous cross-modal state responses through time-delay-aware alignment. At the same time, it dynamically adjusts the contribution of each modality to the fusion result by combining cross-modal consistency and the data quality of the modality itself, thereby reducing the impact of abnormal, missing or low-quality modalities and improving the accuracy and stability of personnel state representation and operational risk identification in complex working environments.

[0143] Example 4 Based on Embodiment 1, this embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps in the personnel state perception method based on time-delay-aware multimodal feature fusion described in Embodiment 1.

[0144] Example 5 Based on Embodiment 1, this embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps in the personnel state perception method based on time-delay-aware multimodal feature fusion described in Embodiment 1.

[0145] The electronic devices proposed in this invention can be mobile terminals or non-mobile terminals. Non-mobile terminals include desktop computers, while mobile terminals include smartphones (such as Android phones, iOS phones, etc.), smart glasses, smartwatches, smart bracelets, tablet computers, laptops, personal digital assistants, and other mobile internet devices capable of wireless communication.

[0146] It should be understood that in this invention, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0147] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0148] In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here. Those skilled in the art will recognize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0149] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0150] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0151] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0153] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for personnel state perception based on time-delay-aware multimodal feature fusion, characterized in that, Includes the following steps: For the personnel to be tested, time-series data of endogenous modalities, behavioral modalities, and exogenous modalities used to identify the personnel's state are acquired and preprocessed. Feature extraction was performed on the preprocessed temporal data of endogenous modalities, behavioral modalities, and exogenous modalities, and the extracted modal features were mapped to a unified task semantic space to obtain endogenous modal semantic features, behavioral modal semantic features, and exogenous modal semantic features. Based on cross-modal cosine similarity, the temporal offset of behavioral modalities and exogenous modalities relative to endogenous modalities is determined, and feature-level temporal alignment of modal semantic features of different modalities is performed based on the temporal offset. Based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality, the dynamic fusion weight of each modality semantic feature is calculated; The time-aligned semantic features of each modality are weighted and fused according to the dynamic fusion weight to obtain multimodal fusion features, and the status or operational risks of the personnel to be detected are identified based on the multimodal fusion features.

2. The personnel state perception method based on time-delay-aware multimodal feature fusion as described in claim 1, characterized in that, Feature extraction is performed on the preprocessed time-series data of endogenous modalities, behavioral modalities, and exogenous modalities, respectively. The extracted modal features are then mapped to a unified task semantic space to obtain endogenous modal semantic features, behavioral modal semantic features, and exogenous modal semantic features. The process includes the following steps: The time series data of endogenous modalities, behavioral modalities, and exogenous modalities are respectively processed by the corresponding feature encoders to extract the corresponding modal features. Semantic mapping networks are set up for endogenous modalities, behavioral modalities and exogenous modalities respectively, and the features of each modality are mapped to a unified task semantic space to obtain modal semantic features of a unified dimension; L2 normalization is performed on the semantic features of each modality after semantic mapping to obtain normalized semantic features of each modality, including endogenous modality features, behavioral modality features and exogenous semantic features.

3. The personnel state perception method based on time-delay-aware multimodal feature fusion as described in claim 1, characterized in that, The feature encoder and semantic mapping network are jointly optimized by task loss and cross-modal semantic alignment loss.

4. The personnel state perception method based on time-delay-aware multimodal feature fusion as described in claim 1, characterized in that, Based on cross-modal cosine similarity, the temporal offsets of behavioral modalities and exogenous modalities relative to endogenous modalities are determined, and feature-level temporal alignment of modal semantic features of different modalities is performed according to the temporal offsets, including the following steps: Using the endogenous modality as the time reference, candidate time offset ranges are set for both behavioral and exogenous modalities. Under different candidate time offset ranges, calculate the average cosine similarity between the corresponding mode and the endogenous mode within the local time range; Based on the average cosine similarity corresponding to different candidate time offsets, the candidate time offset corresponding to the one with the highest average cosine similarity is determined as the target time offset. The feature sequences of behavioral and external modalities are adjusted in time based on the target time offset to obtain the temporally aligned semantic features of each modality.

5. The personnel state perception method based on time-delay-aware multimodal feature fusion as described in claim 1, characterized in that, Based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality, the dynamic fusion weights of the semantic features of each modality are calculated, including the following steps: Calculate the cosine similarity between the semantic features of each modality after time alignment, map it to a set interval, and calculate the cross-modal consistency parameter of each modality; Calculate the corresponding data quality parameters based on the integrity, continuity, and stability of each modal data. The cross-modal consistency parameter and data quality parameter are weighted to calculate the credibility score of each modality, and the fusion weight of each modality is calculated based on the credibility score.

6. The personnel state perception method based on time-delay-aware multimodal feature fusion as described in claim 5, characterized in that, For the k-th time window, calculate the pairwise cosine similarity between the time-aligned endogenous modality, behavioral modality, and exogenous modality, and map the cosine similarity to the interval [0,1]. Based on the similarity between each mode and the other two modes, the cross-modal consistency parameter of the corresponding mode is calculated. The calculation formula is as follows: ; ; ; in, This represents the cross-modal consistency parameter of the endogenous modes within the k-th time window; The cross-modal consistency parameter represents the behavioral modality within the k-th time window; This represents the cross-modal consistency parameter between internal and external modes within the k-th time window; Represents the normalized cosine similarity between endogenous modalities and behavioral modalities; This represents the normalized cosine similarity between endogenous and exogenous modes. It represents the normalized cosine similarity between the behavioral modality and the external modality.

7. The personnel state perception method based on time-delay-aware multimodal feature fusion as described in claim 1, characterized in that, The acquired time series data for each modality are preprocessed, including: For the acquired modal time series data, the timestamps of different data sources are corrected based on a preset unified clock reference; For the timestamp-corrected data, resample the data at different sampling frequencies according to the preset target sampling frequency; The resampled modal time series data are divided into sliding time windows according to the preset time window length and sliding step size, so that the same time window contains endogenous state data, behavioral state data and external working condition data respectively. For data from different modalities within each time window, standardization is performed separately within each modality.

8. A personnel status perception system based on time-delay-aware multimodal feature fusion, characterized in that, include: The multimodal time series data acquisition module is configured to acquire time series data of endogenous modalities, behavioral modalities, and exogenous modalities for identifying the state of the person to be detected, and to perform preprocessing. The feature extraction and mapping module is configured to extract features from the preprocessed endogenous modality, behavioral modality and exogenous modality time series data respectively, and map the extracted different modality features to a unified task semantic space to obtain endogenous modality semantic features, behavioral modality semantic features and exogenous modality semantic features; The time alignment module is configured to determine the time offset of behavioral modalities and exogenous modalities relative to endogenous modalities based on cross-modal cosine similarity, and to perform feature-level time alignment of modal semantic features of different modalities according to the time offset. The weight calculation module is configured to calculate the dynamic fusion weights of each modality's semantic features based on the consistency of cross-modal semantic features after time alignment and the data quality parameters of each modality's data. The fusion module is configured to perform weighted fusion of the time-aligned semantic features of each modality according to the dynamic fusion weight to obtain multimodal fusion features, and to identify the status or operational risks of the personnel to be detected based on the multimodal fusion features.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps in the personnel state perception method based on time-delay-aware multimodal feature fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps in the personnel state perception method based on time-delay-aware multimodal feature fusion as described in any one of claims 1-7.