Unsafe behavior identification method and device based on cross-modal alignment and medium

Abstract

The invention provides an unsafe behavior identification method and device based on cross-modal alignment, and a medium. The method comprises the following steps: respectively obtaining inertial measurement unit data and human skeleton key point data of a scaffolding worker; performing cross-modal alignment training on an inertial measurement unit encoder and a human skeleton key point encoder by using the inertial measurement unit data and the human skeleton key point data, so that the inertial measurement unit encoder and the human skeleton key point encoder can align semantic information of heterogeneous modals in the same feature space; only either an inertial measurement unit encoder or a human skeleton key point encoder is used to analyze the input single modal data to identify the unsafe behavior of the worker. Through the innovative design of the cross-modal alignment training framework and the high-performance encoder, the robustness, flexibility and economy of the system in actual deployment are remarkably enhanced while the recognition precision of the unsafe behaviors of the scaffolding workers is improved.

Description

Technical Field

[0001] This invention relates to the technical field of the intersection of construction engineering and artificial intelligence, specifically to a method, apparatus and medium for identifying unsafe behaviors based on cross-modal alignment, and more particularly to a method, apparatus and storage medium for identifying unsafe behaviors of scaffolding workers based on cross-modal alignment. Background Technology

[0002] The construction industry has long been considered a high-risk industry, with accident and injury rates significantly higher than most other sectors. Statistics show that scaffolding, as a critical work platform widely used in construction projects, involves a large number of workers' daily operations; however, safety violations related to scaffolding are frequent, and the causes of these accidents are often directly related to unsafe worker behaviors. Currently, safety monitoring at construction sites mainly relies on manual inspections by safety management personnel. In highly dynamic and complex real-world work scenarios, it is difficult to achieve comprehensive, real-time, and accurate behavioral monitoring and risk warnings. Therefore, how to automatically, accurately, and robustly identify unsafe behaviors during scaffolding construction has become a key technical challenge for improving construction safety levels.

[0003] Existing automatic identification methods mainly rely on two data modalities: visual modalities based on computer vision and sensing modalities based on inertial measurement units (IMUs). Computer vision-based methods extract features from video images or key points (KPs) on the human skeleton and use deep learning models (such as CNNs and GCNs) to identify behaviors such as not wearing a safety helmet or illegal climbing. However, this method is easily affected by environmental factors such as occlusion, lighting changes, and cluttered backgrounds in actual construction sites, resulting in insufficient stability. Wearable sensing methods based on IMUs collect signals such as acceleration and angular velocity and use recurrent neural networks (RNNs / LSTMs) or time-series-image transformation methods (such as Gram angle fields) for behavior classification. They can effectively perceive local motion dynamics, but their ability to model the spatial topology of human posture is weak, making it difficult to distinguish between behaviors with similar dynamics but different postures.

[0004] To combine the advantages of two modalities, existing research attempts to achieve multimodal information fusion. For example, combining visual models with IMU models through decision-level fusion (such as soft voting) can improve recognition performance to some extent. However, these methods have significant limitations: First, fusion often remains at the decision level, failing to achieve deep interaction and semantic alignment between modalities at the feature representation level, resulting in a single-modal encoder being unable to learn complementary knowledge from the other modality; second, in practical deployments, systems typically require both visual and IMU devices to be available simultaneously, and if one modality fails due to device malfunction, occlusion, or signal loss, system performance will drop sharply; furthermore, inertial measurement unit signals and skeletal keypoint data differ significantly in spatiotemporal structure and data distribution, and direct feature matching can easily lead to semantic shifts or representation collapse. Moreover, during training, the visual modality, with its greater information content, tends to dominate the learning process, suppressing the feature representation of the IMU modality and affecting the overall robustness and generalization ability of the model.

[0005] In summary, existing technologies for identifying unsafe behaviors of scaffolding workers still have the following prominent shortcomings: (1) Single-modal methods have poor environmental adaptability, visual modal is easily affected by occlusion, and IMU modal lacks spatial structure perception; (2) The multimodal fusion method failed to achieve semantic alignment of the feature layer and strongly relied on the availability of dual-modal synchronization, resulting in low deployment reliability in real-world scenarios; (3) The data differences between heterogeneous modalities are large, making alignment difficult, and there is a modal suppression phenomenon, which restricts the full play of the cross-modal knowledge complementarity effect.

[0006] Therefore, there is an urgent need for a behavior recognition method that can achieve deep cross-modal semantic alignment during the training phase and support independent and efficient operation of a single modality during the inference phase, so as to improve the accuracy, robustness and deployability of the system in real and complex construction site environments. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a method, apparatus, and medium for identifying unsafe behaviors based on cross-modal alignment.

[0008] According to the present invention, a method for identifying unsafe behaviors based on cross-modal alignment is provided, the method comprising: Training phase: Acquire inertial measurement unit (IMU) data and human skeleton key point data of scaffolding workers respectively; use the IMU data and human skeleton key point data to perform cross-modal alignment training on the IMU encoder and the human skeleton key point encoder, so that the IMU encoder and the human skeleton key point encoder can align the semantic information of heterogeneous modalities in the same feature space. Inference phase: Using only the inertial measurement unit encoder or the human skeleton keypoint encoder, the input single-modal data is analyzed to identify unsafe behaviors of the worker.

[0009] Preferably, the inertial measurement unit encoder is a time-frequency domain fusion encoder TFusion, which includes: Time-domain module: Used to receive signals from the inertial measurement unit and process them through a state-space model to extract long-range time-series features; the long-range time-series in capturing the raw inertial signal depends on continuous dynamic changes; Given a multi-channel inertial measurement unit signal ,in, It is a signal column vectors, The length of the inertial measurement unit signal. For the number of channels, Represents the set of real numbers. Represents a set of L rows and C columns of real numbers; For each channel signal Apply reversible instance normalization RevIN; the normalization process is as follows:

[0010] in and Represent The mean and variance, and These are learnable affine parameters used to adaptively adjust the normalized feature distribution. The numerical stability constant is used; the normalized signal is divided into... A length of Non-overlapping patch blocks form a patch-level input sequence. ,in Through a linear projection layer, each patch is mapped to... A 3D latent space is constructed and positional encodings are superimposed. To preserve sequence order information, the final input sequence is generated. ; Frequency domain module: used to convert the inertial measurement unit (IMU) signal into a time-frequency image and extract frequency domain features through a convolutional neural network; to model the frequency modes of the IMU signal; and to process the original IMU signal. The STFT is mapped to the time-frequency domain using a short-time Fourier transform; the real and imaginary parts of the STFT output are then concatenated along the channel dimension to obtain the spectrum. ,in The number of frequency components, The time frame number; ResNet is used in the spectrogram feature extraction stage. 50 serves as the basic backbone network; given the first Layer input feature mapping The residual link is represented as:

[0011] in This represents a residual mapping consisting of convolution, normalization, and nonlinear activation. Its learnable parameters, For the first Residual block output of the layer; Fusion module: used to fuse the time-domain features output by the time-domain module and the frequency-domain features output by the frequency-domain module to form a comprehensive characterization of the inertial measurement unit.

[0012] Preferably, the human skeletal keypoint encoder is a multi-scale spatiotemporal graph convolutional encoder (MSTGCN), which includes: Multi-scale convolution module: used to extract hierarchical temporal features at different time scales from the input skeletal keypoint sequence; Spatiotemporal graph convolution module: used to perform spatiotemporal joint modeling of the hierarchical temporal features at each time scale, and generate a spatiotemporal representation of the skeleton by combining the spatial topology of the human skeleton graph.

[0013] Preferably, the cross-modal alignment training employs a three-stage training strategy with layered freezing, including: Phase 1: Using the inertial measurement unit data and the human skeleton key point data respectively, independently perform supervised pre-training on the inertial measurement unit encoder and the human skeleton key point encoder; The second stage involves freezing the parameters of the inertial measurement unit encoder and the human skeleton keypoint encoder, optimizing the shared projection head through comparative learning loss, and mapping the features extracted by the inertial measurement unit encoder and the human skeleton keypoint encoder to the same latent space to achieve semantic alignment. The third stage involves freezing the parameters of the inertial measurement unit encoder, the human skeleton keypoint encoder, and the projection head, and training an independent classifier. The input of the classifier is the concatenation of the original output features of the encoder and the aligned features output by the projection head.

[0014] Preferably, the contrastive learning loss in the second stage adopts bidirectional InfoNCE loss, using the inertial measurement unit features and KP features of the same behavioral sample as positive sample pairs, and the features of different behavioral samples as negative sample pairs. The optimization maximizes the similarity of positive sample pairs in the latent space and minimizes the similarity of negative sample pairs.

[0015] The present invention also provides an apparatus for identifying unsafe behavior based on cross-modal alignment, the apparatus being applied to the above-described method for identifying unsafe behavior based on cross-modal alignment, the apparatus comprising: Data acquisition module: Configured to acquire inertial measurement unit data and human skeletal key point data of scaffold workers during the training phase; Training module: configured to perform cross-modal alignment training on the inertial measurement unit encoder and the human skeleton keypoint encoder using the inertial measurement unit data and the human skeleton keypoint encoder, so that the inertial measurement unit encoder and the human skeleton keypoint encoder can align the semantic information of heterogeneous modalities in the same feature space; Inference module: Configured to, during the inference phase, call only either the inertial measurement unit encoder or the human skeleton keypoint encoder to analyze the input single-modal data to identify unsafe behaviors of the worker.

[0016] Preferably, the inertial measurement unit encoder is a time-frequency domain fusion encoder TFusion, which includes a time domain module, a frequency domain module, and a fusion module; the human skeleton keypoint encoder is a multi-scale spatiotemporal graph convolution encoder MSTGCN, which includes a multi-scale convolution module and a spatiotemporal graph convolution module.

[0017] Preferably, the training module is further configured to execute a three-stage training strategy of tiered freezing, including: Independent pre-training phase: The inertial measurement unit encoder and the human skeleton key point encoder are independently pre-trained using the inertial measurement unit data and the human skeleton key point encoder, respectively. Cross-modal alignment stage: Freeze the parameters of the inertial measurement unit encoder and the human skeleton keypoint encoder, optimize the shared projection head by contrastive learning loss, and map the features extracted by the inertial measurement unit encoder and the human skeleton keypoint encoder to the same latent space to achieve semantic alignment; Classifier fine-tuning stage: Freeze the parameters of the inertial measurement unit encoder, the human skeleton keypoint encoder and the projection head, and train an independent classifier. The input of the classifier is the concatenation of the original output features of the encoder and the aligned features output by the projection head.

[0018] Preferably, the inference module is further configured to identify unsafe behaviors by utilizing the robustness obtained through the cross-modal alignment training when there is noise, occlusion or partial loss in the single modality data.

[0019] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for identifying unsafe behavior based on cross-modal alignment described above.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. The TFusion encoder proposed in this invention achieves an F1-score of 72.62% by relying solely on inertial measurement unit signals after cross-modal alignment training, which is approximately 9.62% higher than the existing best method (StarNet, 63.00%) and more than 30% higher than traditional machine learning methods. The MSTGCN encoder achieves a single-modal F1-score of 83.46% under the same framework, which is nearly 15% higher than the existing ST-GCN method. Even when using only a single sensor in actual deployment, this invention can still achieve high-precision recognition and significantly reduce missed detections and false alarms. 2. This invention achieves semantic alignment between the IMU and KP modalities in the feature space through three-stage cross-modal alignment training, enabling the single-modal encoder to absorb complementary information from the other modality, thereby enhancing feature representation and discrimination capabilities. Experiments show that the aligned IMU model remains stable under noise, occlusion and other interference, demonstrating stronger environmental adaptability and generalization ability. 3. This invention integrates IMU and human skeleton key point data during the training phase, and allows the use of only one modality during the inference phase. This effectively addresses real-world scenarios such as visual blind spots, camera failure, and IMU not being worn or disconnected. The system can still recognize data even when relying solely on the IMU or solely on visual key points. The system can maintain high recognition accuracy even when some modalities are missing, significantly improving the system's availability and reliability.

[0021] 4. This invention does not require that dual-mode equipment be available simultaneously at the construction site. The deployment method can be flexibly selected according to the actual situation, reducing equipment investment, calibration and maintenance costs.

[0022] 5. This invention, through an innovative cross-modal alignment training framework and high-performance encoder design, not only improves the accuracy of identifying unsafe behaviors of scaffolding workers, but also significantly enhances the robustness, flexibility, and economy of the system in actual deployment, providing an efficient and reliable technical solution for intelligent safety monitoring at construction sites. Attached Figure Description

[0023] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a diagram of the overall architecture of CIKP in this invention; Figure 2This is a schematic diagram of the three-stage cross-modal alignment training strategy of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0025] Example 1 Reference Figure 1 and Figure 2 According to the present invention, a method for identifying unsafe behaviors based on cross-modal alignment is provided, the method comprising: Training phase: Acquire inertial measurement unit (IMU) data and human skeleton key point data of scaffolding workers respectively; use the IMU data and human skeleton key point data to perform cross-modal alignment training on the IMU encoder and the human skeleton key point encoder, so that the IMU encoder and the human skeleton key point encoder can align the semantic information of heterogeneous modalities in the same feature space. Inference phase: Using only the inertial measurement unit encoder or the human skeleton keypoint encoder, the input single-modal data is analyzed to identify unsafe behaviors of the worker.

[0026] The inertial measurement unit encoder is a time-frequency domain fusion encoder, TFusion, which includes: Time-domain module: Used to receive signals from the inertial measurement unit (IMU) and process them through a state-space model to extract long-range time-series features; the long-range time sequence in capturing the raw inertial signal depends on continuous dynamic changes; given multi-channel IMU signals... ,in The length of the inertial measurement unit signal. The number of channels; the signal for each channel. Apply reversible instance normalization RevIN; the normalization process is as follows:

[0027] in and Represent The mean and variance, and These are learnable affine parameters used to adaptively adjust the normalized feature distribution. The numerical stability constant is used; the normalized signal is divided into... A length of Non-overlapping patch blocks form a patch-level input sequence. ,in Through a linear projection layer, each patch is mapped to... A 3D latent space is constructed and positional encodings are superimposed. To preserve sequence order information, the final input sequence is generated. ; Frequency domain module: used to convert the inertial measurement unit (IMU) signal into a time-frequency image and extract frequency domain features through a convolutional neural network; to model the frequency modes of the IMU signal; and to process the original IMU signal. The STFT is mapped to the time-frequency domain using a short-time Fourier transform; the real and imaginary parts of the STFT output are then concatenated along the channel dimension to obtain the spectrum. ,in The number of frequency components, The time frame number; ResNet is used in the spectrogram feature extraction stage. 50 serves as the basic backbone network; given the first Layer input feature mapping The residual link is represented as:

[0028] in This represents a residual mapping consisting of convolution, normalization, and nonlinear activation. Its learnable parameters, For the first Residual block output of the layer; Fusion module: used to fuse the time-domain features output by the time-domain module and the frequency-domain features output by the frequency-domain module to form a comprehensive characterization of the inertial measurement unit.

[0029] The human skeletal keypoint encoder is a multi-scale spatiotemporal graph convolutional encoder (MSTGCN), which includes: Multi-scale convolution module: used to extract hierarchical temporal features at different time scales from the input skeletal keypoint sequence; Spatiotemporal graph convolution module: used to perform spatiotemporal joint modeling of the hierarchical temporal features at each time scale, and generate a spatiotemporal representation of the skeleton by combining the spatial topology of the human skeleton graph.

[0030] Cross-modal alignment training employs a three-stage training strategy with layered freezing, including: Phase 1: Using the inertial measurement unit data and the human skeleton key point data respectively, independently perform supervised pre-training on the inertial measurement unit encoder and the human skeleton key point encoder; The second stage involves freezing the parameters of the inertial measurement unit encoder and the human skeleton keypoint encoder, optimizing the shared projection head through comparative learning loss, and mapping the features extracted by the inertial measurement unit encoder and the human skeleton keypoint encoder to the same latent space to achieve semantic alignment. The third stage involves freezing the parameters of the inertial measurement unit encoder, the human skeleton keypoint encoder, and the projection head, and training an independent classifier. The input of the classifier is the concatenation of the original output features of the encoder and the aligned features output by the projection head.

[0031] The second-stage contrastive learning loss employs bidirectional InfoNCE loss, using the inertial measurement unit features and KP features of the same behavioral sample as positive sample pairs, and the features of different behavioral samples as negative sample pairs. The optimization aims to maximize the similarity of positive sample pairs in the latent space and minimize the similarity of negative sample pairs.

[0032] The present invention also provides an apparatus for identifying unsafe behavior based on cross-modal alignment, the apparatus being applied to the above-described method for identifying unsafe behavior based on cross-modal alignment, the apparatus comprising: Data acquisition module: Configured to acquire inertial measurement unit data and human skeletal key point data of scaffold workers during the training phase; Training module: configured to perform cross-modal alignment training on the inertial measurement unit encoder and the human skeleton keypoint encoder using the inertial measurement unit data and the human skeleton keypoint encoder, so that the inertial measurement unit encoder and the human skeleton keypoint encoder can align the semantic information of heterogeneous modalities in the same feature space; Inference module: Configured to, during the inference phase, call only either the inertial measurement unit encoder or the human skeleton keypoint encoder to analyze the input single-modal data to identify unsafe behaviors of the worker.

[0033] The inertial measurement unit encoder is a time-frequency domain fusion encoder TFusion, which includes a time domain module, a frequency domain module, and a fusion module; the human skeleton keypoint encoder is a multi-scale spatiotemporal graph convolution encoder MSTGCN, which includes a multi-scale convolution module and a spatiotemporal graph convolution module.

[0034] The training module is further configured to implement a three-stage training strategy with tiered freezing, including: Independent pre-training phase: The inertial measurement unit encoder and the human skeleton key point encoder are independently pre-trained using the inertial measurement unit data and the human skeleton key point encoder, respectively. Cross-modal alignment stage: Freeze the parameters of the inertial measurement unit encoder and the human skeleton keypoint encoder, optimize the shared projection head by contrastive learning loss, and map the features extracted by the inertial measurement unit encoder and the human skeleton keypoint encoder to the same latent space to achieve semantic alignment; Classifier fine-tuning stage: Freeze the parameters of the inertial measurement unit encoder, the human skeleton keypoint encoder and the projection head, and train an independent classifier. The input of the classifier is the concatenation of the original output features of the encoder and the aligned features output by the projection head.

[0035] The inference module is also configured to identify unsafe behaviors by utilizing the robustness obtained through the cross-modal alignment training, in the event that the single-modal data is noisy, occluded, or partially missing.

[0036] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for identifying unsafe behavior based on cross-modal alignment described above.

[0037] Example 2 Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.

[0038] This invention aims to address the technical pain points in the identification of unsafe behaviors at scaffolding construction sites, such as strong occlusion interference, complex scene dynamics, insufficient robustness of single-modal monitoring, and reliance on the simultaneous availability of dual devices for multimodal methods. It provides a cross-modal alignment pre-training method and system based on IMU inertial sensing data and human skeletal key point (KP) data. By establishing a heterogeneous modal semantic feature alignment mechanism during the training phase and injecting complementary knowledge into the single-modal model, it enables automatic, accurate, and stable identification of unsafe behaviors in scaffolding operations even when relying solely on single-modal input during the inference phase. This reduces missed detections and false alarms, and improves the reliability and deployability of on-site safety monitoring.

[0039] This invention proposes a deep learning framework, CIKP, which aims to enhance the representation capability of single modalities by mining complementary information between heterogeneous modalities. Figure 1As shown, the framework consists of two feature encoders designed for specific modalities: TFusion, a time-frequency domain fusion encoder for processing IMU signals, and MSTGCN, a multi-scale spatiotemporal graph convolutional encoder for processing visual skeleton sequences. These two encoders are optimized through a three-stage cross-modal alignment strategy with hierarchical freezing, progressively completing feature initialization, latent space alignment, and classifier fine-tuning. To achieve the three-stage training objective of hierarchical freezing, three independent functional modules are built in parallel at the backend of each encoder: a primary classifier (Classifier1) for initial supervised learning, a projection head for cross-modal feature alignment, and a refined classifier (Classifier2) for final inference decisions. This design achieves deep fusion of multiple modalities, ultimately endowing each modal branch with independent and robust discriminative capabilities during the inference stage.

[0040] The TFusion encoder employs a parallel dual-stream architecture, comprising a time-domain module and a frequency-domain module. The time-domain module introduces the state-space model HARMamba to model long-range temporal dependencies, while the frequency-domain branch utilizes the Short-Time Fourier Transform (STFT) to convert the signal into a spectrogram and extracts local frequency-domain features through a convolutional neural network. Finally, the two are fused to generate a comprehensive representation. This time-frequency complementary design can simultaneously capture both global evolutionary trends and fine-grained periodic patterns in the signal. For the visual skeleton modality, the MSTGCN encoder first uses convolutional layers to extract hierarchical information at different time scales, and then models the topological structure of human joints through the STGCN module. This structure can fully integrate spatial geometric features and multi-scale temporal context, thereby accurately representing the dynamic limb behavior of construction workers in complex work scenarios.

[0041] To maximize the complementary advantages between the two modalities, the CIKP framework employs a hierarchical, three-stage cross-modal alignment strategy for system-level optimization. This strategy first establishes basic representation capabilities through Classifier 1 and single-modal supervised pre-training, followed by the cross-modal alignment stage. In this stage, the framework utilizes a shared projector to map heterogeneous IMU dynamic features and skeletal spatial structure features to the same common latent subspace, thereby bridging modal differences at the feature manifold level. Finally, CIKP adjusts the decision boundary for specific behavior categories by freezing the encoder and optimizing the independent task classifier (Classifier 2). This design establishes an asymmetric training and deployment paradigm: even though the model relies on multimodal data for joint representation learning during training, inference only requires single-modal data input to the corresponding encoder. This mechanism allows any single-modal branch to effectively utilize complementary semantic priors obtained from heterogeneous modalities during the alignment stage, thus maintaining robust behavior recognition performance even in the absence of other modal inputs.

[0042] TFusion Time-Frequency Domain Fusion Encoder: Construction workers' behavior typically manifests as highly non-stationary and complex movements. Their IMU sensor data not only contains trajectory information that dynamically evolves over time but also significant frequency domain features reflecting the periodicity of the movements and energy distribution. Traditional encoders often struggle to simultaneously capture these two types of heterogeneous information, limiting their ability to represent complex movements. To address this, we designed the TFusion encoder, employing a parallel dual-stream architecture to extract deep features from two complementary perspectives in the time and frequency domains. Figure 1 As shown, the encoder consists of a temporal block that processes the raw sensor signals and a frequency block that processes the time-frequency images. The two branches operate independently and ultimately generate a comprehensive representation containing all motion dynamics through feature fusion.

[0043] The time-domain module is designed to capture long-range temporal dependencies and continuous dynamic changes in raw inertial signals. Given a multi-channel IMU signal... ,in The length of the IMU signal. This refers to the number of channels. To mitigate the distribution drift caused by different subjects and acquisition environments, the signal of each channel was first analyzed. Apply Reversible Instance Normalization (RevIN). The normalization process is shown in the following formula:

[0044] in and Represent The mean and variance, and These are learnable affine parameters used to adaptively adjust the normalized feature distribution. This is the numerical stability constant. The normalized signal is divided into... A length of Non-overlapping patch blocks form a patch-level input sequence. ,in Through a linear projection layer, each patch is mapped to... A 3D latent space is constructed and positional encodings are superimposed. To preserve sequence order information, the final input sequence is generated. .

[0045] The generated sequence They were then sent to Deep feature extraction is performed using stacked HARMamba modules. For example... Figure 2As shown, the core of the HARMamba module lies in introducing a bidirectional state-space model, aiming to overcome the limitation of traditional unidirectional models that cannot utilize future context information. Specifically, given the first... HARMamba input sequence of the layer First, it undergoes layer normalization to obtain Subsequently, the sequence is passed through two linear projection layers to generate the main input stream used for temporal modeling. and the signal flow used for gating mechanisms .enter The sequence passes through a one-dimensional causal convolutional layer to extract local short-range features, and then enters the core state-space model through the SiLU activation function. The SSM layer utilizes a discretized selective scanning mechanism to recursively model the sequence in both ascending and descending order, thereby generating a feature representation containing bidirectional context. and :

[0046] To integrate bidirectional information, the output features of the two branches are first summed element-wise, and then the signal flow is utilized. Modulation is achieved through a gating mechanism. Finally, the fused features are mapped back to the original dimensions via a linear projection layer, serving as the output of the current HARMamba layer.

[0047] This bidirectional design, while maintaining linear computational complexity, significantly enhances the model's ability to represent complex temporal patterns of actions, ultimately yielding a temporal representation. .

[0048] To supplement the ability of time-domain features to characterize unsafe behaviors, this work further constructs a frequency-domain module to model the frequency modes of the IMU signal. To this end, we first... The output is mapped to the time-frequency domain using a Short-Time Fourier Transform (STFT). To preserve complete phase information, the real and imaginary parts of the STFT output are concatenated along the channel dimension to obtain the spectrum. ,in The number of frequency components, This represents the number of time frames. In the spectrogram feature extraction stage, we employ ResNet. ResNet-50 serves as the foundational backbone network. It consists of stacked residual blocks of multiple levels. By introducing residual connections within each residual block, it effectively mitigates the vanishing gradient problem during deep network training, thus enabling the network to achieve greater depth. Layer input feature mapping The residual link can be represented as:

[0049] in This represents a residual mapping consisting of convolution, normalization, and nonlinear activation. Its learnable parameters, For the first The residual block output of the layer.

[0050] While ResNet-50 provides a deep feature extraction framework, its standard convolutional operators process all neighboring pixels within the receptive field equally, often failing to accurately depict the complex spatial dependencies in the spectrogram. Therefore, we replace all standard convolutional layers in the backbone network with Optimal Weighted Convolution (wConv), constructing a lightweight and efficient wResNet-50. This differs from traditional convolutions that rely solely on learnable weights. To aggregate information, wConv introduces a spatial density function. This aims to explicitly model the spatial structure within the convolutional kernel. Specifically, given the input spectral features... wConv first calculates the density matrix. With the original convolution kernel weights The Hadamard product generates the modulated kernel weights. Then, convolution operations are performed:

[0051]

[0052] in This indicates element-wise multiplication. Through this mechanism, It can dynamically adjust the importance of pixels based on their relative positions within the receptive field. This design significantly improves the network's ability to discriminate key time-frequency patterns and its robustness without adding any trainable parameters, ultimately outputting frequency domain features. .

[0053] Multi-Scale Spatial-Temporal Graph Convolutional Encoder: While IMU signals can capture the overall motion state of workers, relying solely on a single IMU worn on the lower back is insufficient to characterize limb movements. Therefore, we introduce visual modalities based on skeletal keypoints as a supplement. The human skeleton naturally possesses graph structure characteristics (joints are viewed as nodes in the graph, and bone connections as edges), making graph convolutional networks (GCNs) an ideal choice for modeling its spatial dependencies. However, unsafe behaviors of scaffolding workers often exhibit significant diversity in the temporal dimension: including both sudden, instantaneous movements (such as falls) and sustained, periodic movements (such as climbing). Inspired by Ang Xu et al., we designed a Multi-Scale Spatial-Temporal Graph Convolutional Network (MSTGCN) for skeletal data. This module aims to construct feature pyramids with different temporal resolutions, enabling the model to simultaneously capture short-term local dynamics and long-term behavioral patterns, thereby generating more discriminative spatiotemporal skeletal representations.

[0054] First, to extract multi-level temporal features from the original skeleton sequence, we constructed a multi-scale convolution (MSC) module. The keypoint input sequence uses... and the corresponding adjacency matrix It means that, among them For frame number, For the number of joints, The coordinate dimension is used. MSC employs a cascaded structure to abstract temporal information layer by layer. Specifically, we designed three sequential convolutional stages, each consisting of a one-dimensional convolutional layer (Conv1D), batch normalization, and a ReLU activation function. Each stage uses a decreasing kernel size. This design constructs a feature pyramid:

[0055] in Representing the Output features at each time scale This refers to the number of frames at that scale. As the number of layers increases... As the number of elements increases, the temporal resolution of the feature map is halved at each subsequent time step, while the receptive field at each time step expands exponentially. This mechanism enables shallow features to... It can preserve fine details of instantaneous movements, as well as deep features. This can effectively summarize long-term behavioral trends.

[0056] After obtaining multi-scale temporal features Subsequently, we utilize a Spatiotemporal Graph Convolutional Network (STGCN) to perform spatiotemporal joint modeling at each scale. The STGCN consists of multiple spatiotemporal convolutional blocks, each containing two temporal convolutional blocks to capture temporal dependencies and one spatial convolutional block to aggregate spatial information. Figure 1 c). First, the temporal convolutional block extracts and integrates local temporal features along the time axis. Given the first... Input feature tensors at each scale The temporal convolutional block applies a kernel size of [size missing] along the time dimension. A one-dimensional convolution. To introduce a non-linear gating mechanism, the number of output channels of the convolution is set to... The output is then split into two branches. and Calculate the linear transformation and gated activation separately. Output after temporal convolution. It can be represented as:

[0057] in Represents element-wise product. It is the Sigmoid activation function. Through this gated linear unit (GLU), the network can adaptively and selectively transmit key timing information.

[0058] Subsequently, the spatial convolutional block runs independently at each time step, aggregating features of neighboring nodes using a graph structure to model spatial dependencies. The spatial convolutional block uses Chebyshev Graph Convolution (ChebGCN) to approximate the spectral graph convolution process. For the temporal convolution output... Input features of a certain frame Output features The calculation is as follows:

[0059] in For the first The learnable parameter matrix of a Chebyshev polynomial. For Chebyshev polynomials, This is the normalized Laplace matrix. , For degree matrix, This is the skeleton adjacency matrix. (Through...) Polynomial approximation of order (taken in this study) This module effectively performs The -hop neighborhood enables feature diffusion and aggregation, ultimately generating a skeleton feature representation containing rich spatiotemporal semantics. .

[0060] A Three-Stage Cross-Modal Alignment Training Strategy: In multimodal alignment tasks, significant modality strength discrepancies often exist between different modalities. Specifically, strong modalities typically possess more discriminative semantic information and exhibit larger gradient magnitudes and faster convergence speeds during training. This leads to an over-reliance on strong modalities in the optimization process under joint training or direct contrastive learning frameworks to quickly reduce loss, and forcibly pulling weak modal features into the strong modal space through alignment constraints. As a result, weak modal encoders struggle to learn their own discriminative structure and may even degenerate into simply fitting the strong modal distribution, resulting in the loss of their unique complementary information. Previous research has shown that modalities relying solely on visual keypoints outperform those relying solely on inertial modalities, while the voting ensemble of both exhibits even higher performance. This indicates that the two modalities are highly complementary, but there are also differences in strength. To address this issue, we design a progressive three-stage cross-modal alignment training strategy. We decouple the training process into three stages: independent representation learning, frozen alignment, and classifier fine-tuning. This aims to preserve the unique modality-specific features of each modality while constructing a shared semantic space.

[0061] In the first stage, our goal is to avoid intermodal interference and ensure that the weak modality (IMU) encoder can form stable feature representations. We construct and train the IMU encoder (TFusion) and the visual encoder (MSTGCN) independently. For each modality branch, we perform standard supervised training using labeled data to minimize the cross-entropy loss. At this stage, the two networks are unaware of each other, each learning to map the original signal to its own high-dimensional feature space. This strategy enables the IMU encoder to extract effective action patterns from the inertial signal, thus providing a highly discriminative single-modal feature foundation for subsequent fusion. For each modality... Given input samples and tags We minimize the standard cross-entropy loss:

[0062] in express Modal encoder, express The first-stage classifier for the modality. After this stage, the encoder is able to extract modality-specific features with semantic discriminative power.

[0063] After obtaining a robust single-modal encoder, this stage aims to establish semantic correspondences between heterogeneous modalities. To prevent the contrastive learning process from disrupting the feature distribution learned in the first stage, we freeze both modal encoders. and The parameters are used to train only the projection head. and The projector maps features of different dimensions to a shared latent space. In this space, we employ bidirectional InfoNCE loss for contrastive learning. For a batch... a sample pair ,in It is an IMU feature. These are the corresponding KP features, we will Treat them as positive sample pairs, and... These are considered negative sample pairs. The contrastive loss from IMU to KP is defined as:

[0064] in For cosine similarity, Here is the temperature coefficient. The total loss is... In this way, each modality has a projective representation that provides cross-modal semantic commonality, while perfectly preserving the original feature structure inside the encoder.

[0065] In the final stage, to fully utilize both modality-specific and modality-shared information, we freeze the encoder and projector head and train a feature-enhanced single-modality classifier (Classifier 2). We then compare the encoder's raw output (modality-specific semantic information) with the projector head's output (aligned common information):

[0066] This design allows the classifier to simultaneously leverage the rich detail of the original features and the semantic consistency of the aligned features. Crucially, this mechanism enables the unimodal branch to benefit from multimodal knowledge during inference, as its projected representation has internalized the shared semantic structure across modalities. Finally, we feed the concatenated features into Classifier 2 and train it again using cross-entropy loss.

[0067] To comprehensively evaluate the effectiveness of the proposed framework, this study used the publicly available SWIT dataset. The dataset consisted of 27 skilled construction workers (23 men and 4 women), representing a diverse range of heights (150–188 cm) and ethnicities to ensure data generalization across different individuals. Data acquisition utilized a wireless IMU sensor with a sampling frequency of 100 Hz, which was fixedly worn on the lower back of the participants. This location, close to the body's center of gravity, most effectively reflects the overall movement characteristics and postural changes of the human body. To obtain accurate ground truth values, a GoPro Hero 12 camera was used to record video at 25 Hz to assist in manual annotation. The dataset contains 10 typical scaffolding work behaviors, covering environmental risk behaviors, violations, ergonomic risk behaviors, and normal walking patterns.

[0068] Table 1. Dataset Description

[0069] The IMU data contains nine channels of signals: 3-axis acceleration, 3-axis angular velocity, and 3-axis gyroscope. Video data was used to estimate pose from camera footage using YOLOv7-pose, extracting the coordinates of 17 key points on the human body. Before inputting the data into the model, a standardized preprocessing procedure was performed. First, the video data was precisely time-synchronized and aligned with the nine-channel IMU signals. Based on the action boundaries determined by video annotations, a specific action clipping and padding strategy was adopted: a 1-second padding time was added before and after the start and end points of each unsafe action annotation. This crucial step aims to capture the transition phases before and after the action and effectively prevent the loss of short-term action information, thus covering the complete action pattern. Subsequently, all data was uniformly divided into fixed time windows of 2 seconds, which served as the standard input to the model. Furthermore, to eliminate the bias caused by uneven class distribution in the original data during model training, the SWIT dataset employed an undersampling strategy for data balancing.

[0070] To comprehensively evaluate the effectiveness and generalization ability of the CIKP method in complex construction environments, we constructed the following four evaluation scenarios: Scenario A: Deployment using only IMU devices This scenario simulates an operating environment limited by the lack of surveillance cameras or the presence of severe visual obstructions (such as dense smoke or blind spots). In these situations, KP modal data is completely unavailable, and the system must rely solely on IMU sensors for inference. This scenario aims to verify whether the alignment mechanism proposed in this paper successfully endows the IMU encoder with a deep perception of action semantics, enabling it to maintain high-precision recognition performance even in the absence of visual modalities.

[0071] Scenario B: Deployment using only visual monitoring This scenario addresses non-contact monitoring needs or responds to unexpected situations where wearable devices malfunction (e.g., due to battery depletion or worker forgetting to wear them). In this case, the IMU data stream is interrupted, and the system relies on visual monitoring. The purpose of this scenario is to evaluate whether the model can accurately capture subtle human motion features solely based on the spatiotemporal topological relationships of visual key points, even in the absence of high-frequency acceleration and angular velocity information.

[0072] Scenario C: Robustness Testing To verify the robustness of the model under non-ideal conditions, we further designed robustness tests for the two modalities. Specifically, these included: (1) IMU robustness test: Gaussian noise or simulated data packet loss was injected into the original signal to test the model's tolerance to sensor drift and transmission errors; (2) KP robustness test: random viewpoint rotation, scale scaling, or node loss was applied to the KP data to simulate camera viewpoint shift and human self-occlusion problems. This scenario aims to demonstrate that CIKP can still maintain stable feature representation and reliable discrimination ability when faced with degraded input quality.

[0073] To objectively and fairly evaluate the performance of the CIKP model and ensure the comparability of experimental results with existing research, we strictly adhered to the evaluation criteria established using the original dataset (SWIT dataset). Specifically, we adopted Macro-averaged Precision, Recall, and F1-score, which are commonly used in multi-class classification tasks, as the main evaluation metrics. Furthermore, to analyze the model's performance under different types of security risks in depth, we also grouped the experimental results for evaluation according to behavioral triggers and urgency.

[0074] For imbalanced datasets or multi-class classification tasks, Macro-average treats each class equally, preventing the results from being dominated by a large number of classes. The specific definitions of each metric are as follows: Precision: Measures the proportion of samples that a model predicts to represent a certain type of risk behavior, but which actually belong to that category.

[0075]

[0076] Recall: Measures the proportion of a certain type of risky behavior that is successfully detected by the model.

[0077]

[0078] F1-score: The harmonic mean of precision and recall, used to comprehensively evaluate the overall performance of a model, especially when a balance needs to be struck between reducing false positives and avoiding false negatives.

[0079]

[0080] in, This indicates that the model correctly predicted the positive class as the positive class. This indicates that a negative class was incorrectly predicted as a positive class. This indicates that a positive class error was incorrectly predicted as a negative class error. In addition to fine-grained evaluations of the 10 specific behavioral categories, we also referenced the classification criteria of the original dataset to divide all behaviors into 5 groups with specific management significance. This allows for a macro-level assessment of the model's ability to perceive different types of safety hazards. As shown in Table 1, this grouping is based on the source and nature of the risk. By calculating the average indicators of these groups, we can gain a clearer understanding of the model's differentiated performance in handling environmental hazards, human violations, and unexpected accidents.

[0081] To comprehensively evaluate the performance advantages of the CIKP model, we compared our proposed method with existing IMU-based action recognition methods. All comparison results are directly cited from the benchmark tests of the original paper on the SWIT dataset. Notably, for the CIKP model, we only used the TFusion branch and did not introduce any visual data. This aims to verify whether cross-modal alignment training successfully endows the single-modal encoder with stronger feature representation capabilities. We divided the baseline methods into two main categories: machine learning methods and deep learning methods. Detailed experimental results are summarized in Table 2.

[0082] Traditional machine learning methods include KNN, Random Forest, SVM, and gradient boosting algorithms such as XGBoost, LightGBM, and CatBoost. Table 2 shows that these methods generally perform poorly when handling complex construction scenarios, with F1-Scores not exceeding 43%. CatBoost and SVM achieved the best performance within their group, with F1-Scores of 42.37% and 41.91%, respectively. However, compared to deep learning methods, machine learning models are limited by their shallow feature extraction capabilities, making it difficult to capture the high-dimensional spatiotemporal dependencies inherent in long-sequence IMU signals.

[0083] In contrast, deep learning methods demonstrated significant performance improvements. We further subdivide these methods into temporal modeling-based methods and image representation-based methods. The latter uses Gramian Angular Fields (GAFs) to encode one-dimensional inertial signals into two-dimensional images, enabling the extraction of spatial texture features using convolutional neural networks. Experimental results show that while temporal models outperform machine learning methods, their F1-Score remains around 47%, indicating a bottleneck in handling complex and varied action patterns with simple temporal memory. Meanwhile, CNN-based methods using GAF image representation demonstrate stronger competitiveness, with the CNN model achieving an F1-Score of 57.08%.

[0084] Our CIKP (TFusion) significantly outperforms other methods. Comparative results show that TFusion, relying solely on IMU input, achieves an F1-Score of 72.62%, a 9.62% improvement over the strongest baseline, StarNet, and over 30% improvement over traditional ML methods. This performance is not only attributed to the time-frequency domain fusion modeling capabilities of the TFusion architecture, but more importantly, it validates the effectiveness of the cross-modal alignment strategy. Through cross-modal alignment during training, TFusion successfully integrates semantic information from the visual modality. Furthermore, CIKP also performs exceptionally well on broader group metrics, achieving a Group F1-Score of 86.51%, further demonstrating the model's robustness and reliability in identifying different risk categories.

[0085] Table 2 Comparison results of CIKP and IMU-based methods

[0086] To verify the effectiveness of the KP encoder (MSTGCN) in CIKP, we compared it with three representative skeleton graph neural network methods. Similarly, in this set of experiments, CIKP uses only a single visual modality (MSTGCN) for inference, without relying on any IMU auxiliary information. The comparison methods include three progressively related methods as baselines: GCN, TGN, and STGCN. Graph Convolutional Networks (GCN) primarily focus on the spatial topology of the skeleton in each frame, neglecting the dynamic relationships between frames. Temporal Graph Networks (TGN) focus on modeling the temporal evolution of joint coordinate trajectories, compensating for the lack of dynamic information in purely spatial modeling. Finally, there is the Spatiotemporal Graph Convolutional Network (ST-GCN), which, as the current mainstream paradigm, aims to capture both the structural features and motion patterns of the skeleton by jointly modeling the spatial and temporal dimensions.

[0087] The experimental results (summarized in Table 3) show a performance improvement trend highly consistent with technological evolution. Specifically, GCN, which relies solely on spatial features, performed the weakest, with an F1-Score of only 53.16%, indicating that the lack of temporal dimension modeling severely restricts the understanding of continuous actions. TGN, which introduces the temporal dimension, significantly improved the F1-Score to 66.84%, verifying the dominant role of temporal information in action recognition. ST-GCN, which further integrates spatiotemporal features, achieved the best performance among the baseline methods (68.58%), but its improvement over TGN was relatively limited, which may be due to its single-scale convolutional receptive field.

[0088] In comparison, our CIKP (MSTGCN) achieves state-of-the-art performance. Compared to the second-best STGCN, MSTGCN boasts an F1-Score of 83.46%, surpassing STGCN by nearly 15%. This advantage is primarily attributed to MSTGCN's introduction of a multi-scale convolutional design, enabling it to adaptively capture action patterns across different time spans. Furthermore, CIKP achieves an astonishing 94.99% in group metrics, demonstrating near-zero misclassification in macro-risk classification. This performance advantage resulting from technological advancement, combined with the feature robustness granted by cross-modal aligned training, solidifies CIKP's leading position in the field of skeleton behavior recognition.

[0089] Table 3 Comparison results of CIKP and KP-based methods

[0090] To explore the differences in performance of different training paradigms when applying cross-modal alignment, and to analyze the specific impact of modal strength differences on model performance, we compared different training paradigms. Using Training from Scratch as a baseline, we compared two mainstream cross-modal alignment strategies: (1) Contrastive Pre-training, which first aligns the encoder through contrastive learning, and then trains the classifier with frozen parameters; and (2) Joint Learning, which directly adds the cross-modal contrastive loss to the classification loss for optimization. By analyzing the training curves and test set performance of each modality under different strategies, we found that both of these conventional paradigms have limitations when facing modal strength differences, making it difficult to achieve ideal complementary alignment.

[0091] First, the "Training from Scratch" results show a significant strength difference between the two modalities. This difference manifests in the gradient dynamics during training: after only one epoch of training, the KP modality achieves a training F1-Score of 0.65, while the IMU branch only achieves 0.40. This stark contrast in training rates indicates that the strong modality (KP) generates stronger and more explicit gradient signals, while the weak modality (IMU) learns relatively slowly. The final test results show that the KP modality's F1-Score is significantly higher than the IMU modality, demonstrating that KP contains richer discriminative information. This imbalance in modality strength is a key reason for the varying performance of different alignment strategies.

[0092] Based on the aforementioned modal differences, we further analyzed the limitations of the two alignment strategies. For the Contrastive Learning strategy, the results show that it failed to effectively preserve the discriminative features unique to each modality. This strategy exhibited the lowest test performance on both modalities, and the training loss was difficult to converge. This confirms that rigid alignment, by forcibly bringing the feature distributions of heterogeneous modalities closer together, disrupts their respective feature spaces. The subsequent freezing operation limited the encoder's adaptability, making it difficult for the classifier to define clear decision boundaries, ultimately resulting in underfitting. Regarding the Joint learning strategy, although the training F1-Score rapidly increased to 1.0, the IMU's test performance significantly decreased, while KP remained almost unchanged. This indicates that during joint optimization, the gradient of the KP modality dominated the update direction of the network weights. The IMU modality passively reduced its loss function under the influence of the KP gradient, rather than actively learning effective discriminative features. This passive optimization led to severe feature collapse and overfitting in the IMU encoder.

[0093] In contrast, our designed three-stage cross-modal alignment training strategy effectively overcomes the optimization problem caused by modal strength differences by integrating independent representation learning, cross-modal alignment, and downstream fine-tuning. As shown in Table 4, our method significantly outperforms the independent training baseline on all evaluation metrics. Specifically, for the IMU modality, our method improves the F1-Score from 66.76% to 72.62% of the baseline, while also achieving an increase in the group-level F1-Score from 80.17% to 86.51%. This indicates that the IMU encoder successfully integrates high-level semantic information from the visual modality, significantly enhancing its feature discrimination ability. More importantly, our strategy also delivers performance improvements for the stronger KP modality. Its F1-Score increases from 75.14% to 83.46%, and the group F1-Score approaches 95%. This result demonstrates that our progressive alignment mechanism not only avoids the destruction of the strong modal feature space caused by forced alignment, but also further enhances the robustness and accuracy of visual representation.

[0094] Table 4. Effects of the Three-Stage Cross-Modal Alignment Training Strategy

[0095] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0096] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for identifying unsafe behaviors based on cross-modal alignment, characterized in that, The method includes: Training phase: Acquire inertial measurement unit (IMU) data and human skeleton key point data of scaffolding workers respectively; use the IMU data and human skeleton key point data to perform cross-modal alignment training on the IMU encoder and the human skeleton key point encoder, so that the IMU encoder and the human skeleton key point encoder can align the semantic information of heterogeneous modalities in the same feature space. Inference phase: Using only the inertial measurement unit encoder or the human skeleton keypoint encoder, the input single-modal data is analyzed to identify unsafe behaviors of the worker.

2. The method for identifying unsafe behavior based on cross-modal alignment according to claim 1, characterized in that, The inertial measurement unit encoder is a time-frequency domain fusion encoder TFusion, which includes: Time-domain module: Used to receive signals from the inertial measurement unit and process them through a state-space model to extract long-range time-series features; the long-range time-series in capturing the raw inertial signal depends on continuous dynamic changes; Given a multi-channel inertial measurement unit signal ,in, It is a signal column vectors, The length of the inertial measurement unit signal. For the number of channels, Represents the set of real numbers. Represents a set of L rows and C columns of real numbers; For each channel signal Apply reversible instance normalization RevIN; the normalization process is as follows: in and Represent The mean and variance, and These are learnable affine parameters used to adaptively adjust the normalized feature distribution. The numerical stability constant is used; the normalized signal is divided into... A length of Non-overlapping patch blocks form a patch-level input sequence. ,in Through a linear projection layer, each patch is mapped to... A 3D latent space is constructed and positional encodings are superimposed. To preserve sequence order information, the final input sequence is generated. ; Frequency domain module: used to convert the inertial measurement unit (IMU) signal into a time-frequency image and extract frequency domain features through a convolutional neural network; to model the frequency modes of the IMU signal; and to process the original IMU signal. The STFT is mapped to the time-frequency domain using a short-time Fourier transform; the real and imaginary parts of the STFT output are then concatenated along the channel dimension to obtain the spectrum. ,in The number of frequency components, The time frame number; ResNet is used in the spectrogram feature extraction stage. 50 serves as the basic backbone network; given the first Layer input feature mapping The residual link is represented as: in This represents a residual mapping consisting of convolution, normalization, and nonlinear activation. Its learnable parameters, For the first Residual block output of the layer; Fusion module: used to fuse the time-domain features output by the time-domain module and the frequency-domain features output by the frequency-domain module to form a comprehensive characterization of the inertial measurement unit.

3. The method for identifying unsafe behavior based on cross-modal alignment according to claim 1, characterized in that, The human skeletal keypoint encoder is a multi-scale spatiotemporal graph convolutional encoder (MSTGCN), which includes: Multi-scale convolution module: used to extract hierarchical temporal features at different time scales from the input skeletal keypoint sequence; Spatiotemporal graph convolution module: used to perform spatiotemporal joint modeling of the hierarchical temporal features at each time scale, and generate a spatiotemporal representation of the skeleton by combining the spatial topology of the human skeleton graph.

4. The method for identifying unsafe behavior based on cross-modal alignment according to claim 1, characterized in that, The cross-modal alignment training employs a three-stage training strategy with layered freezing, including: Phase 1: Using the inertial measurement unit data and the human skeleton key point data respectively, independently perform supervised pre-training on the inertial measurement unit encoder and the human skeleton key point encoder; The second stage involves freezing the parameters of the inertial measurement unit encoder and the human skeleton keypoint encoder, optimizing the shared projection head through comparative learning loss, and mapping the features extracted by the inertial measurement unit encoder and the human skeleton keypoint encoder to the same latent space to achieve semantic alignment. The third stage involves freezing the parameters of the inertial measurement unit encoder, the human skeleton keypoint encoder, and the projection head, and training an independent classifier. The input of the classifier is the concatenation of the original output features of the encoder and the aligned features output by the projection head.

5. The method for identifying unsafe behavior based on cross-modal alignment according to claim 4, characterized in that, The second-stage contrastive learning loss employs bidirectional InfoNCE loss, using the inertial measurement unit features and KP features of the same behavioral sample as positive sample pairs, and the features of different behavioral samples as negative sample pairs. The optimization aims to maximize the similarity of positive sample pairs in the latent space and minimize the similarity of negative sample pairs.

6. A device for identifying unsafe behaviors based on cross-modal alignment, characterized in that, The apparatus is used in the method for identifying unsafe behavior based on cross-modal alignment as described in any one of claims 1-5, and the apparatus comprises: Data acquisition module: Configured to acquire inertial measurement unit data and human skeletal key point data of scaffold workers during the training phase; Training module: configured to perform cross-modal alignment training on the inertial measurement unit encoder and the human skeleton keypoint encoder using the inertial measurement unit data and the human skeleton keypoint encoder, so that the inertial measurement unit encoder and the human skeleton keypoint encoder can align the semantic information of heterogeneous modalities in the same feature space; Inference module: Configured to, during the inference phase, call only either the inertial measurement unit encoder or the human skeleton keypoint encoder to analyze the input single-modal data to identify unsafe behaviors of the worker.

7. The apparatus for identifying unsafe behavior based on cross-modal alignment according to claim 6, characterized in that, The inertial measurement unit encoder is a time-frequency domain fusion encoder TFusion, which includes a time domain module, a frequency domain module, and a fusion module; the human skeleton keypoint encoder is a multi-scale spatiotemporal graph convolution encoder MSTGCN, which includes a multi-scale convolution module and a spatiotemporal graph convolution module.

8. The apparatus for identifying unsafe behavior based on cross-modal alignment according to claim 6, characterized in that, The training module is further configured to execute a three-stage training strategy with tiered freezing, including: Independent pre-training phase: The inertial measurement unit encoder and the human skeleton key point encoder are independently pre-trained using the inertial measurement unit data and the human skeleton key point encoder, respectively. Cross-modal alignment stage: Freeze the parameters of the inertial measurement unit encoder and the human skeleton keypoint encoder, optimize the shared projection head by contrastive learning loss, and map the features extracted by the inertial measurement unit encoder and the human skeleton keypoint encoder to the same latent space to achieve semantic alignment; Classifier fine-tuning stage: Freeze the parameters of the inertial measurement unit encoder, the human skeleton keypoint encoder and the projection head, and train an independent classifier. The input of the classifier is the concatenation of the original output features of the encoder and the aligned features output by the projection head.

9. The apparatus for identifying unsafe behavior based on cross-modal alignment according to claim 6, characterized in that, The inference module is further configured to identify unsafe behaviors by utilizing the robustness obtained through the cross-modal alignment training when there is noise, occlusion, or partial loss in the single-modal data.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for identifying unsafe behavior based on cross-modal alignment as described in any one of claims 1 to 5.

Images