Building construction site dangerous behavior identification method and system based on machine learning

Abstract

The invention belongs to the technical field of building construction safety control, and particularly discloses a building construction site dangerous behavior recognition method and system based on machine learning, and the method comprises the steps: obtaining a single-view continuous video stream of a construction site, segmenting the single-view continuous video stream into a time sequence video frame sequence, and extracting an initial spatial feature sequence through a spatial feature encoder; a pre-trained virtual visual angle projection and feature compensation module encodes the visual angle implicit vector and maps the visual angle implicit vector to a visual angle invariant feature space, and a sequential context is combined to compensate occlusion missing features to generate an enhanced feature sequence; a time sequence memory alignment module captures long-term and short-term time sequence dependence and outputs time sequence consistency characteristics; and finally, outputting a current dangerous behavior classification result through the classification prediction head, and outputting a future dangerous intention probability curve through the time sequence prediction head. The method does not need an additional camera, can accurately cope with a shielding scene, predicts the danger in advance, and improves the construction safety monitoring efficiency and reliability.

Description

Technical Field

[0001] This invention belongs to the field of optical component defect detection, specifically relating to a method and system for identifying hazardous behaviors at construction sites based on machine learning. Background Technology

[0002] At construction sites, dangerous behaviors such as climbing, falling, crossing boundaries, and throwing objects can easily lead to safety accidents. Real-time and accurate identification of such behaviors is crucial to ensuring construction safety. Currently, construction sites mostly rely on ordinary fixed surveillance cameras to collect video and use video recognition technology to detect dangerous behaviors, but this faces many technical bottlenecks in practical applications.

[0003] Construction workers are often obscured by scaffolding, piles of materials, etc., resulting in incomplete human postures in the images. Traditional pure visual recognition models (such as CNN) rely on complete human features, and are prone to missing key points and incomplete features when occluded, leading to misjudgments or missed judgments. Multi-camera fusion solutions can alleviate the occlusion problem, but they have drawbacks such as high cost, difficult deployment, and the need for spatial synchronization. Skeleton point tracking methods are prone to skeleton breakage or misconnection when occluded, and single-frame classification methods cannot use temporal context to complete the information.

[0004] In the existing technology, there is a lack of a dangerous behavior identification scheme that can effectively deal with occluded scenes and infer the behavior of occluded workers by relying solely on a single-view video stream, which makes it difficult to meet the needs of low-cost and highly robust safety monitoring at construction sites. Summary of the Invention

[0005] To address this issue, the present invention provides a method and system for identifying hazardous behaviors at construction sites based on machine learning, thereby solving the aforementioned technical problems.

[0006] This invention provides a method for identifying hazardous behaviors at construction sites based on machine learning, comprising the following steps: A single-view continuous video stream of the construction site is acquired and segmented into a temporal video frame sequence; the visual features of each video frame are extracted using a spatial feature encoder to obtain an initial spatial feature sequence. The initial spatial feature sequence is input into a pre-trained virtual view projection and feature compensation module. The view encoder encodes the current view conditions to obtain the view latent vector. The virtual projector then performs a conditional transformation on the initial spatial features based on the view latent vector, mapping them to the view invariant feature space. Simultaneously, the temporal context information is combined to compensate for the features missing due to occlusion, generating an enhanced feature sequence containing the compensated features. The enhanced feature sequence is input into the temporal memory alignment module, which consists of ConvLSTM and Transform. The module performs temporal alignment and information fusion on the enhanced feature sequence and outputs a temporal consistency feature that integrates long and short-term context information. Based on the temporal consistency feature, the behavior recognition and classification are performed using a classification prediction head, and the classification result of whether there is dangerous behavior in the current sequence is output. The dangerous intent in the future time period is predicted using a temporal prediction head, and the dangerous intent probability curve is output.

[0007] Preferably, the compensation for features lost due to occlusion specifically includes: During the model training phase, a training set containing samples of the same dangerous behavior under different viewpoints is constructed. The virtual viewpoint projection and feature compensation module learns to synthesize an estimated view feature corresponding to the complete viewpoint from the features of the surrounding unoccluded frames of the occluded frame and the viewpoint latent vector. This compensation process is constrained by the viewpoint consistency loss function, which forces the feature vectors of the same dangerous behavior obtained after compensation under different viewpoints or different occlusion states to be close to each other in the feature space.

[0008] Preferably, the view consistency loss function adopts a distance-based loss function or a contrastive loss function to minimize the distance between compensated features of the same behavior category under different view conditions, while maximizing the distance between compensated features of different behavior categories.

[0009] Preferably, the temporal memory alignment module adopts a hybrid architecture of ConvLSTM and temporal Transformer, wherein the ConvLSTM first processes the enhanced feature sequence to output short-term memory features, and then inputs this short-term memory feature sequence into the temporal Transformer for long-range dependency modeling.

[0010] Preferably, before inputting the video stream into the spatial feature encoder, a data augmentation step is included, which includes synthesizing occlusion, specifically: randomly adding a mask of random shape to the human body region of the training video frame, or using an existing object image for coverage, to simulate a real occlusion scene.

[0011] Preferably, when extracting visual features, the spatial feature encoder extracts key point information of the human skeleton in parallel as an auxiliary feature channel, and then fuses the initial spatial features with the auxiliary feature channel before inputting them into the virtual view projection and feature compensation module.

[0012] Preferably, the initial spatial features are fused with the auxiliary feature channels, specifically by concatenating the initial spatial feature vector after dimension matching with the adapted auxiliary feature vector according to the channel dimension to obtain a concatenated feature vector with a dimension twice that of the initial spatial feature vector. The SE channel attention module first performs global average pooling on the concatenated feature vectors to obtain the channel description vectors, and then outputs the channel attention weights through a fully connected layer. The fused feature vector is obtained by multiplying the features of each channel of the concatenated feature vector with the corresponding attention weights.

[0013] Preferably, the time-series prediction head is used to perform dangerous intent prediction, and its output is a probability curve of dangerous behavior occurring within a future time window Δt; the training loss function of this branch is a weighted cross-entropy loss, which assigns higher weights to future time points that are closer to the current time.

[0014] Preferably, the training process of the temporal memory alignment module includes temporal frame loss reconstruction, specifically including: During the training phase, features of some frames in the enhanced feature sequence are randomly masked or discarded, forcing the temporal memory alignment module to reconstruct the features of the discarded frames based on the features of the remaining frames.

[0015] In another aspect, this application also provides a machine learning-based system for identifying hazardous behaviors at construction sites, comprising: The spatial feature sequence acquisition module is used to acquire a single-view continuous video stream from the construction site and segment it into a temporal video frame sequence; the spatial feature encoder is used to extract the visual features of each video frame to obtain the initial spatial feature sequence. The enhanced feature sequence generation module is used to input the initial spatial feature sequence into the pre-trained virtual view projection and feature compensation module. The view encoder encodes the current view conditions to obtain the view latent vector, and the virtual projector performs conditional transformation on the initial spatial features based on the view latent vector, mapping it to the view invariant feature space. At the same time, it combines temporal context information to compensate for the features missing due to occlusion, generating an enhanced feature sequence containing the compensated features. The temporal consistency feature output module is used to input the enhanced feature sequence to the temporal memory alignment module. This module consists of ConvLSTM and Transform, which performs temporal alignment and information fusion on the enhanced feature sequence and outputs a temporal consistency feature that integrates long and short-term context information. The dangerous behavior identification module is used to identify and classify behaviors based on the temporal consistency features using a classification prediction head, outputting a classification result indicating whether dangerous behavior exists in the current sequence, and predicting dangerous intentions in future time periods using a temporal prediction head, outputting a dangerous intention probability curve.

[0016] This invention utilizes existing single-view monitoring equipment without the need for additional cameras. It completes missing features due to occlusion through virtual view projection and feature compensation modules, fuses long and short-term temporal dependencies through temporal memory alignment modules (ConvLSTM and temporal Transformer), and combines data augmentation (synthetic occlusion), human skeleton key point auxiliary feature fusion, and temporal frame loss reconstruction training. This not only solves the problems of false positives and false negatives under occlusion and high costs of multiple cameras in traditional models, but also outputs the probability curve of dangerous intent in advance. It balances recognition accuracy, robustness, and low cost, providing an efficient and practical solution for safety monitoring at construction sites. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating a machine learning-based method for identifying hazardous behaviors at construction sites, as provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the process for generating occlusion feature compensation and enhancement feature sequences provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the feature fusion process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a machine learning-based hazardous behavior identification system for construction sites, provided as an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] like Figure 1 As shown in the figure, this invention discloses a method 100 for identifying hazardous behaviors at construction sites based on machine learning, comprising the following steps: S1. Acquire a single-view continuous video stream of the construction site and segment it into a temporal video frame sequence; use a spatial feature encoder to extract the visual features of each video frame to obtain the initial spatial feature sequence. S2, the initial spatial feature sequence is input into the pre-trained virtual view projection and feature compensation module. The view encoder encodes the current view conditions to obtain the view latent vector. The virtual projector performs conditional transformation on the initial spatial features based on the view latent vector and maps them to the view invariant feature space. At the same time, the temporal context information is combined to compensate for the features missing due to occlusion and generate an enhanced feature sequence containing the compensated features. S3, the enhanced feature sequence is input to the temporal memory alignment module. This module captures local short-term temporal dependencies through the Convolutional Long Short-Term Memory Network (ConvLSTM) and captures global long-term temporal dependencies through the Temporal Transformer. It performs temporal alignment and information fusion on the enhanced feature sequence and outputs temporal consistency features that fuse long and short-term contextual information. S4. Based on the temporal consistency feature, the behavior is identified and classified using a classification prediction head, and the classification result of whether there is dangerous behavior in the current sequence is output. The dangerous intent in the future time period is predicted using a temporal prediction head, and the dangerous intent probability curve is output.

[0021] In some embodiments, for step S1, the video stream source is a regular fixed surveillance camera already deployed at the construction site, without the need for additional equipment. A single-view continuous RGB video stream is acquired, and the video frame rate is controlled at 10-30fps to balance feature extraction efficiency and recognition accuracy, and to adapt to the performance of conventional monitoring equipment at the construction site.

[0022] Frame sequence segmentation uses a video framing algorithm to divide a continuous video stream into discrete temporal video frame sequences in chronological order. The time interval between the frame sequences is consistent with the original video frame rate, ensuring that the continuity of action between frames is not lost. For example, for a 20fps video stream, one frame is extracted every 0.05 seconds to form an ordered set of video frames.

[0023] In one embodiment, an efficient 2D visual backbone network is selected as the spatial feature encoder, such as ResNet-50, ResNet-101, or a lightweight EfficientNet (e.g., EfficientNet-B3). Taking ResNet-50 as an example, its classic residual block structure is retained, the fully connected classification layer of the original network is removed, and the convolutional and pooling layers are retained to construct an encoder architecture suitable for feature extraction from construction site video frames. This architecture can effectively extract visual information such as intra-frame texture and contour, while also possessing a certain degree of anti-interference capability.

[0024] The segmented temporal video frame sequence is input frame by frame into the constructed spatial feature encoder. The encoder extracts features from each frame through multiple convolutional operations. For example, it first performs downsampling and preliminary feature extraction on the input frame using a convolutional layer with 7×7 kernels, and then performs feature enhancement through multiple residual blocks. Finally, it outputs the spatial feature tensor corresponding to each frame (e.g., with dimensions of 7×7×C or 14×14×C, where C is the number of feature channels, and the specific dimension is determined by the encoder backbone network model and the input frame size). The feature tensors of all frames are arranged in chronological order to form an initial spatial feature sequence. This sequence contains the basic visual features of each video frame, providing a data foundation for subsequent feature compensation and temporal modeling.

[0025] In some embodiments, for step S2, the virtual view projection and feature compensation module is first pre-trained. Specifically, this includes constructing a training dataset containing different dangerous behaviors at the construction site (such as climbing, falling, crossing boundaries, throwing, and collaborative lifting). The dataset needs to cover samples of the same dangerous behavior under different virtual viewpoints. By performing geometric enhancement (such as random affine transformation, slight horizontal / vertical transformation, and perspective transformation) on real dangerous behavior video samples, different camera angles are simulated. At the same time, synthetic occlusion processing is performed (using random shape masks or existing object images at the construction site to occlude the edges, torso, legs, head, and other areas of the human body in the video frame), simulating the occlusion scene of the real construction site, ensuring that the training samples can cover multiple occlusion situations and viewpoint changes under a single viewpoint. Subsequently, the constructed training dataset is input into the virtual viewpoint projection and feature compensation module for training. For example, during the training process, the module iteratively learns the feature mapping rules of the same dangerous behavior under different viewpoints and different occlusion states, while optimizing the module parameters by combining the viewpoint consistency loss function (described in detail later) until the module can stably achieve the compensation of occlusion features and the extraction of viewpoint-invariant features, thus completing the pre-training.

[0026] Next, the viewpoint latent vector encoding specifically includes inputting the initial spatial feature sequence obtained in step S1 into the pre-trained virtual viewpoint projection and feature compensation module. The viewpoint encoder in this module first encodes the viewpoint conditions of the current video stream. Specifically, For example, the viewpoint encoder employs a lightweight neural network architecture (e.g., composed of 2-3 fully connected layers). It takes the shooting angle-related parameters of the current video frame (which can be obtained from the installation parameters of the surveillance camera or indirectly represented by viewpoint features obtained through analysis of initial spatial features) as input. Through nonlinear transformation, it converts the viewpoint conditions into a low-dimensional viewpoint latent vector (e.g., a vector with 128 or 256 dimensions). This latent vector accurately represents the features of the current viewpoint, providing conditions for viewpoint-invariant mapping of subsequent features.

[0027] Subsequently, the initial spatial features undergo conditional transformation and viewpoint-invariant mapping. Specifically, the virtual projector performs conditional transformation on each frame of features in the initial spatial feature sequence based on the encoded viewpoint latent vector. The virtual projector uses a conditional batch normalization layer or a lightweight FiLM (Feature-wise Linear Modulation) layer to implement the transformation: using the viewpoint latent vector as modulation parameters, the mean, variance, or feature channel weights of the initial spatial features are adjusted to eliminate differences in feature distribution under different viewpoints, mapping the initial spatial features to a viewpoint-invariant feature space. For example, the FiLM layer linearly scales and shifts each channel of the initial spatial features, with the scaling factor and shift calculated from the viewpoint latent vector, ensuring that the mapped features are no longer affected by viewpoint changes, retaining only the essential features of the behavior itself. Next, as Figure 2 The schematic diagram shown illustrates the process of generating occlusion feature compensation and feature enhancement sequences. Exemplarily, it specifically includes: S201, Extracting Temporal Context Information: Specifically, the module extracts the features of the unoccluded frames before and after the occluded frame (by analyzing the continuity of the feature sequence, frames with missing or abrupt features are identified as occluded frames) from the initial spatial feature sequence as temporal context information. For example, if the t-th frame is an occluded frame, the features of the (t-1)th and (t-2)th frames (forward context) and the (t+1)th and (t+2)th frames (backward context) are extracted to construct the context feature set of the occluded frame. S202, Occlusion Feature Compensation. Specifically, by combining temporal context information and viewpoint latent vectors, the module synthesizes the "estimated complete view features" of the corresponding occluded frame through the learned feature mapping rules. That is, based on the action trend of the behavior in the context features (such as the forward context showing the human body moving up and the backward context showing the human body moving down, it is inferred that the action of the occluded frame may be crossing) and the viewpoint features represented by the viewpoint latent vectors, the missing features of the occluded area are supplemented, so that the compensated features can fully reflect the essential features of the behavior. S203, forming an enhanced feature sequence: Specifically, all frames (including view-invariant features of unoccluded frames and compensated features of occluded frames) are rearranged in chronological order to form an enhanced feature sequence containing compensated features. This sequence eliminates the influence of viewpoint differences and compensates for feature loss caused by occlusion, providing high-quality feature data for subsequent temporal modeling.

[0028] Optionally, during the above feature compensation and mapping process, the module is constrained by a viewpoint consistency loss function. This loss function can be a distance-based loss function (such as Euclidean distance loss) or a contrastive loss function (such as NT-Xent loss): for the same dangerous behavior, after compensation, the feature vectors are forced to be close to each other in the viewpoint-invariant feature space (minimizing the distance between feature vectors). Meanwhile, for the compensated feature vectors of different dangerous behaviors, the distance between them in the feature space is maximized to ensure that the features have good class separability, further improving the module's compensation accuracy for occlusion features and the extraction accuracy of behavioral features.

[0029] In some embodiments, for step S3, the temporal memory alignment module adopts a hybrid architecture of ConvLSTM (Convolutional Long Short-Term Memory Network) and Temporal Transformer, which work in series. That is, ConvLSTM first captures local short-term temporal dependencies, and then Temporal Transformer captures global long-term temporal dependencies, fully combining the advantages of the two models to achieve efficient temporal modeling of enhanced feature sequences.

[0030] Specifically, a standard ConvLSTM network structure is adopted, in which convolutional operations are used in the input gate, forget gate, and output gate instead of the fully connected operations of traditional LSTM, preserving the spatial structure information of the features. The number of network layers is set to 2-3 layers, and the kernel size of each layer is 3×3 or 5×5. The feature dimension of the hidden layer is consistent with the dimension of the features in each frame of the enhanced feature sequence, ensuring that the features can be smoothly transferred.

[0031] The enhanced feature sequence obtained in step S2 is divided into multiple local temporal segments according to a time window (e.g., a window size of 5-10 frames), and each segment is input into the ConvLSTM network. ConvLSTM processes the inter-frame features within the local segments through a gating mechanism. Specifically, the forget gate selectively discards irrelevant historical features, the input gate selectively retains the key features of the current frame, and the output gate generates the hidden state features of the current frame, thereby capturing the continuity of actions and the pattern of feature changes between frames within a short local time range, and outputting a short-term memory feature sequence containing local temporal information.

[0032] Specifically, the temporal Transformer adopts the encoder part of the encoder-decoder structure, which contains multiple Transformer coding layers (such as 6-8 layers). Each coding layer consists of a multi-head self-attention mechanism and a feedforward neural network, while introducing relative position coding (to enhance robustness to time offsets and adapt to video frame drops or asynchronous situations that may occur on the construction site).

[0033] The short-term memory feature sequence output by ConvLSTM is input into the temporal Transformer. The temporal Transformer uses a multi-head self-attention mechanism to calculate the attention weight between any two frames of features in the sequence. The higher the weight, the stronger the correlation between the two frames of features. This captures the evolutionary pattern and potential intent of actions over a long period of time (such as the entire video segment) in the enhanced feature sequence. The feedforward neural network then performs a nonlinear transformation on the features processed by the attention mechanism to further enhance the expressive power of the features and output a feature sequence that integrates long-term and short-term contextual information.

[0034] Next, based on the relative position encoding in the temporal Transformer, the feature sequence fused with long and short-term context information is temporally aligned. By calculating the relative offset of each frame feature in the temporal dimension, the timestamp representation of the feature is adjusted to eliminate temporal misalignment caused by video frame drops, frame rate fluctuations, or changes in motion speed, ensuring that the temporal correlation of features between frames is accurate. Subsequently, global average pooling or attention-weighted fusion is used to integrate the information of the time-aligned feature sequence. For example, if attention-weighted fusion is used, the fusion weight is calculated based on the importance of each frame's features to behavior recognition (e.g., frames containing key poses of dangerous actions are given higher weights). The features of all frames are then summed according to their weights to obtain a single temporal consistency feature. This feature fully integrates the long and short-term contextual information of the enhanced feature sequence, and can comprehensively characterize the behavioral features of the entire video segment.

[0035] In some embodiments, for step S4, a classification prediction head is first constructed. The classification prediction head adopts a neural network architecture consisting of 2-3 fully connected layers. The dimension of the input layer is consistent with the dimension of the temporal consistency feature. The intermediate layer adopts the ReLU activation function to enhance the non-linear expression capability. The output layer adopts the Softmax activation function. The output dimension corresponds to the number of behavior categories (such as including categories such as "normal behavior", "climbing", "falling", "crossing the boundary", "throwing", "cooperative lifting", etc., and the output dimension is 6 or 7). Next, the current dangerous behavior is classified. Specifically, the temporal consistency features obtained in step S3 are input into the classification prediction head. The prediction head performs progressive mapping and transformation of the features through a fully connected layer, and finally outputs the probability value of each behavior category through the Softmax activation function. The category with the highest probability value is selected as the behavior classification result of the current video sequence. For example, if the category corresponding to the highest probability is a dangerous behavior category (such as "climbing" or "falling"), then the current sequence is determined to have a dangerous behavior; if it is a "normal behavior" category, then the current sequence is determined to have no dangerous behavior, thus completing the identification of the current dangerous behavior.

[0036] Subsequently, a temporal prediction head is constructed. For example, the temporal prediction head adopts a temporal convolutional network (TCN) or a fully connected network architecture with a temporal dimension. The input layer receives temporal consistency features and a partial short-term memory feature sequence (to retain the temporal information of recent actions). The output layer dimension corresponds to the number of future time window divisions (e.g., the future prediction time Δt is set to 0.5-2 seconds, the time window is divided at 0.1-second intervals, and the output dimension is 5-20). Next, to predict future dangerous intentions, temporal consistency features and recent short-term memory (SNSM) feature sequences are input into the temporal prediction head. The prediction head learns the temporal evolution pattern of actions through temporal modeling (such as causal convolution operations in TCN), predicting the probability of dangerous behavior occurring at each time point within a future time window Δt, thus forming a dangerous intention probability curve. For example, if the predicted probability of "crossing the line" behavior is 0.9 in 0.5 seconds and 0.3 in 1 second, the probability curve will peak at 0.5 seconds, intuitively reflecting the likelihood of danger occurring at different future time points.

[0037] Optionally, the training loss function of the time series prediction head adopts weighted cross-entropy loss, that is, the closer the future time point is to the current time, the higher the weight is assigned (e.g., the weight of the future 0.1 seconds is 1.0, the weight of the future 0.2 seconds is 0.9, and so on), which encourages the model to prioritize the accuracy of short-term hazard prediction, so as to buy time for construction site safety management personnel to intervene in dangerous behaviors in a timely manner.

[0038] Preferably, in one embodiment, before inputting the video stream into the spatial feature encoder, a data augmentation step is further included, the data augmentation step including synthetic occlusion, specifically: randomly adding a mask of random shape to the human body region of the training video frame, or using an existing object image for coverage, to simulate a real occlusion scene.

[0039] Specifically, firstly, training video frames are selected from the constructed training dataset of dangerous behaviors at construction sites. Training video frames containing worker work scenarios are selected based on the following criteria: the human target in the frame is clear, the behavior and actions are identifiable, and the video frames cover different work periods (such as day and night), different types of work (such as scaffolders and steelworkers), and different work areas (such as scaffolding areas, material stacking areas, and edge work areas). This ensures that the subsequent synthesized occluded scenes are representative and adaptable to the diverse environment of real construction sites. Next, the human body regions are located. Specifically, open-source 2D human detection algorithms (such as the YOLO series algorithms and the Faster R-CNN algorithm) can be used to detect human body regions on the selected training video frames. The algorithm identifies key features of the human body within the frame (such as the head, torso, and limb contours) and outputs the bounding box coordinates of each human target (represented by the pixel coordinates of the frame image, such as the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2)), accurately locating the human body regions of all workers within the frame, thus adding a defined range for subsequent occlusion.

[0040] Subsequently, random shape masks are generated. Specifically, various random shape mask materials are generated based on computer graphics technology, covering the morphological characteristics of common obstructions on construction sites. These include irregular polygons (simulating material pile edge obstruction), rectangles (simulating scaffolding horizontal / vertical pole obstruction), circles (simulating tools or small equipment obstruction), and arcs (simulating tower crane booms or pipe obstruction). Each shape mask is set with a different size range (e.g., the width of a rectangular mask ranges from 20-80 pixels, and the height ranges from 30-100 pixels, with the specific size dynamically adjusted according to the size of the human body area), and the mask transparency is set to completely opaque (all RGB channel values ​​are 0) to ensure realistic obstruction effects.

[0041] Existing images are acquired and preprocessed. Images of common occlusion objects at construction sites (such as scaffolding pipes, rebar piles, cement bags, safety nets, construction tools, etc.) are collected. The acquired images are preprocessed by first cropping the images to remove the background and retain the main body of the objects; then, the object images are scaled to different sizes (consistent with the size range of the random shape mask) and uniformly converted to the same image resolution and color space as the training video frames (such as RGB color space, 256×256 pixel resolution); finally, the preprocessed object images are edge-smoothing to avoid obvious image edge fragmentation when adding occlusions and improve the realism of the occlusion scene.

[0042] In one embodiment, the obscured area is randomly selected. First, based on the needs of identifying dangerous behaviors at the construction site, the located human body area is divided into several key parts, including the head (associated with head protection, falling from heights and other dangerous behaviors), the torso (associated with body posture, whether carrying prohibited items and other dangerous behaviors), the upper limbs (associated with throwing, climbing and other dangerous behaviors), the lower limbs (associated with crossing, unstable standing and other dangerous behaviors), and the edge area (associated with whether the body is partially exposed to dangerous areas). A random sampling algorithm is used to select 1-3 parts from the defined key human body parts as the areas to be occluded. The sampling probability is set according to the occlusion frequency of the actual construction site. For example, the sampling probability of the edge area and lower limb area is set to 30% because they are easily occluded by piles of materials and scaffolding; the sampling probability of the torso and upper limb area is set to 25%; and the sampling probability of the head area is set to 20% because the occlusion frequency is low, to ensure that the synthetic occlusion scene is consistent with the actual situation. Subsequently, within the selected area to be occluded, the placement coordinates of the occlusion material are randomly generated. Using the human body part bounding box as the range, the pixel coordinates of the upper left corner of the occlusion material are randomly determined to ensure that the occlusion material falls completely within the area to be occluded, and the occlusion area accounts for 30%-60% of the area to be occluded (this avoids both insufficient training effect due to too small an occlusion area and complete loss of behavioral features due to too large an occlusion area).

[0043] In one embodiment, for random shape mask occlusion, a mask of one shape is randomly selected from a prepared random shape mask material library, and the mask size is adjusted according to the size of the area to be occluded. For example, the mask is scaled to the size corresponding to the preset occlusion area by a bilinear interpolation algorithm to ensure the matching degree between the mask and the area to be occluded. By using image pixel replacement, the adjusted random shape mask is superimposed on the occlusion position of the training video frame. That is, the pixel value of the mask area directly replaces the pixel value of the corresponding position in the training video frame to achieve the occlusion effect. If the occlusion area involves the boundary between the human body and the background, edge feathering is performed (the transparency gradient of the mask edge range of 5-10 pixels is adjusted) to make the mask blend naturally with the background of the video frame and reduce the traces of artificial synthesis.

[0044] In one embodiment, for existing object image occlusion, an existing object image is selected based on the scene relevance of the part to be occluded. For example, if the part to be occluded is a lower limb and is located in a scaffolding area, an image of scaffolding steel pipes is preferred; if the part to be occluded is an upper limb and is located in a material stacking area, an image of a pile of steel bars or cement bags is preferred. After selection, the object image size is adjusted using a bilinear interpolation algorithm to meet the preset occlusion area requirements. The adjusted object image is superimposed onto the occluded position. In addition to pixel replacement, optional lighting adaptation processing is also performed. Lighting parameters (such as brightness, contrast, and color temperature) within a 10-20 pixel range around the occluded position in the training video frame are extracted. Through image brightness adjustment and contrast correction algorithms, the lighting effect of the object image is made consistent with the background of the video frame. If there is a shadow in the video frame, a shadow effect with the corresponding direction and intensity is added below the object image (achieved through Gaussian blur and brightness reduction) to further enhance the realism of the occlusion.

[0045] After completing the synthesis of occlusion for a single frame of training video, record the occlusion information for that frame, including the occlusion method (random shape mask / existing object image), occlusion location (head / torso / upper limb / lower limb / edge region), size and position coordinates of the occluded material, and occlusion area percentage, etc. Simultaneously, a "training mask" for the frame is generated. For example, the training mask is a binary image with the same size as the video frame, where the pixel values ​​of the occluded areas are set to 1, and the pixel values ​​of the unoccluded areas are set to 0. The training mask is associated with and stored in conjunction with the synthesized occluded video frame. During subsequent model training, the training mask is used to guide the model to identify occluded areas, assisting the model in learning to compensate for the features of occluded areas from the features of unoccluded areas, and improving the model's robustness to occluded scenes.

[0046] Preferably, in one embodiment, when the spatial feature encoder extracts visual features, it extracts key point information of the human skeleton in parallel as an auxiliary feature channel, and then fuses the initial spatial features with the auxiliary feature channel before inputting them into the virtual view projection and feature compensation module.

[0047] Specifically, a 2D human pose estimation model such as HRNet (adapting to occlusion and multiple poses, ensuring joint accuracy) is selected and deployed in parallel with the spatial feature encoder. Both share the same video frame input stream. For example, a confidence threshold of 0.6-0.7 is set to filter false detections.

[0048] Extract the coordinates of 17-25 joints, fill in the missing key joints, normalize the coordinates, and construct a 2n+m dimensional auxiliary feature vector by combining the relative distances and angles between joints, where n is the number of joints. Specifically, arrange all the normalized joint coordinates (x'1, y'1, x'2, y'2, ..., x'n, y'n, where n is the number of joints) in a fixed order to form a basic vector of dimension 2n. At the same time, calculate the relative distances (such as the distance between shoulders, the distance from elbow to wrist, and the distance from hip to knee) and angles (such as the angle between shoulder, elbow, and wrist, and the angle between hip, knee, and ankle) between key joints. After normalizing these distance and angle values, supplement them into the basic vector to form a skeletal key point auxiliary feature vector of dimension 2n+m (where m is the number of relative distance and angle features, usually set to 10-15). This vector can completely represent the skeletal structure and posture features of the human body.

[0049] In one embodiment, as described in the preceding embodiments, after processing the video frames, the spatial feature encoder outputs an initial spatial feature tensor for each frame. A combined strategy of feature concatenation and attention-weighted fusion is employed to achieve deep fusion of the initial spatial features and auxiliary features. Figure 3 The feature fusion process shown is illustrated below. S301, the initial spatial feature vector (denoted as F_vis) after dimension matching and the adaptation auxiliary feature vector (denoted as F_skel) are concatenated along the channel dimension; the resulting concatenated feature vector F_concat=[F_vis,F_skel] has a dimension that is twice the dimension of the initial spatial feature vector (e.g., 4096 dimensions for ResNet-50). This step preserves the complete information of both types of features, avoiding the loss of key details from a single feature. S302 introduces a channel attention mechanism (such as a Squeeze-and-Excitation, SE) module to learn attention weights on the concatenated feature vector F_concat to highlight feature channels that are more important for identifying dangerous behaviors. The specific operations of the SE module include: Squeeze operation: Performs global average pooling on F_concat, compressing the features of each channel into a single value, and obtaining a channel description vector (the dimension of which is the number of channels in the concatenated feature vector). Excitation operation: The channel description vector is non-linearly transformed by two fully connected layers (the middle layer has a dimension of 1 / 16 of the number of channels and the activation function is ReLU; the output layer has the same dimension as the number of channels and the activation function is Sigmoid), and the attention weight of each channel is output (the value range is 0-1). S303, multiply each channel feature of the concatenated feature vector F_concat with its corresponding attention weight to obtain the weighted fused feature vector F_fuse. In this vector, important channels related to behavioral texture and contour in the initial spatial features, as well as important channels related to skeletal pose and joint movement in the auxiliary features, are given higher weights. This allows for a more accurate representation of the combined visual and pose features of the human body in the video frame, providing more comprehensive input feature support for the subsequent virtual viewpoint projection and feature compensation modules.

[0050] Preferably, in one embodiment, the training process of the temporal memory alignment module includes a temporal frame loss reconstruction step: during the training phase, the features of some frames in the enhanced feature sequence are randomly masked or discarded, forcing the temporal memory alignment module to reconstruct the features of the discarded frames based on the features of the remaining frames, thereby enhancing the model's completion and robustness against temporal occlusion.

[0051] Specifically, after obtaining the enhanced feature sequence from the virtual perspective projection and feature compensation module, the sequence is first segmented according to a fixed time window (the window length is set to 8-16 frames to adapt to the continuous action duration of dangerous behaviors at the construction site, such as climbing and crossing boundaries, which usually take 1-3 seconds to complete, corresponding to 10-30 frames of a 10-30fps video, and the 8-16 frame window can cover key action segments) to obtain several sub-sequences; Each subsequence needs to be labeled with a timestamp of the frame (calculated based on the original video frame rate; for example, in a 20fps video, each frame interval is 0.05 seconds, and the timestamps are 0, 0.05, 0.1 seconds, etc.) to ensure that the temporal correlation between subsequent frame loss and reconstruction is traceable. At the same time, the feature values ​​of each subsequence are normalized (mapping the feature values ​​to the [0,1] interval) to avoid the difference in feature value range affecting the accuracy of frame loss reconstruction.

[0052] Based on potential frame loss scenarios in construction site videos (such as random frame loss due to monitoring equipment lag, or continuous frame loss due to network transmission delays), two types of frame loss parameters are set: Random single-frame dropping parameters: The frame dropping ratio is set to 10%-20% (simulating low-frequency random frame dropping). 1-3 frames are randomly selected from each subsequence (the specific number is determined proportionally according to the length of the subsequence, such as 1 frame for an 8-frame subsequence and 2-3 frames for a 16-frame subsequence) as frames to be dropped. Parameters for consecutive frame drops: The length of the dropped frame segment is set to 2-4 frames (to simulate consecutive frame drops caused by short-term equipment failure), the frame drop frequency is set to a maximum of 1 consecutive frame drop segment in each subsequence (to avoid excessive loss of feature information due to too many dropped frames, making reconstruction impossible), and the starting position of the consecutive dropped frame segment is randomly determined in the subsequence.

[0053] A binary mask matrix is ​​used to implement frame dropping. The dimension of the mask matrix is ​​the same as the dimension of the enhanced feature subsequence (e.g., if the subsequence has 16 frames × 2048 dimensional features, then the mask matrix is ​​16 × 2048). For a given frame to be dropped (including random single frames and consecutive multiple frames), all feature dimension values ​​of the corresponding frame in the mask matrix are set to 0 (i.e., the "mask"). For frames to be retained, all feature dimension values ​​of the corresponding frame in the mask matrix are set to 1. The generated mask matrix is ​​multiplied element-wise with the original enhanced feature subsequence to obtain the "feature subsequence after frame dropping," in which the features of the frames to be dropped are set to zero, and only the valid features of the non-dropped frames are retained. After frame loss, frames in the feature subsequence are labeled to distinguish between "valid frames" (non-discarded frames, features not set to zero) and "masked frames" (discarded frames, features set to zero), and the position index of the masked frame in the subsequence is recorded (e.g., frame 3, frames 8-10, etc.). This labeling information is used for subsequent reconstruction loss calculation, clarifying the target frame position that the model needs to reconstruct, and ensuring targeted optimization during training.

[0054] Subsequently, the feature subsequence after frame loss is input into the temporal memory alignment module. For example, the module reconstructs the features of the discarded frames according to the following logic: ConvLSTM Local Information Transfer: ConvLSTM utilizes the preserved valid frame features and, through a gating mechanism (input gate, forget gate, output gate), transfers local temporal information (such as the feature change trend of adjacent valid frames before and after the mask frame) to the time step corresponding to the mask frame, generating the "local reconstruction features" of the mask frame (focusing on feature completion of short-term inter-frame dependencies). Temporal Transformer Global Information Fusion: The Temporal Transformer captures the global temporal dependencies between all valid frames and mask frames in the entire subsequence (such as the action evolution patterns of valid frames far before and after the mask frame) through a multi-head self-attention mechanism. It corrects and supplements the local reconstruction features output by ConvLSTM to generate the "global optimized reconstruction features" of the mask frame. This feature integrates local short-term dependencies and global long-term dependencies, and is closer to the true features of the discarded frames.

[0055] For example, the reconstruction loss function uses the mean squared error (MSE) loss function to calculate the difference between the "global optimized reconstruction features" and the "original features of the discarded frames" (i.e., the non-zeroed features of the corresponding frames in the enhanced feature subsequence); The reconstruction loss is weighted and summed with the original loss of the temporal memory alignment module (the reconstruction loss weight is set to 0.3-0.5 to ensure that its influence on model training is moderate and does not mask the losses of other core tasks), resulting in the total loss. The total loss is then backpropagated to all parameter layers of the temporal memory alignment module (including the gating parameters of ConvLSTM, the attention weight parameters of the temporal Transformer, etc.) through a backpropagation algorithm (such as the Adam optimizer), iteratively updating the parameters so that the module gradually improves its ability to reconstruct the features of discarded frames based on the features of the remaining frames, ultimately enhancing the completion and robustness against temporal occlusion (such as "temporal dimension occlusion" caused by dropped video frames).

[0056] Figure 4 A machine learning-based hazardous behavior recognition system 400 for construction sites is illustrated. This device embodiment is similar to... Figure 1 Corresponding to the illustrated method embodiments, the specific methods include: The spatial feature sequence acquisition module 401 is used to acquire a single-view continuous video stream of the construction site and divide it into a temporal video frame sequence; the spatial feature encoder is used to extract the visual features of each video frame to obtain the initial spatial feature sequence. The enhanced feature sequence generation module 402 is used to input the initial spatial feature sequence into the pre-trained virtual view projection and feature compensation module, encode the current view conditions through the view encoder to obtain the view latent vector, and use the virtual projector to perform condition transformation on the initial spatial features based on the view latent vector, mapping it to the view invariant feature space. At the same time, it combines temporal context information to compensate for the features missing due to occlusion, and generates an enhanced feature sequence containing the compensated features. The temporal consistency feature output module 403 is used to input the enhanced feature sequence to the temporal memory alignment module. This module captures local short-term temporal dependencies through the Convolutional Long Short-Term Memory Network (ConvLSTM) and captures global long-term temporal dependencies through the Temporal Transformer. It performs temporal alignment and information fusion on the enhanced feature sequence and outputs temporal consistency features that fuse long and short-term contextual information. The dangerous behavior identification module 404 is used to identify and classify behaviors based on the temporal consistency features using a classification prediction head, outputting a classification result indicating whether dangerous behavior exists in the current sequence, and to predict dangerous intentions in future time periods using a temporal prediction head, outputting a dangerous intention probability curve.

[0057] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying dangerous behavior in a construction site based on machine learning, characterized by, The method comprises the following steps: obtaining a single-view continuous video stream of a construction site and dividing the video stream into a time sequence of video frames; extracting visual features of each video frame by using a spatial feature encoder to obtain an initial spatial feature sequence; inputting the initial spatial feature sequence into a pre-trained virtual view projection and feature compensation module, encoding the current view condition by using a view encoder to obtain a view hidden vector, and using a virtual projector to perform conditional transformation on the initial spatial features based on the view hidden vector, map to a view-invariant feature space, and compensate for the missing features due to occlusion by combining the time context information, to generate an enhanced feature sequence containing compensated features; inputting the enhanced feature sequence into a time memory alignment module composed of a ConvLSTM and a Transform, performing time alignment and information fusion on the enhanced feature sequence, and outputting a time consistency feature fused with long-term and short-term context information; based on the time consistency feature, performing behavior recognition classification by using a classification prediction head to output a classification result of whether there is dangerous behavior in the current sequence, and performing dangerous intent prediction in the future period by using a time prediction head to output a dangerous intent probability curve.

2. The method according to claim 1, wherein the compensation for the missing features due to occlusion comprises: in the model training phase, a training set containing samples of the same dangerous behavior under different views is constructed; the virtual view projection and feature compensation module learns to synthesize an estimated view feature corresponding to a complete view from the features of the surrounding unoccluded frames of the occluded frame and the view hidden vector; the compensation process is constrained by a view consistency loss function, which forces the feature vectors obtained after compensation of the same dangerous behavior under different views or different occlusion states to be close to each other in the feature space.

3. The method according to claim 2, wherein the view consistency loss function adopts a distance-based loss function or a contrastive loss function to minimize the distance between the compensated features of the same behavior category under different view conditions, and maximize the distance between the compensated features of different behavior categories. including: the time memory alignment module adopts a hybrid architecture of ConvLSTM and time Transformer, wherein the ConvLSTM first processes the enhanced feature sequence to output short-term memory features, and then inputs the short-term memory feature sequence into the time Transformer for long-range dependency modeling.

4. The method of claim 1, wherein the method is based on machine learning.

5. The method according to claim 1, wherein before inputting the video stream into the spatial feature encoder, a data augmentation step is further included, and the data augmentation step includes synthetic occlusion, specifically: randomly adding a mask of random shape to the human body region of the training video frame, or using an existing object image to cover, to simulate a real occlusion scene. ​ ​ ​ 6. The machine learning-based construction site dangerous behavior recognition method according to claim 1, characterized in that, the spatial feature encoder extracts human skeleton key point information as an auxiliary feature channel in parallel when extracting visual features, and fuses the initial spatial features and the auxiliary feature channel before inputting them into the virtual perspective projection and feature compensation module.

7. The machine learning-based construction site dangerous behavior recognition method according to claim 6, characterized in that, the initial spatial features are fused with the auxiliary feature channel, specifically, the initial spatial feature vectors and the adaptive auxiliary feature vectors are spliced by channel dimension after dimension matching to obtain a spliced feature vector with a dimension of twice the initial spatial feature vector; the SE channel attention module first globally averages the spliced feature vector to obtain a channel description vector, and then outputs a channel attention weight through a fully connected layer; the channel features of the spliced feature vector are multiplied by the corresponding attention weight to obtain a fused feature vector.

8. The machine learning-based construction site dangerous behavior recognition method according to claim 1, characterized in that, the time sequence prediction head is used to perform dangerous intention prediction, and its output is a probability curve of dangerous behavior occurrence in a future Δt time window; the training loss function of this branch is a weighted cross-entropy loss, and the closer the future time point to the current time, the higher the weight.

9. The machine learning-based construction site dangerous behavior recognition method according to claim 1, characterized in that, the training process of the time sequence memory alignment module includes time sequence frame loss reconstruction, specifically including, in the training stage, randomly mask or discard the features of part of the frames in the enhanced feature sequence, and force the time sequence memory alignment module to reconstruct the features of the discarded frames according to the features of the remaining frames. 10.A machine learning based construction site dangerous behavior recognition system, characterized in that, including: a spatial feature sequence acquisition module for acquiring a single-view continuous video stream of a construction site and dividing it into a time sequence video frame sequence; extracting visual features of each video frame using a spatial feature encoder to obtain an initial spatial feature sequence; an enhanced feature sequence generation module for inputting the initial spatial feature sequence into a pre-trained virtual perspective projection and feature compensation module, encoding the current perspective condition through a perspective encoder to obtain a perspective hidden vector, and using a virtual projector to conditionally transform the initial spatial features based on the perspective hidden vector to map to a perspective-invariant feature space, while combining the time context information to compensate for the missing features due to occlusion, generating an enhanced feature sequence containing compensated features; a time sequence consistency feature output module for inputting the enhanced feature sequence into a time sequence memory alignment module composed of ConvLSTM and Transform, which performs time alignment and information fusion on the enhanced feature sequence, and outputs a time sequence consistency feature that fuses long and short-term context information. The dangerous behavior recognition module is configured to perform behavior recognition classification based on the time sequence consistency feature through a classification prediction head, output a classification result of whether the current sequence has dangerous behavior, and perform dangerous intention prediction in a future time period through a time sequence prediction head, and output a dangerous intention probability curve.

Images