An intelligent construction site safety monitoring and abnormal behavior detection method

Abstract

The application provides a kind of intelligent construction site safety monitoring and abnormal behavior detection method, it is related to behavior identification technical field, including: based on video frame data, through target detection algorithm, personnel and cigarette in the construction site multi-modal data are positioned, to output first detection result;Based on the first detection result, video frame data and construction site multi-modal data in the preset time period, utilize 3D CNN to combine attention mechanism, to output second detection result;Based on the second detection result and construction area security level information, by isolated forest and self-encoder algorithm, get comprehensive abnormal score.

Description

Technical Field

[0001] This application belongs to the field of behavior recognition technology, specifically relating to a method for intelligent construction site safety monitoring and abnormal behavior detection. Background Technology

[0002] With the rapid development of computer vision and deep learning technologies, video surveillance systems are increasingly widely used in industrial safety. From early rule-based manual analysis to today's automatic identification of dangerous behaviors such as smoking and not wearing a safety helmet using machine learning algorithms, the intelligence level of video surveillance systems is constantly improving. In particular, the application of advanced algorithms such as 3D convolutional neural networks (3D CNN), attention mechanisms, isolated forests, and autoencoders in recent years has made the detection of abnormal behaviors in complex scenes more accurate and efficient.

[0003] Existing construction site safety monitoring systems primarily rely on target detection algorithms to locate personnel and specific objects, such as using models like YOLOv5s to process video frame data to detect cigarettes or other hazardous materials. However, these systems typically only provide basic target localization capabilities and cannot effectively distinguish between normal behavior and potentially risky behavior. Furthermore, traditional methods fall short for behavioral analysis that requires combining time-series information.

[0004] Most existing technologies rely solely on information from a single video frame for judgment, lacking the ability to understand continuous actions and failing to accurately capture dynamic behavioral characteristics. Traditional detection methods often ignore the influence of surrounding environmental factors on behavior, leading to a high false positive rate. Current systems cannot differentiate the assessment of abnormal behavior based on the safety levels of different construction site areas, limiting the effectiveness of safety management. Summary of the Invention

[0005] This application provides a method for encrypted and isolated computation based on tenant identifiers to solve one of the aforementioned technical problems.

[0006] The technical solution adopted in this application is as follows:

[0007] This application provides a method for intelligent construction site safety monitoring and abnormal behavior detection, including:

[0008] Based on video frame data, a target detection algorithm is used to locate people and cigarettes in the construction site multimodal data to output the first detection result;

[0009] Based on the first detection result, video frame data within a preset time period, and construction site multimodal data, a second detection result is output using 3DCNN combined with an attention mechanism.

[0010] Based on the second detection result and the safety level information of the construction site area, a comprehensive anomaly score is obtained through the isolated forest and autoencoder algorithms.

[0011] According to one embodiment of this application, the step of locating people and cigarettes in the construction site multimodal data based on video frame data using a target detection algorithm to output a first detection result specifically involves: using the lightweight target detection algorithm YOLOv5s to analyze each frame of the image and identify the positions of people and cigarettes within it;

[0012] YOLOv5s predicts the location, confidence score, and class label for each object;

[0013] Based on the set thresholds and rules, targets that meet the conditions are filtered out to output the first detection result containing information on the location of people and cigarettes, specifically including bounding box coordinates, confidence scores, and category labels.

[0014] According to one embodiment of this application, the step of using 3D CNN combined with an attention mechanism to output a second detection result based on the first detection result, video frame data within a preset time period and construction site multimodal data specifically involves: aligning the bounding box information in the first detection result with the corresponding video frame data, and merging other types of multimodal data as input to the 3D convolutional neural network;

[0015] The 3D convolutional neural network considers feature changes in spatial and temporal dimensions to capture changes in hand posture and cigarette position over time, and highlights key actions through an attention mechanism.

[0016] The presence of smoking behavior is determined through a comprehensive evaluation based on pre-defined behavioral judgment rules.

[0017] The smoking behavior and its detailed description are output as the second detection result.

[0018] According to one embodiment of this application, the step of obtaining a comprehensive anomaly score based on the second detection result and the safety level information of the construction site area through the isolated forest and autoencoder algorithms specifically involves combining the detailed behavioral description in the second detection result with the safety level information of the construction site area as input data.

[0019] The input data is processed using both the isolated forest and autoencoder algorithms, and anomaly scores are generated for each.

[0020] Based on the specific needs of the construction site, the weight parameters are dynamically adjusted to balance the importance of the two algorithms, isolated forest and autoencoder.

[0021] The abnormal scores from the two algorithms are combined into a comprehensive abnormal score by weighted summation.

[0022] According to one embodiment of this application, before locating personnel and cigarettes in the construction site multimodal data based on video frame data using a target detection algorithm to output a first detection result, the method further includes: preprocessing the video frame data, specifically:

[0023] The raw RGB video frames from the camera are denoised using Gaussian filtering.

[0024] Adjusting the brightness and contrast of an image to improve its quality;

[0025] Apply data augmentation techniques to increase the diversity of the model training set;

[0026] The video frame data is obtained by outputting the data.

[0027] According to one embodiment of this application, after obtaining the comprehensive anomaly score based on the second detection result and the construction site area safety level information using the isolated forest and autoencoder algorithms, the method further includes:

[0028] Based on the safety level information of the construction site area, set the first alarm threshold and the second alarm threshold;

[0029] The comprehensive anomaly score is compared with the first alarm threshold and the second alarm threshold;

[0030] If the overall anomaly score is greater than or equal to the first alarm threshold, a level one alarm is triggered;

[0031] If the overall anomaly score is greater than or equal to the second alarm threshold, a level two alarm is triggered.

[0032] According to one embodiment of this application, it also includes:

[0033] Use the smoking behavior from the first detection result as input;

[0034] Activation maps of the hand-to-mouth region and existing high-temperature points were generated using a convolutional neural network.

[0035] Based on the generated activation map, calculate the activation values ​​for the hand-to-mouth region;

[0036] Examine the high-temperature regions in the infrared images of the construction site multimodal data, and calculate the overlap rate between the high-temperature regions and the high-activation value regions in the activation map;

[0037] When both the activation value and the overlap rate are greater than a preset threshold, it is determined to be a high-confidence smoking behavior.

[0038] A computer program product containing instructions that, when run on a device, causes the device to perform the steps in implementing the method.

[0039] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the method.

[0040] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method.

[0041] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0042] This application introduces an advanced object detection algorithm that can quickly and accurately identify people and their held objects (such as cigarettes) in videos, laying a solid foundation for subsequent analysis. This step solves the problem of single-dimensional analysis and provides preliminary behavioral clues.

[0043] This application utilizes 3D CNNs to effectively process spatiotemporal information, and combines this with an attention mechanism to enhance the model's focus on key regions, thereby improving its ability to understand complex behavioral patterns. This step overcomes the shortcomings of traditional methods in handling continuous actions and significantly improves the accuracy of behavior recognition.

[0044] This application integrates data from different sources and applies advanced anomaly detection algorithms (such as isolated forests and autoencoders) to quantitatively assess potential risks based on the specific safety level of a construction site, achieving personalized and refined safety management. This process fills the gaps in existing technologies regarding comprehensive assessment systems, helps reduce false alarm rates, and improves safety management efficiency. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart illustrating a method for intelligent construction site safety monitoring and abnormal behavior detection provided in an embodiment of this application. Detailed Implementation

[0047] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0048] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0049] In this application, unless otherwise expressly specified and limited, the "above" or "below" of the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0050] Example 1

[0051] like Figure 1 As shown, a method for intelligent construction site safety monitoring and abnormal behavior detection includes:

[0052] A method for intelligent construction site safety monitoring and abnormal behavior detection includes:

[0053] S100. Based on video frame data, locate personnel and cigarettes in the construction site multimodal data using a target detection algorithm, and output the first detection result.

[0054] As mentioned above, various types of data are collected from the construction site, including RGB images (for capturing personnel movements), infrared images (for detecting high-temperature areas), sound data (for capturing the sound of lighters igniting and exhalation), and smoke concentration data (for monitoring local air particulate matter concentration). This data is then preprocessed: Gaussian filtering is used to remove noise from the image and sound data. Brightness and contrast are adjusted in the image data, and data augmentation operations (such as rotation, scaling, and cropping) are performed. Smoking behavior is labeled based on whether positive and negative samples meet specific rules, thus constructing a training dataset.

[0055] The lightweight object detection algorithm YOLOv5s was used to locate people and cigarettes in video frames. The YOLOv5s loss function consists of three parts: bounding box loss, confidence loss, and class loss, designed to optimize model performance. Specifically:

[0056] Bounding box loss: EIoU (Extended Intersection over Union) is used. Compared to CIoU (Complete Intersection over Union), EIoU can more accurately measure the differences in bounding box shape and performs better in complex scenes. Confidence loss: Measures the degree of confidence that the predicted bounding box contains an object. Classification loss: Ensures classification accuracy.

[0057] A YOLOv5s model was trained using labeled data, and its overall loss function was optimized. The optimized model improved the accuracy of person detection from 89.2% to 93.7%, and the accuracy of cigarette detection from 63.5% to 78.4%. The specific judgment rules are as follows:

[0058] The confidence level of the personnel detection bounding box must be greater than 0.8, and the category loss must be greater than 0.9.

[0059] The confidence level of the cigarette detection bounding box must be greater than 0.7, and the overlap rate with the area of ​​a person's hand must be greater than 60%.

[0060] After the above steps, the system will output the "first detection result," which is the preliminary location information of the people and cigarettes appearing in the video frame. This result forms the basis for subsequent advanced analysis such as behavior recognition and abnormal behavior detection. It not only improves the accuracy and real-time performance of detection but also provides crucial evidence for further determining whether smoking has occurred. Furthermore, by combining multimodal data (such as visual, infrared, sound, and smoke concentration), the system's adaptability and robustness are further enhanced, enabling it to work effectively even in complex construction site environments.

[0061] For example, suppose that within a specific time period, a camera captures a sequence of video frames, an infrared sensor detects localized temperature changes, a microphone records ambient sounds, and a smoke sensor measures the concentration of particulate matter in the air. The video frames are then subjected to Gaussian filtering for noise reduction, and brightness and contrast are adjusted to improve image quality. The audio data is also denoised to more clearly identify key sounds (such as the sound of a lighter igniting). This preprocessed data provides high-quality input for subsequent object detection.

[0062] The system first selected a video frame as an example, showing a worker standing in a corner. The YOLOv5s algorithm began scanning the entire image, attempting to find all possible human silhouettes and cigarette shapes. In this frame, the algorithm successfully identified a human silhouette and marked a bounding box. Based on the optimized model parameters, the bounding box had a confidence score of 0.85, exceeding the set threshold of 0.8, indicating that this was a reliable person detection result.

[0063] Next, near the person's hand, the algorithm also detected a small object with a shape similar to a cigarette. The bounding box confidence score of this object was 0.75, which exceeded the set threshold of 0.7. Furthermore, the position of this small object overlapped with the area of ​​the person's hand by 65%, which met the requirement of an overlap rate of over 60%.

[0064] Based on the above analysis, the system outputs the "first detection result," indicating that a worker was found in this image frame, and he / she appears to be holding a cigarette. This result includes not only the specific location information (bounding box coordinates) of the person and the cigarette, but also their respective confidence scores and category labels (confirming whether it is a person or a cigarette).

[0065] The results of this initial location will be used in subsequent steps, such as behavior recognition. The system will further analyze this data, combining it with information from other sensors (such as high-temperature points displayed in infrared images, and the sound of a lighter igniting in audio data) to determine whether the worker is actually smoking. If it is ultimately determined that the behavior is indeed smoking, the system will also assess whether it occurred in a no-smoking area, and then decide whether to trigger an alarm mechanism.

[0066] It is important to note that, based on this embodiment, after initially determining the location of the person and the cigarette, further analysis of specific behavioral patterns can be conducted. For example, a 3D convolutional neural network (3D CNN) combined with an attention mechanism can be used to capture changes in hand-to-mouth movements, thereby more accurately determining whether smoking has occurred. By tracking the continuous movements of a specific object using video frame sequences, not only can smoking be identified, but other potentially unsafe behaviors can also be analyzed, such as not wearing a helmet or not using protective equipment.

[0067] Based on this embodiment, a comprehensive anomaly score can be calculated using isolated forest and autoencoder algorithms, based on the first detection result and other sensor data (such as infrared images, sound data, and smoke concentration), to identify and issue warnings for abnormal behavior in high-risk areas. Once abnormal behavior (such as smoking in a no-smoking area) is detected, an alarm mechanism is immediately triggered to notify on-site management personnel to take measures to prevent safety accidents from occurring.

[0068] S200: Based on the first detection result, video frame data within a preset time period, and construction site multimodal data, a second detection result is output using a 3D CNN combined with an attention mechanism.

[0069] As described above, the location information of people and cigarettes, along with their confidence scores, are obtained from object detection algorithms (such as YOLOv5s). A continuous video frame containing the first detection results is selected to capture the temporal features of the action. In addition to RGB images, infrared images, audio data, and smoke concentration data are also included, providing information from different perspectives regarding potential smoking behavior.

[0070] The bounding box information from the first detection result is aligned with the corresponding video frame data, and other types of multimodal data (such as infrared images, audio data, and smoke concentration data) are merged as input to the 3D CNN. These data will collectively form a comprehensive dataset for subsequent action recognition.

[0071] Using a 3D convolutional neural network to process integrated multimodal data, this network can simultaneously consider feature variations in both the spatial dimension (data provided by RGB images and other sensors) and the temporal dimension (through consecutive video frame sequences). The 3D CNN can capture changes in hand pose and the cigarette's position over time, which is crucial for recognizing smoking behavior.

[0072] By incorporating an attention mechanism into 3D CNNs, the model's focus on key regions (such as the motion path from hand to mouth, high-temperature points, etc.) is enhanced, thereby improving its sensitivity to these important cues. The attention mechanism can be implemented using the Self-Attention method, emphasizing the relationship between the query matrix (Q), key matrix (K), and value matrix (V), allowing the model to focus more intently on important local features.

[0073] Based on the results of multimodal data fusion, specific behavioral judgment rules are applied to comprehensively assess whether smoking behavior exists. For example:

[0074] A visual confidence score greater than 0.8 indicates a periodic hand gesture of holding a cigarette.

[0075] Infrared confidence level greater than 0.7 (indicating the presence of a high-temperature point that lasts for more than a specific duration);

[0076] The sound confidence level is greater than 0.6 (the sound of a lighter igniting and the sound of exhalation were detected).

[0077] The smoke concentration gradient exceeds the set threshold (e.g., it increases by 50 μg / m³ within 60 seconds after the action occurs).

[0078] After the above steps, the system will output a second detection result regarding whether smoking behavior has occurred. This result not only includes a binary judgment (i.e., whether smoking behavior has occurred), but may also include specific behavioral details, such as the specific time point of the action, the duration, and the body parts involved.

[0079] This second detection result provides a foundation for subsequent abnormal behavior detection, early warning, and recording.

[0080] For example, on a construction site, various sensors and cameras are installed to monitor site safety. One day, the system uses an object detection algorithm (such as YOLOv5s) to perform a preliminary analysis of a video and outputs the "first detection result," which indicates that a worker was found in a certain frame of the image, and he appears to be holding a cigarette in his hand.

[0081] First test result

[0082] Personnel location: bounding box coordinates (x1, y1, x2, y2), confidence level 0.87.

[0083] Cigarette location: bounding box coordinates (x3, y3, x4, y4), with an overlap rate of 65% with the area of ​​the person's hand, and a confidence level of 0.76.

[0084] Next, based on this preliminary result, the system will combine video frame data and other multimodal data (infrared images, sound data, smoke concentration data) within a preset time period, and use 3D CNN combined with an attention mechanism to perform more in-depth behavior recognition in order to determine whether smoking behavior exists.

[0085] The system selects a sequence of video frames from several consecutive seconds containing the aforementioned individuals and their suspected cigarettes as input. Simultaneously, it collects infrared images (monitoring for high-temperature spots near hands or cigarettes), sound data (capturing the sound of a lighter igniting or exhalation), and smoke concentration data (monitoring changes in airborne particulate matter concentration) within the same time period.

[0086] Using a 3D CNN to process these video frame sequences, the network is able to simultaneously consider feature variations in both the spatial dimension (data provided by RGB images and other sensors) and the temporal dimension (through consecutive video frame sequences). For example, in this video, the 3D CNN can capture the worker's hand holding a cigarette and periodically raising it to his mouth, while also detecting temperature changes in the vicinity of the hand or cigarette.

[0087] An attention mechanism is added to the 3D CNN to enhance the model's focus on key regions (such as the action path from hand to mouth, high-temperature points, etc.). The attention mechanism emphasizes these key regions, making the model more focused on important local features. For example, if the hand-to-mouth region frequently shows high activation values, it indicates that this region is an important basis for the model to judge smoking behavior.

[0088] Based on the results of multimodal data fusion, specific behavioral judgment rules are applied to comprehensively assess whether smoking behavior exists:

[0089] Visual data: The action of holding a cigarette was detected, and the frequency of the hand periodically raising to the mouth was about 0.5 to 1 time per second, with a confidence level greater than 0.8.

[0090] Infrared data: Transient high temperature point (temperature > 300℃ and lasting > 1 second) detected near the hand or cigarette, with a confidence level greater than 0.7.

[0091] Sound data: The sound of a lighter igniting (signal-to-noise ratio >15dB) and subsequent periodic exhalation sounds were detected, with a confidence level greater than 0.6.

[0092] Smoke concentration data: The local PM2.5 concentration rises above the threshold (e.g., 100 μg / m³) within 60 seconds after the action is taken, and the gradient exceeds the set threshold.

[0093] Based on the above analysis, the system confirmed that the worker was indeed smoking and output detailed second detection results:

[0094] Confirm the presence of smoking behavior. This includes the start time (e.g., starting from the 5th second), duration (e.g., lasting 15 seconds), body parts involved (hand to mouth movements), and other relevant characteristics (such as the specific location of the infrared high-temperature point, sound characteristics, and trends in smoke concentration).

[0095] It is important to note that, based on this embodiment, in addition to identifying smoking behavior, other complex behavioral patterns, such as not wearing a safety helmet or not using protective equipment, can also be identified using 3D CNN and attention mechanisms. This can be achieved by training the model to recognize different action sequences. By utilizing historical data and current behavioral patterns, the system can predict workers' potential future behaviors. For example, if a worker frequently violates regulations, the system can issue an early warning to prevent potential safety accidents.

[0096] Based on this embodiment, once abnormal behavior (such as smoking in a no-smoking area) is detected, an alarm mechanism is immediately triggered to notify on-site management personnel to take action. This not only improves response speed but also effectively prevents safety accidents. Personalized safety behavior reports are provided to each worker to help them understand their behavioral habits and provide improvement suggestions. This feedback mechanism helps improve workers' safety awareness.

[0097] Building upon this embodiment, in addition to existing RGB images, infrared images, sound data, and smoke concentration data, other types of construction site environmental data, such as weather conditions and work progress, can be integrated to provide a more comprehensive safety assessment. Generative Adversarial Networks (GANs) or other data augmentation techniques can be used to create more training samples, improving the model's generalization ability and robustness. This is particularly useful for handling complex construction site environments.

[0098] S300. Based on the second detection result and the safety level information of the construction site area, a comprehensive anomaly score is obtained through the isolated forest and autoencoder algorithms.

[0099] As described above, the results regarding the presence of smoking behavior are derived from analysis using 3D CNN combined with an attention mechanism. These include, but are not limited to:

[0100] Specific details of smoking behavior, such as the timing and duration of the action. Indicators including visual confidence level, infrared confidence level, sound confidence level, and smoke concentration gradient. Safety level information for different areas within the construction site, such as higher safety levels for high-risk areas like fuel storage areas and material storage areas.

[0101] The feature information from the second detection result (such as hand gestures holding a cigarette, sound characteristics, changes in smoke concentration, etc.) is combined with the safety level information of the construction site area as input data for subsequent processing. This information will be used to assess whether a specific behavior constitutes abnormal behavior in a specific area.

[0102] For each input sample, the Isolation Forest algorithm is used to compute its anomaly score. This algorithm excels at identifying instances that are significantly different from other samples, i.e., potentially anomalous behavioral patterns. The Isolation Forest algorithm generates an anomaly score, representing the likelihood that a sample is identified as anomalous. This score reflects the uniqueness or deviation of the behavioral pattern from the overall dataset.

[0103] An autoencoder reconstructs the input data and calculates the difference between the original input and the reconstructed output as another anomaly score. This difference reflects whether there are parts of the data that do not conform to the expected pattern. The autoencoder algorithm also produces an anomaly score to measure how much the input data deviates from the normal pattern. This method is particularly suitable for capturing complex and subtle changing patterns.

[0104] Depending on the application scenario (e.g., no-smoking area or regular area), the weight parameter λ is dynamically adjusted to balance the importance of the isolated forest and autoencoder algorithms.

[0105] In no-smoking zones, more emphasis may be placed on spatial violations, so the Isolation Forest algorithm is given a higher weight (e.g., λ=0.8) to emphasize anomaly detection capabilities;

[0106] In normal scenarios, more attention may be paid to abnormal behavior patterns. In this case, the weight of the autoencoder algorithm is relatively higher (e.g., λ=0.4), emphasizing the ability to recognize patterns.

[0107] Combining the two abnormal scores mentioned above, a comprehensive abnormal score is calculated by weighted summation. This score represents the degree of abnormality of the entire behavioral pattern.

[0108] Set appropriate alarm thresholds based on the safety level of the construction site area. For example:

[0109] In no-smoking areas (such as oil storage areas), a low threshold (e.g., 0.6) is set, and a level one alarm is triggered when the overall anomaly score exceeds this threshold;

[0110] In the normal zone, a higher threshold is set (e.g., 0.8), and a level 2 alarm is only triggered when the overall abnormal score is significantly higher than the normal range.

[0111] The final output is a numerical comprehensive anomaly score, which reflects the degree of anomaly of a specific behavior in the overall site environment.

[0112] Based on the comparison between the comprehensive anomaly score and the preset threshold, it is determined whether to trigger an alarm and its level, thereby achieving effective monitoring and management of smoking behavior among construction site personnel.

[0113] For example, suppose a worker is initially identified by the system as potentially smoking at a certain point in time, and a "second detection result" is output. The system needs to further determine whether this behavior constitutes a high-risk abnormal behavior, especially whether the behavior occurred in a high-risk area (such as an oil storage area).

[0114] Second detection result (from 3D CNN + attention mechanism module):

[0115] Visual confidence level: 0.82 (the hand holding the cigarette is obvious)

[0116] Infrared confidence level: 0.75 (temperature rise near the hand)

[0117] Sound confidence level: 0.68 (lighter ignition sound and exhalation sound detected)

[0118] Smoke concentration gradient change: +60μg / m³ (particulate matter concentration increases after smoking)

[0119] Safety level information for construction site area:

[0120] Current location: Oil warehouse (no-smoking area)

[0121] Safety level: Extremely high risk area

[0122] Preset alarm threshold: Overall anomaly score > 0.6 → Trigger Level 1 alarm.

[0123] The various indicators in the second detection result are converted into a standardized feature vector:

[0124] Eigenvector = [0.82, 0.75, 0.68, 0.6]

[0125] These values ​​represent the confidence level or intensity of change in visual, infrared, sound, and smoke aspects, respectively.

[0126] Isolation Forest is an unsupervised anomaly detection algorithm that excels at identifying "rare" behaviors that differ significantly from the majority of samples.

[0127] Inputting the above features into the Isolation Forest model, the model outputs a normalized anomaly score:

[0128] Isolated forest anomaly score = 0.72

[0129] This behavior is significantly off-center from the overall behavior distribution, especially considering that it occurred in a no-smoking area, so it scores medium to high.

[0130] An autoencoder is a reconstructive neural network model used to measure the deviation between input data and a normal pattern.

[0131] Given the same feature vector, the autoencoder attempts to reconstruct it and calculates the reconstruction error (the further it deviates from the normal pattern, the higher the error).

[0132] Autoencoder anomaly score = 0.65

[0133] This behavior deviates from the normal operating pattern in some dimensions (such as smoke concentration), but not as extreme as the isolated forest's assessment.

[0134] Since the current location is a high-risk no-smoking area, the system will place greater emphasis on the risk of spatial violations, thus assigning a higher weight to the Isolation Forest algorithm, for example:

[0135] λ = 0.75 (Isolation Forest Weight)

[0136] 1 - λ = 0.25 (autoencoder weights)

[0137] Final calculation:

[0138] Overall anomaly score = 0.75 × 0.72 + 0.25 × 0.65 = 0.7025

[0139] According to the preset rules:

[0140] If the overall anomaly score is greater than 0.6, a Level 1 alarm will be triggered.

[0141] If the value is greater than 0.8, a level 2 alarm will be triggered.

[0142] Overall anomaly score = 0.7025 > 0.6 → Trigger Level 1 alarm

[0143] The system immediately notified on-site management personnel via voice broadcast, recorded the incident, and generated a safety management report.

[0144] It is important to note that, based on this embodiment, multiple alarm levels (such as low, medium, and high) can be set according to different ranges of comprehensive anomaly scores, and different response measures can be configured for each level. For example, for behaviors with low scores, the system can notify management personnel via SMS or email; while for behaviors with high scores, an on-site voice warning will be triggered immediately. The alarm thresholds are dynamically adjusted according to different time periods or special events (such as holidays or peak construction periods) to ensure the sensitivity and accuracy of the system.

[0145] Building upon this embodiment, real-time data stream processing frameworks (such as Apache Kafka and Apache Flink) can be introduced to perform real-time analysis of multimodal data, enabling faster responses. This helps to take action as soon as abnormal behavior is detected, reducing potential risks. Providing personalized safety behavior reports to each worker helps them understand their behavioral habits and provides improvement suggestions. This feedback mechanism helps to improve workers' safety awareness.

[0146] Building upon this embodiment, the model can be retrained and optimized periodically as new data accumulates, ensuring its adaptability to changes in the construction site environment and maintaining high accuracy. For example, online learning methods can be used to enable the model to automatically adjust parameters during runtime. Interpretable AI (XAI) technologies, such as LIME (Local Interpretable Model-agnostic Explanations) or SHAP (SHapley Additive explanations), can be used to generate easily understandable explanations, helping managers better understand and trust model decisions.

[0147] According to one embodiment of this application, the step of locating people and cigarettes in the construction site multimodal data based on video frame data using a target detection algorithm to output a first detection result specifically involves: using the lightweight target detection algorithm YOLOv5s to analyze each frame of the image and identify the positions of people and cigarettes within it;

[0148] YOLOv5s predicts the location, confidence score, and class label for each object;

[0149] Based on the set thresholds and rules, targets that meet the conditions are filtered out to output the first detection result containing information on the location of people and cigarettes, specifically including bounding box coordinates, confidence scores, and category labels.

[0150] As described above, continuous RGB image frames are acquired from cameras installed on the construction site.

[0151] In addition to RGB images, it also includes infrared images, sound data, and smoke concentration data, but these are mainly used for subsequent behavior recognition and anomaly detection stages.

[0152] Before inputting video frames into YOLOv5s, some preprocessing operations are usually required, such as resizing the image (YOLOv5s has specific input size requirements) and normalizing pixel values. These operations help improve the model's performance and stability.

[0153] First, load the pre-trained YOLOv5s model. This model is optimized to reduce computational resource requirements while maintaining high accuracy, making it ideal for real-time applications.

[0154] For each frame of video image, YOLOv5s performs the following operations:

[0155] Convolutional Neural Networks (CNNs) are used to extract feature maps from images. Multiple candidate bounding boxes are generated on these feature maps, and for each bounding box, a corresponding confidence score (i.e., the probability that the bounding box contains an object) and a class label (e.g., "person" or "cigarette") are predicted. To remove redundant bounding boxes with excessive overlap, YOLOv5s employs non-maximum suppression, retaining only the bounding box with the highest score.

[0156] Set a confidence threshold (e.g., 0.8) based on actual needs. Only when the confidence score of the bounding box exceeds this threshold is the object within that bounding box considered a valid detection result. For different object types (such as people and cigarettes), additional classification accuracy requirements can be set (e.g., a class loss greater than 0.9 for people detection bounding boxes; a class loss greater than 0.7 for cigarette detection bounding boxes). Considering that cigarettes are often associated with the hand region of a person, rules can be set to check whether the cigarette bounding box has sufficient overlap with the person's hand region (e.g., an overlap rate exceeding 60%), thereby further confirming the effectiveness of the detection.

[0157] After the above steps, the system will output the first detection result, which includes the location information of the person and the cigarette. Specifically, this includes:

[0158] The precise location of each detected object (person or cigarette) in the image is usually represented by four values ​​(x_min, y_min, x_max, y_max), which correspond to the coordinates of the upper left and lower right corners of the bounding box, respectively.

[0159] The confidence score represents the degree of certainty that each bounding box contains the corresponding object, ranging from 0 to 1, with a higher confidence level being closer to 1.

[0160] Identify the object type corresponding to each bounding box, such as "person" or "cigarette".

[0161] According to one embodiment of this application, the step of using 3D CNN combined with an attention mechanism to output a second detection result based on the first detection result, video frame data within a preset time period and construction site multimodal data specifically involves: aligning the bounding box information in the first detection result with the corresponding video frame data, and merging other types of multimodal data as input to the 3D convolutional neural network;

[0162] The 3D convolutional neural network considers feature changes in spatial and temporal dimensions to capture changes in hand posture and cigarette position over time, and highlights key actions through an attention mechanism.

[0163] The presence of smoking behavior is determined through a comprehensive evaluation based on pre-defined behavioral judgment rules.

[0164] The smoking behavior and its detailed description are output as the second detection result.

[0165] As described above, the location information of people and cigarettes and their confidence scores are obtained from target detection algorithms (such as YOLOv5s).

[0166] A pre-selected segment of consecutive video frames containing the first detection result is used to capture the temporal features of the action. In addition to RGB images, infrared images, audio data, and smoke concentration data are also included, providing information from different perspectives about potential smoking behavior.

[0167] The bounding box information (positions of people and cigarettes) from the first detection result is aligned with the corresponding video frame data. This means that for each video frame, the specific locations of people and cigarettes are determined. Infrared images (providing temperature change information), sound data (capturing the sound of a lighter igniting or exhalation), and smoke concentration data (monitoring changes in airborne particulate matter concentration) are used as input to the 3D CNN. This creates a comprehensive dataset, providing rich information for subsequent behavior recognition.

[0168] 3D CNNs can process information in both spatial and temporal dimensions simultaneously. They use convolutional operations to slide a window in three-dimensional space, thereby capturing changes in hand pose and the position of a cigarette over time.

[0169] For example, in a few seconds of video, a 3D CNN can identify the movement pattern of a worker holding a cigarette and periodically raising it to his mouth.

[0170] At the same time, it can also detect temperature changes, sound characteristics, and trends in smoke concentration near the hand or cigarette.

[0171] Adding an attention mechanism to a 3D CNN enhances the model's focus on key regions (such as the action path from hand to mouth, high-temperature points, etc.). This helps improve sensitivity to important cues. The attention mechanism can be implemented using the Self-Attention method, emphasizing the relationship between the query matrix (Q), key matrix (K), and value matrix (V), allowing the model to focus more on important local features. For example, if the hand-to-mouth region frequently shows high activation values, this indicates that this region is an important basis for the model to judge smoking behavior.

[0172] Based on the results of multimodal data fusion, specific behavioral judgment rules are applied to comprehensively assess whether smoking behavior exists. These rules may include:

[0173] The action of holding a cigarette in the hand was detected, and the frequency of the hand periodically raising to the mouth was about 0.5 to 1 time per second, with a confidence level greater than 0.8.

[0174] A transient high-temperature point (temperature > 300°C and duration > 1 second) is detected on the hand or near a cigarette with a confidence level greater than 0.7.

[0175] The sound of a lighter igniting (signal-to-noise ratio >15dB) and subsequent periodic exhalation sounds were detected with a confidence level greater than 0.6.

[0176] If the local PM2.5 concentration rises above the threshold (e.g., 100 μg / m³) within 60 seconds after the action is taken, the gradient exceeds the set threshold.

[0177] Based on the above analysis, the system confirms whether smoking behavior has occurred and outputs detailed second detection results. These results include not only a binary judgment (i.e., whether smoking behavior occurred), but may also include specific behavioral details, such as:

[0178] Start time: For example, starting from the 5th second.

[0179] Duration: For example, it lasted for 15 seconds.

[0180] Body parts involved: Hand to mouth movements.

[0181] Other relevant characteristics include the specific location of the infrared high-temperature point, sound characteristics, and the trend of smoke concentration changes.

[0182] According to one embodiment of this application, the step of obtaining a comprehensive anomaly score based on the second detection result and the safety level information of the construction site area through the isolated forest and autoencoder algorithms specifically involves combining the detailed behavioral description in the second detection result with the safety level information of the construction site area as input data.

[0183] The input data is processed using both the isolated forest and autoencoder algorithms, and anomaly scores are generated for each.

[0184] Based on the specific needs of the construction site, the weight parameters are dynamically adjusted to balance the importance of the two algorithms, isolated forest and autoencoder.

[0185] The abnormal scores from the two algorithms are combined into a comprehensive abnormal score by weighted summation.

[0186] As described above, the behavioral feature data from the second detection result is combined with the regional security level information to form a unified multi-dimensional feature vector, which serves as the input to the model. An example input feature vector is shown below:

[0187] [Visual confidence level: 0.82, Infrared confidence level: 0.75, Sound confidence level: 0.68, Smoke concentration change: +60μg / m³, Area safety level: High risk (1.0)]

[0188] Note: Regional security levels can be mapped to numerical values ​​(e.g., low risk = 0.2, medium risk = 0.5, high risk = 1.0) to facilitate model processing.

[0189] Anomaly scores are obtained using Isolation Forest, an unsupervised anomaly detection algorithm that excels at identifying data points that deviate from the majority of samples in a high-dimensional space. It is particularly well-suited for detecting "spatial violation" anomalies, such as smoking behavior occurring in a high-risk area.

[0190] Output: A normalized anomaly score (typically in the range [0, 1], with scores closer to 1 indicating a stronger anomaly).

[0191] If a behavior occurs in an oil tank area (a high-risk area), even if the behavior itself is not particularly complex, it will be given a high anomaly score by the Isolation Forest.

[0192] Isolated forest anomaly score = 0.73

[0193] An autoencoder is used to obtain a reconstruction error score. An autoencoder is a deep learning model that is trained to reconstruct normal behavioral patterns. For data that deviates from the normal pattern, the reconstruction error is larger, thus it can be judged as an anomaly. It is suitable for discovering "abnormal behavioral patterns," where the behavior occurs in a normal area, but the action pattern is very atypical (such as frequent and repetitive inhalation, occlusion behavior).

[0194] Output: Also a normalized anomaly score (range [0, 1], the larger the score, the more abnormal the anomaly).

[0195] If a worker smokes in a regular rest area but smokes very frequently and does so while dodging, the autoencoder will consider this behavior abnormal.

[0196] Autoencoder anomaly score = 0.64

[0197] Based on the specific application scenario and management strategy of the construction site, the weights of the two algorithms are dynamically adjusted so that the final comprehensive anomaly score better reflects actual needs.

[0198] Scene type, isolated forest weight (λ), autoencoder weight (1-λ)

[0199] No-smoking area (high-risk area) 0.8 0.2

[0200] Regular work area 0.4 0.6

[0201] Integrated monitoring area 0.60.4

[0202] For example, if smoking is detected in a no-smoking area (such as an oil depot), the system will pay more attention to spatial violations and therefore give higher weight to isolated forests.

[0203] The final comprehensive anomaly score is:

[0204] Overall Anomaly Score = λ × Isolated Forest Score + (1-λ) × Autoencoder Score

[0205] Example: If λ = 0.75 (no-smoking area), the isolated forest score is 0.73, and the autoencoder score is 0.64, then:

[0206] Overall anomaly score = 0.75 × 0.73 + 0.25 × 0.64 = 0.7175

[0207] The overall anomaly score is a value between 0 and 1, reflecting how abnormal the behavior is in the current construction site environment.

[0208] This score can be used for subsequent alarm judgment, event classification management, historical trend analysis, etc.

[0209] According to one embodiment of this application, before locating personnel and cigarettes in the construction site multimodal data based on video frame data using a target detection algorithm to output a first detection result, the method further includes: preprocessing the video frame data, specifically:

[0210] The raw RGB video frames from the camera are denoised using Gaussian filtering.

[0211] Adjusting the brightness and contrast of an image to improve its quality;

[0212] Apply data augmentation techniques to increase the diversity of the model training set;

[0213] The video frame data is obtained by outputting the data.

[0214] As mentioned above, noise in the image is removed, especially random noise caused by camera sensor or environmental factors. A Gaussian filter is used to perform a convolution operation on each frame of the RGB image. Gaussian filtering is a linear smoothing filter that calculates a new pixel value by weighting the pixel values ​​around the pixel according to a Gaussian distribution. This filtering method effectively reduces high-frequency noise in the image while preserving the main structural features of the image as much as possible. The image after Gaussian filtering is smoother, reducing target detection errors caused by noise.

[0215] Improving the overall visual quality of an image makes target objects (such as people and cigarettes) easier to identify. The overall brightness of an image can be altered by increasing or decreasing the brightness value of each pixel (i.e., the values ​​of the three RGB channels). For images that are too dark or too bright, adjusting the brightness appropriately can make parts hidden in shadows or overexposed areas clearer. Contrast refers to the degree of difference between the brightest and darkest areas of an image. Adjusting the contrast can make the boundaries between different objects in an image more distinct, helping to improve the accuracy of object detection algorithms. Optimized images have better visual quality, helping object detection algorithms to more accurately identify the location of people and cigarettes.

[0216] Increasing the diversity and size of the training dataset helps prevent overfitting and improves the model's generalization ability. Common data augmentation techniques include, but are not limited to:

[0217] Rotation: Randomly rotate the image by a certain angle to simulate shooting from different perspectives.

[0218] Scaling and panning: Enlarge, shrink, or pan an image to simulate the effect of shooting at different distances.

[0219] Flip: Flip the image horizontally or vertically to increase the diversity of the samples.

[0220] Color jittering: Randomly adjusts the color attributes of an image (such as hue, saturation, brightness, etc.) to simulate different lighting conditions.

[0221] Cropping: Randomly cropping sub-images from the original image to serve as new training samples helps the model learn different local features of the target object.

[0222] The diverse image samples generated by the aforementioned data augmentation techniques not only increase the size of the training set but also improve the model's adaptability to different scene changes.

[0223] After the above preprocessing steps, the original video frame data was significantly improved. Noise interference was removed, image quality and detail were enhanced, and data augmentation techniques increased the diversity of training samples. This high-quality video frame data provides a solid foundation for subsequent object detection tasks, helping to improve the accuracy and reliability of person and cigarette location recognition.

[0224] According to one embodiment of this application, after obtaining the comprehensive anomaly score based on the second detection result and the construction site area safety level information using the isolated forest and autoencoder algorithms, the method further includes:

[0225] Based on the safety level information of the construction site area, set the first alarm threshold and the second alarm threshold;

[0226] The comprehensive anomaly score is compared with the first alarm threshold and the second alarm threshold;

[0227] If the overall anomaly score is greater than or equal to the first alarm threshold, a level one alarm is triggered;

[0228] If the overall anomaly score is greater than or equal to the second alarm threshold, a level two alarm is triggered.

[0229] As mentioned above, the comprehensive anomaly score is a numerical value derived from the combined processing of the Isolation Forest and Autoencoder algorithms, representing the degree of anomaly of a certain behavior within its environment. This score is typically a value between 0 and 1, with higher scores indicating more anomalous behavior. The construction site area safety level information includes the safety level classification of various areas within the construction site (such as high-risk areas, medium-risk areas, low-risk areas, etc.), which determines the alarm strategies to be adopted for different areas.

[0230] Based on the safety level information of the construction site areas, two alarm thresholds are set for each area:

[0231] First alarm threshold (Threshold 1): When the overall anomaly score reaches or exceeds this threshold, the system will trigger a Level 1 alarm. A Level 1 alarm usually indicates a serious security risk that requires immediate action.

[0232] The second alarm threshold (Threshold 2): When the overall anomaly score reaches or exceeds this higher threshold, the system will trigger a second-level alarm. A second-level alarm may indicate an extremely critical situation, requiring immediate emergency measures to prevent an accident from occurring.

[0233] These thresholds can be determined based on the specific site's safety management regulations and historical data analysis. For example, in high-risk areas (such as oil storage areas), the first alarm threshold can be set to 0.6, and the second alarm threshold to 0.8; while in ordinary work areas, these two thresholds may be set to 0.7 and 0.9, respectively.

[0234] The calculated comprehensive anomaly score is compared with the pre-set first alarm threshold and second alarm threshold.

[0235] If the overall anomaly score is greater than or equal to the first alarm threshold, a level one alarm is triggered.

[0236] If the overall anomaly score is greater than or equal to the second alarm threshold, a second-level alarm is triggered.

[0237] Level 1 Alarm: When the overall anomaly score meets the conditions for a Level 1 alarm, the system will initiate corresponding response measures, such as sending an SMS notification to the on-site management personnel, playing an audio warning to prompt workers to stop dangerous behavior, and recording the details of the event for subsequent review.

[0238] Level 2 alarm: If the criteria for a Level 2 alarm are met, in addition to the Level 1 alarm measures mentioned above, more stringent intervention measures may be included, such as automatically shutting down relevant equipment, forcibly evacuating personnel, or directly contacting the emergency service team.

[0239] According to one embodiment of this application, it also includes:

[0240] Use the smoking behavior from the first detection result as input;

[0241] Activation maps of the hand-to-mouth region and existing high-temperature points were generated using a convolutional neural network.

[0242] Based on the generated activation map, calculate the activation values ​​for the hand-to-mouth region;

[0243] Examine the high-temperature regions in the infrared images of the construction site multimodal data, and calculate the overlap rate between the high-temperature regions and the high-activation value regions in the activation map;

[0244] When both the activation value and the overlap rate are greater than a preset threshold, it is determined to be a high-confidence smoking behavior.

[0245] As mentioned above, the location information of people and cigarettes obtained from object detection algorithms (such as YOLOv5s) and their confidence scores are of particular interest to the parts marked as smoking behavior.

[0246] Construction site multimodal data, including RGB images and infrared images, is used to provide additional information to assist in judgment.

[0247] Select relevant frames or segments identified as smoking behavior from the first detection results, and extract the key frames of the hand holding the cigarette to the mouth as input to the convolutional neural network (CNN) for subsequent processing.

[0248] Using a pre-trained convolutional neural network (which may be a model fine-tuned specifically for this task), the selected hand-to-mouth region is analyzed to generate an activation map of that region.

[0249] Activation maps show the key areas that the model focuses on when making predictions, which can help understand the basis of the model's decisions.

[0250] During this process, areas that may have high temperatures will also be identified, which are usually associated with smoking behavior (such as the heat generated when a lighter is lit).

[0251] Based on the generated activation map, the activation value of the hand-to-mouth region is calculated. This value reflects the model's confidence in the presence of smoking behavior in that region.

[0252] For example, if the hand-to-mouth area in the activation map shows a high level of activation, it indicates that the model believes that the area is highly likely to be where smoking is taking place.

[0253] Using infrared image data provided by the construction site, high-temperature areas in the image were located. Infrared images can capture the thermal radiation emitted by objects, making them ideal for detecting localized temperature increases, such as the high temperatures generated when smoking.

[0254] Calculate the overlap rate between these high-temperature regions and the high-activation regions in the previously generated activation map. The higher the overlap rate, the stronger the correlation between the two, and the more likely it is that the temperature rise is indeed caused by smoking.

[0255] Activation threshold: For example, 0.7. When the activation value of the area from the hand to the mouth exceeds this value, it is considered that there are significant signs of smoking in that area.

[0256] Overlap rate threshold: For example, 60%. When the overlap rate between the high temperature region and the high activation value region in the activation map reaches or exceeds this ratio, the existence of smoking behavior is further confirmed.

[0257] If both of the above conditions are met (i.e., the activation value is greater than 0.7 and the overlap rate is greater than 60%), the system will ultimately determine the behavior as "high-confidence smoking behavior".

[0258] Suppose a camera at a construction site captures a video clip where a worker is detected by the YOLOv5s algorithm to be holding what appears to be a cigarette, and his movement pattern matches the characteristics of smoking. Next:

[0259] The system selects keyframes containing the worker's hand-to-mouth movement with a cigarette, uses a convolutional neural network to generate an activation map of the hand-to-mouth region, and simultaneously identifies potential high-temperature locations.

[0260] Analysis revealed that the activation value of the area from the hand to the mouth was 0.78, exceeding the preset activation threshold (0.7), indicating that there were indeed significant signs of smoking in this area.

[0261] Meanwhile, the system examined infrared images within the same time period and discovered obvious high-temperature points, which largely overlapped with high-activation value areas in the activation image.

[0262] The calculated overlap rate was 65%, which exceeded the preset overlap rate threshold (60%).

[0263] Because the activation value (0.78) is greater than the activation value threshold (0.7) and the overlap rate (65%) exceeds the overlap rate threshold (60%), the system ultimately determines that this is a high-confidence smoking behavior.

[0264] A computer program product containing instructions that, when run on a device, causes the device to perform the steps in implementing the method.

[0265] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the method.

[0266] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method.

[0267] For any parts not mentioned in this application, existing technologies may be used or referenced.

[0268] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0269] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for intelligent construction site safety monitoring and abnormal behavior detection, characterized in that, include: Based on video frame data, a target detection algorithm is used to locate people and cigarettes in the construction site multimodal data to output the first detection result; Based on the first detection result, video frame data within a preset time period, and construction site multimodal data, a second detection result is output using a 3D CNN combined with an attention mechanism, specifically: Align the bounding box information in the first detection result with the corresponding video frame data, and merge other types of multimodal data as input to the 3D convolutional neural network; The 3D convolutional neural network considers feature changes in spatial and temporal dimensions to capture changes in hand posture and cigarette position over time, and highlights key actions through an attention mechanism. The presence of smoking behavior is determined through a comprehensive evaluation based on pre-defined behavioral judgment rules. The smoking behavior and its detailed description are output as the second detection result; Based on the second detection result and the safety level information of the construction site area, a comprehensive anomaly score is obtained through the isolated forest and autoencoder algorithms, specifically: The behavioral details described in the second test result are combined with the safety level information of the construction site area as input data; The input data is processed using both the isolated forest and autoencoder algorithms, and anomaly scores are generated for each. Based on the specific needs of the construction site, the weight parameters are dynamically adjusted to balance the importance of the two algorithms, isolated forest and autoencoder. The abnormal scores from the two algorithms are combined into a comprehensive abnormal score by weighted summation.

2. The method according to claim 1, characterized in that, The method involves using video frame data and a target detection algorithm to locate personnel and cigarettes in the multimodal data of the construction site, and outputting a first detection result, specifically as follows: The lightweight object detection algorithm YOLOv5s was used to analyze each frame of the image to identify the location of people and cigarettes. YOLOv5s predicts the location, confidence score, and class label for each object; Based on the set thresholds and rules, targets that meet the conditions are filtered out to output the first detection result containing information on the location of people and cigarettes, specifically including bounding box coordinates, confidence scores, and category labels.

3. The method according to claim 1, characterized in that, Before locating personnel and cigarettes in the construction site multimodal data using a target detection algorithm based on video frame data to output the first detection result, the method further includes: preprocessing the video frame data, specifically: The raw RGB video frames from the camera are denoised using Gaussian filtering. Adjusting the brightness and contrast of an image to improve its quality; Apply data augmentation techniques to increase the diversity of the model training set; The video frame data is obtained by outputting the data.

4. The method according to claim 1, characterized in that, After obtaining the comprehensive anomaly score based on the second detection result and the construction site area safety level information, using the isolated forest and autoencoder algorithms, the method further includes: Based on the safety level information of the construction site area, set the first alarm threshold and the second alarm threshold; The comprehensive anomaly score is compared with the first alarm threshold and the second alarm threshold; If the overall anomaly score is greater than or equal to the first alarm threshold, a level one alarm is triggered; If the overall anomaly score is greater than or equal to the second alarm threshold, a level two alarm is triggered.

5. The method according to claim 1, characterized in that, Also includes: Use the smoking behavior from the first detection result as input; Activation maps of the hand-to-mouth region and existing high-temperature points were generated using a convolutional neural network. Based on the generated activation map, calculate the activation values ​​for the hand-to-mouth region; Examine the high-temperature regions in the infrared images of the construction site multimodal data, and calculate the overlap rate between the high-temperature regions and the high-activation value regions in the activation map; When both the activation value and the overlap rate are greater than a preset threshold, it is determined to be a high-confidence smoking behavior.

6. A computer program product containing instructions, characterized in that, when run on a device, This causes the device to perform the steps of the method as described in any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-5.

Images