Small engineering safety risk identification method and system based on multi-modal data fusion

Abstract

The invention discloses a small engineering safety risk identification method and system based on multi-modal data fusion. According to the method, the CLIP pre-training model is combined with the LoRA technology, so that effective alignment of four-dimensional multi-modal data is realized, project text features, field images and videos, personnel portrait records and environment spatio-temporal data can be fused and analyzed in the same feature space, and the information barrier of a single data type is broken. According to a weighted loss function designed for the high risk, the learning priority of the model for fatal hidden dangers such as high falling and electric shock is enhanced, the situation that a common sample dilutes the feature weight of a high-risk sample is avoided, the high risk identification sensitivity is greatly improved, and through the combination of a two-stage training strategy and contrast learning, the high risk identification accuracy is improved. The model not only has a general building safety knowledge basis, but also can accurately adapt to fragmented scene requirements of small-sized projects, so that the small-sized projects turn from passive risk handling to active prevention, and the overall safety is improved.

Description

Technical Field

[0001] This invention belongs to the field of engineering risk identification technology, specifically a method and system for identifying safety risks in small-scale engineering projects based on multimodal data fusion. Background Technology

[0002] Safety risk identification for small-scale engineering projects is a fundamental management task that systematically identifies, analyzes, and evaluates potential hazards and unsafe factors for projects with small investment scale, relatively simple structure, and short construction period. Despite their limited size, small-scale projects also face common risks such as falls from heights, falling objects, mechanical injuries, electric shocks, and collapses, and accidents frequently occur due to inadequate on-site management, insufficient safety investment, and weak safety awareness among personnel. This identification process emphasizes addressing risks from four dimensions: unsafe acts of people, unsafe conditions of equipment, adverse environmental factors, and management deficiencies. It comprehensively investigates potential hazards at each stage of construction through methods such as on-site inspections, safety checklists, and job hazard analysis. Its core objective is to proactively identify and develop targeted control measures before risks escalate into accidents, making it a primary step in implementing the principle of "safety first, prevention foremost" and building a safety production defense line for small-scale engineering projects.

[0003] However, existing technologies often struggle to achieve effective cross-modal alignment of multimodal data, and the fragmentation of information from different types of data leads to incomplete risk identification; the lack of integration of general and small-project-specific scenario data during training makes it difficult to adapt to fragmented scenario requirements; thus reducing the accuracy of real-time risk identification and early warning. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for identifying safety risks in small-scale engineering projects based on multimodal data fusion in order to solve the problems mentioned above.

[0005] The technical solution adopted in this invention is as follows: a method for identifying safety risks in small-scale engineering projects based on multimodal data fusion, comprising the following steps:

[0006] S1: Collect basic project information, IoT device sensing data, personnel qualification and behavior records, and environmental dynamic data to provide comprehensive raw materials for subsequent analysis.

[0007] S2: Clean and denoise the collected video, image, text, and sensor numerical data, and bind the multi-source data of the same work area to form a unified risk feature library;

[0008] S3: Based on the CLIP cross-modal pre-trained model, cross-modal feature alignment of four-dimensional multimodal data is performed through LoRA low-rank adaptation technology; for high-risk risks such as falls from heights and electric shocks, which account for more than 85%, a weighted loss function based on Focal Loss is designed, which assigns three times the loss weight of ordinary samples to high-risk samples, so that the model learns fatal hazard features first.

[0009] A two-stage training strategy is adopted: the first stage uses a general building safety dataset to complete transfer learning; the second stage imports fragmented scenario data specific to small projects for fine-tuning, while introducing comparative learning to allow the model to learn the subtle differences between compliant operations and potential hazard scenarios; finally, the knowledge of the large model is transferred to a lightweight model to achieve real-time recognition.

[0010] S4: Using a trained model, analyze the real-time images and sensor data transmitted by drones and robot dogs to identify potential hazards on site; at the same time, link the data with the mobile phones of the operation and monitoring personnel, combine location to judge violations, and generate a real-time risk list.

[0011] S5: Push early warnings and handling suggestions based on risk levels, record the handling process and results, and accumulate data for system optimization;

[0012] S6: Feedback is provided on the early warning response, and the feedback results are used to update the risk feature database and adjust the model parameters. Then, the optimized model and data are fed back to the preceding steps, and the entire small-scale engineering safety risk identification process based on multimodal data fusion can be completed.

[0013] In a preferred embodiment, step S1 first involves collecting basic project information, including the overall project scale, specific work types, and a pre-defined database of high-risk scenarios. This information forms the initial reference framework for the model to identify risk scenarios. Next, data from IoT devices is collected, including videos taken by drones at the work site, detection data obtained by robotic dogs in concealed areas, real-time operating parameters uploaded by construction equipment sensors, and location information and operation records uploaded by personnel via mobile phones. This data reflects the dynamic changes on-site in real time. Simultaneously, personnel qualification and behavior files are compiled, including qualification certificates from construction units, personnel safety training completion status, and records of past violations, providing crucial evidence for the model to assess personnel operational risks. Finally, dynamic environmental data is acquired, including real-time updated meteorological information, natural disaster warning signals, and detailed information on current geological conditions. This data helps the model determine the potential impact of the external environment on construction safety. All collected information is integrated into comprehensive raw materials for subsequent risk analysis by the model, ensuring that the model can conduct accurate analysis based on complete information.

[0014] In a preferred embodiment, step S2 first involves cleaning and denoising the data, including video images, text, and sensor values. This removes blurry and invalid frames from the video, interference noise from the image, erroneous information from the text record, and abnormal fluctuations in the sensor values, ensuring that each piece of data has real and valid analytical value. Next, multi-source data from the same work area are bound together. Images of the exterior walls captured by the drone are matched with wall crack detection information obtained by the robot, and personnel location information is matched with the operating status of nearby equipment, forming an interconnected organic whole from previously scattered data of different types. Finally, these processed and correlated data are integrated to extract key features reflecting safety risks, including the structural state of the work area, equipment operational stability, deviations from regulations in personnel activity trajectories, and the degree of influence of environmental factors. These features are categorized and stored to form a unified risk feature library, serving as the basic material for model training. This ensures that the model can learn accurate risk identification logic from comprehensive and correlated features, providing strong support for subsequent early warning analysis.

[0015] In a preferred embodiment, in step S3, the four-dimensional multimodal data is first processed for modal adaptation: the project scale / type and personnel qualification / behavioral records are directly input into CLIP's text encoder as text; the image and video frames collected by the drone / machine dog are used to extract visual features through CLIP's visual encoder; and the environmental and meteorological sensor data are converted into structured text descriptions and also input into the text encoder.

[0016] Next, a LoRA low-rank adaptation module is inserted into each linear layer of the CLIP Transformer encoder, introducing two low-rank matrices A (dimension d×r) and B (dimension r×d, where r is much smaller than d), adjusting the original linear layer output to Wx + BAx (W is the pre-trained original weight matrix, and x is the input feature vector); by training and optimizing the parameters of A and B, different modal features are aligned in the same semantic space without changing the original model weights.

[0017] The formula for the weighted cross-modal contrast loss is:

[0018] ;

[0019] In the formula:

[0020] L is the total loss value of the weighted cross-modal contrastive loss function, which is used to quantify the learning error of the model in semantic association of multimodal data. The smaller the loss, the better the model's alignment effect on "image scene-text description-sensor features".

[0021] i is the index of a single sample in the current training batch, with a value ranging from 1 to N.

[0022] N represents the total number of samples in the current training batch, including high-risk, medium-risk, and low-risk multimodal data samples in small-scale engineering scenarios.

[0023] w i The risk weight for the i-th sample is set according to the severity of the accident corresponding to the sample: samples with fatal risks such as falls from heights and electric shocks are assigned 3, and samples with ordinary violations are assigned 1. The model is forced to learn the modal association of fatal hazards first by weight tilting.

[0024] s i The cross-modal similarity score for the i-th sample is calculated from the cosine similarity between the visual features and text features output by the CLIP model. A higher score indicates stronger semantic consistency between the two modalities.

[0025] τ is a temperature parameter used to adjust the steepness of the similarity score distribution. Its value is fixed at 0.07 to balance the model's sensitivity in distinguishing between positive and negative samples.

[0026] j is the index of a single sample in the comparison sample set, ranging from 1 to M, including positive samples that match the i-th sample and negative samples that do not match.

[0027] M is the total number of the comparison sample set, that is, the number of samples compared with the i-th sample, including 1 positive sample and M-1 negative samples (M is twice the number of samples in the current batch).

[0028] In a preferred embodiment, in step S3, a Focal Loss-based weighted loss function is designed during the model training phase. This function assigns a loss weight three times that of ordinary violation samples to high-risk samples, forcing the model to prioritize learning fatal hazard features. Specifically, this includes:

[0029] First, classify the samples in the training dataset according to their risk level. Scenarios that directly cause injury or death, such as falls from heights, electric shocks, and collapses, are marked as high-risk samples, while ordinary violations (such as not wearing reflective clothing) are marked as low-risk samples.

[0030] Then, based on Focal Loss, a sample weight factor is introduced. When the sample is high-risk, the weight is set to 3, and when it is low-risk, the weight is set to 1. At the same time, the focusing parameter γ of Focal Loss (with a value of 2) is retained to suppress the loss contribution of easily classified samples, so that the model pays more attention to those high-risk hidden danger scenarios that are difficult to classify during training.

[0031] In this way, the model will prioritize correcting the prediction error of high-risk samples during the iteration process, improve the recognition and recall rate of fatal risks such as falls from heights and electric shocks, solve the problem of insufficient model learning for high-risk samples due to the low proportion of high-risk samples in traditional loss functions, and fit the scenario characteristics of small-scale projects that are "small, scattered, but high-risk".

[0032] The formula for weighted focusing loss is:

[0033] ;

[0034] In the formula:

[0035] L is the total loss value of the weighted focusing loss function, which quantifies the learning error of the model on high-risk samples. The smaller the loss, the higher the model's recognition accuracy for high-risk scenarios.

[0036] i is the index of a single sample in the training batch, with a value ranging from 1 to N.

[0037] N represents the total number of samples in the current training batch, including high-risk and low-risk multimodal data samples in small engineering scenarios (such as the combination of "not wearing a safety helmet + risk of falling from a height" samples).

[0038] w i As the risk weight of the samples, high-risk samples (falls from heights, electric shocks, etc.) are assigned a weight of 3, while ordinary violation samples are assigned a weight of 1. The weighting is adjusted to emphasize the proportion of losses from high-risk samples.

[0039] α i This is the category balancing factor, used to balance the differences in sample size between different risk categories. Its value ranges from 0 to 1, with 0.7 for high-risk categories and 0.3 for low-risk categories.

[0040] p i is the predicted probability of the model for the i-th sample, that is, the confidence level of the model in judging that the sample belongs to the high-risk category, and the value ranges from 0 to 1.

[0041] γ is the focusing parameter, typically set to 2, which is used to reduce the loss contribution of easily classified samples and allow the model to focus on high-risk scenarios that are difficult to classify (such as the safety hazards of occlusion).

[0042] This formula, by combining weights and focusing parameters, not only solves the problem of the low proportion of high-risk samples in small-scale projects, but also enhances the model's ability to learn about fatal hazards, demonstrating significant scenario adaptability and innovation.

[0043] In a preferred embodiment, in step S3, during the two-stage training strategy: the first stage is based on a general building safety dataset, and transfer learning is carried out using a pre-trained visual model. The model learns basic safety features (such as general visual patterns of safety helmets and scaffolding protection) through cross-entropy loss.

[0044] The second phase involves importing fragmented data specific to small-scale projects below the limit (including hidden hazard scenarios unique to small-scale projects such as temporary workers not wearing safety helmets, simple scaffolding without protection, and unauthorized temporary electrical connections) for fine-tuning. At the same time, a contrastive learning mechanism is introduced—compliant samples (such as wearing safety helmets correctly during exterior wall work) and hazard samples (such as not wearing safety helmets during exterior wall work) in the same work scenario are paired, and the similarity of their feature spaces is calculated. By using contrastive loss, the model is forced to learn subtle differences (such as the presence or absence of safety helmets, and the absence of scaffold guardrails), so that compliant and hazard samples are further apart in the feature space, ultimately improving the accuracy of hazard identification in small-scale projects.

[0045] The formula for the loss in the second-stage contrastive learning is:

[0046] ;

[0047] In the formula:

[0048] L contrast The contrastive learning loss value quantifies the model's ability to distinguish between compliant and potentially hazardous scenarios; the smaller the loss, the higher the discrimination.

[0049] N represents the number of sample pairs in the current batch (the number of compliance-hazard pairings).

[0050] i represents the sample pair index, which takes values ​​from 1 to N;

[0051] s i,pos This represents the cosine similarity of features between compliant and potentially problematic samples in the i-th pair of samples;

[0052] τ represents the temperature parameter, which adjusts the steepness of the similarity distribution, and has a value of 0.07.

[0053] M represents the total number of samples in the comparison sample set (the total number of positive and negative samples corresponding to each pair of samples).

[0054] j represents the index of the comparison sample, which ranges from 1 to MM;

[0055] s i,j This represents the feature cosine similarity between the i-th pair of samples and the j-th comparison sample;

[0056] ∑ represents summation, summing the losses over all samples in the batch.

[0057] Second stage fine-tuning of the joint loss formula

[0058] ;

[0059] In the formula:

[0060] L total This indicates the total loss after fine-tuning in the second stage, balancing the contributions of transfer learning and contrastive learning;

[0061] α represents the transfer learning loss weight, with a value of 0.3 (preserving the learning results of general features);

[0062] L transfer This represents the cross-entropy loss (basic security feature recognition loss) of the first stage of transfer learning.

[0063] β represents the weight of the contrastive learning loss, with a value of 0.7 (to enhance the difference recognition in small engineering scenarios).

[0064] L contrast This represents the learning loss compared to the comparison.

[0065] These two formulas, through a two-stage weight allocation and comparative learning mechanism, specifically address the pain points of "fragmented samples and strong hidden dangers" in small-scale engineering scenarios, and are the core innovative design adapted to the safety identification of small and scattered projects.

[0066] In a preferred embodiment, in step S4, the trained model continuously receives real-time images of the work area transmitted by the drone, data on concealed areas collected by the robot dog, and real-time operating parameters of the construction equipment sensors. It analyzes this data in real time, identifying behaviors and states that do not comply with safety regulations from the images and capturing abnormal fluctuations in equipment operation from the sensor data. Simultaneously, the model links with personnel's mobile phone data and combines it with location information to determine whether personnel operations meet safety requirements. It categorizes and sorts the identified hazards and violations according to risk level, generating a real-time risk list containing the location of the hazard, the type of violation, the scope of impact, and the risk level. This list is then simultaneously pushed to the operation and maintenance management platform for relevant personnel to view and handle in a timely manner.

[0067] In a preferred embodiment, in step S5, the early warning push parameters corresponding to the risk level classification include: For high-risk situations, the push targets all on-site workers, project managers, and safety supervision personnel; the push channels include real-time SMS, platform pop-ups, and on-site audio-visual alarm linkage; the push frequency remains continuous until the risk is confirmed and eliminated. For medium-risk situations, the push targets are regional operation managers and on-site team members; the push channels combine platform-directed notifications and on-site broadcasts; the rectification deadline is set at 24 hours, and the early warning level will be automatically upgraded if rectification is not completed within the time limit. For low-risk situations, the push targets are limited to on-site team leaders; the push channel is internal work group messages; and the feedback deadline for handling suggestions is 48 hours. The parameter requirements for the handling process record include: a complete record of the early warning reception time, the identity information of the handling personnel, the specific measures taken, the duration of the measures, the risk elimination verification time and results, and the record format must be uniformly structured text, stored in a dedicated system handling log library, with a storage period of not less than three years. The system optimization operation conditions are as follows: when the success rate of handling the same risk type is continuously lower than the preset threshold, the model algorithm parameters are triggered to iterate; when the cumulative number of handled cases reaches the set value, the risk feature library is updated; the optimization frequency adopts a combination of quarterly regular optimization and temporary optimization when new risk types appear, to ensure that the model always adapts to the needs of on-site safety management.

[0068] In a preferred embodiment, in step S6, the risk feature library update is triggered by setting conditions. When new features appear in the feedback of the same type of risk or the accuracy of the original feature recognition is lower than a preset threshold, the update is initiated. Valid risk features are added, outdated features are deleted, and feature weights are adjusted. The update frequency is monthly or temporary when the cumulative feedback reaches a specified number. Model parameter adjustments are based on core indicators such as accuracy, false alarm rate, and false negative rate in the feedback data. When the false alarm rate exceeds the threshold, the recognition threshold parameters are adjusted. When the risk recognition accuracy is insufficient, the weight of the corresponding feature training samples is increased. After adjustment, the performance must be verified by test data before it can be officially launched. The optimized model is synchronously pushed to the preceding data collection stage to update the collection rules of the drone robot dog and the feature extraction logic of data preprocessing, realizing a closed-loop iteration of the entire process and ensuring that the system continuously improves its recognition and early warning capabilities.

[0069] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0070] 1. This invention achieves effective alignment of four-dimensional multimodal data by combining a CLIP pre-trained model with LoRA technology. This allows for the fusion and analysis of project text features, on-site images and videos, personnel profiles, and environmental spatiotemporal data within the same feature space, breaking down information barriers between single data types and comprehensively capturing various risk signals in safety scenarios. The weighted loss function designed for high-risk scenarios strengthens the model's learning priority for fatal hazards such as falls and electric shocks, preventing ordinary samples from diluting the feature weights of high-risk samples and significantly improving the sensitivity of high-risk risk identification.

[0071] 2. In this invention, a two-stage training strategy combined with contrastive learning allows the model to possess both a general foundation in building safety knowledge and the ability to accurately adapt to the fragmented needs of small-scale projects. Transfer learning from a general dataset lays the foundation for the model's basic recognition capabilities, while fine-tuning for specific small-scale project scenarios familiarizes the model with unique hazards such as improper operation by temporary workers and lack of protection in simple facilities. Contrastive learning further refines the understanding of differences between compliant and hazardous scenarios, improving recognition accuracy. Finally, the knowledge from the large model is transferred to a lightweight model, adapted to the computing power of edge devices, enabling real-time on-site recognition without relying on cloud computing. This allows safety warnings to reach on-site personnel instantly, helping small-scale projects shift from passively responding to risks to proactively preventing them. Attached Figure Description

[0072] Figure 1 This is a schematic diagram illustrating the overall process principle of the present invention;

[0073] Figure 2 This is an overall system block diagram of the present invention;

[0074] Figure 3 This is a schematic diagram of the process principle of step S1 in this invention;

[0075] Figure 4 This is a schematic diagram of the process principle of step S2 in this invention;

[0076] Figure 5 This is a schematic diagram of the process principle of step S3 in this invention;

[0077] Figure 6 This is a schematic diagram of the process principle of step S4 in this invention;

[0078] Figure 7 This is a schematic diagram of the process principle of step S5 in this invention;

[0079] Figure 8 This is a schematic diagram of the process principle of step S6 in this invention. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0081] Example:

[0082] Reference Figure 1-8 A method for identifying safety risks in small-scale engineering projects based on multimodal data fusion includes the following steps:

[0083] S1: Collect basic project information, IoT device sensing data, personnel qualification and behavior records, and environmental dynamic data to provide comprehensive raw materials for subsequent analysis.

[0084] S2: Clean and denoise the collected video, image, text, and sensor numerical data, and bind the multi-source data of the same work area to form a unified risk feature library;

[0085] S3: Based on the CLIP cross-modal pre-trained model, cross-modal feature alignment of four-dimensional multimodal data is performed through LoRA low-rank adaptation technology; for high-risk risks such as falls from heights and electric shocks, which account for more than 85%, a weighted loss function based on Focal Loss is designed, which assigns three times the loss weight of ordinary samples to high-risk samples, so that the model learns fatal hazard features first.

[0086] A two-stage training strategy is adopted: the first stage uses a general building safety dataset to complete transfer learning; the second stage imports fragmented scenario data specific to small projects for fine-tuning, while introducing comparative learning to allow the model to learn the subtle differences between compliant operations and potential hazard scenarios; finally, the knowledge of the large model is transferred to a lightweight model to achieve real-time recognition.

[0087] S4: Using a trained model, analyze the real-time images and sensor data transmitted by drones and robot dogs to identify potential hazards on site; at the same time, link the data with the mobile phones of the operation and monitoring personnel, combine location to judge violations, and generate a real-time risk list.

[0088] S5: Push early warnings and handling suggestions based on risk levels, record the handling process and results, and accumulate data for system optimization;

[0089] S6: Feedback is provided on the early warning response, and the feedback results are used to update the risk feature database and adjust the model parameters. Then, the optimized model and data are fed back to the preceding steps, and the entire small-scale engineering safety risk identification process based on multimodal data fusion can be completed.

[0090] In step S1, basic project information is first collected, covering the overall scale of the project, specific work types, and a pre-defined database of high-risk scenarios. This constitutes the initial reference framework for the model to identify risk scenarios. Next, IoT device sensing data is collected, including videos taken by drones at the work site, detection data obtained by robotic dogs in concealed areas, real-time operating parameters uploaded by construction equipment sensors, and location information and operation records uploaded by personnel via mobile phones. This data can reflect the dynamic changes on site in real time. At the same time, personnel qualification and behavior files are compiled, including qualification certificates of construction units, personnel safety training completion status, and records of past violations, providing key evidence for the model to assess personnel operational risks. Finally, dynamic environmental data is acquired, including real-time updated meteorological information, natural disaster early warning signals, and detailed information on current geological conditions. This data helps the model determine the potential impact of the external environment on construction safety. All collected information will be integrated into comprehensive raw materials for the model's subsequent risk analysis, ensuring that the model can conduct accurate analysis based on complete information.

[0091] In step S2, the data, including video images, text, and sensor values, are first cleaned and denoised to remove blurry and invalid frames from the video, interference noise from the images, errors in the text records, and abnormal fluctuations in the sensor values, ensuring that each piece of data has real and effective analytical value. Next, multi-source data from the same work area are linked together, with drone-captured exterior wall images and wall crack detection information obtained by the robot dog being matched, and personnel location information being matched with the operational status of nearby equipment, thus forming an interconnected organic whole from previously scattered data of different types. Finally, these processed and correlated data are integrated to extract key features reflecting safety risks, including the structural state of the work area, equipment operational stability, deviations from regulations in personnel activity trajectories, and the degree of influence of environmental factors. These features are categorized and stored to form a unified risk feature library, serving as the basic material for model training. This ensures that the model can learn accurate risk identification logic from comprehensive and correlated features, providing strong support for subsequent early warning analysis.

[0092] In step S3, modal adaptation processing is first performed on the four-dimensional multimodal data: the project scale / type and personnel qualification / behavioral records are directly input into CLIP's text encoder as text; the image and video frames collected by drones / machine dogs are used to extract visual features through CLIP's visual encoder; and environmental and meteorological sensor data are converted into structured text descriptions and also input into the text encoder.

[0093] Next, a LoRA low-rank adaptation module is inserted into each linear layer of the CLIP Transformer encoder, introducing two low-rank matrices A (dimension d×r) and B (dimension r×d, where r is much smaller than d), adjusting the original linear layer output to Wx + BAx (W is the pre-trained original weight matrix, and x is the input feature vector); by training and optimizing the parameters of A and B, different modal features are aligned in the same semantic space without changing the original model weights.

[0094] The formula for the weighted cross-modal contrast loss is:

[0095] ;

[0096] In the formula:

[0097] L is the total loss value of the weighted cross-modal contrastive loss function, which is used to quantify the learning error of the model in semantic association of multimodal data. The smaller the loss, the better the model's alignment effect on "image scene-text description-sensor features".

[0098] i is the index of a single sample in the current training batch, with a value ranging from 1 to N.

[0099] N represents the total number of samples in the current training batch, including high-risk, medium-risk, and low-risk multimodal data samples in small-scale engineering scenarios.

[0100] w i The risk weight for the i-th sample is set according to the severity of the accident corresponding to the sample: samples with fatal risks such as falls from heights and electric shocks are assigned 3, and samples with ordinary violations are assigned 1. The model is forced to learn the modal association of fatal hazards first by weight tilting.

[0101] s i The cross-modal similarity score for the i-th sample is calculated from the cosine similarity between the visual features and text features output by the CLIP model. A higher score indicates stronger semantic consistency between the two modalities.

[0102] τ is a temperature parameter used to adjust the steepness of the similarity score distribution. Its value is fixed at 0.07 to balance the model's sensitivity in distinguishing between positive and negative samples.

[0103] j is the index of a single sample in the comparison sample set, ranging from 1 to M, including positive samples that match the i-th sample and negative samples that do not match.

[0104] M is the total number of the comparison sample set, that is, the number of samples compared with the i-th sample, including 1 positive sample and M-1 negative samples (M is twice the number of samples in the current batch).

[0105] In step S3, a Focal Loss-based weighted loss function is designed during the model training phase. This function assigns a loss weight three times that of ordinary violation samples to high-risk samples, forcing the model to prioritize learning fatal hazard features. Specifically, this includes:

[0106] First, classify the samples in the training dataset according to their risk level. Scenarios that directly cause injury or death, such as falls from heights, electric shocks, and collapses, are marked as high-risk samples, while ordinary violations (such as not wearing reflective clothing) are marked as low-risk samples.

[0107] Then, based on Focal Loss, a sample weight factor is introduced. When the sample is high-risk, the weight is set to 3, and when it is low-risk, the weight is set to 1. At the same time, the focusing parameter γ of Focal Loss (with a value of 2) is retained to suppress the loss contribution of easily classified samples, so that the model pays more attention to those high-risk hidden danger scenarios that are difficult to classify during training.

[0108] In this way, the model will prioritize correcting the prediction error of high-risk samples during the iteration process, improve the recognition and recall rate of fatal risks such as falls from heights and electric shocks, solve the problem of insufficient model learning for high-risk samples due to the low proportion of high-risk samples in traditional loss functions, and fit the scenario characteristics of small-scale projects that are "small, scattered, but high-risk".

[0109] The formula for weighted focusing loss is:

[0110] ;

[0111] In the formula:

[0112] L is the total loss value of the weighted focusing loss function, which quantifies the learning error of the model on high-risk samples. The smaller the loss, the higher the model's recognition accuracy for high-risk scenarios.

[0113] i is the index of a single sample in the training batch, with a value ranging from 1 to N.

[0114] N represents the total number of samples in the current training batch, including high-risk and low-risk multimodal data samples in small engineering scenarios (such as the combination of "not wearing a safety helmet + risk of falling from a height" samples).

[0115] w i As the risk weight of the samples, high-risk samples (falls from heights, electric shocks, etc.) are assigned a weight of 3, while ordinary violation samples are assigned a weight of 1. The weighting is adjusted to emphasize the proportion of losses from high-risk samples.

[0116] α i This is the category balancing factor, used to balance the differences in sample size between different risk categories. Its value ranges from 0 to 1, with 0.7 for high-risk categories and 0.3 for low-risk categories.

[0117] p iis the predicted probability of the model for the i-th sample, that is, the confidence level of the model in judging that the sample belongs to the high-risk category, and the value ranges from 0 to 1.

[0118] γ is the focusing parameter, typically set to 2, which is used to reduce the loss contribution of easily classified samples and allow the model to focus on high-risk scenarios that are difficult to classify (such as the safety hazards of occlusion).

[0119] This formula, by combining weights and focusing parameters, not only solves the problem of the low proportion of high-risk samples in small-scale projects, but also enhances the model's ability to learn about fatal hazards, demonstrating significant scenario adaptability and innovation.

[0120] In step S3, during the two-stage training strategy: the first stage is based on a general building safety dataset, and uses a pre-trained visual model to carry out transfer learning. The model learns basic safety features (such as general visual patterns of safety helmets and scaffolding protection) through cross-entropy loss.

[0121] The second phase involves importing fragmented data specific to small-scale projects below the limit (including hidden hazard scenarios unique to small-scale projects such as temporary workers not wearing safety helmets, simple scaffolding without protection, and unauthorized temporary electrical connections) for fine-tuning. At the same time, a contrastive learning mechanism is introduced—compliant samples (such as wearing safety helmets correctly during exterior wall work) and hazard samples (such as not wearing safety helmets during exterior wall work) in the same work scenario are paired, and the similarity of their feature spaces is calculated. By using contrastive loss, the model is forced to learn subtle differences (such as the presence or absence of safety helmets, and the absence of scaffold guardrails), so that compliant and hazard samples are further apart in the feature space, ultimately improving the accuracy of hazard identification in small-scale projects.

[0122] The formula for the loss in the second-stage contrastive learning is:

[0123] ;

[0124] In the formula:

[0125] L contrast The contrastive learning loss value quantifies the model's ability to distinguish between compliant and potentially hazardous scenarios; the smaller the loss, the higher the discrimination.

[0126] N represents the number of sample pairs in the current batch (the number of compliance-hazard pairings).

[0127] i represents the sample pair index, which takes values ​​from 1 to N;

[0128] s i,pos This represents the cosine similarity of features between compliant and potentially problematic samples in the i-th pair of samples;

[0129] τ represents the temperature parameter, which adjusts the steepness of the similarity distribution, and has a value of 0.07.

[0130] M represents the total number of samples in the comparison sample set (the total number of positive and negative samples corresponding to each pair of samples).

[0131] j represents the index of the comparison sample, which ranges from 1 to MM;

[0132] s i,j This represents the feature cosine similarity between the i-th pair of samples and the j-th comparison sample;

[0133] ∑ represents summation, summing the losses over all samples in the batch.

[0134] Second stage fine-tuning of the joint loss formula

[0135] ;

[0136] In the formula:

[0137] L total This indicates the total loss after fine-tuning in the second stage, balancing the contributions of transfer learning and contrastive learning;

[0138] α represents the transfer learning loss weight, with a value of 0.3 (preserving the learning results of general features);

[0139] L transfer This represents the cross-entropy loss (basic security feature recognition loss) of the first stage of transfer learning.

[0140] β represents the weight of the contrastive learning loss, with a value of 0.7 (to enhance the difference recognition in small engineering scenarios).

[0141] L contrast This represents the learning loss compared to the comparison.

[0142] These two formulas, through a two-stage weight allocation and comparative learning mechanism, specifically address the pain points of "fragmented samples and strong hidden dangers" in small-scale engineering scenarios, and are the core innovative design adapted to the safety identification of small and scattered projects.

[0143] In step S4, the trained model continuously receives real-time images of the work area transmitted by the drone, data on concealed areas collected by the robot dog, and real-time operating parameters of the construction equipment sensors. It analyzes this data in real time, identifying behaviors and states that do not comply with safety regulations from the images and capturing abnormal fluctuations in equipment operation from the sensor data. At the same time, the model links with personnel's mobile phone data and combines it with location information to determine whether personnel operations meet safety requirements. It categorizes and sorts the identified hazards and violations according to risk level, generating a real-time risk list that includes the location of the hazard, the type of violation, the scope of impact, and the risk level. This list is then pushed to the operation and maintenance management platform for relevant personnel to view and handle in a timely manner.

[0144] In step S5, the early warning push parameters corresponding to the risk level classification include: High-risk risks are pushed to all on-site workers, project managers, and safety supervision personnel through real-time SMS, platform pop-ups, and on-site audio-visual alarms, with continuous triggering until the risk is confirmed and eliminated; medium-risk risks are pushed to regional work managers and on-site team members through a combination of platform-directed notifications and on-site broadcasts, with a 24-hour rectification deadline; failure to complete rectification within the time limit will automatically escalate the early warning level; low-risk risks are limited to on-site team leaders, with internal work group messages and a 48-hour feedback deadline for handling suggestions. The parameters for recording the handling process include: complete recording of the early warning reception time, the identity information of the handling personnel, the specific measures taken, the duration of the measures, the risk elimination verification time and results, and a uniform structured text format. These records are stored in a dedicated system handling log database for a period of no less than three years. The system optimization operation conditions are as follows: when the success rate of handling the same risk type is continuously lower than the preset threshold, the model algorithm parameters are triggered to iterate; when the cumulative number of handled cases reaches the set value, the risk feature library is updated; the optimization frequency adopts a combination of quarterly regular optimization and temporary optimization when new risk types appear, to ensure that the model always adapts to the needs of on-site safety management.

[0145] In step S6, the risk feature library update is triggered by setting conditions. When new features appear in the feedback of the same type of risk or the accuracy of the original feature recognition is lower than the preset threshold, the update is initiated. Valid risk features are added, outdated features are deleted, and feature weights are adjusted. The update frequency is monthly or temporary when the cumulative feedback reaches a specified number. Model parameter adjustments are based on core indicators such as accuracy, false alarm rate, and false negative rate in the feedback data. When the false alarm rate exceeds the threshold, the recognition threshold parameters are adjusted. When the risk recognition accuracy is insufficient, the weight of the corresponding feature training samples is increased. After adjustment, the performance must be verified by test data before it can be officially launched. The optimized model is synchronously pushed to the preceding data collection stage to update the collection rules of the drone robot dog and the feature extraction logic of data preprocessing, realizing a closed-loop iteration of the entire process and ensuring that the system continuously improves its recognition and early warning capabilities.

[0146] From the above, we can conclude that:

[0147] This invention achieves effective alignment of four-dimensional multimodal data by combining a CLIP pre-trained model with LoRA technology. This allows for the fusion and analysis of project text features, on-site images and videos, personnel profiles, and environmental spatiotemporal data within the same feature space, breaking down information barriers between single data types and comprehensively capturing various risk signals in safety scenarios. A weighted loss function designed for high-risk scenarios strengthens the model's learning priority for fatal hazards such as falls and electric shocks, preventing ordinary samples from diluting the feature weights of high-risk samples and significantly improving the sensitivity of high-risk risk identification.

[0148] This invention combines a two-stage training strategy with contrastive learning to enable the model to possess both a general foundation in building safety knowledge and the ability to accurately adapt to the fragmented needs of small-scale projects. Transfer learning from a general dataset lays the foundation for the model's basic recognition capabilities, while fine-tuning for specific small-project scenarios familiarizes the model with unique hazards such as improper temporary worker operations and inadequate protective measures in simple facilities. Contrastive learning further refines the understanding of differences between compliant and hazardous scenarios, improving recognition accuracy. Finally, the knowledge from the large model is transferred to a lightweight model, adapted to the computing power of edge devices, enabling real-time on-site recognition without relying on cloud computing. This allows safety warnings to reach on-site personnel instantly, helping small-scale projects shift from passively responding to risks to proactively preventing them.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying safety risks in small-scale engineering projects based on multimodal data fusion, characterized in that: The method includes the following steps: S1: Collect basic project information, IoT device sensing data, personnel qualification and behavior records, and environmental dynamic data; S2: Clean and denoise the collected video, image, text, and sensor numerical data, and bind the multi-source data of the same work area to form a unified risk feature library; S3: Based on the CLIP cross-modal pre-trained model, cross-modal feature alignment of four-dimensional multimodal data is performed through LoRA low-rank adaptation technology; a weighted loss function based on Focal Loss is designed for high-risk samples, and high-risk samples are given a loss weight of 3 times that of ordinary samples. S4: Use the trained model to analyze data and identify potential hazards on site; at the same time, link the data with the mobile phones of the operation monitoring personnel, combine location to judge violations, and generate a real-time risk list. S5: Push early warnings and handling suggestions based on risk levels, record the handling process and results, and accumulate data for system optimization; S6: Provide feedback on the early warning and response procedures, and use the feedback results to update the risk feature database and adjust the model parameters.

2. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S1, basic project information is first collected, including the overall scale of the project, specific work types, and a pre-defined database of high-risk scenarios. This information forms the initial reference framework for the model to identify risk scenarios. Next, IoT device sensing data is collected, including videos taken by drones at the work site, detection data obtained by robot dogs in concealed areas, real-time operating parameters uploaded by construction equipment sensors, and location information and operation records uploaded by personnel via mobile phones. At the same time, personnel qualification and behavior files are compiled, including qualification certificates of construction units, personnel safety training completion status, and records of past violations, providing key evidence for the model to assess personnel operational risks. Finally, dynamic environmental data is acquired, including real-time updated meteorological information, natural disaster early warning signals, and detailed information on current geological conditions.

3. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S2, the video image, text, and sensor numerical data are first cleaned and denoised to remove blurry and invalid frames from the video, interference noise from the image, erroneous information from the text record, and abnormal fluctuations in the sensor values. Next, multi-source data from the same work area are bound together, and the exterior wall images captured by the drone and the wall crack detection information obtained by the robot dog are matched and the personnel positioning information is matched with the operating status of nearby equipment. Finally, these processed and associated data are integrated to extract key features that reflect safety risks, including the structural status of the work area, the stability of equipment operation, the deviation of personnel activity trajectories from regulations, and the degree of influence of environmental factors. These features are classified and stored to form a unified risk feature library.

4. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S3, a two-stage training strategy is adopted: the first stage uses a general building safety dataset to complete transfer learning; the second stage imports fragmented scenario data specific to small projects for fine-tuning, while introducing comparative learning to allow the model to learn the subtle differences between compliant operations and hidden danger scenarios; finally, the knowledge of the large model is transferred to the lightweight model to achieve real-time recognition. First, modal adaptation processing is performed on the four-dimensional multimodal data: project scale / type and personnel qualification / behavioral records are directly input into CLIP's text encoder as text; image and video frames collected by drones / robot dogs are used to extract visual features through CLIP's visual encoder; environmental and meteorological sensor data are converted into structured text descriptions and then input into the text encoder. Next, a LoRA low-rank adaptation module is inserted into each linear layer of the CLIP Transformer encoder, introducing two low-rank matrices A and B, and adjusting the output of the original linear layer to Wx + BAx; by training and optimizing the parameters of A and B, different modal features are aligned in the same semantic space without changing the weights of the original model. The formula for the weighted cross-modal contrast loss is: ； In the formula: L is the total loss value of the weighted cross-modal contrastive loss function; i is the index of a single sample in the current training batch, and its value ranges from 1 to N; N represents the total number of samples in the current training batch, including high-risk, medium-risk, and low-risk multimodal data samples in small-scale engineering scenarios. w i The risk weight for the i-th sample is set according to the severity of the accident corresponding to the sample: 3 for fatal risk samples of falls from heights and electric shocks, and 1 for ordinary violation samples. The model is forced to learn the modal association of fatal hazards first by weight tilting. s i The cross-modal similarity score for the i-th sample is calculated from the cosine similarity between the visual features and text features output by the CLIP model. A higher score indicates stronger semantic consistency between the two modalities. τ is a temperature parameter used to adjust the steepness of the similarity score distribution. Its value is fixed at 0.07 to balance the model's sensitivity in distinguishing between positive and negative samples. j is the index of a single sample in the comparison sample set, with a value ranging from 1 to M, including positive samples that match the i-th sample and negative samples that do not match. M is the total number of the comparison sample set, that is, the number of samples compared with the i-th sample, including 1 positive sample and M-1 negative samples.

5. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S3, a weighted loss function based on Focal Loss is designed during the model training phase. By assigning a loss weight of 3 times that of ordinary violation samples to high-risk samples, the model is forced to prioritize learning fatal hazard features. Specifically, it includes: First, classify the samples in the training dataset according to their risk level. Scenarios that directly cause injury or death due to falls from heights, electric shocks, or collapses are marked as high-risk samples, while ordinary violations are marked as low-risk samples. Then, based on Focal Loss, a sample weight factor is introduced. When the sample is high-risk, the weight is set to 3, and when it is low-risk, the weight is set to 1. At the same time, the focusing parameter γ of Focal Loss is retained to suppress the loss contribution of easily classified samples, so that the model pays more attention to those high-risk hidden danger scenarios that are difficult to classify during training. The formula for weighted focusing loss is: ； In the formula: L is the total loss value of the weighted focusing loss function, which quantifies the learning error of the model on high-risk samples; i is the index of a single sample in the training batch, and its value ranges from 1 to N; N represents the total number of samples in the current training batch, including high-risk and low-risk multimodal data samples in small-scale engineering scenarios; w i As the risk weight for the samples, high-risk samples are assigned a weight of 3, and ordinary violation samples are assigned a weight of 1. The weighting is adjusted to emphasize the proportion of losses in high-risk samples. α i This is the category balancing factor, used to balance the differences in sample size between different risk categories. Its value ranges from 0 to 1, with 0.7 for high-risk categories and 0.3 for low-risk categories. p i The predicted probability of the model for the i-th sample is the confidence level of the model in judging that the sample belongs to the high-risk category, and the value ranges from 0 to 1. γ is the focusing parameter, typically set to 2, which is used to reduce the loss contribution of easily classified samples, allowing the model to focus on high-risk scenarios that are difficult to classify.

6. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S3, during the two-stage training strategy: the first stage is based on a general building safety dataset, and uses a pre-trained visual model to carry out transfer learning, and uses cross-entropy loss to enable the model to learn basic safety features. The second stage involves importing fragmented data specific to small-scale projects below the limit for fine-tuning, while introducing a comparative learning mechanism to pair compliant samples and potential hazard samples in the same work scenario, calculate the similarity of their feature spaces, and force the model to learn subtle differences through comparative loss. The formula for the loss in the second-stage contrastive learning is: ； In the formula: L contrast This represents the comparative learning loss value, quantifying the model's ability to distinguish between compliant and potentially hazardous scenarios; N represents the number of sample pairs in the current batch; i represents the sample pair index, which takes values ​​from 1 to N; s i,pos This represents the cosine similarity of features between compliant and potentially problematic samples in the i-th pair of samples; τ represents the temperature parameter, which adjusts the steepness of the similarity distribution, and has a value of 0.

07. M represents the total number of samples in the comparison sample set; j represents the index of the comparison sample, which ranges from 1 to MM; s i,j This represents the feature cosine similarity between the i-th pair of samples and the j-th comparison sample; ∑ represents summation, summing the losses over all samples in the batch; Second-stage fine-tuning of the joint loss formula: ； In the formula: L total This indicates the total loss after fine-tuning in the second stage, balancing the contributions of transfer learning and contrastive learning; α represents the weight of the transfer learning loss, with a value of 0.3; L transfer This represents the cross-entropy loss in the first stage of transfer learning; β represents the weight of the contrastive learning loss, with a value of 0.7; L contrast This represents the learning loss compared to the comparison.

7. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S4, the trained model continuously receives real-time images of the work area transmitted back by the drone, data of the concealed area collected by the robot dog, and real-time operating parameters of the construction equipment sensors. It analyzes these data in real time and identifies behaviors and states that do not comply with safety regulations from the images.

8. The method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in claim 1, characterized in that: In step S5, the early warning push parameters corresponding to the risk level classification include: for high-risk risks, the push targets cover all on-site workers, project managers, and safety supervision personnel, and the push channels include real-time SMS, platform forced pop-ups, and on-site sound and light alarm linkage, with the push frequency maintained to be continuously triggered until the risk is confirmed to be eliminated; for medium-risk risks, the push targets are regional operation managers and on-site team members, and the push channels combine platform targeted notifications with on-site broadcasts, with a rectification deadline set at 24 hours, and the early warning level will be automatically upgraded if rectification is not completed within the time limit; for low-risk risks, the push targets are limited to on-site team leaders, and the push channel is internal work group messages, with a feedback deadline of 48 hours for handling suggestions.

9. The small-scale engineering safety risk identification system based on multimodal data fusion as described in claim 1, characterized in that: In step S6, the risk feature library update is set with trigger conditions. When new features appear in the same type of risk feedback or the accuracy of the original feature identification is lower than the preset threshold, the update is initiated. Valid risk features are added, outdated features are deleted, and feature weights are adjusted. The update frequency is either monthly or temporary when the cumulative feedback reaches a specified number.

10. A small-scale engineering safety risk identification system based on multimodal data fusion, characterized in that: The system executes the method for identifying safety risks in small-scale engineering projects based on multimodal data fusion as described in any one of claims 1-9 during operation.

Images