Construction safety penetration type supervision method and system based on multi-modal data fusion

Abstract

The invention provides a construction safety penetration type supervision method and system based on multi-modal data fusion. The method comprises the steps of collecting construction site multi-modal safety data through Internet of Things sensing equipment; registering the data with spatial coordinates of a geographic information system, mapping the data to a gridding management unit bound with a project multi-level management architecture, and establishing association between the data and a spatial position and management responsibility unit; fusing and structuring the multi-modal data, inputting a risk identification model to obtain a safety risk and a grade and a position thereof, associating the safety risk and the grade and the position thereof to a specific engineering part, and generating early warning information when the risk exceeds a threshold value; and based on a preset index, pushing the early warning information to a corresponding person in charge in a grading manner, receiving disposal feedback, and performing associated storage for tracing management responsibility and performing closed-loop optimization on the risk model. According to the invention, intelligent identification, accurate positioning, penetration type pushing and closed-loop management of construction safety risks can be realized, and the real-time performance, accuracy and cooperation efficiency of safety supervision are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of engineering construction technology, and in particular to a method and system for penetrating supervision of construction safety based on multimodal data fusion. Background Technology

[0002] In recent years, with the continuous expansion of my country's railway construction scale, the characteristics of long construction cycles, complex construction environments, and numerous participating units have made the complexity and challenges of construction safety supervision and management increasingly prominent. Traditionally, the safety management of railway construction projects mainly relies on a management model of manual inspection, paper records, and hierarchical reporting. Under this model, the collection and transmission of safety information are characterized by lag, dispersion, and subjectivity, making it difficult for project management to grasp the real-time status and potential risks of the front-line work site in a timely, accurate, and complete manner. This information asymmetry and insufficient penetration lead to safety management decisions often being based on delayed or indirect reports, making it difficult to achieve accurate risk warnings and rapid emergency responses. Although some projects have introduced information technology such as video surveillance and sensors, these technologies are mostly deployed in isolation, and the data has not been effectively integrated and connected, failing to form a systematic and intelligent perception and analysis of multi-dimensional safety elements such as "people, machines, environment, and management."

[0003] In existing technologies, several digital and intelligent solutions for engineering safety management have been proposed. For example, IoT sensors are used to monitor specific structural or environmental parameters, or video surveillance systems are used for remote visual supervision. At the data application level, some systems attempt to record and query safety information using databases, or trigger alarms through simple rule engines. However, these existing solutions still have significant limitations in practical applications. First, they are often "siloed" systems targeting a single risk source or specific business process, lacking systematic integration and fusion of safety data across the entire construction site. For example, sensor data, video streams, business ledgers, inspection reports, and other heterogeneous data from multiple sources are highly fragmented, making correlation analysis difficult and creating information silos. Second, existing technologies rely heavily on preset static thresholds or manual inspection for safety risk identification, with limited intelligence levels. They struggle to deeply mine the causes, evolution patterns, and correlations of risks from massive, dynamic data, and cannot achieve proactive risk prediction and precise risk location. Finally, at the information flow level, security warnings or notifications are usually broadcast or distributed hierarchically, failing to be deeply integrated with the project management organizational structure and personnel roles and responsibilities. This results in key information not reaching the relevant decision-makers and implementers accurately and in a timely manner, leading to low efficiency in emergency response coordination and difficulty in penetrating security responsibilities to the most basic management units. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and system for penetrating supervision of construction safety based on multimodal data fusion, in order to solve the problems of low digital integration, weak intelligent risk identification capability, and insufficient penetration of safety information transmission in the existing technology.

[0005] On the one hand, this invention provides a method for penetrating supervision of construction safety based on multimodal data fusion, the method comprising: Multimodal safety data is collected by IoT sensing devices deployed at the construction site, and the safety data is preprocessed. The preprocessed safety data is registered with the spatial coordinates in the geographic information system and mapped to a pre-defined grid management unit that is bound to the multi-level management architecture of the construction project, thereby establishing the association between the safety data, spatial location, and management responsibility unit. The mapped safety data is subjected to multimodal fusion and structured processing to form input data; the input data is identified based on a pre-trained risk identification model to obtain the safety risks, their levels, and location information of the construction site, and the safety risks are associated with specific engineering parts in the grid management unit; when the identified risk level or frequency of occurrence exceeds a preset threshold, safety warning information is generated. Based on the risk level, scope of impact, and urgency of the safety warning information, the safety warning information is pushed to the responsible persons corresponding to the preset user roles. The responsible persons are given feedback information on their handling, and the handling feedback information is associated with the corresponding safety warning information, the responsible entity, and the engineering part for storage. This is used to trace management responsibility and optimize the risk identification model in a closed loop.

[0006] In some embodiments of the present invention, the Internet of Things sensing device includes environmental sensors, structural monitoring sensors, device status sensors, smart wearable devices, mobile terminal applications, and video image acquisition devices; The multimodal security data includes at least environmental status data, equipment operation data, personnel behavior data, and video image data; The preprocessing includes cleaning, compression, and format normalization.

[0007] In some embodiments of the present invention, the method further includes constructing a security sample database for training the risk identification model, the construction step comprising: Collect multimodal historical security data, including historical sensor data, historical video image data, and historical text record data of security management business from the IoT sensing devices; The historical sensor data is cleaned and converted into a format to generate structured time-series data; The historical video image data is used to perform target detection and semantic segmentation using computer vision algorithms to identify personnel, safety equipment, devices and hazardous elements, and to annotate bounding boxes and classification labels to generate standard visual data. The historical text record data is processed using natural language processing technology to extract event type, time, location, personnel, equipment, and cause elements, and event level and classification are labeled to generate structured text event data. The security sample database is constructed based on the structured time-series data, the standard visual data, and the structured text event data.

[0008] In some embodiments of the present invention, the security sample database is constructed based on the structured time-series data, the standard visual data, and the structured text event data, including: Acquire business data for construction projects, including project information, work site information, and personnel file information; A data association model is constructed, which is based on the entity-relationship model. The entities include personnel, equipment, engineering parts, events, and sensor nodes, and the relationships include membership relationships, monitoring relationships, triggering relationships, and spatial location relationships. Based on the data association model, the structured time-series data is associated with the corresponding device or engineering part entity according to the sensor node ID and acquisition timestamp, and stored in the time-series database; the business data is stored in the relational database; the standard visual data and the structured text event data are associated with the corresponding entity according to the corresponding tag information, and the corresponding feature vectors are stored in the sample feature library. Based on the data association model, a unified data access interface is established for the time-series database, the relational database, and the sample feature library. The data access interface supports cross-database association queries based on entity, time, space, and event type, forming the secure sample database.

[0009] In some embodiments of the present invention, the pre-trained risk identification model is trained through the following steps: A training sample set is extracted from the security sample database. The training sample set contains multiple samples, each of which includes an image feature vector extracted from the standard visual data, a text feature vector extracted from the structured text event data, and a time series and state feature vector extracted from the structured time series data. A machine learning model is constructed, wherein the input layer of the machine learning model is adapted to the feature dimensions of the sample, and the output layer corresponds to the classification, level, and location information prediction of the security risk. The machine learning model is trained using the training sample set, and the model parameters are adjusted by optimizing the loss function until the machine learning model achieves a preset accuracy in identifying security risks on the validation set, thus obtaining the trained risk identification model.

[0010] In some embodiments of the present invention, the method further includes a targeted warning triggering step performed before generating the security warning information: By using text mining and natural language processing techniques, historical event reports and historical safety data are analyzed to extract key safety risk factors and form safety risk terms. Each safety risk term is associated with at least the risk type, typical characteristics, and corresponding construction location type. Statistical analysis of current and historical safety data is conducted to identify the frequency and severity distribution of various safety risks in specific types of construction sites. When a safety risk is identified to occur more frequently or more severely than a preset threshold in a specific type of construction site, the site is determined to have a predisposing safety risk. Based on the analysis results of the aforementioned potential safety risks, early warning information is generated and output for the specific type of construction site.

[0011] In some embodiments of the present invention, the security warning information is pushed to responsible persons corresponding to preset user roles in a tiered manner, including: The system pre-defines three levels of user roles: decision-making, management, and on-site execution, and defines security management responsibilities and information processing permissions for each role. An information classification and scoring model is constructed. The information classification and scoring model calculates the importance score and classification result of the warning information based on three core indicators: risk level, scope of impact, and urgency. Based on the classification results and the responsibilities and permissions of the user roles, the following actions are taken respectively: Warning information corresponding to a low risk based on the importance score is pushed to the field grid worker role in the field execution layer; warning information corresponding to a relatively high risk based on the importance score is pushed to the safety supervision specialist role in the management layer; and warning information corresponding to a major risk based on the importance score is pushed to the safety supervision manager role in the decision-making layer. Based on project progress, organizational changes, or the feedback information, dynamically adjust the permission configuration of the user role and / or the scoring weight of the information classification and scoring model.

[0012] In some embodiments of the present invention, when the security warning information is pushed in a tiered manner, the method further includes establishing an information reception confirmation and processing feedback mechanism: The safety warning information is pushed through preset channels, including a dedicated management application, a management platform, and a web interface; when pushing the safety warning information, the construction site safety situation map and statistical analysis results related to the safety warning information are displayed in a visual form. The push interface provides an entry point for the responsible person to confirm the receipt of information; the corresponding rectification task work order is generated in the safety warning information, and the processing status and result feedback of the work order by the responsible person are tracked; the information receipt confirmation operation, the processing status and result feedback of the work order are used as part of the incremental sample or related information and fed back to the system to optimize the risk identification model and the information classification and scoring model to form a closed-loop management.

[0013] On the other hand, the present invention provides a construction safety penetration monitoring system based on multimodal data fusion, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the system implements the steps of any of the methods mentioned above.

[0014] On the other hand, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods mentioned above.

[0015] This invention effectively solves the problems of information silos, delayed risk identification, and poor accountability in the safety supervision of complex engineering construction sites such as railways. Through multimodal data fusion and intelligent analysis, the system realizes a shift from a passive response to a proactive early warning management model, significantly improving the timeliness and accuracy of risk detection. Based on the precise mapping of spatial grids and organizational structures, it ensures that safety responsibilities penetrate to the most basic engineering parts and responsible persons, achieving direct and closed-loop management instructions. Simultaneously, the feedback-driven model optimization mechanism enables the system to continuously learn and evolve. This invention can significantly improve the overall safety level of construction sites, optimize the allocation efficiency of safety management resources, and achieve clear definition, full-process traceability, and effective implementation of safety responsibilities.

[0016] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0017] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the steps of a construction safety penetration monitoring method based on multimodal data fusion in one embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the data acquisition and preprocessing process in one embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of a multimodal data fusion processing flow in one embodiment of the present invention.

[0021] Figure 4 This is a structural block diagram of a construction safety penetration monitoring system based on multimodal data fusion in one embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0023] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0024] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0025] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0026] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0027] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0028] To address the shortcomings of existing technologies, such as low digital integration, weak intelligent risk identification capabilities, and insufficient penetration in safety information transmission, this invention provides a method for penetrating construction safety supervision based on multimodal data fusion, such as... Figure 1 As shown, the method includes the following steps S101~S104: Step S101: Collect multimodal safety data through IoT sensing devices deployed at the construction site, and preprocess the safety data.

[0029] Step S102: Register the preprocessed safety data with the spatial coordinates in the geographic information system and map it to the preset grid management unit that is bound to the multi-level management architecture of the construction project, thereby establishing the association between safety data, spatial location, and management responsibility unit.

[0030] Step S103: Perform multimodal fusion and structured processing on the mapped safety data to form input data; identify the input data based on the pre-trained risk identification model to obtain the safety risks, their levels, and location information of the construction site, and associate the safety risks with specific engineering parts in the grid management unit; when the identified risk level or frequency of occurrence exceeds the preset threshold, generate safety warning information.

[0031] Step S104: Based on the risk level, scope of impact, and urgency of the safety warning information, push the safety warning information to the responsible persons corresponding to the preset user roles, receive the handling feedback information from the responsible persons, and associate and store the handling feedback information with the corresponding safety warning information, responsible entities, and engineering parts for tracing management responsibilities and optimizing the risk identification model in a closed loop.

[0032] The core objective of step S101 is to build a three-dimensional, fully covered Internet of Things (IoT) sensing network at the construction site to achieve real-time, automatic, and multi-dimensional data collection of the status of four core safety elements: "people, machines, environment, and management" (personnel, machinery and equipment, environment, and engineering structure), and to immediately perform preliminary localized data processing.

[0033] First, we will explain the deployment and data collection of IoT sensing devices.

[0034] In some embodiments, IoT sensing devices include environmental sensors, structural monitoring sensors, device status sensors, smart wearable devices, mobile terminal applications, video image acquisition devices, etc. Specifically: Environmental sensors, such as those for temperature and humidity, wind speed and direction, noise, and dust (PM2.5 / 10), are used to collect real-time construction environment data.

[0035] Structural monitoring sensors, such as strain gauges, hydrostatic levels, inclinometers, and crack gauges, are deployed in tunnel supports, key bridge sections, and high-support systems to monitor structural safety data such as deformation, stress, and settlement in real time.

[0036] Equipment status sensors, such as vibration sensors, temperature sensors, pressure sensors, current and voltage sensors, are installed on large construction equipment such as gantry cranes, bridge erecting machines, and tunnel boring machines to monitor the health status of the equipment in real time.

[0037] Smart wearable devices, such as smart safety helmets worn by construction workers that integrate UWB (ultra-wideband) or Bluetooth positioning modules, can track their precise location and movement trajectory in real time.

[0038] Video image acquisition equipment, such as AI video analysis cameras, is deployed at key construction locations, such as the edge of the foundation pit, high-altitude work surfaces, tunnel entrances, and material storage areas, to collect video streams in real time.

[0039] The aforementioned devices connect to the field-deployed edge computing gateway or IoT gateway via wired (e.g., RS485, Ethernet) or wireless (e.g., LoRa, 4G / 5G, Wi-Fi) methods, forming a perception layer network. For example... Figure 2 As shown, the terminal perception layer, as the bottom layer, transmits the collected raw data stream upwards.

[0040] Then, the collected multimodal security data is preprocessed at the edge computing node or gateway.

[0041] In some embodiments, preprocessing includes the following processing: Perform data cleaning on the security data to filter out invalid data such as abnormal signals and obvious errors, such as the jump value of the sensor at the moment of power failure.

[0042] For high-frequency data, such as vibration data several times per second, downsampling or packaging can be performed to reduce network transmission load.

[0043] Data from devices with different manufacturers and protocols is uniformly converted into a standardized data format defined within the system, such as JSON or a specific binary protocol.

[0044] Based on the preset location information of the device deployment, a preliminary location label, such as GPS coordinates or relative coordinates, is attached to the collected data packets.

[0045] Based on the above explanation, the collected multimodal security data can be divided into the following four categories, each corresponding to different physical world elements: Structured time-series data: Numerical data from various sensors, exhibiting strong temporal sequence characteristics. Examples include temperature, wind speed, displacement, vibration amplitude, and current values.

[0046] Spatiotemporal trajectory data: Location data from smart wearable devices, including person ID, three-dimensional coordinates (x, y, z), and timestamp, forming a continuous behavioral trajectory.

[0047] Unstructured visual data: Video streams or image snapshots from video image acquisition devices, existing in the form of pixel matrices, containing rich semantic information, such as human actions, object states, etc.

[0048] Semi-structured / unstructured status data: Some device alarm signals or simple status indicators can be regarded as an extension of structured data.

[0049] In summary, step S101 successfully transformed the complex and dynamic physical safety status of the construction site into a high-quality, standardized multimodal digital data stream in real time by deploying a three-dimensional sensing network composed of multiple types of IoT devices and relying on edge computing nodes to perform standardized preprocessing.

[0050] In step S102, the standardized data stream with preliminary location tags generated in step S101 is precisely mapped to a unified geographic information space and encapsulated in a grid according to a preset management logic, thereby binding each piece of data with a specific physical location and management person in charge.

[0051] First, the preprocessed safety data is registered with spatial coordinates in a Geographic Information System (GIS). Specifically, the preliminary location labels attached to the preprocessed safety data, such as the preset coordinates of the equipment installation location and the spatial parameters of the video camera, are matched and calibrated with the precise coordinates in the GIS. For example, the ultra-wideband positioning coordinates of the smart safety helmet are transformed to an engineering coordinate system, or the pixel coordinates of the personnel identified by the AI ​​camera are calculated into real-world coordinates using the camera's intrinsic and extrinsic parameters. After registration, each piece of safety data obtains precise geographic coordinates within the same engineering space.

[0052] Then, the registered safety data is mapped to a pre-defined grid management unit bound to the multi-level management architecture of the construction project, establishing the association between the safety data and spatial location as well as the management responsibility unit. Specifically: Based on geographic information system maps and combined with a multi-level management architecture for engineering projects, such as "construction project - construction section - work area - work point - specific engineering location", the entire construction area is dynamically divided into management grids of different levels. Among them, the smallest grid usually corresponds to a specific engineering location, such as a bridge pier abutment or the Xth lining section of a tunnel.

[0053] Each grid cell is associated and bound to the responsible entity (construction unit, work team, safety officer, etc.) in the project management organizational structure during system initialization. This binding relationship can be dynamically updated as the project progresses and personnel are adjusted.

[0054] Based on the registered precise coordinates, through spatial calculations such as point-in-polygon determination, it is automatically assigned to the corresponding smallest management grid cell.

[0055] Based on the aforementioned chain operation of coordinates → grid → responsible entity, the association between security data and spatial location, security data and management grid, and security data and responsible unit is established at the data level.

[0056] Step S103 involves deep integration and intelligent analysis of spatially mapped, multi-source security data to automatically identify security risks and automate the entire process from risk identification to location and early warning.

[0057] First, a risk identification model is built and trained to automatically identify security risks.

[0058] In some embodiments, a security sample database for training the risk identification model is first constructed, including: Collect multimodal historical security data, including historical sensor data from IoT sensing devices, historical video image data, and historical text records from security management operations.

[0059] Historical sensor data is cleaned and converted to remove obviously erroneous data and transformed into structured time-series data with a unified internal system format.

[0060] For historical video image data, computer vision (CV) algorithms are used for target detection and semantic segmentation to identify personnel, safety equipment, equipment and hazardous state elements, and to label them with bounding boxes and classification labels. For example, the bounding boxes are labeled with category labels such as "personnel", "safety helmet" and "excavator", and the "personnel" box is labeled with the attribute label "not wearing a safety helmet". The overall scene is labeled with the "hazardous state" label.

[0061] Historical text records are processed using Natural Language Processing (NLP) technology to extract event type, time, location, personnel, equipment, and cause elements. Event levels are then assigned and categorized to generate structured text event data. Event levels can be categorized as major events, ordinary events, etc. Event categories can be divided according to professional fields. Process status can also be tagged, such as "rectified" or "pending verification," for subsequent management purposes.

[0062] In some embodiments, the standardized data described above are stored and managed in an integrated manner according to their characteristics and uses, in a way that can efficiently support relational queries and model training.

[0063] Obtain business data for construction projects, including project information, work site information, and personnel file information.

[0064] A data association model is constructed based on the entity-relationship (ER) model. Entities include people, equipment, engineering parts, events, and sensor nodes, while relationships include membership, monitoring, triggering, and spatial location relationships.

[0065] Based on the data association model, structured time-series data is associated with the corresponding equipment or engineering entity according to the sensor node ID and the acquisition timestamp, and stored in the time-series database.

[0066] Based on the data association model, business data is stored in a relational database.

[0067] Based on the data association model, standard visual data and structured text event data are associated with corresponding entities according to their corresponding label information, and the corresponding feature vectors are stored in the sample feature library.

[0068] Based on the data association model, a unified data access interface is established for time-series databases, relational databases, and sample feature libraries. This data access interface supports cross-database association queries based on entities, time, space, and event types, ultimately forming a secure sample database.

[0069] In some embodiments, after constructing the security sample database, the risk identification model is trained, including: A training sample set is extracted from a security sample database. This training sample set contains multiple samples, each of which includes an image feature vector extracted from standard visual data, a text feature vector extracted from structured text event data, and a time series and state feature vector extracted from structured time series data.

[0070] Construct a machine learning model whose input layer is adapted to the feature dimensions of the samples, and whose output layer corresponds to the prediction of the classification, level, and location information of security risks.

[0071] The machine learning model is trained using a training sample set. The model parameters are adjusted by optimizing the loss function until the machine learning model achieves a preset accuracy in identifying security risks on the validation set, thus obtaining the trained risk identification model.

[0072] In some embodiments, risk identification is performed using a pre-trained risk identification model, including: Based on the spatiotemporal scope of the event or real-time data stream to be analyzed, such as analyzing the data of work site No. 3 at the current moment, the system uses a unified data access interface to query and extract all multimodal data related to the target from time-series databases, relational databases, and sample feature libraries. For example, to analyze the risks of "aerial work platforms," ​​the system will simultaneously retrieve real-time video frames (visual data) from AI cameras in the area, the location and trajectory of smart safety helmets worn by personnel in the area (spatiotemporal data), vibration and tilt sensor readings of the platform equipment (time-series data), and the personnel qualification information of the work team (business data).

[0073] Furthermore, for unstructured visual data, pre-built lightweight feature extraction models, such as MobileNet and YOLO, can be invoked for real-time analysis, converting video frames into feature vectors describing their content. All modalities of data are ultimately integrated into a unified, structured feature representation (such as a feature tensor), which serves as input to the risk identification model.

[0074] The above input data is fed into the risk identification model. After calculation and reasoning, the model outputs the type, level, and location information of the safety risk. Risk types include falls from heights, being struck by objects, mechanical injuries, and collapses. Risk levels are categorized as low risk, moderate risk, significant risk, and major risk, typically determined based on the probability values ​​output by the model or by a combination of preset rules.

[0075] After identification, the system automatically performs spatial matching between the risk location information identified by the model and the gridded management units established in step S102. Through spatial query of the geographic information system, it determines which specific engineering part grid the risk point falls within, thereby realizing the association between the risk and the specific engineering part.

[0076] The system analyzes the risk levels identified by the model, or statistically analyzes the frequency of a certain type of risk occurring in a specific area or time period, and compares this data with preset thresholds. For example, if three instances of not wearing a safety helmet are identified consecutively within 30 minutes in the same grid (frequency exceeding the threshold), or if the tunnel arch settlement value reaches the yellow warning level (level exceeding the threshold), the system automatically generates a structured safety warning message when the trigger conditions are met. This message includes the risk type, level, location (grid cell ID), time, associated data evidence (such as images, data curves), and the primary responsible unit / person identified through grid association.

[0077] In some embodiments, the warning triggering mechanism further includes: By using text mining and natural language processing techniques, historical event reports and historical safety data are analyzed to extract key safety risk factors and form safety risk terms. Each safety risk term is associated with at least the risk type, typical characteristics, and corresponding construction location type.

[0078] Statistical analysis of current and historical safety data is used to identify the frequency and severity distribution of various safety risks in specific types of construction sites.

[0079] When a safety risk is identified to occur more frequently or more severely than a preset threshold in a specific type of construction site, the site is determined to have a predisposing safety risk.

[0080] Based on the analysis results of potential safety risks, early warning information is generated and output for specific types of construction sites.

[0081] Step S104 accurately and promptly pushes each intelligently identified security risk to the corresponding responsible person, tracks their handling process, and feeds the results back to the system for optimization, ultimately forming a complete management chain of perception → analysis → decision-making → execution → learning.

[0082] A three-tiered user role architecture—decision-making, management, and on-site execution—is pre-built, and security management responsibilities and information processing permissions are defined for each role.

[0083] An information classification and scoring model is constructed. This model calculates the importance score and classification result of the warning information based on three core indicators: risk level, scope of impact, and urgency.

[0084] Based on the hierarchical results and the responsibilities and permissions of user roles, the following actions will be taken respectively: The importance score is used to correspond to low-risk early warning information, which is then pushed to the field grid member role in the field execution layer.

[0085] The importance score is used to identify early warning information indicating a significant risk, which is then pushed to the safety supervisors in the management team.

[0086] The importance score is used to correspond to early warning information of major risks, which is then pushed to the safety supervision person in the decision-making level.

[0087] Based on project progress, organizational changes, or feedback information, the permission configuration of user roles and / or the scoring weights of the information classification and scoring model can be dynamically adjusted.

[0088] In some embodiments, security alerts can be pushed to the corresponding users' terminals via various channels such as integrated instant messaging interfaces, dedicated security management applications, management platforms, and web interfaces, in the form of pop-ups, SMS messages, and application notifications. The information content is structured and includes risk details, location (grid), requirements, time limits, and one-click access to the handling entry (such as an order processing button).

[0089] In some embodiments, an information reception confirmation and processing feedback mechanism is established, including: The push notification interface provides an entry point for the responsible person to confirm receipt of the information.

[0090] Corresponding rectification task work orders are generated by associating them with safety early warning information, and the processing status and feedback of the responsible persons for the work orders are tracked. The information reception confirmation operation, the processing status and feedback of the work orders are used as part of the incremental sample or related information and fed back to the system (which can be stored in the safety sample database) to optimize the risk identification model and information classification and scoring model, so as to form a closed-loop management.

[0091] In some embodiments, the handling feedback information (result feedback) is stored in association with the source security early warning information, including: Each safety warning message is assigned a unique ID, and each feedback record is associated with the corresponding safety warning message ID when stored. Simultaneously, through the grids and responsible entities already associated with the safety warning messages, feedback records can also be indirectly but definitively linked to specific engineering locations and responsible entities. Ultimately, a relationship chain is formed in the database table: safety warning record → feedback record ← grid / responsible entity.

[0092] Based on the aforementioned associated storage, the system can perform panoramic queries at any time, realizing full-process digital traceability and auditability of management responsibilities, which serves as an objective basis for implementing accountability and assessment.

[0093] Corresponding to the above methods, the present invention also provides a construction safety penetration monitoring system based on multimodal data fusion, such as... Figure 4As shown, the system includes a data acquisition and preprocessing module, a sample database construction module, a secure data intelligent analysis module, and a data hierarchical and precise push module. This system is used to execute the steps of the methods described above.

[0094] Corresponding to the above method, the present invention also provides another construction safety penetration monitoring system based on multimodal data fusion, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the aforementioned method.

[0095] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0096] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0097] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0098] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for penetrating supervision of construction safety based on multimodal data fusion, characterized in that, The method includes: Multimodal safety data is collected by IoT sensing devices deployed at the construction site, and the safety data is preprocessed. The preprocessed safety data is registered with the spatial coordinates in the geographic information system and mapped to a pre-defined grid management unit that is bound to the multi-level management architecture of the construction project, thereby establishing the association between the safety data, spatial location, and management responsibility unit. The mapped safety data is subjected to multimodal fusion and structured processing to form input data; the input data is identified based on a pre-trained risk identification model to obtain the safety risks, their levels, and location information of the construction site, and the safety risks are associated with specific engineering parts in the grid management unit; when the identified risk level or frequency of occurrence exceeds a preset threshold, safety warning information is generated. Based on the risk level, scope of impact, and urgency of the safety warning information, the safety warning information is pushed to the responsible persons corresponding to the preset user roles. The responsible persons are given feedback information on their handling, and the handling feedback information is associated with the corresponding safety warning information, the responsible entity, and the engineering part for storage. This is used to trace management responsibility and optimize the risk identification model in a closed loop.

2. The construction safety penetration monitoring method based on multimodal data fusion according to claim 1, characterized in that, The IoT sensing devices include environmental sensors, structural monitoring sensors, equipment status sensors, smart wearable devices, mobile terminal applications, and video image acquisition devices. The multimodal security data includes at least environmental status data, equipment operation data, personnel behavior data, and video image data; The preprocessing includes cleaning, compression, and format normalization.

3. The construction safety penetration monitoring method based on multimodal data fusion according to claim 1, characterized in that, The method further includes constructing a security sample database for training the risk identification model, the construction steps of which include: Collect multimodal historical security data, including historical sensor data, historical video image data, and historical text record data of security management business from the IoT sensing devices; The historical sensor data is cleaned and converted into a format to generate structured time-series data; The historical video image data is used to perform target detection and semantic segmentation using computer vision algorithms to identify personnel, safety equipment, devices and hazardous elements, and to annotate bounding boxes and classification labels to generate standard visual data. The historical text record data is processed using natural language processing technology to extract event type, time, location, personnel, equipment, and cause elements, and event level and classification are labeled to generate structured text event data. The security sample database is constructed based on the structured time-series data, the standard visual data, and the structured text event data.

4. The construction safety penetration monitoring method based on multimodal data fusion according to claim 3, characterized in that, The security sample database is constructed based on the structured time-series data, the standard visual data, and the structured text event data, including: Acquire business data for construction projects, including project information, work site information, and personnel file information; A data association model is constructed, which is based on the entity-relationship model. The entities include personnel, equipment, engineering parts, events, and sensor nodes, and the relationships include membership relationships, monitoring relationships, triggering relationships, and spatial location relationships. Based on the data association model, the structured time-series data is associated with the corresponding device or engineering part entity according to the sensor node ID and acquisition timestamp, and stored in the time-series database; the business data is stored in the relational database; the standard visual data and the structured text event data are associated with the corresponding entity according to the corresponding tag information, and the corresponding feature vectors are stored in the sample feature library. Based on the data association model, a unified data access interface is established for the time-series database, the relational database, and the sample feature library. The data access interface supports cross-database association queries based on entity, time, space, and event type, forming the secure sample database.

5. The construction safety penetration monitoring method based on multimodal data fusion according to claim 3, characterized in that, The pre-trained risk identification model is trained through the following steps: A training sample set is extracted from the security sample database. The training sample set contains multiple samples, each of which includes an image feature vector extracted from the standard visual data, a text feature vector extracted from the structured text event data, and a time series and state feature vector extracted from the structured time series data. A machine learning model is constructed, wherein the input layer of the machine learning model is adapted to the feature dimensions of the sample, and the output layer corresponds to the classification, level, and location information prediction of the security risk. The machine learning model is trained using the training sample set, and the model parameters are adjusted by optimizing the loss function until the machine learning model achieves a preset accuracy in identifying security risks on the validation set, thus obtaining the trained risk identification model.

6. The construction safety penetration monitoring method based on multimodal data fusion according to claim 1, characterized in that, The method also includes a targeted warning triggering step performed before generating the security warning information: By using text mining and natural language processing techniques, historical event reports and historical safety data are analyzed to extract key safety risk factors and form safety risk terms. Each safety risk term is associated with at least the risk type, typical characteristics, and corresponding construction location type. Statistical analysis of current and historical safety data is conducted to identify the frequency and severity distribution of various safety risks in specific types of construction sites. When a safety risk is identified to occur more frequently or more severely than a preset threshold in a specific type of construction site, the site is determined to have a predisposing safety risk. Based on the analysis results of the aforementioned potential safety risks, early warning information is generated and output for the specific type of construction site.

7. The construction safety penetration monitoring method based on multimodal data fusion according to claim 1, characterized in that, The security warning information is pushed to the responsible persons corresponding to the preset user roles in a tiered manner, including: The system pre-defines three levels of user roles: decision-making, management, and on-site execution, and defines security management responsibilities and information processing permissions for each role. An information classification and scoring model is constructed. The information classification and scoring model calculates the importance score and classification result of the warning information based on three core indicators: risk level, scope of impact, and urgency. Based on the classification results and the responsibilities and permissions of the user roles, the following actions are taken respectively: Warning information corresponding to a low risk based on the importance score is pushed to the field grid worker role in the field execution layer; warning information corresponding to a relatively high risk based on the importance score is pushed to the safety supervision specialist role in the management layer; and warning information corresponding to a major risk based on the importance score is pushed to the safety supervision manager role in the decision-making layer. Based on project progress, organizational changes, or the feedback information, dynamically adjust the permission configuration of the user role and / or the scoring weight of the information classification and scoring model.

8. The method for penetrating supervision of construction safety based on multimodal data fusion according to claim 1 or 7, characterized in that, When pushing the security warning information in a tiered manner, the method further includes establishing an information reception confirmation and processing feedback mechanism: The safety warning information is pushed through preset channels, including a dedicated management application, a management platform, and a web interface; when pushing the safety warning information, the construction site safety situation map and statistical analysis results related to the safety warning information are displayed to the responsible person in a visual form. The push interface provides an entry point for the responsible person to confirm receipt of information. The corresponding rectification task work order is generated in the safety warning information, and the processing status and result feedback of the responsible person for the work order are tracked. The information reception confirmation operation, the processing status and result feedback of the work order are fed back to the system as part of the incremental sample or related information to optimize the risk identification model and the information classification and scoring model, so as to form a closed-loop management.

9. A construction safety penetration monitoring system based on multimodal data fusion, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images