Construction site intelligent monitoring method based on multi-modal data fusion

Abstract

The invention discloses a construction site intelligent monitoring method based on multi-modal data fusion, and the method comprises the following steps: collecting and preprocessing multi-modal data of a construction site, and obtaining a multi-source data standardization set; through time sequence alignment, space registration and semantic coding, multi-modal unified space-time semantic representation is constructed; constructing a construction intention priori map, generating intention embedding corresponding to the multi-modal unified space-time semantic representation, and obtaining a construction intention priori set; constructing an intention-driven inverse evolution model, and outputting a construction behavior recognition result; and executing space-time risk assessment based on the construction behavior recognition result, and generating a risk level assessment result. According to the method, the multi-modal data and the intention-driven inverse evolution model are fused, intelligent construction behavior identification and space-time risk assessment are realized, and the method has the advantages of high accuracy, strong interpretation and early warning timeliness.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring, and in particular to an intelligent monitoring method for construction sites based on multimodal data fusion. Background Technology

[0002] Current intelligent monitoring systems at construction sites primarily rely on video surveillance, acoustic and environmental sensors, personnel positioning, and equipment operation logs as data sources. They employ rule-based thresholds or general classification models for alarms, with some systems providing multi-source data fusion and large-screen visualizations. However, common solutions often remain at a shallow level of fusion, at the signal or feature level, lacking systematic and standardized processing of the collected data within a unified clock and construction site reference coordinate system. This results in incomplete time alignment, spatial registration, and semantic coding links, as well as insufficient cross-source consistency, leading to limited stability and robustness in recognition under complex working conditions.

[0003] On the other hand, existing technologies generally lack a priori map of construction intentions centered on construction task plans, job roles, and equipment conditions, as well as its numerical intention embedding. They lack a reverse evolution mechanism that infers potential behaviors from intentions and aligns them with observations in time and space and interprets deviations. Risk assessments are mostly based on single alarms or coarse-grained regional statistics, and have not formed a spatiotemporal risk output with gridded, sliding time window aggregation and hierarchical threshold mapping in the construction site reference coordinate system, making it difficult to achieve early detection, rapid location, and interpretable risk level assessment.

[0004] Therefore, how to provide an intelligent monitoring method for construction sites based on multimodal data fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an intelligent monitoring method for construction sites based on multimodal data fusion. This invention integrates multimodal data with an intent-driven inverse evolution model to achieve intelligent identification of construction behavior and spatiotemporal risk assessment, and has the advantages of high accuracy, strong interpretability and timely early warning.

[0006] A construction site intelligent monitoring method based on multimodal data fusion according to an embodiment of the present invention includes the following steps:

[0007] Multimodal data from the construction site is collected, and the data undergoes format standardization, time reference standardization, and spatial coordinate correction to obtain a standardized set of multi-source data.

[0008] Based on the normalized set of multi-source data, temporal alignment, spatial registration and semantic encoding are performed to construct a unified spatiotemporal semantic representation of multiple modalities;

[0009] Based on the construction task plan, job role information and equipment condition information, a construction intention prior graph is constructed, and an intention embedding corresponding to the multimodal unified spatiotemporal semantic representation is generated to obtain the construction intention prior set.

[0010] An intent-driven inverse evolution model is constructed, comprising an intent generation layer, an inverse evolution prediction layer, a spatiotemporal alignment layer, and a deviation interpretation layer, and outputs construction behavior recognition results.

[0011] Based on the results of construction behavior identification, a spatiotemporal risk assessment is performed, and a risk level assessment result is generated by combining spatial location distribution and temporal evolution trend.

[0012] Optionally, the generation of the multi-source data normalization set specifically includes:

[0013] Acquire video data, acoustic data, environmental sensor data, positioning data, and device operation data, record the sampling time marker and acquisition source identifier respectively, and remove empty frames, silent segments, out-of-bounds readings, and invalid logs to obtain the original multimodal data set;

[0014] The original multimodal dataset is quality checked, and abnormal amplitudes are clipped, short-term missing data are imputed, and duplicate samples are merged to generate a clean dataset.

[0015] The time base specification is applied to the cleaned dataset, the time stamps are offset and resampled to a common time step according to a unified clock, and a time-aligned set is obtained.

[0016] Spatial coordinate correction is performed on the time alignment set to map the video imaging coordinates, positioning coordinates and equipment installation coordinates to the construction site reference coordinate system. Spatial consistency processing is performed through the rotation and translation parameters corresponding to the spatial coordinate source to obtain the spatial correction set.

[0017] The spatial correction set is formatted uniformly, the encoding format and resolution of video frames are standardized, the sampling rate and frame shift of acoustic data are unified, the field names and unit dimensions of environmental sensor data and equipment operation data are standardized, and a formatted uniform set is generated.

[0018] Amplitude standardization and dimension normalization are performed across sources on the format-unified set. The center value and scale coefficient are calculated according to the source and channel respectively. The numerical channels are centered and scaled to generate the amplitude specification set.

[0019] The consistency of time and space labels is checked on the amplitude standard set. It is checked whether multiple source samples exist at the same time step and under the same reference coordinate and whether the source labels are consistent. Sample segments that fail the consistency check are removed. For segments that can be restored, their time and space labels are completed to obtain the multi-source data standard set.

[0020] Optionally, the generation of the multimodal unified spatiotemporal semantic representation specifically includes:

[0021] The normalized set of multi-source data is segmented into sliding segments according to a unified time base, and a fixed-length time window and a fixed step are set to generate a window sequence arranged in chronological order, thus obtaining a time-series segment set.

[0022] The data source with the highest sampling accuracy is used as the reference source. The alignment offset of each source in the time series segment set relative to the reference source is determined and summarized to form a time lag set.

[0023] Based on the time lag set, each source is resampled and indexed and rearranged. Consistent time step markers are generated at the window level. Time masks are generated for missing time steps while keeping the source identifier unchanged, resulting in a time-series aligned set.

[0024] Spatial registration initialization is performed based on the temporal alignment set. The reference coordinate system of the construction site is selected as the target coordinate system. An initial parameter set is constructed using the prior relationship between the camera's external parameters, the positioning equipment's installation attitude, and the equipment's installation coordinates, thus obtaining the initial value set for spatial registration.

[0025] Based on the initial set of spatial registration values, the pose parameters from each source to the reference coordinate system are optimized, the optimized pose parameter set is output, and the video imaging coordinates, positioning coordinates and equipment installation coordinates are uniformly mapped to the reference coordinate system to obtain the spatial registration set.

[0026] The spatial registration set is semantically encoded according to its source. The encoder extracts spatial semantic vectors, time-frequency semantic vectors, working condition semantic vectors, and motion semantic vectors, and aggregates them to form a source-specific semantic representation set.

[0027] Cross-source semantic consistency weights are calculated for the semantic representation set from different sources, and the set of semantic consistency weights is determined based on the similarity with the unified reference coordinates within the same time step.

[0028] The semantic representation sets from different sources are weighted and fused based on the semantic consistency weight set, and a fused semantic vector sequence is generated under a unified time step and a unified reference coordinate to form a multimodal unified spatiotemporal semantic representation.

[0029] Optionally, the generation of the prior set of construction intentions specifically includes:

[0030] The system acquires construction task plans, job role information, and equipment condition information. It breaks down the construction task plans into work processes and time periods to obtain planned time period information. It marks the job role information with responsibility domains and work permissions. It divides the equipment condition information into status levels and marks the equipment as available. It generates task element sets, role element sets, and condition element sets.

[0031] Based on the task element set, role element set, and working condition element set, construct the node and relationship types of the construction intention prior graph, and generate the construction intention prior graph and relationship initialization set;

[0032] The relationship weights and time period validity weights are calculated for the initial set of relationships. The comprehensive weight between each node is obtained by weighting the plan dependency score, equipment availability score, and spatial constraint consistency score according to the set weights. The relationship weights are normalized by combining the plan time period information, equipment availability markers, and area occupancy conflict events to generate a weighted set of relationships.

[0033] Based on the multimodal unified spatiotemporal semantic representation, the prior graph of construction intention is generated by time step and construction site reference coordinate system. Node state injection and relation weight update are performed to obtain the intention alignment graph.

[0034] On the intent alignment graph, task nodes and role nodes are jointly expanded by time step to generate an intent candidate set consisting of task sequence, role participation and device support. The prior score of each intent candidate is calculated based on the weighted relationship set to obtain the intent candidate score set.

[0035] Based on the intent candidate score set and node state, intent embedding generation is performed at each time step. The intent candidate set is mapped to a fixed-dimensional vector representation and normalized to obtain the intent embedding sequence.

[0036] The node states of the intent embedding sequence and the intent alignment graph are encapsulated together as a construction intent prior set.

[0037] Optionally, the generation of the construction behavior recognition result specifically includes:

[0038] Construct an intention-driven inverse evolution model, including an intention generation layer, an inverse evolution prediction layer, a spatiotemporal alignment layer and a bias interpretation layer, and a behavior integration unit;

[0039] In the intent generation layer, the prior set of construction intents is input into the intent embedding mapping module and transformed into a semantic tensor. The semantic alignment module is used to achieve time step pairing with the multimodal unified spatiotemporal semantic representation. Based on the intent enhancement module, the salience of each intent node is adjusted according to the weighted relation set to generate an intent seed sequence and intent constraint set containing intent goals, constraints and trigger thresholds.

[0040] In the inverse prediction layer, the intention seed sequence and the intention constraint set are input into the state inversion module. Based on the intention target, the module performs reverse reasoning from future behavior to the current moment to generate a potential behavior state set. The time-series path generation module establishes behavior transfer links based on the potential behavior state set and the construction task logic to obtain a time-series path set. The consistency evaluation module calculates the path consistency score for each time step in the time-series path set. The path consistency score is determined by the time period validity weight and the intention matching degree. After integration, a predicted behavior trajectory set is obtained.

[0041] In the spatiotemporal alignment layer, the set of predicted behavior trajectories and the unified spatiotemporal semantic representation of multimodality are input into the time alignment module. Local offset search is performed on each time step to determine the time pairing of the predicted behavior trajectory and the unified spatiotemporal semantic representation. The spatial mapping module calculates the corresponding region and location boundary of the predicted behavior trajectory in space based on the construction site reference coordinate system. The confidence evaluation module outputs the alignment confidence set by combining the time offset and spatial deviation to form the alignment pairing sequence.

[0042] In the deviation interpretation layer, the alignment pairing sequence is input into the intention deviation analysis module to calculate the semantic difference between the intention target and the observed behavior at each time step. The spatial deviation measurement module calculates the spatial deviation distance under the reference coordinates of the construction site. The comprehensive deviation fusion module weights and fuses the semantic difference and spatial deviation distance according to preset weights to obtain the deviation index set and the behavioral event candidate set.

[0043] The deviation index set, alignment confidence set, and path consistency score are input into the behavior integration unit, which includes an event chain construction module and a behavior classification module. The event chain construction module establishes candidate event links based on time step continuity and spatial adjacency. The behavior classification module performs threshold determination and category identification on the event links and outputs construction behavior identification results. The construction behavior identification results include behavior category, occurrence time step, and the range of reference coordinates of the construction site.

[0044] Optionally, the generation of the risk level assessment result specifically includes:

[0045] The system receives the construction behavior identification results, analyzes the behavior category, occurrence time step and construction site reference coordinate range according to time step, and generates a set of event entries containing event identifier, behavior category, start and end time steps and spatial boundaries. At the same time, it establishes a behavior category weight table and a risk classification threshold set based on historical construction event samples to form an evaluation parameter set.

[0046] Based on the construction site reference coordinate system, the construction site is spatially discretized and divided into a set of spatial grids. The event item set is then projected onto the corresponding spatial grid according to its spatial boundary. The event item index of each spatial grid at each time step is accumulated to obtain the spatial event index set.

[0047] By setting a fixed-length time window and a fixed step, sliding aggregation is performed on the spatial event index set to obtain the event aggregation set in each spatial grid and each time window.

[0048] Spatial risk intensity is calculated based on the set of evaluation parameters and the set of event aggregation to form a spatial risk intensity sequence. The spatial risk intensity is obtained by weighted summation of the behavioral category weights and duration steps of events in each spatial grid and time window.

[0049] The spatial risk intensity sequence is normalized and combined with the spatial risk intensity difference between adjacent time windows to generate a standardized risk score sequence. Based on the risk classification threshold set, the standardized risk score sequence is mapped to a risk level sequence, and a risk heat map is constructed to generate a risk level assessment result consisting of spatial location range, corresponding time window, and a list of key behavior categories.

[0050] The beneficial effects of this invention are:

[0051] This invention constructs a full-link intelligent monitoring system from data acquisition, semantic unification, intent modeling to risk assessment by introducing multimodal data fusion and intent-driven inverse evolution mechanisms, achieving dynamic, comprehensive, and interpretable risk identification and assessment at construction sites. At the data layer, this method forms a unified multimodal spatiotemporal semantic representation through time benchmark standardization, spatial coordinate correction, and semantic encoding, significantly improving the alignment accuracy and fusion consistency between heterogeneous data. At the intelligent identification layer, it introduces a priori maps of construction intent and an inverse evolution reasoning model, enabling reverse inference of behavioral paths from construction task objectives and spatiotemporal alignment and deviation interpretation of observed data, thereby achieving intention-driven proactive identification and behavioral understanding. At the risk assessment layer, through gridded spatiotemporal analysis and hierarchical threshold mapping based on behavioral identification results, it outputs risk heat and level results in continuous space, making risk warnings regional, trend-based, and interpretable. In summary, this invention not only improves the accuracy and robustness of construction site monitoring but also realizes a shift from "passive monitoring" to "proactive inference," effectively enhancing safety management and decision support capabilities under complex working conditions. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a flowchart of a construction site intelligent monitoring method based on multimodal data fusion proposed in this invention;

[0054] Figure 2This is a schematic diagram of the intent-driven inverse evolution model of an intelligent construction site monitoring method based on multimodal data fusion proposed in this invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0056] refer to Figures 1-2 A construction site intelligent monitoring method based on multimodal data fusion includes the following steps:

[0057] Collect multimodal data from the construction site, including video data, acoustic data, environmental sensor data, positioning data, and equipment operation data. Perform format unification, time reference standardization, and spatial coordinate correction on the multimodal data to obtain a standardized set of multi-source data.

[0058] Based on the normalized set of multi-source data, temporal alignment, spatial registration and semantic encoding are performed to construct a unified spatiotemporal semantic representation of multiple modalities;

[0059] Based on the construction task plan, job role information and equipment condition information, a construction intention prior graph is constructed, and an intention embedding corresponding to the multimodal unified spatiotemporal semantic representation is generated to obtain the construction intention prior set.

[0060] An intent-driven inverse evolution model is constructed, comprising an intent generation layer, an inverse evolution prediction layer, a spatiotemporal alignment layer, and a deviation interpretation layer, and outputs construction behavior recognition results.

[0061] Based on the results of construction behavior identification, a spatiotemporal risk assessment is performed, and a risk level assessment result is generated by combining spatial location distribution and temporal evolution trend.

[0062] In this embodiment, the generation of the multi-source data normalization set specifically includes:

[0063] Acquire video data, acoustic data, environmental sensor data, positioning data, and device operation data, record the sampling time marker and acquisition source identifier respectively, and remove empty frames, silent segments, out-of-bounds readings, and invalid logs to obtain the original multimodal data set;

[0064] The original multimodal dataset is quality checked, and abnormal amplitudes are clipped, short-term missing data are imputed, and duplicate samples are merged to generate a clean dataset.

[0065] The time base specification is applied to the cleaned dataset, the time stamps are offset and resampled to a common time step according to a unified clock, and a time-aligned set is obtained.

[0066] Spatial coordinate correction is performed on the time alignment set to map the video imaging coordinates, positioning coordinates and equipment installation coordinates to the construction site reference coordinate system. Spatial consistency processing is performed through the rotation and translation parameters corresponding to the spatial coordinate source to obtain the spatial correction set.

[0067] The spatial correction set is formatted uniformly, the encoding format and resolution of video frames are standardized, the sampling rate and frame shift of acoustic data are unified, the field names and unit dimensions of environmental sensor data and equipment operation data are standardized, and a formatted uniform set is generated.

[0068] Amplitude standardization and dimension normalization are performed across sources on the format-unified set. The center value and scale coefficient are calculated according to the source and channel respectively. The numerical channels are centered and scaled to generate the amplitude specification set.

[0069] The consistency of time and space labels is checked on the amplitude standard set. It is checked whether multiple source samples exist at the same time step and under the same reference coordinate and whether the source labels are consistent. Sample segments that fail the consistency check are removed. For segments that can be restored, their time and space labels are completed to obtain the multi-source data standard set.

[0070] In this embodiment, the generation of the multimodal unified spatiotemporal semantic representation specifically includes:

[0071] The normalized set of multi-source data is segmented into sliding segments according to a unified time base, and a fixed-length time window and a fixed step are set to generate a window sequence arranged in chronological order, thus obtaining a time-series segment set.

[0072] The data source with the highest sampling accuracy is used as the reference source. The alignment offset of each source in the time series segment set relative to the reference source is determined and summarized to form a time lag set.

[0073] Based on the time lag set, each source is resampled and indexed and rearranged. Consistent time step markers are generated at the window level. Time masks are generated for missing time steps while keeping the source identifier unchanged, resulting in a time-series aligned set.

[0074] Spatial registration initialization is performed based on the temporal alignment set. The reference coordinate system of the construction site is selected as the target coordinate system. An initial parameter set is constructed using the prior relationship between the camera's external parameters, the positioning equipment's installation attitude, and the equipment's installation coordinates, thus obtaining the initial value set for spatial registration.

[0075] Based on the initial set of spatial registration values, the pose parameters from each source to the reference coordinate system are optimized, the optimized pose parameter set is output, and the video imaging coordinates, positioning coordinates and equipment installation coordinates are uniformly mapped to the reference coordinate system to obtain the spatial registration set.

[0076] The spatial registration set is semantically encoded according to its source. The encoder extracts spatial semantic vectors, time-frequency semantic vectors, working condition semantic vectors, and motion semantic vectors, and aggregates them to form a source-specific semantic representation set.

[0077] The generation of each semantic vector specifically includes:

[0078] Video data is used to extract spatial semantic vectors through a visual encoder, acoustic data is used to extract time-frequency semantic vectors through an acoustic encoder, environmental sensing data and equipment operation data are used to extract working condition semantic vectors through a numerical encoder, and positioning data is used to extract motion semantic vectors through a trajectory encoder.

[0079] Cross-source semantic consistency weights are calculated for the semantic representation set from different sources, and the set of semantic consistency weights is determined based on the similarity with the unified reference coordinates within the same time step.

[0080] The semantic representation sets from different sources are weighted and fused based on the semantic consistency weight set, and a fused semantic vector sequence is generated under a unified time step and a unified reference coordinate to form a multimodal unified spatiotemporal semantic representation.

[0081] In this embodiment, the generation of the prior set of construction intentions specifically includes:

[0082] The system acquires construction task plans, job role information, and equipment condition information. It breaks down the construction task plans into work processes and time periods to obtain planned time period information. It marks the job role information with responsibility domains and work permissions. It divides the equipment condition information into status levels and marks the equipment as available. It generates task element sets, role element sets, and condition element sets.

[0083] Based on the task element set, role element set, and working condition element set, construct the node and relationship types of the construction intention prior graph, and generate the construction intention prior graph and relationship initialization set;

[0084] The relationship initialization set includes a set of nodes such as task nodes, role nodes, device nodes, region nodes and security boundary nodes, task dependency relationships, role-to-task execution relationships, device-to-task support relationships, region-to-task spatial constraint relationships and security boundary-to-role compliance relationships, as well as conflict constraint edges generated by analyzing the time period overlap relationship of task elements to identify region occupancy conflict events.

[0085] The relationship weights and time period validity weights are calculated for the initial set of relationships. The comprehensive weight between each node is obtained by weighting the plan dependency score, equipment availability score, and spatial constraint consistency score according to the set weights. The relationship weights are normalized by combining the plan time period information, equipment availability markers, and area occupancy conflict events to generate a weighted set of relationships.

[0086] The generation of the plan dependency score, equipment availability score, and spatial constraint consistency score specifically includes:

[0087] Based on the dependencies between tasks in the task element set and the degree of overlap of planned time periods, a plan dependency score is calculated to reflect the degree of temporal sequence conformity and dependency tightness between tasks. Based on the equipment availability flag corresponding to the equipment node, a value of 1 is assigned when the equipment is running normally, a value of 0.01 is assigned when the equipment is under maintenance or offline, and an intermediate value is assigned when the performance is degraded, to obtain the equipment availability score. Based on the spatial location and time period occupied by the regional node, the spatial overlap ratio and time period overlap ratio of different tasks in the same region are compared to obtain the spatial constraint consistency score.

[0088] Based on the multimodal unified spatiotemporal semantic representation, the prior graph of construction intention is generated by time step and construction site reference coordinate system. Node state injection and relation weight update are performed to obtain the intention alignment graph.

[0089] On the intent alignment graph, task nodes and role nodes are jointly expanded by time step to generate an intent candidate set consisting of task sequence, role participation and device support. The prior score of each intent candidate is calculated based on the weighted relationship set to obtain the intent candidate score set.

[0090] Based on the intent candidate score set and node state, intent embedding generation is performed at each time step. The intent candidate set is mapped to a fixed-dimensional vector representation and normalized to obtain the intent embedding sequence.

[0091] The node states of the intent embedding sequence and the intent alignment graph are encapsulated together as a construction intent prior set.

[0092] In this embodiment, the generation of the construction behavior identification result specifically includes:

[0093] Construct an intention-driven inverse evolution model, including an intention generation layer, an inverse evolution prediction layer, a spatiotemporal alignment layer and a bias interpretation layer, and a behavior integration unit;

[0094] In the intent generation layer, the prior set of construction intents is input into the intent embedding mapping module and transformed into a semantic tensor. The semantic alignment module is used to achieve time step pairing with the multimodal unified spatiotemporal semantic representation. Based on the intent enhancement module, the salience of each intent node is adjusted according to the weighted relation set to generate an intent seed sequence and intent constraint set containing intent goals, constraints and trigger thresholds.

[0095] In the inverse prediction layer, the intention seed sequence and the intention constraint set are input into the state inversion module. Based on the intention target, the module performs reverse reasoning from future behavior to the current moment to generate a potential behavior state set. The time-series path generation module establishes behavior transfer links based on the potential behavior state set and the construction task logic to obtain a time-series path set. The consistency evaluation module calculates the path consistency score for each time step in the time-series path set. The path consistency score is determined by the time period validity weight and the intention matching degree. After integration, a predicted behavior trajectory set is obtained.

[0096] In the spatiotemporal alignment layer, the set of predicted behavior trajectories and the unified spatiotemporal semantic representation of multimodality are input into the time alignment module. Local offset search is performed on each time step to determine the time pairing of the predicted behavior trajectory and the unified spatiotemporal semantic representation. The spatial mapping module calculates the corresponding region and location boundary of the predicted behavior trajectory in space based on the construction site reference coordinate system. The confidence evaluation module outputs the alignment confidence set by combining the time offset and spatial deviation to form the alignment pairing sequence.

[0097] In the deviation interpretation layer, the alignment pairing sequence is input into the intention deviation analysis module to calculate the semantic difference between the intention target and the observed behavior at each time step. The spatial deviation measurement module calculates the spatial deviation distance under the reference coordinates of the construction site. The comprehensive deviation fusion module weights and fuses the semantic difference and spatial deviation distance according to preset weights to obtain the deviation index set and the behavioral event candidate set.

[0098] The deviation index set, alignment confidence set, and path consistency score are input into the behavior integration unit, which includes an event chain construction module and a behavior classification module. The event chain construction module establishes candidate event links based on time step continuity and spatial adjacency. The behavior classification module performs threshold determination and category identification on the event links and outputs construction behavior identification results. The construction behavior identification results include behavior category, occurrence time step, and the range of reference coordinates of the construction site.

[0099] In this embodiment, the generation of the risk level assessment result specifically includes:

[0100] The system receives the construction behavior identification results, analyzes the behavior category, occurrence time step and construction site reference coordinate range according to time step, and generates a set of event entries containing event identifier, behavior category, start and end time steps and spatial boundaries. At the same time, it establishes a behavior category weight table and a risk classification threshold set based on historical construction event samples to form an evaluation parameter set.

[0101] The behavior category weight table is determined based on the frequency of occurrence, severity of consequences, and expert correction coefficient of historical construction event samples, reflecting the degree of impact of different behavior categories on overall risk. The risk classification threshold set is set according to the historical risk score distribution and safety management requirements, dividing the risk into high-risk, medium-risk, and low-risk level ranges.

[0102] Based on the construction site reference coordinate system, the construction site is spatially discretized and divided into a set of spatial grids. The event item set is then projected onto the corresponding spatial grid according to its spatial boundary. The event item index of each spatial grid at each time step is accumulated to obtain the spatial event index set.

[0103] By setting a fixed-length time window and a fixed step, sliding aggregation is performed on the spatial event index set to obtain the event aggregation set in each spatial grid and each time window.

[0104] Spatial risk intensity is calculated based on the set of evaluation parameters and the set of event aggregation to form a spatial risk intensity sequence. The spatial risk intensity is obtained by weighted summation of the behavioral category weights and duration steps of events in each spatial grid and time window.

[0105] The spatial risk intensity sequence is normalized and combined with the spatial risk intensity difference between adjacent time windows to generate a standardized risk score sequence. Based on the risk classification threshold set, the standardized risk score sequence is mapped to a risk level sequence, and a risk heat map is constructed to generate a risk level assessment result consisting of spatial location range, corresponding time window, and a list of key behavior categories.

[0106] Example 1:

[0107] To verify the feasibility of this invention in practice, it was applied to a large-scale integrated construction site in a coastal city. This construction area covers a vast area with numerous overlapping work processes, involving complex scenarios with multiple types of equipment such as tower cranes, concrete pump trucks, and excavators operating collaboratively. Construction workers are widely distributed and work across extended periods, making it difficult to promptly identify potential risks using traditional methods relying on video inspections and manual safety patrols. This invention deploys video acquisition equipment, acoustic pickup devices, environmental sensor nodes, positioning base stations, and equipment operation monitoring terminals in this scenario to form a multimodal data acquisition network, enabling full-time, all-element monitoring of the construction site.

[0108] During application, the collected multimodal data first undergoes format standardization, temporal reference standardization, and spatial coordinate correction to automatically generate a standardized multi-source data set. The system then integrates video, acoustic, environmental, and equipment data in a unified coordinate system through temporal alignment and spatial registration, forming a unified spatiotemporal semantic representation that reflects the overall state of the construction site. Combining construction task plans, job role information, and equipment operating condition data, a priori set of construction intentions is further generated to reflect the target state and constraints of each work phase, providing a priori cognitive foundation for subsequent behavior recognition.

[0109] Subsequently, the constructed intent-driven inverse evolution model achieves a deep understanding of the work process during operation. The model first generates intent goals and constraints based on a prior set of construction intentions. Then, through an inverse evolution prediction mechanism, it infers possible behavioral paths from the future goal state, gradually generating potential behavioral sequences. By aligning with real-time multimodal semantic information, the model can identify the actual trajectory and temporal location differences of the work behavior, and calculates semantic differences and spatial deviations during the deviation interpretation stage, thereby outputting the construction behavior identification result. This identification result covers the behavior category, occurrence time, and location range in the construction site's reference coordinates, achieving automated judgment and semantic description of the work activity.

[0110] During continuous operation, the system utilizes the results of construction behavior recognition to perform spatiotemporal risk assessment. By projecting the identified behavior items spatially onto a grid-based partition of the construction site, and combining this with a set of behavior category weights and risk thresholds established from historical event data, the system calculates the risk intensity of behaviors in each time period and each area. After normalization and dynamic sliding window aggregation, the system ultimately generates risk heat distribution and risk level assessment results, forming a visual representation of the on-site safety situation.

[0111] Long-term monitoring and comparison results of this embodiment demonstrate that the present invention can effectively integrate multimodal information in complex construction environments, achieving dynamic understanding of construction intentions and accurate identification of behaviors, significantly reducing misjudgments and omissions. Its reasoning mechanism, based on an intention-driven inverse evolution model, enables the system not only to identify "what behavior occurred," but also to infer "why it occurred" and "whether it met expectations," thereby achieving higher spatiotemporal consistency and interpretability in the risk assessment phase. Ultimately, construction managers can identify hidden safety hazards in advance based on risk level assessment results, optimize construction scheduling and work configuration, and achieve a shift from post-event response to pre-event prevention, significantly improving the intelligence level and operational reliability of on-site safety management.

[0112] Table 1. Performance comparison between intelligent monitoring methods for construction sites based on multimodal data fusion and traditional methods.

[0113] Evaluation metrics (standardized definitions) Single video + rule threshold Multi-source early fusion + general classifier Multimodal general large model solution This invention Behavior recognition accuracy (%) 78.6 83.1 86.4 92.1 Behavioral recognition recall rate (%) 72.4 81.5 84.2 90.3 F1（%） 75.4 82.3 85.3 91.2 False alarm rate (%) 11.8 9.7 8.9 6.1 Missed report rate (%) 27.6 18.5 15.8 9.7 Warning lead time (min, median) 1–2 3–4 4–6 6–9 Spatiotemporal alignment error (m@95th percentile) 2–3 1.5–2.0 1.2–1.8 0.7–1.1 Processing latency (s, end-to-end) 2.8–3.6 3.1–4.0 4.5–6.2 3.2–4.1 Missing modal robustness (F1 reduction, lack of acoustics) −18% −11% −8% −4% High dust masking robustness (F1 reduction) −22% −14% −10% −6% Robustness to low light at night (F1 reduction) −25% −16% −12% −7% Risk popularity positioning deviation (m@mean) 3.2 2.4 1.9 1.1 Consistency of risk classification (consistency rate with manual judgment, %) 74.5 81.2 85.6 91.8 Offline replay consistency (online / replay result consistency rate, %) 88.1 91.7 93.4 96.2 Interpretability rating (expert subjective scale, 0–5) 2.1 2.8 3.4 4.5

[0114] According to the analysis in Table 1, in terms of overall recognition capability, this invention achieves 91.2% on F1, which is about 6 percentage points higher than the "multimodal general large model scheme" and nearly 9 percentage points higher than the "multi-source early fusion + general classifier". This advantage comes first from the three-layer consistency link of S1-S2: timing alignment and spatial registration are completed under a unified clock and construction site reference coordinate system, and semantic encoding and weighted fusion are completed through cross-source semantic consistency weights. This enables different sources to form a computable unified spatiotemporal semantic representation on the same time step and the same spatial reference, which significantly reduces misjudgments and omissions caused by time drift, coordinate inconsistencies and semantic fragmentation.

[0115] In terms of "early warning lead time," this invention achieves a median lead time of 6–9 minutes, further advancing compared to general large-scale models. This improvement is directly related to the "construction intent prior map + inverse evolution prediction" of S3–S4: first, the intent embedding is generated from the task plan, job roles, and equipment conditions; then, possible behavioral paths are deduced from the target intent through inverse reasoning, and spatiotemporal alignment and deviation interpretation are performed with real-time observations. When on-site behavior tends to deviate from the intent, the model captures the signal of "what will happen" earlier, rather than waiting until the anomaly has become fully explicit before issuing an alert, thus increasing the lead time without sacrificing recall.

[0116] Regarding robustness in complex and degraded scenarios (lack of acoustics, high dust levels, low nighttime illumination), the present invention exhibits the smallest F1 reduction: only about 4% in the absence of acoustics, about 6% in dust obstruction, and about 7% in low nighttime illumination. This is due to intent-driven cross-modal compensation: when a source degrades or is missing, the model does not simply rely on the "representation similarity" of the remaining modalities, but instead searches for alternative evidence chains in the temporal path space that conform to the plan dependencies and device availability based on the intent seed and constraint set. This allows behavioral-level cues to compensate for signal-level gaps, reducing the fragile dependence on a single modality.

[0117] The reduction of the "spatiotemporal alignment error" to the order of 0.7–1.1 meters and the decrease of the "risk heat map positioning deviation" to approximately 1.1 meters demonstrate that this invention maintains high spatial consistency in the link from behavior recognition to risk mapping. The key lies in the spatiotemporal alignment layer and the deviation interpretation layer of S4: the former ensures precise alignment between the trajectory and observation through local offset search and mapping to the construction site's reference coordinates, while the latter constructs interpretable deviation indices based on semantic differences and spatial deviations, making risk heat map more stable during gridded mapping and reducing "heat zone drift."

[0118] The simultaneous optimization of both the "false alarm rate" and the "false alarm rate" (false alarms to approximately 6% and false alarms to approximately 9.7%) demonstrates the positive effect of the interpretable closed loop on the threshold strategy: the semantic differences and spatial deviations of the bias interpretation layer provide auditable evidence for alarms, and the behavior integration unit constructs event chains based on temporal continuity and spatial adjacency, reducing occasional false alarms caused by isolated fragments and avoiding missing persistent risks due to insufficient evidence at a single point.

[0119] In terms of "consistency in risk grading" and "consistency in offline replay," this invention achieves approximately 91.8% and 96.2%, respectively. The former benefits from S5's gridded spatiotemporal aggregation and grading threshold mapping: the identification results are first aggregated within a unified coordinate grid and sliding time window, and then the heat and level are generated based on the behavior category weight table and the grading threshold set, which is consistent with human cognition and can reproduce the same conclusions in the review; the latter shows that the entire process from data preprocessing to intent reasoning, from alignment to grading, is repeatable and traceable, and is suitable for long-term operation and accountability review in real production.

[0120] In summary, the significant improvements of this invention come from three aspects: First, the layer-by-layer consistency of data, semantics, and intent reduces the structural noise of multimodal fusion; second, the intent-driven reverse evolution path reasoning elevates "what happened" to "why it happened and whether it deviated from expectations," thus benefiting advance notice and recall; third, interpretable bias and gridded risk mapping naturally translate behavior recognition into spatiotemporally continuous and hierarchical management signals, reducing the fluctuations of subjective thresholds and on-site experience, making early warnings more stable and accurate.

[0121] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1.A construction site intelligent monitoring method based on multi-modal data fusion, characterized in that, The method comprises the following steps: Collecting multi-modal data of the construction site, performing format unification, time reference specification and space coordinate correction on the multi-modal data to obtain a multi-source data standardized set; Performing time alignment, space registration and semantic coding based on the multi-source data standardized set to construct a multi-modal unified space-time semantic representation; Constructing a construction intention prior atlas according to a construction task plan, post role information and equipment working condition information, generating an intention embedding corresponding to the multi-modal unified space-time semantic representation to obtain a construction intention prior set; Constructing an intention-driven inverse evolution model, the model comprising an intention generation layer, an inverse evolution prediction layer, a space-time alignment layer and a deviation explanation layer, and outputting a construction behavior recognition result; Performing space-time risk assessment based on the construction behavior recognition result, and generating a risk level assessment result in combination with the spatial position distribution and the time evolution trend. 2.The construction site intelligent monitoring method based on multi-modal data fusion of claim 1, wherein, The generation of the multi-source data standardized set specifically comprises: Obtaining video data, acoustic data, environmental sensing data, positioning data and equipment operation data, recording sampling time marks and collection source identifiers respectively, eliminating empty frames, silent segments, out-of-bound readings and invalid logs to obtain an original multi-modal data set; Performing quality review on the original multi-modal data set, completing abnormal amplitude clipping, short-time missing interpolation and repeated sample merging to generate a cleaned data set; Performing time reference specification on the cleaned data set, offset correcting the time marks according to a unified clock and resampling to a common time step to obtain a time alignment set; Performing space coordinate correction on the time alignment set, mapping video imaging coordinates, positioning coordinates and equipment installation coordinates to a construction site reference coordinate system, and performing space consistency processing through rotation and translation parameters corresponding to the space coordinate sources to obtain a space correction set; Performing format unification on the space correction set, specifying the encoding format and resolution of video frames, unifying the sampling rate and frame shift of acoustic data, standardizing the field name and unit dimension of environmental sensing data and equipment operation data to generate a format unified set; Implementing cross-source amplitude standardization and dimension normalization on the format unified set, calculating the center value and scale coefficient according to the source and channel respectively, and performing centering and equal-scale processing on the numerical channel to generate an amplitude specification set; Performing consistency review of the time marks and space marks of the amplitude specification set, checking whether the multi-source samples at the same time step and the same reference coordinate exist simultaneously and have consistent source identifiers, eliminating sample segments that do not pass the consistency check, and completing the time and space marks of the segments with restoration conditions to obtain the multi-source data standardized set. 3.The construction site intelligent monitoring method based on multi-modal data fusion of claim 1, wherein, The generation of the multi-modal unified space-time semantic representation specifically comprises: Sliding segmenting the multi-source data standardized set according to a unified time reference, setting a fixed length time window and a fixed step, generating a window sequence arranged in time sequence to obtain a time sequence segmentation set; Determining the alignment offset of each source relative to the reference source in the time sequence segmentation set, and forming a time lag set by summarizing; The time series alignment set is obtained by resampling and index rearrangement of each source according to the time lag set, generating consistent time step markers at the window level, generating time masks for missing time steps and keeping the source identification unchanged; The spatial registration initialization is performed based on the time series alignment set, the construction site reference coordinate system is selected as the target coordinate system, the initial parameter set is constructed by using the prior relationship of the camera external parameter, the positioning device installation attitude and the device installation coordinate, and the spatial registration initial value set is obtained; On the basis of the spatial registration initial value set, the pose parameters of each source to the reference coordinate system are optimized, the optimized pose parameter set is output, and the video imaging coordinates, positioning coordinates and device installation coordinates are uniformly mapped to the reference coordinate system, and the spatial registration set is obtained; The spatial registration set is semantically coded according to the source, the spatial semantic vector, the time-frequency semantic vector, the working condition semantic vector and the motion semantic vector are extracted through the encoder, and the source-specific semantic representation set is aggregated; The cross-source semantic consistency weight is calculated for the source-specific semantic representation set, and the semantic consistency weight set is determined according to the similarity in the same time step and under the unified reference coordinate; The source-specific semantic representation set is fused according to the semantic consistency weight set, and the fused semantic vector sequence is generated under the unified time step and the unified reference coordinate, and the multi-modal unified space-time semantic representation is formed. 4.The construction site intelligent monitoring method based on multi-modal data fusion of claim 1, wherein, The generation of the construction intention prior set specifically includes: Construction task plans, post role information and equipment working condition information are obtained, the construction task plans are decomposed and time period is divided to obtain plan time period information, the post role information is marked for responsibility domain and operation permission, and the equipment working condition information is divided into state levels and marked for equipment availability, to generate a task element set, a role element set and a working condition element set; The nodes and relationship types of the construction intention prior graph are constructed according to the task element set, the role element set and the working condition element set, and the construction intention prior graph and the relationship initialization set are generated; The relationship weight and the time period effectiveness weight are calculated for the relationship initialization set, the comprehensive weight between each node is obtained by weighting the plan dependence score, the equipment availability score and the spatial constraint consistency score according to the set weight, and the relationship weight is normalized by combining the plan time period information, the equipment availability mark and the area occupation conflict event, to generate a weighted relationship set; Based on the multi-modal unified space-time semantic representation, the intention alignment graph is obtained by performing node state injection and relationship weight updating on the construction intention prior graph according to the time step and the construction site reference coordinate system; The task nodes and role nodes are jointly expanded according to the time step on the intention alignment graph, an intention candidate set composed of task sequences, role participation and equipment support is generated, and the prior score of each intention candidate is calculated according to the weighted relationship set, to obtain an intention candidate score set; The intention embedding sequence is obtained by performing intention embedding generation on each time step according to the intention candidate score set and the node state, which maps the intention candidate set to a fixed-dimensional vector representation and performs normalization processing. The intention embedding sequence and the node state of the intention alignment graph are jointly encapsulated as the construction intention prior set. 5.The construction site intelligent monitoring method based on multi-modal data fusion of claim 1, wherein, The construction behavior recognition result generation specifically includes: An intention-driven reverse evolution model is constructed, including an intention generation layer, a reverse evolution prediction layer, a space-time alignment layer, a deviation explanation layer, and a behavior integration unit; In the intention generation layer, the construction intention priori set is input into the intention embedding mapping module to be converted into a semantic tensor. The semantic alignment module is used to realize time step pairing with the multi-modal unified space-time semantic representation. The intention reinforcement module is used to adjust the saliency of each intention node based on the weighted relationship set to generate an intention seed sequence and an intention constraint set containing intention targets, constraint conditions, and trigger thresholds; In the reverse evolution prediction layer, the intention seed sequence and the intention constraint set are input into the state inversion module to perform reverse reasoning from future behavior to the current time according to the intention target, generate a set of potential behavior states, and establish a behavior transition link according to the set of potential behavior states and the construction task logic to obtain a set of time sequence paths. The consistency evaluation module calculates the path consistency score for each time step in the set of time sequence paths, which is determined by the time period effectiveness weight and the intention matching degree. After integration, a set of predicted behavior trajectories is obtained; In the space-time alignment layer, the set of predicted behavior trajectories and the multi-modal unified space-time semantic representation are input into the time alignment module to perform local offset search on each time step, determine the time pairing of the predicted behavior trajectory and the unified space-time semantic representation, and calculate the corresponding area and position boundary of the predicted behavior trajectory in space according to the construction site reference coordinate system by the space mapping module. The confidence evaluation module outputs a set of alignment confidence sets to form an alignment pairing sequence by integrating the time offset and the spatial deviation amount; In the deviation explanation layer, the alignment pairing sequence is input into the intention deviation analysis module to calculate the semantic difference amount between the intention target and the observed behavior at each time step. The spatial deviation measure module calculates the spatial deviation distance under the construction site reference coordinate. The comprehensive deviation fusion module weights and fuses the semantic difference amount and the spatial deviation distance according to the preset weight to obtain a deviation index set and a behavior event candidate set; The deviation index set, the alignment confidence set, and the path consistency score are input into the behavior integration unit, including an event chain construction module and a behavior classification module. The event chain construction module establishes candidate event links according to the time step continuity and spatial adjacency. The behavior classification module performs threshold determination and class identification on the event link to output the construction behavior recognition result, which includes the behavior class, the occurrence time step, and the construction site reference coordinate position range. 6.The construction site intelligent monitoring method based on multi-modal data fusion of claim 1, wherein, The risk level evaluation result generation specifically includes: The construction behavior recognition result is received, and the behavior class, occurrence time step, and construction site reference coordinate position range are analyzed by time step to generate an event entry set containing event identification, behavior class, start and end time steps, and spatial boundaries. Meanwhile, the behavior class weight table and the risk classification threshold set are established based on historical construction event samples to form an evaluation parameter set; According to the construction site reference coordinate system, the construction site is discretized in space, divided into a spatial grid set, and the event entry set is projected onto the corresponding spatial grid according to its spatial boundary. The event entry index of each spatial grid at each time step is accumulated to obtain a spatial event index set; A fixed-length time window and a fixed step are set, and a sliding aggregation is performed on the spatial event index set to obtain an event aggregation set within each spatial grid and each time window; Based on the evaluation parameter set and the event aggregation set, the spatial risk intensity is calculated to form a spatial risk intensity sequence. The spatial risk intensity is obtained by weighted summation of the behavior category weight and the duration step of the events in each spatial grid and time window; The spatial risk intensity sequence is normalized and combined with the spatial risk intensity difference between adjacent time windows to generate a standardized risk score sequence. According to the risk classification threshold set, the standardized risk score sequence is mapped to a risk level sequence, and a risk heat map is constructed to generate a risk level evaluation result composed of spatial location range, corresponding time window and key behavior category list.

Images