Supervision work abnormity tracking method and system based on artificial intelligence

Abstract

The invention relates to the technical field of engineering supervision, in particular to a supervision work abnormity tracking method and system based on artificial intelligence, and constructs a structured input data set by synchronously collecting and uniformly preprocessing video monitoring data, environment sensing data and supervision log text data of a construction site. Personnel states, environment change features and text risk semantics are obtained through feature extraction algorithms such as target detection, posture recognition and semantic analysis, and are integrated to form a multi-modal abnormal feature set. An abnormal node set is established through a clustering algorithm, abnormal node feature vectors are constructed based on behavior attributes, spatial positions and risk semantics of nodes, a weighted adjacency matrix is constructed by using a neighbor retrieval algorithm, and quantitative modeling of a space-time association relationship between potential risk nodes is realized. Node risk weights are obtained through matrix multiplication and addition and normalization calculation, multi-level abnormal grade division is performed according to the risk weights, differential tracking alarms are generated, and the alarm accuracy and response efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of engineering supervision technology, and more particularly to an artificial intelligence-based method and system for tracking anomalies in supervision work. Background Technology

[0002] Construction sites present inherent risks that are both sudden and hidden due to the complexity of procedures, the diversity of responsible parties, and the dynamic nature of the work environment. Current supervision methods primarily rely on manual inspections, on-site spot checks, and paper or electronic logs. However, the limited coverage of supervisors, long inspection cycles, and delayed information feedback make it easy for risks to go undetected, resulting in blind spots in monitoring and missed violations.

[0003] Meanwhile, manual recording is highly subjective, and textual information is difficult to structure, making it impossible to fully uncover potential risk correlations. Existing technologies use video surveillance systems and environmental sensors to assist in supervision, but they process data from different sources separately, lacking a unified multimodal fusion mechanism. This prevents the correlation analysis of personnel behavior, changes in environmental parameters, and log semantic information, resulting in significant deficiencies in cross-dimensional anomaly identification within the supervision system. Furthermore, existing monitoring and analysis methods are mostly based on threshold judgments or empirical rules, making it difficult to identify abnormal states with continuous, gradual, and contextual characteristics. Their ability to provide early warnings of complex risks is limited, leading to severe alarm generalization, high false alarm rates, and impacting the efficiency of construction management decisions and personnel response. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an artificial intelligence-based method and system for tracking anomalies in supervisory work.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An AI-based method for tracking anomalies in supervisory work includes the following steps: Acquire comprehensive supervision data from the construction site, including video surveillance data, environmental sensor data, and supervision log text data; perform denoising, normalization, and time synchronization processing on the comprehensive supervision data to obtain a structured input dataset; The video surveillance data in the structured input dataset is processed by target detection and behavior recognition to obtain personnel status data; the environmental sensor data is processed by time-series feature extraction to obtain environmental change feature data; and the supervision log text data is processed by semantic recognition and keyword extraction to obtain text risk feature data. By integrating personnel status data, environmental change characteristic data, and text risk characteristic data, a multimodal anomaly feature set is obtained; Based on the multimodal anomaly feature set, an anomaly node set is established through a clustering algorithm, an anomaly node feature vector is constructed based on node attributes, and a weighted adjacency matrix is ​​constructed based on a nearest neighbor retrieval algorithm. Based on the weighted adjacency matrix and the feature vector of the abnormal node, matrix multiplication and addition operations and normalization are performed to obtain the node risk weight value and the corresponding abnormal event risk assessment result. The risk assessment results are classified into anomaly levels, and anomaly tracking alarms are generated based on the anomaly levels.

[0006] Furthermore, obtaining comprehensive supervision data from the construction site includes the following steps: Video stream data is collected by fixed cameras deployed at the construction site and frame extraction processing is performed to obtain a continuous video frame sequence. Environmental monitoring data is obtained by collecting temperature, humidity, gas concentration and noise signals through environmental sensors installed at the construction site. The supervisor's log text data is obtained by retrieving the supervisor's written records through the electronic log system.

[0007] Furthermore, the denoising, normalization, and time synchronization processing of the comprehensive supervision data includes: Perform frame mean filtering and edge-preserving enhancement processing on video surveillance data; Perform sliding window smoothing and extreme value truncation on environmental sensor data; Perform character cleaning and format standardization processing on the text data of the supervision log; Based on timestamp information and spatial coordinates, video frames, sensor data, and text records are time-synchronized and spatially registered to obtain a structured input dataset.

[0008] Furthermore, the step of performing target detection and behavior recognition processing on the video surveillance data in the structured input dataset to obtain personnel status data includes the following steps: The video surveillance data in the structured input dataset is processed by a target detection model based on the YOLO algorithm to locate people, resulting in people detection data containing location coordinates and bounding box information; Based on the personnel detection data, the OpenPose algorithm is used to extract key points in the personnel area to obtain personnel pose data. Based on the personnel posture data, behavior classification and recognition are performed using an action classification and matching model to obtain personnel status data.

[0009] Furthermore, the step of extracting time-series features from environmental sensing data to obtain environmental change feature data includes the following steps: The environmental sensor data in the structured input dataset is subjected to time series fitting processing based on an autoregressive moving average model to obtain trend change data of environmental parameters; Based on the trend change data, outlier detection processing is performed using the local outlier factor algorithm to obtain outlier deviation data of environmental parameters. Based on the outlier deviation data, volatility analysis is performed using an exponentially weighted moving average to obtain the risk volatility data of environmental parameters; Based on the risk fluctuation data, the abnormal amplitude and frequency of environmental parameters are labeled with attributes to obtain environmental change characteristic data including the direction of change, fluctuation intensity and abnormal cycle.

[0010] Furthermore, the semantic recognition and keyword extraction processing of the supervision log text data includes the following steps: The supervision log text data is semantically encoded based on a pre-trained BERT language model to obtain text embedding data containing context semantic vectors; Based on the text embedding data, named entity recognition is performed using a conditional random field to obtain semantic recognition data containing risk entity tags; By calculating keyword weights using semantic recognition data based on the TF-IDF algorithm, the weight ranking data of each risk phrase in the text is obtained; Based on the weighted ranking data, attribute annotation processing is performed on high-weight risk phrases to obtain text risk feature data containing risk category, frequency of occurrence, and semantic context position.

[0011] Further, the step of establishing an abnormal node set based on the multimodal abnormal feature set using a clustering algorithm, constructing an abnormal node feature vector based on node attributes, and constructing a weighted adjacency matrix based on a nearest neighbor retrieval algorithm includes the following steps: Based on the personnel status data, environmental change feature data, and text risk feature data in the multimodal anomaly feature set, the DBSCAN algorithm is used to cluster the same time slices and adjacent spatial regions to obtain a set of anomaly nodes containing spatial centers, behavioral attributes, and risk attributes. Based on the set of abnormal nodes, the spatial coordinates, behavioral attributes and risk semantic attributes of each abnormal node are vectorized and encoded to obtain the corresponding abnormal node feature vector. The KD-Tree nearest neighbor retrieval algorithm is used to calculate the adjacency relationship of the spatial locations of nodes in the abnormal node set, and spatial adjacency data containing the spatial distance between nodes is obtained. Based on the spatial adjacency data and the personnel status data and environmental change characteristic data corresponding to the abnormal nodes, the spatiotemporal weight data between nodes are jointly calculated using the Gaussian radial basis function and the dynamic time warping algorithm. Based on the spatiotemporal weighted data and the text risk feature data corresponding to the abnormal nodes, the text risk feature vectors are compared using the cosine similarity algorithm to generate a weighted adjacency matrix.

[0012] Furthermore, performing matrix multiplication and addition operations and normalization calculations on the weighted adjacency matrix includes the following steps: Based on the weighted adjacency matrix, the feature vectors of abnormal nodes are processed by multiplication and addition to obtain initial risk score data containing the interaction effects of multiple nodes; The initial risk score data is subjected to amplitude scaling based on the minimum-maximum normalization algorithm to obtain standardized risk score data. The probability distribution is normalized using the Softmax function to obtain the risk assessment results.

[0013] Furthermore, the abnormality levels include no abnormality, preventative abnormality, and emergency abnormality.

[0014] An AI-based system for tracking anomalies in supervisory work, applied to any of the aforementioned AI-based methods for tracking anomalies in supervisory work, is characterized by comprising: The data acquisition module is used to acquire comprehensive supervision data from the construction site, including video surveillance data, environmental sensor data, and supervision log text data; and to perform noise reduction, normalization, and time synchronization processing on the comprehensive supervision data to obtain a structured input dataset. The data feature processing module is used to perform target detection and behavior recognition processing on the video surveillance data in the structured input dataset to obtain personnel status data; to perform time-series feature extraction processing on the environmental sensor data to obtain environmental change feature data; and to perform semantic recognition and keyword extraction processing on the supervision log text data to obtain text risk feature data. The feature integration module is used to integrate personnel status data, environmental change feature data, and text risk feature data to obtain a multimodal anomaly feature set; The adjacency matrix construction module is used to establish an abnormal node set based on the multimodal abnormal feature set through a clustering algorithm, construct an abnormal node feature vector based on node attributes, and construct a weighted adjacency matrix based on a nearest neighbor retrieval algorithm. The risk assessment module is used to perform matrix multiplication and addition operations and normalization processing based on the weighted adjacency matrix and the feature vector of abnormal nodes to obtain the node risk weight value and the corresponding abnormal event risk assessment result. The tracking and alarm module is used to classify the risk assessment results into anomaly levels and generate anomaly tracking alarms based on the anomaly levels.

[0015] The beneficial effects of this invention are as follows: By synchronously collecting and uniformly preprocessing video surveillance data, environmental sensor data, and supervision log text data, a structured input dataset is established, achieving consistency of supervision data in both time and spatial dimensions. Furthermore, feature extraction algorithms using target detection, pose recognition, and semantic parsing are employed to obtain personnel status, environmental change trends, and textual risk semantics, respectively. These features are then integrated to form a multimodal anomaly feature set, overcoming the shortcomings of traditional supervision models, such as unstructured text, fragmented data, and inability to perform cross-domain analysis. An anomaly node set is established through clustering algorithms, and feature vectors are constructed using node behavioral attributes, spatial locations, and risk semantics. Combined with a nearest neighbor retrieval algorithm, a weighted adjacency matrix is ​​generated, quantifying the spatiotemporal correlation between different risk nodes. Based on matrix multiplication and addition and normalization operations, this scheme further calculates node risk weights, compensating for the shortcomings of traditional threshold-based judgments in identifying continuous and contextual latent anomalies, and enabling the analysis of anomaly propagation chains and risk accumulation effects. Finally, based on the risk assessment results, multi-level anomaly classification is carried out, and differentiated tracking alarms are generated to improve alarm accuracy, reduce false alarm rate and management costs caused by generalized alarms, thereby supporting supervisors to intervene in and refine the supervision of multi-dimensional complex risks in advance, and ultimately improve the safety management level and risk handling efficiency of the construction site. Attached Figure Description

[0016] Figure 1 This is a flowchart of the steps of an artificial intelligence-based method for tracking anomalies in supervision work according to the present invention.

[0017] Figure 2 The flowchart of the steps in this invention is as follows: based on the multimodal anomaly feature set, an anomaly node set is established through a clustering algorithm; an anomaly node feature vector is constructed based on node attributes; and a weighted adjacency matrix is ​​constructed based on a nearest neighbor retrieval algorithm. Detailed Implementation

[0018] Please see Figures 1-2 As shown, this invention relates to an artificial intelligence-based method for tracking anomalies in supervisory work, comprising the following steps: Acquire comprehensive supervision data from the construction site, including video surveillance data, environmental sensor data, and supervision log text data; perform denoising, normalization, and time synchronization processing on the comprehensive supervision data to obtain a structured input dataset; The video surveillance data in the structured input dataset is processed by target detection and behavior recognition to obtain personnel status data; the environmental sensor data is processed by time-series feature extraction to obtain environmental change feature data; and the supervision log text data is processed by semantic recognition and keyword extraction to obtain text risk feature data. By integrating personnel status data, environmental change characteristic data, and text risk characteristic data, a multimodal anomaly feature set is obtained; Based on the multimodal anomaly feature set, an anomaly node set is established through a clustering algorithm, an anomaly node feature vector is constructed based on node attributes, and a weighted adjacency matrix is ​​constructed based on a nearest neighbor retrieval algorithm. Based on the weighted adjacency matrix and the feature vector of the abnormal node, matrix multiplication and addition operations and normalization are performed to obtain the node risk weight value and the corresponding abnormal event risk assessment result. The risk assessment results are classified into anomaly levels, and anomaly tracking alarms are generated based on the anomaly levels.

[0019] In some embodiments, comprehensive supervision data is first acquired through multimodal sensing devices deployed at the construction site. Video surveillance data captures personnel work behavior and dynamic changes in the area; environmental sensor data reflects real-time operating parameters such as temperature, humidity, gas concentration, and noise intensity; and supervision log text data records descriptions of violations, rectification suggestions, and hazard markings during the construction process. Because the three types of data have different sources, inconsistent time bases, and significantly different noise characteristics, visual flicker noise, signal abrupt changes, and redundant semantic symbols are removed through frame mean filtering, sliding smoothing, and string cleaning. Synchronization registration is then performed based on a unified timestamp to obtain a spatially, temporally, and semantically consistent structured input dataset. Subsequently, the system incorporates the YOLO detection model and OpenPose skeletal keypoint recognition technology into the video surveillance data. This not only locates personnel positions using bounding boxes but also encodes their movement trajectories using posture vectors, enabling the identification of potentially dangerous behaviors such as crossing safety barriers or leaning against scaffolding. Simultaneously, it utilizes an autoregressive moving average model to extract trend fluctuations in environmental data and captures abnormal deviations through a local outlier algorithm. This allows it to identify issues that are difficult to detect using traditional threshold triggering mechanisms, such as the gradual accumulation of gas leaks or a slow, abnormal rise in temperature. Furthermore, semantic encoding based on the BERT language model and conditional random field entity recognition technology are used to perform semantic association mining on the supervision log text. This allows implicit soft expressions such as "scaffolding swaying" and "welding residue" in the text to be given structured risk labels, overcoming the limitation of existing technologies where textual risks cannot be used for quantitative analysis. By horizontally concatenating personnel posture vectors, environmental fluctuation indicators, and textual semantic risk embedding vectors using a feature mapping model, a multimodal anomaly feature set is constructed, solving the problem of isolated data processing in existing technologies that leads to the inability to identify cross-dimensional coupled risks. Subsequently, the DBSCAN clustering algorithm is used to construct anomaly node sets based on high-risk features that are in the same time slice and spatially adjacent. Anomaly node feature vectors are then vectorized based on node spatial coordinates, behavioral action types, and risk semantic attributes. Unlike traditional methods that rely solely on a single indicator to construct anomaly entities, this approach can create a high-dimensional risk profile for each node. Furthermore, the KD-Tree nearest neighbor retrieval algorithm, combined with Gaussian radial basis functions and dynamic time warping, is used to construct a weighted adjacency matrix between nodes. This enables a quantitative representation of risk spreading over time, across regions, and in chain-like propagation, overcoming the weakness of existing technologies in failing to reflect the spatiotemporal dependence of anomaly events. Based on this, by performing matrix multiplication and addition operations on the weighted adjacency matrix and node feature vectors, the system calculates the interactive risk distribution of each node. Combining minimum-maximum normalization and Softmax probability mapping, the risk assessment results are made comparable and interpretable. Ultimately, this embodiment classifies anomalies into different levels based on risk assessment values ​​and generates differentiated tracking alarms. It can issue early warnings to supervisors when potential hazards are in their nascent stage, significantly reducing false alarms, avoiding generalized alarms, and improving the efficiency of supervisory resource allocation.Compared with existing supervision methods that rely on fixed thresholds, manual judgment, or single-modal analysis, this invention has the ability to perform cross-modal fusion, spatiotemporal propagation modeling, and contextual semantic association.

[0020] Furthermore, obtaining comprehensive supervision data from the construction site includes the following steps: Video stream data is collected by fixed cameras deployed at the construction site and frame extraction processing is performed to obtain a continuous video frame sequence. Environmental monitoring data is obtained by collecting temperature, humidity, gas concentration and noise signals through environmental sensors installed at the construction site. The supervisor's log text data is obtained by retrieving the supervisor's written records through the electronic log system.

[0021] In some embodiments, multiple fixed industrial monitoring cameras are first deployed at key locations in the construction area. These cameras acquire continuous video stream data at a constant frame rate. The video acquisition module then performs frame extraction processing, discretizing the video stream into a sequence of image frames with a uniform frame rate, thus avoiding inter-frame time drift caused by differences in encoding and decoding between different cameras. Simultaneously, a timestamp and spatial location information are added to each frame during the extraction process to construct a temporal-spatial reference coordinate system for subsequent behavior recognition. For environmental monitoring, multi-mode environmental sensor nodes deployed in different construction functional zones are used to collect physical quantities such as temperature, humidity, combustible gas concentration, and noise intensity in real time. These sensors output time-series numerical signals with a fixed sampling period. However, due to factors such as vibration, electromagnetic interference, and air turbulence during construction, sensor data is prone to sudden spikes or sensor drift. Therefore, the system uses a data acquisition interface to uniformly buffer the sensor outputs and bind sampling timestamps for subsequent time-series model processing. In terms of collecting supervision text data, data is called through the API interface of the electronic log system to incrementally capture the text records entered by supervisors in real time through mobile terminals or PCs. This process not only obtains the basic text content, but also extracts metadata such as log creation time, input work station number and component association tags at the same time.

[0022] Furthermore, the denoising, normalization, and time synchronization processing of the comprehensive supervision data includes: Perform frame mean filtering and edge-preserving enhancement processing on video surveillance data; Perform sliding window smoothing and extreme value truncation on environmental sensor data; Perform character cleaning and format standardization processing on the text data of the supervision log; Based on timestamp information and spatial coordinates, video frames, sensor data, and text records are time-synchronized and spatially registered to obtain a structured input dataset.

[0023] In some embodiments, the video surveillance data is first denoised using a frame mean filtering algorithm. This algorithm suppresses random noise by calculating the pixel-by-pixel mean of pixel values ​​across multiple consecutive frames, while maintaining the continuity of the dynamic scene. Based on this, an edge-preserving enhancement algorithm is used to enhance the gradient of structural boundaries, resulting in higher gradient responses for human silhouettes, edges of dangerous components, and safety barriers in the construction area. This facilitates faster convergence of the subsequent pose recognition model during key point localization. For example, when dust or insufficient lighting at the construction site causes image blurring, edge-preserving enhancement effectively improves feature saliency, thereby reducing the probability of misjudging human poses. For environmental sensing data, which is susceptible to random electromagnetic interference, mechanical vibration, and air turbulence, resulting in instantaneous gaps or spike noise, the system introduces a sliding window smoothing algorithm to perform window averaging on the sensing time series and weaken short-period jumps. Simultaneously, an extreme value truncation mechanism is used to prune data exceeding the confidence interval threshold to avoid sparse extreme values ​​misleading the time series analysis model. For example, when a gas sensor experiences a sudden spike in reading due to a workpiece impact, traditional processing methods might mistakenly identify this as a leak. This invention avoids false alarms and maintains the stability of the sequence trend by using extreme value truncation. Regarding the processing of supervisor log text data, the text is entered by multiple people and often contains redundant symbols, typos, and inconsistent formatting. The system uses a character cleaning module to filter redundant punctuation, spaces, and abnormal coded characters, and performs formatting and regularization based on regular expressions to ensure the text conforms to the semantic model input specifications, improving the feature purity of subsequent semantic parsing. Based on the timestamp information attached during acquisition, video frames, sensor data, and text records are synchronized in time, aligning all data to a unified time series, ensuring consistent mapping of behavior, environment, and text under the same working condition. Furthermore, spatial registration is performed using spatial coordinates to match the video monitoring range, sensor deployment location, and the workstation indicated by the log, thereby achieving comparability of cross-regional data within the same spatial reference system. For example, when a person performs a dangerous action in a specific scaffolding area, spatial registration can automatically associate this with noise anomalies in the corresponding area and the log description of "scaffolding loosening." The structured input dataset obtained after the above processing has multimodal feature consistency, temporal continuity and spatial traceability, providing a stable, high-dimensional and fusionable data foundation for subsequent abnormal node construction and risk propagation modeling.

[0024] Furthermore, the step of performing target detection and behavior recognition processing on the video surveillance data in the structured input dataset to obtain personnel status data includes the following steps: The video surveillance data in the structured input dataset is processed by a target detection model based on the YOLO algorithm to locate people, resulting in people detection data containing location coordinates and bounding box information; Based on the personnel detection data, the OpenPose algorithm is used to extract key points in the personnel area to obtain personnel pose data. Based on the personnel posture data, behavior classification and recognition are performed using an action classification and matching model to obtain personnel status data.

[0025] In some embodiments, a YOLO-based one-stage object detection model is first invoked to detect people in the image. This model divides the input frame into grid regions using a deep convolutional neural network and predicts the bounding box offset, object confidence, and class probability for each region. It can complete position regression and class discrimination in a single forward propagation, ensuring high robustness of person detection even in complex backgrounds, lighting variations, or partial occlusion. Through this process, the system obtains person detection data containing position coordinates, bounding box width and height, and confidence scores, providing accurate regions of interest for subsequent pose estimation. Based on the extracted person detection boxes, the system further employs the OpenPose pose keypoint extraction algorithm to perform skeletal-level point inference of the person's limb structure. This algorithm predicts the affinity field between human keypoint heatmaps and limb joints using a multi-branch convolutional structure, and achieves keypoint association through maximum response mapping and vector field tracking, thus accurately inferring the position of each body part even when multiple people appear simultaneously or when limbs are partially occluded. In high-altitude work scenarios at construction sites, OpenPose can accurately identify dangerous postures such as deep bending, clinging to structural surfaces, or extending arms beyond safe limits, providing fine-grained human motion characteristics for behavior identification. After obtaining the coordinate set of key points of personnel, a time-series vector of actions is constructed, and behavior recognition is performed based on an action classification and matching model. This model can use Bi-LSTM or temporal convolutional network (TCN) structures to model the dynamic changes of key points over time. By analyzing the angle changes, velocity gradients, and relative displacement features between key points, it classifies typical behaviors at construction sites. For example, the model can distinguish between two visually similar but different risk levels: "avoiding sideways passage" and "illegally climbing over guardrails." Furthermore, through a pre-trained action template matching mechanism, the model can capture subtle skews in the early stages of action evolution. For example, if a welder's angle of holding the welding torch deviates from the standard posture, it may indicate the risk of welding slag falling, thus achieving proactive identification. Finally, the spatial location information of the target detection, the skeletal features formed by the posture key points, and the behavior classification labels are fused to form personnel status data. This data not only includes the area where the human body is located, the posture pattern, and the action category, but also records the risk confidence of the behavior and the confidence score of the violation of the norm.

[0026] Furthermore, the step of extracting time-series features from environmental sensing data to obtain environmental change feature data includes the following steps: The environmental sensor data in the structured input dataset is subjected to time series fitting processing based on an autoregressive moving average model to obtain trend change data of environmental parameters; Based on the trend change data, outlier detection processing is performed using the local outlier factor algorithm to obtain outlier deviation data of environmental parameters. Based on the outlier deviation data, volatility analysis is performed using an exponentially weighted moving average to obtain the risk volatility data of environmental parameters; Based on the risk fluctuation data, the abnormal amplitude and frequency of environmental parameters are labeled with attributes to obtain environmental change characteristic data including the direction of change, fluctuation intensity and abnormal cycle.

[0027] In some embodiments, the environmental sensor data in the structured input dataset is first subjected to time series fitting processing based on an Autoregressive Moving Average (ARMA) model to construct the asymptotic trend of environmental parameters over time. The ARMA model, by linearly combining historical sampled values ​​and residual terms, can effectively fit the periodic changes and slow drift trends of environmental parameters at the construction site, such as the gradual increase in temperature due to welding operations or the continuous accumulation of dust concentration due to cutting operations. Compared to traditional averaging methods, the ARMA model can predict short-term fluctuations more precisely, enabling the system to distinguish between normal environmental disturbances and potential abnormal upward trends during construction. Through this step, the system generates trend change data of environmental parameters, providing a continuous benchmark for subsequent anomaly detection. After acquiring the trend change data, the Local Outlier Factor (LOF) algorithm is used to perform anomaly detection. By calculating the outlier degree of each data point in its local density, anomalies that significantly deviate from normal density clusters in the neighborhood distribution are identified, making it suitable for capturing noise-type risks in the dynamic context of construction sites. For example, when localized combustion at a construction site generates short-term high concentrations of gas or sudden noise impacts, LOF can identify abnormal density distributions and obtain outlier data for environmental parameters, avoiding missed detections due to single-threshold judgments. Subsequently, an Exponentially Weighted Moving Average (EWMA) model is constructed based on the outlier data for fluctuation analysis. The EWMA model assigns higher weights to recent data, achieving a balance between sensitivity and inertia in the time-series response. When an environmental parameter continuously and slowly deviates from the normal range, the EWMA curve will gradually rise, reflecting the continuous accumulation of risk; for example, a slow increase in combustible gas concentration in the welding area may indicate insufficient ventilation. Finally, this step outputs risk fluctuation data for environmental parameters, reflecting the intensity and continuity of risk evolution over time. Based on this, attribute labeling is performed on the abnormal amplitude and frequency of environmental parameters according to the risk fluctuation data. The attribute labeling process uses feature encoding based on the fluctuation intensity range, the duration of exceeding safety limits, and the risk direction (increase or decrease), enabling the system to output environmental change characteristic data containing key fields such as change direction, fluctuation intensity, and abnormal period. For example, in a high-temperature environment, if the temperature continues to rise and fluctuates significantly, the system will label it as a "high-intensity short-cycle anomaly"; while a slow increase in dust concentration may be labeled as a "low-intensity long-cycle latent accumulation risk".

[0028] Furthermore, the semantic recognition and keyword extraction processing of the supervision log text data includes the following steps: The supervision log text data is semantically encoded based on a pre-trained BERT language model to obtain text embedding data containing context semantic vectors; Based on the text embedding data, named entity recognition is performed using a conditional random field to obtain semantic recognition data containing risk entity tags; By calculating keyword weights using semantic recognition data based on the TF-IDF algorithm, the weight ranking data of each risk phrase in the text is obtained; Based on the weighted ranking data, attribute annotation processing is performed on high-weight risk phrases to obtain text risk feature data containing risk category, frequency of occurrence, and semantic context position.

[0029] In some embodiments, a Chinese engineering scenario corpus is first constructed. The original text from the electronic log system is denoised, segmented, and word-segmented before being input into a pre-trained BERT language model to complete semantic encoding, resulting in a dynamic semantic vector for each token under contextual conditions. To reduce the extra-domain bias of colloquial records, abbreviations, and professional terms, the embodiment adopts a continued pre-training strategy on the basis of BERT. Self-supervised optimization is performed using an incremental corpus consisting of "supervision terminology glossary + engineering safety regulations + historical hazard rectification records" to make the embedding vectors more closely resemble the distribution in the construction supervision field. At the vector level, the system performs multi-head attention aggregation on the [CLS] sentence-level vectors and token-level context vectors, retaining the weights of key prompts within the sentence (such as "welding slag", "loose scaffolding", "open flame operation"). Subsequently, the system performs named entity recognition by concatenating a Conditional Random Field (CRF) layer on top of the BERT output sequence. It uses the BIO or BILOU annotation system to label sequences such as "risk objects (e.g., personnel / equipment / components)," "risk actions (e.g., climbing / illegal electrification)," and "risk conditions (e.g., high temperature / oxygen deficiency / excessive dust)." During the training phase, variable-length batch processing and Viterbi decoding are employed, with a negative log-likelihood (NLL) loss function. Adversarial training (FGM) is introduced on engineering-related few-sample labels to improve the model's robustness to misspellings and synonym substitutions. For the labeled entities and relational fragments output by the CRF, the system further performs keyword weight calculation based on TF-IDF: TF is obtained by standardizing the frequency of entity occurrences in the same log, and IDF is obtained by statistically analyzing historical corpora across projects and work sites. To avoid "common but non-risk" words crowding out weights, the implementation introduces a stop list from the engineering safety dictionary and performs word-form merging on synonymous risk phrases (e.g., "unprotected edges" and "missing edge protection") to form a sparse, interpretable weight ranking vector. Finally, the system performs attribute annotation on high-weight risk phrases based on weight ranking. The attribute fields include: risk category (person, machine, material, method, environment), frequency of occurrence (normalized by shift / work site), semantic context position (relative sentence index and trigger word window), triplet of involved objects and actions, and an alignable identifier with the site spatiotemporal stamp. For example, the log "East side scaffolding swaying, working layer not wearing safety belts, welding slag cleaning delayed" is identified by BERT+CRF as the entity <scaffolding / component> <swaying / state> <not wearing safety belts / behavior> <welding slag / material> <delayed cleaning / action>. After TF-IDF weighting, "not wearing safety belts" and "scaffolding swaying" receive higher scores. The attribute annotation classifies them into "personnel protection deficiency / high risk" and "abnormal edge support / medium risk" respectively and records the sentence index so as to align with video behavior and sensor fluctuations in subsequent multimodal fusion.

[0030] Further, the step of establishing an abnormal node set based on the multimodal abnormal feature set using a clustering algorithm, constructing an abnormal node feature vector based on node attributes, and constructing a weighted adjacency matrix based on a nearest neighbor retrieval algorithm includes the following steps: Based on the personnel status data, environmental change feature data, and text risk feature data in the multimodal anomaly feature set, the DBSCAN algorithm is used to cluster the same time slices and adjacent spatial regions to obtain a set of anomaly nodes containing spatial centers, behavioral attributes, and risk attributes. Based on the set of abnormal nodes, the spatial coordinates, behavioral attributes and risk semantic attributes of each abnormal node are vectorized and encoded to obtain the corresponding abnormal node feature vector. The KD-Tree nearest neighbor retrieval algorithm is used to calculate the adjacency relationship of the spatial locations of nodes in the abnormal node set, and spatial adjacency data containing the spatial distance between nodes is obtained. Based on the spatial adjacency data and the personnel status data and environmental change characteristic data corresponding to the abnormal nodes, the spatiotemporal weight data between nodes are jointly calculated using the Gaussian radial basis function and the dynamic time warping algorithm. Based on the spatiotemporal weighted data and the text risk feature data corresponding to the abnormal nodes, the text risk feature vectors are compared using the cosine similarity algorithm to generate a weighted adjacency matrix.

[0031] In some embodiments, the local distribution characteristics of personnel status data, environmental change feature data, and text risk feature data are first analyzed within the same time slice, and the DBSCAN density clustering algorithm is used to cluster abnormal patterns appearing in adjacent spatial regions. DBSCAN evaluates the density distribution of each feature point within its ε-neighborhood, grouping high-density areas into anomaly clusters while excluding noise points. This feature is suitable for scenarios with irregular anomaly distributions, non-convex cluster shapes, and frequent noise at construction sites. For example, when multiple construction workers illegally climb on the same floor, corresponding sensors detect a slow temperature rise, and the log shows a "insufficient protection in the isolation zone" description, DBSCAN can group them into the same anomaly node based on a density threshold, thus obtaining a set of anomaly nodes containing spatial center, behavioral attributes, and risk semantic attributes. After obtaining the set of anomaly nodes, the system vectorizes the basic attributes of each node. Spatial coordinates utilize (x, y, z) positional information within a three-dimensional component reference frame. Behavioral attributes are achieved through one-hot or distributed encoding of skeletal angles, limb velocity gradients, and action categories from posture features. Risk semantic attributes are formed by compressing keyword vectors extracted from the supervisor's log text using pre-trained word vectors (such as Word2Vec or BERT token-level embedding) to create fixed-dimensional semantic features. This multi-dimensional fusion allows the abnormal node feature vector to express the node's spatial localization, behavioral risk orientation, and semantically implicit risk factors, achieving a semantic-behavioral-environment ternary fusion representation. Subsequently, the KD-Tree nearest neighbor retrieval algorithm is used to construct a fast spatial index within the abnormal node set, efficiently retrieving the three-dimensional Euclidean distance between nodes to obtain spatial adjacency data containing the spatial distance between nodes. KD-Tree significantly improves the efficiency of high-dimensional spatial nearest neighbor queries by constructing a binary spatial partitioning structure, enabling rapid identification of neighborhood relationships even with a large number of nodes. For example, if a node on an adjacent floor is far from another node, it will not be included in a strongly correlated adjacency relationship. A preliminary physical adjacency framework between nodes is established using spatial adjacency data. After obtaining the spatial adjacency data, spatiotemporal weights are further calculated based on the personnel status data and environmental change characteristic data corresponding to the nodes. First, the distance decay model between nodes is performed using the Gaussian radial basis function (RBF), with the connection weight between more distant nodes decreasing exponentially, thus simulating the limitations of spatial risk propagation. Second, the Dynamic Time Warping (DTW) algorithm is introduced to compare the sequence similarity between the node behavior and environmental change curves. DTW aligns the temporal trend through elastic deformation of the time axis, capturing coupling relationships with time displacement, such as "abnormal personnel behavior occurs first, while environmental abnormalities appear later." The combination of these two methods constitutes the spatiotemporal weight data between nodes, achieving a joint characterization of the time-borne and spatial diffusion of risk.Finally, to incorporate risk semantic similarity into the risk propagation graph, cosine similarity is calculated based on the text risk feature vectors corresponding to abnormal nodes. This compares whether the risk description directions between nodes are consistent or have similar contexts. For example, when two different regional logs both show the "loose guardrail" message, the semantic similarity will be significantly improved. The system integrates spatiotemporal weighted data with semantic similarity through a weighted summation method to generate a final weighted adjacency matrix, thereby establishing a complex risk association graph structure that can simultaneously characterize spatial adjacency, temporal coupling, and semantic consistency. Furthermore, performing matrix multiplication and addition operations and normalization calculations on the weighted adjacency matrix includes the following steps: Based on the weighted adjacency matrix, the feature vectors of abnormal nodes are processed by multiplication and addition to obtain initial risk score data containing the interaction effects of multiple nodes; The initial risk score data is subjected to amplitude scaling based on the minimum-maximum normalization algorithm to obtain standardized risk score data. The probability distribution is normalized using the Softmax function to obtain the risk assessment results.

[0032] In some embodiments, matrix multiplication and addition operations are first performed to simulate the interactive propagation impact of anomalies between nodes. Specifically, the system scans the weighted adjacency matrix row by row, using the spatiotemporal weights between nodes as propagation coefficients, applying them to the corresponding anomaly node feature values ​​and performing linear superposition to generate initial risk score data containing multi-hop propagation effects. To avoid excessive risk diffusion during propagation, the system sets self-loop reinforcement at diagonal positions during the calculation process, ensuring that each node retains a certain proportion of its own state weight when calculating its own risk, thus reflecting the stability of "in-situ risk". In addition, the system adopts a residual injection strategy, that is, after each round of propagation superposition, a portion of the baseline risk value is added, so that the node risk value is not completely diluted during deep propagation. After obtaining the initial risk score data, since the risk output dimensions differ between different construction areas and different time batches, direct comparison will lead to attribution bias. Therefore, the system performs amplitude scaling processing on the risk scores. The amplitude scaling process comprises three core steps: First, the highest and lowest scores in the current batch of data are identified and used as boundaries; second, all node scores are mapped proportionally to a unified interval to mitigate bias caused by extreme distributions; and finally, quantile pruning is used to suppress the pull of a few extreme outliers on the overall distribution, making the risk assessment closer to the real construction environment. After this step, the risk scores of each node are normalized to a directly comparable scale, providing a consistent foundation for subsequent distribution modeling. After amplitude scaling, the standardized risk scores undergo probability distribution normalization. This step treats the risk values ​​of all nodes as competing weights in the same event set and maps them to a probability form, so that nodes with higher risks within the same hazard category have higher probability distribution weights. This probabilistic processing can be adjusted by setting the distribution "sharpness" parameter according to business needs, thereby controlling the prominence of risk differences: if the construction scenario has low tolerance for high-risk actions, the parameter setting tends to be "sharp" to strengthen the priority warning level of high-risk nodes; if the scenario focuses on broad-coverage warnings, the setting tends to be "smooth" to increase the alert coefficient of potential hazards.

[0033] Furthermore, the abnormality levels include no abnormality, preventative abnormality, and emergency abnormality.

[0034] Specifically, when the risk probability of a node is in the low confidence interval, and its personnel status, environmental parameters, and text semantics do not show significant deviations, the system marks the node as having no anomalies, indicating that the construction status is under safe operating conditions and requires no additional intervention. Secondly, when the node's risk probability is in the medium confidence interval and exhibits a continuous but slow shift in its time-series trend (e.g., slightly non-standard personnel actions, a slow increase in ambient temperature, or ambiguous risk warnings in logs), such risks often have potential for evolution. These are classified as preventative anomalies, and alarms are used to prompt supervisors to take necessary risk mitigation measures, preventing further accumulation of hidden dangers from the source. Finally, when the node's risk probability is in the high confidence interval and exhibits strong consistency across modal characteristics (e.g., obvious violations by personnel, a sharp deterioration in environmental parameters, or clear hazard expressions in logs), this combined pattern indicates that the risk may have reached the triggering conditions. The system classifies this as an emergency anomaly and triggers a mandatory alarm mechanism to promote immediate response and prevent safety accidents.

[0035] This invention also includes an AI-based system for tracking anomalies in supervisory work, applied to any of the preceding AI-based methods for tracking anomalies in supervisory work, characterized in that it includes: The data acquisition module is used to acquire comprehensive supervision data from the construction site, including video surveillance data, environmental sensor data, and supervision log text data; and to perform noise reduction, normalization, and time synchronization processing on the comprehensive supervision data to obtain a structured input dataset. The data feature processing module is used to perform target detection and behavior recognition processing on the video surveillance data in the structured input dataset to obtain personnel status data; to perform time-series feature extraction processing on the environmental sensor data to obtain environmental change feature data; and to perform semantic recognition and keyword extraction processing on the supervision log text data to obtain text risk feature data. The feature integration module is used to integrate personnel status data, environmental change feature data, and text risk feature data to obtain a multimodal anomaly feature set; The adjacency matrix construction module is used to establish an abnormal node set based on the multimodal abnormal feature set through a clustering algorithm, construct an abnormal node feature vector based on node attributes, and construct a weighted adjacency matrix based on a nearest neighbor retrieval algorithm. The risk assessment module is used to perform matrix multiplication and addition operations and normalization processing based on the weighted adjacency matrix and the feature vector of abnormal nodes to obtain the node risk weight value and the corresponding abnormal event risk assessment result. The tracking and alarm module is used to classify the risk assessment results into anomaly levels and generate anomaly tracking alarms based on the anomaly levels.

[0036] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for tracking anomalies in supervision work based on artificial intelligence, characterized in that, Includes the following steps: Acquire comprehensive supervision data from the construction site, including video surveillance data, environmental sensor data, and supervision log text data; perform denoising, normalization, and time synchronization processing on the comprehensive supervision data to obtain a structured input dataset; The video surveillance data in the structured input dataset is processed by target detection and behavior recognition to obtain personnel status data; the environmental sensor data is processed by time-series feature extraction to obtain environmental change feature data; and the supervision log text data is processed by semantic recognition and keyword extraction to obtain text risk feature data. By integrating personnel status data, environmental change characteristic data, and text risk characteristic data, a multimodal anomaly feature set is obtained; Based on the multimodal anomaly feature set, an anomaly node set is established through a clustering algorithm, an anomaly node feature vector is constructed based on node attributes, and a weighted adjacency matrix is ​​constructed based on a nearest neighbor retrieval algorithm. Based on the weighted adjacency matrix and the feature vector of the abnormal node, matrix multiplication and addition operations and normalization are performed to obtain the node risk weight value and the corresponding abnormal event risk assessment result. The risk assessment results are classified into anomaly levels, and anomaly tracking alarms are generated based on the anomaly levels.

2. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 1, characterized in that, The process of obtaining comprehensive supervision data from the construction site includes the following steps: Video stream data is collected by fixed cameras deployed at the construction site and frame extraction processing is performed to obtain a continuous video frame sequence. Environmental monitoring data is obtained by collecting temperature, humidity, gas concentration and noise signals through environmental sensors installed at the construction site. The supervisor's log text data is obtained by retrieving the supervisor's written records through the electronic log system.

3. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 1, characterized in that, The denoising, normalization, and time synchronization processing of the comprehensive supervision data includes: Perform frame mean filtering and edge-preserving enhancement processing on video surveillance data; Perform sliding window smoothing and extreme value truncation on environmental sensor data; Perform character cleaning and format standardization processing on the text data of the supervision log; Based on timestamp information and spatial coordinates, video frames, sensor data, and text records are time-synchronized and spatially registered to obtain a structured input dataset.

4. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 1, characterized in that, The process of performing target detection and behavior recognition on the video surveillance data in the structured input dataset to obtain personnel status data includes the following steps: The video surveillance data in the structured input dataset is processed by a target detection model based on the YOLO algorithm to locate people, resulting in people detection data containing location coordinates and bounding box information; Based on the personnel detection data, the OpenPose algorithm is used to extract key points in the personnel area to obtain personnel pose data. Based on the personnel posture data, behavior classification and recognition are performed using an action classification and matching model to obtain personnel status data.

5. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 1, characterized in that, The step of extracting time-series features from environmental sensor data to obtain environmental change feature data includes the following steps: The environmental sensor data in the structured input dataset is subjected to time series fitting processing based on an autoregressive moving average model to obtain trend change data of environmental parameters; Based on the trend change data, outlier detection processing is performed using the local outlier factor algorithm to obtain outlier deviation data of environmental parameters. Based on the outlier deviation data, volatility analysis is performed using an exponentially weighted moving average to obtain the risk volatility data of environmental parameters; Based on the risk fluctuation data, the abnormal amplitude and frequency of environmental parameters are labeled with attributes to obtain environmental change characteristic data including the direction of change, fluctuation intensity and abnormal cycle.

6. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 1, characterized in that, The semantic recognition and keyword extraction processing of the supervision log text data includes the following steps: The supervision log text data is semantically encoded based on a pre-trained BERT language model to obtain text embedding data containing context semantic vectors; Based on the text embedding data, named entity recognition is performed using a conditional random field to obtain semantic recognition data containing risk entity tags; By calculating keyword weights using semantic recognition data based on the TF-IDF algorithm, the weight ranking data of each risk phrase in the text is obtained; Based on the weighted ranking data, attribute annotation processing is performed on high-weight risk phrases to obtain text risk feature data containing risk category, frequency of occurrence, and semantic context position.

7. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 1, characterized in that, The steps of establishing an abnormal node set based on the multimodal abnormal feature set using a clustering algorithm, constructing an abnormal node feature vector based on node attributes, and constructing a weighted adjacency matrix based on a nearest neighbor retrieval algorithm include the following: Based on the personnel status data, environmental change feature data, and text risk feature data in the multimodal anomaly feature set, the DBSCAN algorithm is used to cluster the same time slices and adjacent spatial regions to obtain a set of anomaly nodes containing spatial centers, behavioral attributes, and risk attributes. Based on the set of abnormal nodes, the spatial coordinates, behavioral attributes and risk semantic attributes of each abnormal node are vectorized and encoded to obtain the corresponding abnormal node feature vector. The KD-Tree nearest neighbor retrieval algorithm is used to calculate the adjacency relationship of the spatial locations of nodes in the abnormal node set, and spatial adjacency data containing the spatial distance between nodes is obtained. Based on the spatial adjacency data and the personnel status data and environmental change characteristic data corresponding to the abnormal nodes, the spatiotemporal weight data between nodes are jointly calculated using the Gaussian radial basis function and the dynamic time warping algorithm. Based on the spatiotemporal weighted data and the text risk feature data corresponding to the abnormal nodes, the text risk feature vectors are compared using the cosine similarity algorithm to generate a weighted adjacency matrix.

8. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 7, characterized in that, Performing matrix multiplication and addition operations and normalization calculations on the weighted adjacency matrix includes the following steps: Based on the weighted adjacency matrix, the feature vectors of abnormal nodes are processed by multiplication and addition to obtain initial risk score data containing the interaction effects of multiple nodes; The initial risk score data is subjected to amplitude scaling based on the minimum-maximum normalization algorithm to obtain standardized risk score data. The probability distribution is normalized using the Softmax function to obtain the risk assessment results.

9. The method for tracking anomalies in supervision work based on artificial intelligence according to claim 7, characterized in that, The abnormality levels include no abnormality, preventative abnormality, and emergency abnormality.

10. An AI-based system for tracking anomalies in supervisory work, applied to the AI-based method for tracking anomalies in supervisory work according to any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire comprehensive supervision data from the construction site, including video surveillance data, environmental sensor data, and supervision log text data. The comprehensive supervision data is subjected to denoising, normalization, and time synchronization processing to obtain a structured input dataset; The data feature processing module is used to perform target detection and behavior recognition processing on the video surveillance data in the structured input dataset to obtain personnel status data; to perform time-series feature extraction processing on the environmental sensor data to obtain environmental change feature data; and to perform semantic recognition and keyword extraction processing on the supervision log text data to obtain text risk feature data. The feature integration module is used to integrate personnel status data, environmental change feature data, and text risk feature data to obtain a multimodal anomaly feature set; The adjacency matrix construction module is used to establish an abnormal node set based on the multimodal abnormal feature set through a clustering algorithm, construct an abnormal node feature vector based on node attributes, and construct a weighted adjacency matrix based on a nearest neighbor retrieval algorithm. The risk assessment module is used to perform matrix multiplication and addition operations and normalization processing based on the weighted adjacency matrix and the feature vector of abnormal nodes to obtain the node risk weight value and the corresponding abnormal event risk assessment result. The tracking and alarm module is used to classify the risk assessment results into anomaly levels and generate anomaly tracking alarms based on the anomaly levels.

Images