Field operation supervision method and system based on image intelligent identification, and electronic equipment

Abstract

The invention discloses a field operation supervision method and system based on image intelligent identification and electronic equipment. The method comprises the following steps: acquiring field operation image data based on image acquisition equipment fusing visible light and infrared light; generating a foreground target image set based on the field operation image data; constructing a personnel action semantic graph and an environment semantic graph based on the foreground target image set, and generating a semantic behavior tag of the operator based on the personnel action semantic graph and the environment semantic graph; obtaining a track sequence of the operator labeled by the semantic behavior label, and determining a process deviation index and an abnormal process identification result based on the track sequence; and performing hierarchical supervision according to the process deviation index and the abnormal process identification result. According to the invention, a traceable and quantifiable safety data system is constructed, and the real-time supervision capability and the safety prevention and control level of a construction site can be comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of on-site operation supervision technology, and more specifically, to an on-site operation supervision method, system, and electronic device based on image intelligent recognition. Background Technology

[0002] With the continuous expansion of on-site operations in construction engineering, energy operation and maintenance, and infrastructure construction, on-site safety supervision and behavior management have become crucial aspects affecting project quality and safe production. Existing on-site supervision methods primarily rely on manual inspections, fixed-point video surveillance, and post-event video playback analysis. These methods have the following shortcomings when facing construction sites with numerous personnel, complex environments, and frequent changes in lighting: Traditional video surveillance often uses fixed cameras and manual review, failing to achieve real-time dynamic identification of multiple targets and areas in complex construction scenarios, making it difficult to detect abnormal behaviors and violations in a timely manner; existing behavior recognition methods are mostly based on single-image or single-modal information, lacking causal relationship modeling between personnel actions and environmental semantics, easily leading to biased behavior judgments; existing systems lack dynamic trajectory clustering analysis mechanisms for asynchronous multi-process behaviors, failing to promptly detect potential process deviations or abnormal behaviors.

[0003] Therefore, a method for on-site operation supervision based on image intelligent recognition is needed. Summary of the Invention

[0004] This invention proposes a method, system, and electronic device for on-site operation supervision based on image intelligent recognition, in order to solve the problem of how to efficiently and accurately supervise on-site operations.

[0005] To address the aforementioned problems, according to one aspect of the present invention, a method for on-site operation supervision based on image intelligent recognition is provided, the method comprising:

[0006] Image data of on-site operations is acquired using image acquisition equipment that integrates visible and infrared light;

[0007] Based on the on-site operation image data, a set of foreground target images is generated;

[0008] Based on the foreground target image set, construct a personnel action semantic map and an environment semantic map, and based on the personnel action semantic map and the environment semantic map, generate semantic behavior labels for the workers;

[0009] Obtain the trajectory sequence of the workers labeled with semantic behavior tags, and determine the process deviation index and abnormal process identification result based on the trajectory sequence;

[0010] Based on the process deviation index and the results of abnormal process identification, hierarchical supervision is carried out.

[0011] Preferably, the on-site operation image data includes: images of the head and limb postures of the workers, images of the wearing status of reflective vests and safety helmets, images of the tools and materials carried, and images of the operating status of construction machinery and equipment and environmental safety signs in the work area.

[0012] Preferably, the process of generating a foreground target image set based on the on-site operation image data includes:

[0013] The on-site operation image data is constructed as a multi-layer graph structure represented by nodes; wherein, the nodes of the multi-layer graph structure include: personnel area nodes, mechanical equipment nodes and environmental background nodes, and a topology-preserving transformation operator is introduced into the multi-layer graph structure to maintain the boundary connectivity of the construction area;

[0014] Define a graph potential field constraint energy function on the multi-layer graph structure;

[0015] A time-series decay parameter and a perturbation adjustment kernel are introduced into the graphical potential field constraint energy function to form a dynamic graphical potential field constraint model;

[0016] The graph potential field constraint model is used to execute the graph potential field constraint segmentation algorithm to output a set of foreground target images; wherein, the set of foreground target images includes: image regions of workers, image regions of construction machinery and image regions of safety signs with topological labels.

[0017] Preferably, the construction of a person action semantic map and an environment semantic map based on the foreground target image set includes:

[0018] The worker region in the foreground target image set is constructed as a multi-channel motion node graph; wherein, the nodes of the motion node graph include: skeletal topology nodes, joint mechanics nodes and tool interaction nodes, and dynamic edges driven by the Euler angle change rate are introduced between the nodes;

[0019] The construction environment area in the foreground target image set is constructed as a multi-dimensional environment node graph; wherein, the nodes of the multi-dimensional environment node graph include: mechanical equipment operation status nodes, safety sign location nodes, and hazard source distribution nodes, and energy edges modeled by thermal field distribution functions are established between the nodes.

[0020] Preferably, the semantic behavior tags of the workers are generated based on the personnel action semantic map and the environment semantic map, including:

[0021] A causal reasoning-driven cross-graph mapping edge is introduced between the action node graph and the environment node graph. Based on the causal alignment mechanism of graph homeomorphism constraint, the key actions of the operators and environmental risk events are matched in time sequence to generate a cross-domain causal semantic alignment graph.

[0022] The cross-domain causal semantic alignment graph is input into a causal graph regularized multi-level contrastive learning network to generate semantic behavior labels for workers.

[0023] Preferably, the causal alignment mechanism based on graph homeomorphism constraints pairs key actions of workers with environmental risk events in a temporal sequence to generate a cross-domain causal semantic alignment graph, including:

[0024] The node time information in the operator action semantic graph and the environment semantic graph is standardized and synchronized, and time-series slicing is performed to extract candidate event sequences with time stamps.

[0025] Based on the candidate event sequence, a cross-graph mapping method with graph structure homeomorphism constraint is adopted to establish a cross-domain correspondence between the action node graph and the environment node graph, and to construct an initial cross-domain causal mapping graph.

[0026] A weighted causal edge set is formed by introducing an edge weight calculation mechanism based on the causal strength function for the node pairs in the cross-domain causal mapping graph.

[0027] Based on the weighted causal edge set, node semantic attributes, graph structure topology information, and temporal dynamic features are integrated to generate a cross-domain causal semantic alignment graph.

[0028] Preferably, the causal strength function includes:

[0029]

[0030] Among them, I c (v i ,v j ) represents the worker's action node v i With environment node v j The causal strength value between them; Δt ij This represents the time difference between personnel action nodes and environmental event nodes; f represents the time-series decay scale parameter; ij Indicates the number of times a worker's actions and a specific environmental event co-occur in adjacent frame sequences; F i With F j They represent nodes v respectively i With node v j Total number of occurrences in the global sequence; d ij Represents node v i With node v j Spatial distance between them; σ d Represents the spatial attenuation parameter; γ ij α represents the semantic consistency weight coefficient; α, β1, and β2 represent global adjustment coefficients used to balance the contribution weights of time, frequency, and spatial factors.

[0031] Preferably, determining the process deviation index and abnormal process identification result based on the trajectory sequence includes:

[0032] The trajectory sequence of workers labeled with semantic behavior tags is constructed into a temporal event graph. The nodes of the temporal event graph include action event nodes, environmental interaction nodes, and safety status nodes. The edges between nodes represent temporal continuity and causal relationship.

[0033] Based on the aforementioned time-series event graph, an asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model the trajectories of multiple personnel. An event intensity function and a background noise term are introduced in the time domain to decouple and cluster the action patterns of different workers.

[0034] In the trajectory clustering results, the process deviation index is calculated based on the temporal distribution and spatial density characteristics of each cluster center;

[0035] Based on the process deviation index and clustering labels, anomaly process identification results are generated using the Bayesian belief propagation mechanism; wherein, the anomaly process identification results include deviation type, deviation magnitude and impact level.

[0036] Preferably, the formula for the asynchronous trajectory clustering algorithm is as follows:

[0037]

[0038] Among them, Λ k (τ) represents the intensity value of the trajectory event of the k-th type of worker at time τ; η k λ represents the normalized intensity coefficient of the k-th type of trajectory; k τ represents the time decay rate parameter of the k-th type of trajectory; 0k δ represents the initial time point of the k-th type of trajectory; k P represents the background noise term of the k-th type trajectory; k ω represents the total number of behavioral events contained in the k-th type of trajectory; kp The semantic weight coefficient of the p-th behavioral event in the k-th trajectory is represented by r*τ; the spatial coordinate position vector of the worker at time τ is represented by r*kp; the spatial center position of the p-th behavioral event in the k-th trajectory is represented by r*kp. The spatial scale parameter represents the k-th type of trajectory.

[0039] Preferably, the hierarchical supervision based on the process deviation index and the abnormal process identification results includes:

[0040] Based on the anomaly identification results, risk values ​​of different dimensions are calculated for the behavioral trajectory of each worker within the current time window;

[0041] The risk values ​​from different dimensions are weighted and summed to determine the real-time risk level label;

[0042] Regulatory decision instructions are determined based on the real-time risk level labels, and supervision is carried out based on the regulatory decision instructions.

[0043] According to another aspect of the present invention, an on-site operation monitoring system based on image intelligent recognition is provided, the system comprising:

[0044] The data acquisition unit is used to acquire on-site operation image data based on image acquisition equipment that fuses visible light and infrared light;

[0045] An image processing unit is used to generate a set of foreground target images based on the on-site operation image data;

[0046] The semantic behavior label determination unit is used to construct a personnel action semantic map and an environment semantic map based on the foreground target image set, and to generate semantic behavior labels for the workers based on the personnel action semantic map and the environment semantic map;

[0047] The identification unit is used to acquire the trajectory sequence of the workers marked with semantic behavior tags, and to determine the process deviation index and abnormal process identification result based on the trajectory sequence;

[0048] The monitoring unit is used to conduct hierarchical monitoring based on the process deviation index and the abnormal process identification results.

[0049] According to another aspect of the present invention, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the steps of a field operation monitoring method based on image intelligent recognition.

[0050] According to another aspect of the present invention, the present invention provides an electronic device, comprising:

[0051] The aforementioned computer-readable storage medium; and

[0052] One or more processors for executing a program in the computer-readable storage medium.

[0053] This invention provides a method, system, and electronic device for on-site operation supervision based on image intelligent recognition. By integrating visible light and infrared image acquisition equipment, it achieves stable image acquisition under day / night, strong light, and occlusion environments, significantly improving the integrity and adaptability of on-site image acquisition. A graph potential field constraint segmentation algorithm based on topology-preserving transformation is employed to maintain the topological continuity of the construction area while suppressing illumination changes and occlusion interference, improving the accuracy and consistency of target segmentation. By constructing personnel action semantic graphs and environmental semantic graphs, and introducing a multi-level contrastive learning network with causal graph regularization, adaptive alignment of cross-graph information is achieved, effectively capturing the causal relationship between worker behavior and environmental state. An asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model and dynamically cluster personnel trajectories, accurately identifying process deviations and abnormal work behaviors in asynchronous scenarios. A hierarchical supervision strategy driven by a fuzzy inference mechanism, combined with process deviation and anomaly identification results, enables adaptive adjustment of early warning prompts and mandatory control, forming a closed-loop management system from risk discovery to safety response. By writing the monitoring results into the safety database in real time and synchronizing them to the monitoring terminal, a traceable and quantifiable safety data system is built, which comprehensively improves the real-time monitoring capabilities and safety control level of the construction site. Attached Figure Description

[0054] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures:

[0055] Figure 1 A flowchart of an image-based intelligent recognition-based on-site operation monitoring method 100 according to an embodiment of the present invention;

[0056] Figure 2 A flowchart illustrating the generation of semantic behavior tags for workers according to an embodiment of the present invention;

[0057] Figure 3 This is a flowchart illustrating the determination of abnormal process identification results according to an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the structure of an image-based intelligent recognition-based on-site operation monitoring system 400 according to an embodiment of the present invention. Detailed Implementation

[0059] Exemplary embodiments of the invention will now be described with reference to the accompanying drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. These embodiments are provided to fully and completely disclose the invention and to fully convey its scope to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the drawings is not intended to limit the invention. In the drawings, the same units / elements are referred to by the same reference numerals.

[0060] Unless otherwise stated, the terms used herein (including technical terms) have their common meaning as understood by one of ordinary skill in the art. Furthermore, it is understood that terms defined in commonly used dictionaries should be understood to have a meaning consistent with the context of their relevant field, and not to be interpreted as having an idealized or overly formal meaning.

[0061] Figure 1 This is a flowchart of an image-based intelligent recognition-based on-site operation monitoring method 100 according to an embodiment of the present invention. Figure 1 As shown, the on-site operation supervision method based on image intelligent recognition provided by this invention achieves stable image acquisition under day and night, strong light, and occlusion environments by integrating visible light and infrared image acquisition devices, significantly improving the integrity and adaptability of on-site image acquisition. A graph potential field constraint segmentation algorithm based on topology-preserving transformation is adopted to maintain the continuity of the topological structure of the construction area while suppressing illumination changes and occlusion interference, improving the accuracy and consistency of target segmentation. By constructing a semantic graph of personnel actions and an semantic graph of the environment, and introducing a multi-level contrastive learning network with causal graph regularization, adaptive alignment of cross-graph information is achieved, effectively capturing the causal relationship between personnel behavior and environmental state. An asynchronous trajectory clustering algorithm driven by a Poisson mixed process is used to model and dynamically cluster personnel trajectories, accurately identifying process deviations and abnormal operation behaviors in asynchronous scenarios. A hierarchical supervision strategy driven by a fuzzy inference mechanism, combined with process deviation and anomaly identification results, achieves adaptive adjustment of early warning prompts and mandatory control, forming a closed-loop management from risk discovery to safety response. The monitoring results are written into the safety database in real time and synchronized to the monitoring terminal to build a traceable and quantifiable safety data system, thereby comprehensively improving the real-time monitoring capabilities and safety control level of the construction site. The on-site operation monitoring method 100 based on image intelligent recognition provided by the embodiments of the present invention starts from step 101. In step 101, on-site operation image data is acquired based on an image acquisition device that integrates visible light and infrared light.

[0062] Preferably, the on-site operation image data includes: images of the head and limb postures of the workers, images of the wearing status of reflective vests and safety helmets, images of the tools and materials carried, and images of the operating status of construction machinery and equipment and environmental safety signs in the work area.

[0063] In this embodiment, an image acquisition device that integrates visible light and infrared light is used to acquire on-site operation image data. The on-site operation image data includes images of the head and limb postures of the workers, images of the wearing status of reflective vests and safety helmets, images of the tools and materials carried, and images of the operating status of construction machinery and equipment and environmental safety signs in the work area.

[0064] Specifically, the acquisition of on-site operational image data is accomplished by deploying multimodal image acquisition equipment that integrates visible light and infrared. This equipment is typically installed at key locations on the construction site, such as entrances and exits, main work areas, high-altitude work areas, and areas surrounding hazardous equipment, to achieve full-coverage dynamic monitoring. Each acquisition terminal includes a visible light image sensing module, an infrared thermal imaging module, a depth sensing module, and an ambient light sensor, enabling it to operate continuously and stably under different lighting and weather conditions.

[0065] 1. Visible Light Image Acquisition Section: Employing a high-resolution CMOS sensor module with a frame rate exceeding 60fps, this section captures clear images of workers' appearance, posture, and the color characteristics of their protective equipment. Automatic exposure and white balance adjustment algorithms ensure high-quality images are obtained even in strong light, shadow, and nighttime conditions. This module is primarily used to identify personnel characteristics, the color mode of reflective vests, the wearing status of safety helmets, and the appearance of tools and construction materials.

[0066] 2. Infrared Thermal Imaging Acquisition Section: The infrared thermal imaging module monitors the surface temperature distribution of objects, enabling the identification of personnel and equipment outlines in environments with insufficient light or severe dust interference. For nighttime or tunnel operations, infrared thermal images can effectively supplement visible light information, detecting high-temperature machinery, welding areas, or potential electrical overheating hazards. This module can also assist in detecting the thermal reflection characteristics of reflective vest materials, improving the accuracy of personnel identification.

[0067] 3. Synchronization and Calibration Mechanism: To achieve spatial and temporal consistency of multimodal images, the system is equipped with a clock synchronization module and a calibration algorithm unit at the device level. Each module acquires images under the control of a unified clock signal and achieves frame-level alignment through an internal synchronization trigger mechanism. Spatial calibration is achieved through feature point matching and perspective transformation, enabling the overlay and fusion of infrared and visible light images at the pixel level, providing accurate input for subsequent potential field constraint segmentation and semantic recognition.

[0068] 4. Content and Data Hierarchy

[0069] Images of head and limb postures of personnel: used to detect whether personnel's movements are standardized, such as whether there are characteristic postures such as climbing, bending over, or squatting.

[0070] Safety protective equipment wearing status image: including the wearing status of safety helmets, reflective vests, protective gloves, etc., used to determine whether they meet safety requirements.

[0071] Image recognition of tools and materials: This function detects tools such as electric drills, welding guns, hooks, and measuring rulers to confirm whether the personnel's work behavior matches their authorized job type.

[0072] Mechanical equipment operation status images: These images detect whether the machinery is in a start-up, stop, or abnormal state by changing continuous frames, and are used to determine the safety of equipment operation.

[0073] Environmental safety signage images: capture visual symbols such as on-site warning signs, no-entry signs, and indicator lights for use in subsequent semantic graph construction of environmental semantic nodes.

[0074] Each data acquisition terminal performs preliminary compression and feature summary extraction through its local edge computing unit, reducing transmission bandwidth consumption. All acquired data is tagged with timestamps, device numbers, and spatial location information, and is packaged and stored in the edge server cache in JSON format. The system maintains low-latency data synchronization with the central monitoring platform via 5G or Wi-Fi 6 communication links, achieving second-level image reporting and local backup redundancy to prevent network outages or frame drops.

[0075] When the system detects changes in lighting, smoke interference, or lens obstruction, it automatically switches to the infrared main channel mode and adjusts the exposure parameters and gain. If a cluster of people or abnormal mechanical vibration is detected in consecutive frames, a high-frequency sampling mode is triggered, increasing the sampling frequency from 30fps to 120fps to ensure complete capture of behavioral details. The system also has an automatic diagnostic function, providing real-time correction and alarms for lens contamination, focus deviation, temperature drift, and other conditions.

[0076] In step 102, a set of foreground target images is generated based on the on-site operation image data.

[0077] Preferably, the process of generating a foreground target image set based on the on-site operation image data includes:

[0078] The on-site operation image data is constructed as a multi-layer graph structure represented by nodes; wherein, the nodes of the multi-layer graph structure include: personnel area nodes, mechanical equipment nodes and environmental background nodes, and a topology-preserving transformation operator is introduced into the multi-layer graph structure to maintain the boundary connectivity of the construction area;

[0079] Define a graph potential field constraint energy function on the multi-layer graph structure;

[0080] A time-series decay parameter and a perturbation adjustment kernel are introduced into the graphical potential field constraint energy function to form a dynamic graphical potential field constraint model;

[0081] The graph potential field constraint model is used to execute the graph potential field constraint segmentation algorithm to output a set of foreground target images; wherein, the set of foreground target images includes: image regions of workers, image regions of construction machinery and image regions of safety signs with topological labels.

[0082] In this invention, the on-site operation image data is preprocessed, and a graph potential field constraint segmentation algorithm based on topology preservation transformation is adopted to suppress occlusion and illumination interference while maintaining the topological structure of the construction site area, thereby generating a set of foreground target images.

[0083] In this embodiment, the step of generating a set of foreground target images includes:

[0084] (1) The on-site operation image data is constructed into a multi-layer graph structure represented by nodes, including personnel area nodes, mechanical equipment nodes and environmental background nodes, and a topology preservation transformation operator is introduced into the graph structure to maintain the boundary connectivity of the construction area.

[0085] Specifically, after the image acquisition module completes the acquisition of raw data, the system first performs structured modeling of the image.

[0086] Each frame of the on-site operation image is divided into several local region blocks, and each region block is regarded as a node. Based on the semantic type of the nodes, they are divided into three categories:

[0087] Personnel area nodes: correspond to key posture areas of the human body, such as head, shoulders, upper limbs and lower limbs;

[0088] Mechanical equipment nodes: These correspond to areas with rigid structures, such as robotic arms, hoisting equipment, and transport vehicles.

[0089] Environmental background nodes: correspond to static background areas such as construction sites, fences, ground, and walls.

[0090] Multiple layers of connecting edges are established between nodes based on spatial adjacency, forming a multi-layer graph structure containing multiple semantic channels.

[0091] The system introduces a topology-preserving transform operator into the graph structure to ensure that the boundaries between different semantic layers maintain connectivity during the transform process. For example, at the boundary between the human region and the background region, the transform operator constrains the adjacency weights of edge pixels, ensuring that the segmented human contour remains continuous and smooth, without being cut off by noise.

[0092] (2) Define a graph potential field constraint energy function on the multi-layer graph structure. The energy function integrates the node gray gradient, boundary curvature features and adjacency weight to realize the energy distribution modeling of local areas of the construction site.

[0093] Specifically, after constructing a multi-layer graph structure, the system further defines a graph potential field constraint model to describe the spatial and visual feature distribution of each node in the image.

[0094] During this stage, the system extracts multiple feature information for each node:

[0095] Gray-scale gradient features: used to reflect local brightness changes and edge direction;

[0096] Boundary curvature features: used to characterize the degree of curvature and structural morphology of an object's contour;

[0097] Adjacency weight feature: reflects the interdependencies between nodes and prevents the segmentation result from destroying the overall structure.

[0098] The system integrates these three types of feature information to form an energy distribution field within the graph structure, which expresses the saliency and boundary importance of each node in the entire image. The construction of this energy distribution can guide subsequent segmentation algorithms to maintain natural boundaries between regions under complex lighting conditions, unaffected by shadows or reflections.

[0099] (3) Introduce a time-series decay parameter and a disturbance adjustment kernel into the graph potential field constraint energy function to form a dynamic graph potential field constraint model.

[0100] Specifically, in traditional static segmentation models, changes in image illumination or dynamic conditions can cause instability in energy distribution. This embodiment extends the graphical potential field model into a dynamic model with time-varying characteristics by introducing a temporal decay parameter and a perturbation adjustment kernel.

[0101] The temporal decay parameter is used to control the degree of inheritance of historical frame information by the graph potential field, so that the energy distribution between adjacent time frames can be smoothly transitioned, thereby maintaining the temporal continuity of the target region.

[0102] The perturbation adjustment kernel is used to cope with sudden disturbances, such as strong light reflection, shading caused by mechanical movement, or local brightness anomalies caused by dust floating. This kernel function can dynamically adjust the local response intensity of the energy field, so that the segmentation remains stable even under noise impact.

[0103] By introducing the above two types of parameters, the system achieves adaptive modeling of temporal changes in images, ensuring the continuity and robustness of the segmentation process in different time periods.

[0104] (4) Based on the dynamic graph potential field constraint model, execute the graph potential field constraint segmentation algorithm to output a set of foreground target images, which includes the image regions of workers, construction machinery and safety signs with topological labels.

[0105] Specifically, after completing the dynamic graph potential field modeling, the system initiates a graph potential field constraint segmentation algorithm to divide the graph structure into regions. During the algorithm's operation, it combines the aforementioned energy distribution and topology preservation rules to progressively update the label state of each node and determine convergence.

[0106] After segmentation, the system outputs a set of foreground target images and attaches topological labels to each region to distinguish different target types:

[0107] Worker image region: Identifies individual boundaries, posture contours, and position coordinates for subsequent behavior recognition;

[0108] Construction machinery image area: preserves the structural form and motion boundaries of the machinery and equipment for judging the operating status of the machinery;

[0109] Safety sign image area: Accurately extract visual elements such as safety signs, warning signs, and light signals within the construction area.

[0110] The output set of foreground targets will be input into the next semantic modeling module to construct a semantic graph of personnel actions and an semantic graph of the environment, thereby enabling behavioral understanding and regulatory analysis of multi-level work scenarios.

[0111] In step 103, a semantic map of human actions and a semantic map of the environment are constructed based on the set of foreground target images, and semantic behavior labels of the workers are generated based on the semantic map of human actions and the semantic map of the environment.

[0112] Preferably, the construction of a person action semantic map and an environment semantic map based on the foreground target image set includes:

[0113] The worker region in the foreground target image set is constructed as a multi-channel motion node graph; wherein, the nodes of the motion node graph include: skeletal topology nodes, joint mechanics nodes and tool interaction nodes, and dynamic edges driven by the Euler angle change rate are introduced between the nodes;

[0114] The construction environment area in the foreground target image set is constructed as a multi-dimensional environment node graph; wherein, the nodes of the multi-dimensional environment node graph include: mechanical equipment operation status nodes, safety sign location nodes, and hazard source distribution nodes, and energy edges modeled by thermal field distribution functions are established between the nodes.

[0115] Preferably, the semantic behavior tags of the workers are generated based on the personnel action semantic map and the environment semantic map, including:

[0116] A causal reasoning-driven cross-graph mapping edge is introduced between the action node graph and the environment node graph. Based on the causal alignment mechanism of graph homeomorphism constraint, the key actions of the operators and environmental risk events are matched in time sequence to generate a cross-domain causal semantic alignment graph.

[0117] The cross-domain causal semantic alignment graph is input into a causal graph regularized multi-level contrastive learning network to generate semantic behavior labels for workers.

[0118] Preferably, the causal alignment mechanism based on graph homeomorphism constraints pairs key actions of workers with environmental risk events in a temporal sequence to generate a cross-domain causal semantic alignment graph, including:

[0119] The node time information in the operator action semantic graph and the environment semantic graph is standardized and synchronized, and time-series slicing is performed to extract candidate event sequences with time stamps.

[0120] Based on the candidate event sequence, a cross-graph mapping method with graph structure homeomorphism constraint is adopted to establish a cross-domain correspondence between the action node graph and the environment node graph, and to construct an initial cross-domain causal mapping graph.

[0121] A weighted causal edge set is formed by introducing an edge weight calculation mechanism based on the causal strength function for the node pairs in the cross-domain causal mapping graph.

[0122] Based on the weighted causal edge set, node semantic attributes, graph structure topology information, and temporal dynamic features are integrated to generate a cross-domain causal semantic alignment graph.

[0123] Preferably, the causal strength function includes:

[0124]

[0125] Among them, I c (v i ,v j ) represents the worker's action node v i With environment node v j The causal strength value between them; Δt ij This represents the time difference between personnel action nodes and environmental event nodes; f represents the time-series decay scale parameter; ij Indicates the number of times a worker's actions and a specific environmental event co-occur in adjacent frame sequences; F i With F j They represent nodes v respectively i With node v j Total number of occurrences in the global sequence; d ij Represents node v i With node v j Spatial distance between them; σ d Represents the spatial attenuation parameter; γ ij α represents the semantic consistency weight coefficient; α, β1, and β2 represent global adjustment coefficients used to balance the contribution weights of time, frequency, and spatial factors.

[0126] In this embodiment, a semantic graph of personnel actions and a semantic graph of the environment are constructed based on the set of foreground target images, and a multi-level contrastive learning network with causal graph regularization is used to align cross-graph information and generate semantic behavior labels for the workers.

[0127] Combination Figure 2 As shown, in this embodiment, the process of generating semantic behavior tags for workers includes:

[0128] (1) Construct a multi-channel motion node graph for the worker region in the foreground target image set. The nodes of the motion node graph include skeletal topology nodes, joint mechanics nodes and tool interaction nodes, and introduce dynamic edges driven by the Euler angle change rate between the nodes.

[0129] Specifically, the worker region is extracted from the foreground target image set, and its pose is estimated and its behavioral structure is analyzed to form a multi-channel action node map.

[0130] This action node diagram consists of three types of nodes:

[0131] Skeletal topology nodes: Key points (such as head, shoulders, elbows, knees, etc.) are extracted from the human posture recognition model to form a skeleton topology structure, which is used to describe the body posture and movement pattern of a person.

[0132] Joint biomechanical nodes: Joint motion trends are calculated using optical flow estimation and joint angle change rate to reflect the intensity, directionality, and stability of the movement.

[0133] Tool interaction node: When contact between the hand and a tool (such as an electric drill, wrench, or welding torch) is detected, an interaction node is generated to record the tool type, holding state, and interaction duration.

[0134] Between these nodes, the system establishes dynamic edges driven by the Euler angle change rate to describe the motion transmission relationships between different joints or interactive objects. For example, when a person bends down to pick up materials, the system automatically creates a dynamic connection edge between the waist and hand nodes to reflect the temporal extension of the action. In this way, the motion node graph not only describes the static posture structure but also captures the continuous changing trend of the action, providing a foundation for subsequent semantic modeling.

[0135] (2) The construction environment area is constructed as a multi-dimensional environmental node diagram. The nodes of the multi-dimensional environmental node diagram include mechanical equipment operation status nodes, safety sign location nodes and hazard source distribution nodes, and energy edges modeled by thermal field distribution functions are established between the nodes.

[0136] Specifically, while extracting personnel action information, the system performs structural modeling of the environmental regions in the foreground image to generate a multi-dimensional environmental node map.

[0137] The graph contains the following main nodes:

[0138] Mechanical equipment operation status node: By detecting the motion trajectory, vibration amplitude and infrared thermal signal of mechanical equipment, its operation, shutdown or abnormal status can be determined;

[0139] Safety sign location nodes: Identify warning signs, traffic lights, notice boards, etc. in the construction area, and record their spatial coordinates and orientation;

[0140] Hazard distribution nodes: Model high-voltage areas, flammable materials, edges, fall risk points, etc. at the construction site to form a risk field distribution map.

[0141] Between nodes, the system establishes energy edges based on the spatial distribution of local thermal fields to characterize the correlation strength between different environmental elements. For example, when high-temperature machinery is near flammable materials, the system establishes a high-weight energy edge between them to reflect the potential risk. Through this mechanism, the environmental node graph can form a dynamic and continuous environmental semantic network at the spatial level, providing rich contextual information for subsequent cross-graph causal alignment.

[0142] (3) Introduce a cross-graph mapping edge driven by causal reasoning between the action node graph and the environment node graph. Using a causal alignment mechanism based on graph homeomorphism constraints, the key actions of the operators and environmental risk events are matched in time sequence to generate a cross-domain causal semantic alignment graph.

[0143] (4) Input the cross-domain causal semantic alignment graph into a multi-level contrastive learning network with causal graph regularization, and generate hierarchical semantic behavior labels, including process deviation labels, environmental risk interaction labels and time sequence anomaly labels.

[0144] Specifically, after completing cross-domain causal semantic alignment, the system inputs the alignment graph into a multi-level contrastive learning network with causal graph regularization to automatically generate semantic behavior labels.

[0145] The network is divided into three functional levels:

[0146] Low-level semantic comparison layer: Compares and analyzes the local features (posture, speed, contact object) of the action node to determine whether the action conforms to the standard procedure;

[0147] Mid-level causal constraint layer: Combining causal edge weights and time window information, it analyzes the triggering relationship between actions and environmental events to identify potential risky behaviors;

[0148] High-level semantic reasoning layer: Integrates information from the entire graph with contextual semantics to classify behaviors and output final labels.

[0149] The system ultimately generates three types of hierarchical semantic behavior labels:

[0150] Process deviation label: Indicates the degree of deviation between the personnel's behavior and the standard operating procedure;

[0151] Environmental risk interaction label: identifies the interaction between behavior and risk source, such as "entering a high-altitude area without wearing a seat belt";

[0152] Timing Anomaly Label: Used to identify situations where there is an unreasonable delay or abnormal triggering between the time of an action and the environmental response.

[0153] These labels will serve as input for subsequent steps, providing semantic data support for security oversight.

[0154] The causal alignment mechanism specifically includes the following steps:

[0155] The node time information in the operator action semantic graph and the environment semantic graph is standardized and synchronized, and time-series slicing is performed to extract candidate event sequences with time stamps.

[0156] Based on the candidate event sequence, a cross-graph mapping method with graph structure homeomorphism constraint is adopted to establish a cross-domain correspondence between the action node graph and the environment node graph, and to construct an initial cross-domain causal mapping graph.

[0157] A weighted edge calculation mechanism based on a causal strength function is introduced for the node pairs in the cross-domain causal mapping graph. The causal strength function is based on the order of actions between nodes, the frequency of interactions, and spatial proximity, and forms a weighted causal edge set.

[0158] Based on the weighted causal edge set, node semantic attributes, graph structure topology information, and temporal dynamic features are integrated to generate a cross-domain causal semantic alignment graph.

[0159] Specifically, after the construction of the action node graph and the environment node graph is completed, the system enters the cross-graph information alignment stage, and establishes the logical connection between the two types of graphs through the causal reasoning mechanism, forming a chain causal relationship of "action-environment-result".

[0160] The system first standardizes the node time information in the two types of graphs, dividing the video frame sequence into continuous temporal slices. Each slice contains one or more candidate event sequences, such as "person raises hand," "equipment starts," and "signal light flashes." The system uses a time alignment module to achieve temporal synchronization between nodes, ensuring that actions and environmental events within the same time window can be directly compared.

[0161] Based on the temporal synchronization results, the system employs a graph structure homeomorphism constraint method to establish a one-to-one or many-to-one correspondence between the action node graph and the environment node graph. For example, when the events "person approaches equipment" and "equipment starts" are detected to be highly overlapping in time, the system establishes a cross-domain mapping edge between them to reflect the potential causal dependency between the action and the environment state. This homeomorphism constraint ensures the structural consistency of the two graphs in terms of topological relationships, avoiding invalid or redundant matching.

[0162] To quantify the strength of causal relationships between different node pairs, the system introduces a causal weighting mechanism on cross-domain mapping edges. This weight is determined by a combination of the following factors:

[0163] Sequence of actions: If an action occurs before an environmental change, it is assigned a higher causal probability;

[0164] Interaction frequency: Nodes with high-frequency interactions (such as repetitive tool operations) are assigned higher relevance;

[0165] Spatial proximity: The closer the action node is to the environment node in space, the higher the causal weight.

[0166] Through this process, the system generates a weighted causal edge set to reflect the potential chain of action between human behavior and environmental events.

[0167] Based on weighted causal edge sets, the system integrates node semantic attributes (such as action type and risk level), graph topology (such as connectivity and hierarchical relationships), and temporal dynamic features to construct a cross-domain causal semantic alignment graph. This graph can intuitively represent the semantic chain of "action trigger—environmental response—risk outcome," for example: "personnel not wearing safety helmets → entering the lifting area → operation of lifting machinery → potential high-risk interaction event." This cross-graph alignment method enables the system to accurately identify the causal logic behind behaviors, providing a basis for intelligent supervision decisions.

[0168] Furthermore, the formula for the causal strength function is as follows:

[0169]

[0170] Among them, I c (v i ,v j ) represents the worker's action node v i With environment node v j The causal strength value between two events is a key indicator used to measure the degree of temporal causal association between them. A larger value indicates a stronger causal relationship between human actions and environmental events; Δt ij This represents the time difference between personnel action nodes and environmental event nodes; This represents the temporal decay scale parameter, reflecting the range of influence of the time difference on the causal strength; f ij Indicates the number of times a worker's actions and a specific environmental event co-occur in adjacent frame sequences; F i With F j They represent nodes v respectively i With node v j The total number of occurrences in the global sequence represents the overall activity level of the node throughout the entire job cycle. This is used to normalize the interaction frequency and prevent high-frequency nodes from having excessively high weights due to frequent occurrences; d ij Represents node v i With node v j The spatial distance between them is calculated based on the pixel coordinate difference between the centroid of the personnel area and the centroid of the equipment or environmental event area in the image; σ d γ represents the spatial attenuation parameter, used to adjust the scale of the influence of distance on causality strength. ij α represents the semantic consistency weight coefficient, used to comprehensively measure the matching degree between action semantics and environment semantics; α, β1 and β2 represent global adjustment coefficients, used to balance the contribution weights of time, frequency and spatial factors.

[0171] In step 104, the trajectory sequence of the workers marked with semantic behavior tags is obtained, and the process deviation index and abnormal process identification result are determined based on the trajectory sequence.

[0172] Preferably, determining the process deviation index and abnormal process identification result based on the trajectory sequence includes:

[0173] The trajectory sequence of workers labeled with semantic behavior tags is constructed into a temporal event graph. The nodes of the temporal event graph include action event nodes, environmental interaction nodes, and safety status nodes. The edges between nodes represent temporal continuity and causal relationship.

[0174] Based on the aforementioned time-series event graph, an asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model the trajectories of multiple personnel. An event intensity function and a background noise term are introduced in the time domain to decouple and cluster the action patterns of different workers.

[0175] In the trajectory clustering results, the process deviation index is calculated based on the temporal distribution and spatial density characteristics of each cluster center;

[0176] Based on the process deviation index and clustering labels, anomaly process identification results are generated using the Bayesian belief propagation mechanism; wherein, the anomaly process identification results include deviation type, deviation magnitude and impact level.

[0177] Preferably, the formula for the asynchronous trajectory clustering algorithm is as follows:

[0178]

[0179] Among them, Λ k (τ) represents the intensity value of the trajectory event of the k-th type of worker at time τ; η k λ represents the normalized intensity coefficient of the k-th type of trajectory; k τ represents the time decay rate parameter of the k-th type of trajectory; 0k δ represents the initial time point of the k-th type of trajectory; k P represents the background noise term of the k-th type trajectory; k ω represents the total number of behavioral events contained in the k-th type of trajectory; kp The semantic weight coefficient of the p-th behavioral event in the k-th trajectory is represented by r*τ; the spatial coordinate position vector of the worker at time τ is represented by r*kp; the spatial center position of the p-th behavioral event in the k-th trajectory is represented by r*kp. The spatial scale parameter represents the k-th type of trajectory.

[0180] In this embodiment, based on the semantic behavior tags, an asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model and compare personnel behavior trajectories, and output process deviation index and abnormal process identification results.

[0181] Combination Figure 3 As shown, in this embodiment, the process of outputting the process deviation index and the abnormal process identification result includes:

[0182] (1) The sequence of worker trajectories labeled with semantic behavior tags is constructed into a time-series event graph. The nodes of the time-series event graph include action event nodes, environmental interaction nodes and safety status nodes. The edges between nodes represent temporal continuity and causal relationship.

[0183] Specifically, the behavioral data of workers with semantic behavioral labels are transformed into a structured time-series trajectory information stream, and a time-series event graph is constructed.

[0184] The node types of this timing event graph include:

[0185] Action event nodes: used to record key human behaviors, such as bending down to pick up an object, operating machinery, walking, observing, or remaining still;

[0186] Environmental interaction nodes: These represent the interaction behaviors of workers with surrounding environmental elements, such as contacting equipment, entering specific areas, and approaching hazardous sources.

[0187] Safety status node: Used to describe the current safety status of personnel, such as the wearing status of protective equipment, body posture stability, or distance from the safety boundary.

[0188] Edges between nodes represent temporal continuity and causal relationships. The system standardizes the timestamps of video frame sequences and semantic tags, connecting event nodes into a chain structure in chronological order; at the same time, causal edges are established when causal logic exists (such as "device startup" being caused by "personnel operation").

[0189] This graph structure not only preserves the temporal evolution characteristics of the operation behavior, but also explicitly reflects the interaction logic between the behavior and the environment, providing an interpretable basis for subsequent asynchronous trajectory modeling.

[0190] (2) Based on the time-series event graph, an asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model the trajectory of multiple personnel. The asynchronous trajectory clustering algorithm decouples and clusters the action occurrence patterns of different workers by introducing an event intensity function and a background noise term in the time domain.

[0191] Specifically, based on the constructed time-series event graph, the system employs a Poisson hybrid process-driven asynchronous trajectory clustering algorithm to model the behavioral trajectories of multiple personnel. Unlike traditional synchronous clustering, this algorithm introduces the concept of "asynchronous," meaning that different workers performing the same task may have different start times and durations of their actions. To address this, the system models events in the time domain using event intensity, treating each person's trajectory as an independent stream of time events, and introduces an event intensity function to decouple it from the background noise term. The event intensity function describes the frequency and trend of specific types of action events on the time axis, thereby identifying the dominant rhythm of personnel actions; the background noise term is used to suppress the interference of irrelevant or random events (such as short pauses or false detections of actions) on the clustering results.

[0192] In this way, the system can distinguish the execution rhythm and behavioral characteristics of different operators on a time scale. Even if they perform similar tasks at different times, they can be identified as asynchronous individuals under the same work process mode. The clustering algorithm ultimately outputs multiple trajectory categories, each corresponding to a standard or non-standard work mode.

[0193] (3) In the trajectory clustering results, the process deviation index is calculated based on the temporal distribution and spatial density characteristics of each cluster center.

[0194] Specifically, after obtaining the trajectory clustering results, the system calculates the process deviation index of each person based on the temporal distribution and spatial density characteristics of each cluster center.

[0195] Specifically, the system analyzes the temporal pattern (i.e., standard operating rhythm) of each cluster center and calculates the degree of deviation of each individual trajectory from this standard:

[0196] If an individual's trajectory is highly synchronized with the cluster center in the time dimension, it indicates that its work behavior conforms to the standard rhythm and has a low degree of deviation.

[0197] If an individual's trajectory shows a significant advance, delay, or interruption, it is determined that there is a time deviation.

[0198] If the spatial distribution of the trajectory deviates significantly from the main work area, such as when personnel stay in a high-risk area for an extended period of time, it will be marked as spatial deviation.

[0199] The system integrates both temporal and spatial deviation characteristics to generate a unified process deviation index. This index measures the degree of matching between personnel operations and process standards, serving as the basic data for identifying abnormal processes.

[0200] (4) Based on the process deviation index and clustering label, the abnormal process identification result is generated using the Bayesian belief propagation mechanism. The abnormal process identification result includes deviation type, deviation magnitude and impact level.

[0201] Specifically, after obtaining the process deviation index, the system further identifies abnormal processes based on clustering labels and deviation information through a Bayesian belief propagation mechanism. The system first inputs the deviation index of each trajectory node into a Bayesian network model. The network performs confidence inference on the current trajectory by statistically analyzing the probability of occurrence and risk impact of different deviation types in historical data. For example, when a combined event of "delayed personnel behavior + premature start-up of environmental equipment" occurs, the system identifies a potential "abnormal operation sequence"; when a situation of "prolonged personnel stay + proximity to high-temperature machinery" occurs, the system outputs a "risk contact deviation" result.

[0202] The final abnormal process identification results include three types of key information:

[0203] Deviation type: Used to describe the nature of the anomaly, such as time lag, disordered sequence, area crossing, risk exposure, etc.

[0204] Deviation magnitude: A quantitative result representing the degree of deviation, reflecting its severity;

[0205] Impact Level: Based on historical accident statistics and risk field intensity, the risk level is automatically determined to be low, medium, or high, which is used to guide on-site intervention.

[0206] The system visualizes the identified abnormal results on the construction supervision terminal and pushes them to the area manager in the form of warning cards, enabling immediate response.

[0207] Furthermore, the formula for the asynchronous trajectory clustering algorithm is as follows:

[0208]

[0209] Among them, Λ k (τ) represents the trajectory event intensity value of the k-th type of worker at time τ, which is used to characterize the event occurrence probability density of this type of trajectory in the asynchronous time domain. It is the core quantitative indicator in the asynchronous trajectory clustering algorithm; η k λ represents the normalized intensity coefficient of the k-th type trajectory, reflecting the overall frequency of human behavior events in this cluster; k The time decay rate parameter represents the k-th type of trajectory, used to describe the trend of the operation behavior gradually decaying with time delay, thereby capturing the differences in the execution rhythm of different processes; τ 0k The initial time point represents the k-th type of trajectory, corresponding to the moment when a semantic behavioral event occurs for the first time within the construction area for this type of worker, such as the time of first entering a specific construction area or starting a certain operation task; δ k The background noise term representing the k-th type of trajectory is used to characterize the impact of atypical events or random disturbances (such as waiting, pausing, equipment shutdown, etc.) on the intensity of the behavior; P k ω represents the total number of behavioral events contained in the k-th type of trajectory. This number is determined by the semantic behavioral label sequence and reflects the complexity of the process; kp The semantic weight coefficient of the p-th behavioral event in the k-th trajectory is dynamically assigned based on the type of human behavior (e.g., handling, detection, command) and its corresponding environmental risk level, to highlight the dominant role of key actions in clustering; r*τ represents the spatial coordinate position vector of the worker at time τ, obtained by topologically preserving mapping from the personnel position information in the foreground target image set; r*kp represents the spatial center position of the p-th behavioral event in the k-th trajectory, corresponding to the spatial positioning result of the event on the construction area topology map; The spatial scale parameter represents the k-th type of trajectory, reflecting the spatial diffusion range of event distribution in this type of trajectory, and is used to control the spatial similarity weight between the behavioral trajectories of different people.

[0210] In step 105, hierarchical supervision is carried out based on the process deviation index and the abnormal process identification results.

[0211] Preferably, the hierarchical supervision based on the process deviation index and the abnormal process identification results includes:

[0212] Based on the anomaly identification results, risk values ​​of different dimensions are calculated for the behavioral trajectory of each worker within the current time window;

[0213] The risk values ​​from different dimensions are weighted and summed to determine the real-time risk level label;

[0214] Regulatory decision instructions are determined based on the real-time risk level labels, and supervision is carried out based on the regulatory decision instructions.

[0215] In this embodiment, hierarchical supervision is performed based on the process deviation index and abnormal process identification results. The hierarchical supervision includes early warning prompts and mandatory control instructions generated based on fuzzy reasoning mechanism, and the supervision results are written into the security database and synchronized to the supervision terminal.

[0216] Specifically, in this embodiment, the risk value is first calculated for the behavioral trajectory of each worker within the current time window based on the aforementioned anomaly identification results. Risk assessment factors include:

[0217] Deviation intensity (such as entering unauthorized areas, failing to follow standard procedures, etc.);

[0218] Scenarios with varying degrees of hazard (e.g., working at heights, under power, lifting, working in confined spaces, etc.);

[0219] Safety protection status (whether a safety helmet, reflective vest, and seat belt are worn, etc.);

[0220] Historical violation records and handling results (used to construct individual risk trend curves).

[0221] The system integrates the above multi-dimensional indicators and obtains a real-time risk level label through weighted evaluation. The risk level is divided into four levels from low to high: safe, suspicious, serious, and urgent, corresponding to green, yellow, orange, and red status lights.

[0222] After the risk level assessment is completed, the system enters the intelligent decision-making phase. This module introduces a fuzzy reasoning mechanism to dynamically judge complex or ambiguous risk situations. For example, when a worker's behavior deviates moderately, but the area they are in is a high-risk area, the system can automatically determine it as a "high-risk warning" rather than a simple "medium-level warning" based on a rule base. The fuzzy rule base can include the following examples: if the behavior deviation is low and the protection status is normal, the monitoring status is maintained; if the deviation is moderate and the environmental hazard is high, a yellow warning is generated; if the deviation is high and accompanied by a violation posture label (such as bending over to pick up an object, bending at height, etc.), an orange mandatory intervention is directly triggered; if entry into a dangerous area or abnormal proximity to equipment is identified, a red emergency control command is output, triggering the alarm device. The output of this module is a set of regulatory decision commands, including three action levels: "prompt," "warning," and "mandatory control."

[0223] The decision results are distributed to multiple execution terminals via the communication scheduling module, including:

[0224] On-site audible and visual alarm device: triggers flashing lights and a buzzer alarm at red or orange levels;

[0225] Security management terminal: Pushes alarm cards containing data such as personnel information, time, risk type, and behavioral images;

[0226] Mobile security monitoring app: Reminds regional managers to intervene in a timely manner via push notifications;

[0227] Cloud-based monitoring center: centrally displays risk status distribution maps of multiple work areas, enabling cross-regional joint monitoring.

[0228] In addition, the system supports automatic video playback based on alarm level, which makes it easy for managers to quickly check the on-site footage and realize the whole chain response of "discovery-judgment-tracing-intervention".

[0229] All monitoring results and execution records are automatically written to the safety database. The database structure includes: personnel ID, employee number and work group information; abnormal behavior tags and risk levels; alarm trigger time, response time and handling results; post-intervention behavior correction status and on-site feedback information.

[0230] The system periodically generates statistical reports and trend analysis curves for safety performance evaluation and risk evolution analysis. When the same person or area triggers medium-to-high level alarms multiple times within a short period, the system can automatically mark it as a "key concern" and push its information to the project safety manager for review or training. If the risk status does not decrease after intervention measures are implemented, the system will automatically escalate the regulatory level and generate secondary instructions to achieve dynamic closed-loop control.

[0231] Through this tiered supervision mechanism, the system not only achieves real-time perception and intelligent judgment of on-site work behavior, but also forms a full-process supervision system of "data collection - risk identification - tiered decision-making - closed-loop feedback", which greatly improves the intelligence, precision and traceability of construction safety supervision.

[0232] Figure 4 This is a schematic diagram of the structure of an image-based intelligent recognition-based on-site operation monitoring system 400 according to an embodiment of the present invention. Figure 4 As shown, the on-site operation supervision system 400 based on image intelligent recognition provided by the embodiment of the present invention includes: a data acquisition unit 401, an image processing unit 402, a semantic behavior label determination unit 403, a recognition unit 404, and a supervision unit 405.

[0233] Preferably, the data acquisition unit 401 is used to acquire on-site operation image data based on an image acquisition device that fuses visible light and infrared light.

[0234] Preferably, the on-site operation image data includes: images of the head and limb postures of the workers, images of the wearing status of reflective vests and safety helmets, images of the tools and materials carried, and images of the operating status of construction machinery and equipment and environmental safety signs in the work area.

[0235] Preferably, the image processing unit 402 is used to generate a set of foreground target images based on the on-site operation image data.

[0236] Preferably, the image processing unit 402 generates a set of foreground target images based on the on-site operation image data, including:

[0237] The on-site operation image data is constructed as a multi-layer graph structure represented by nodes; wherein, the nodes of the multi-layer graph structure include: personnel area nodes, mechanical equipment nodes and environmental background nodes, and a topology-preserving transformation operator is introduced into the multi-layer graph structure to maintain the boundary connectivity of the construction area;

[0238] Define a graph potential field constraint energy function on the multi-layer graph structure;

[0239] A time-series decay parameter and a perturbation adjustment kernel are introduced into the graphical potential field constraint energy function to form a dynamic graphical potential field constraint model;

[0240] The graph potential field constraint model is used to execute the graph potential field constraint segmentation algorithm to output a set of foreground target images; wherein, the set of foreground target images includes: image regions of workers, image regions of construction machinery and image regions of safety signs with topological labels.

[0241] Preferably, the semantic behavior label determination unit 403 is used to construct a personnel action semantic map and an environment semantic map based on the foreground target image set, and to generate semantic behavior labels for the workers based on the personnel action semantic map and the environment semantic map.

[0242] Preferably, the semantic behavior label determination unit 403 constructs a person action semantic map and an environment semantic map based on the foreground target image set, including:

[0243] The worker region in the foreground target image set is constructed as a multi-channel motion node graph; wherein, the nodes of the motion node graph include: skeletal topology nodes, joint mechanics nodes and tool interaction nodes, and dynamic edges driven by the Euler angle change rate are introduced between the nodes;

[0244] The construction environment area in the foreground target image set is constructed as a multi-dimensional environment node graph; wherein, the nodes of the multi-dimensional environment node graph include: mechanical equipment operation status nodes, safety sign location nodes, and hazard source distribution nodes, and energy edges modeled by thermal field distribution functions are established between the nodes.

[0245] Preferably, the semantic behavior label determination unit 403 generates semantic behavior labels for the workers based on the personnel action semantic map and the environmental semantic map, including:

[0246] A causal reasoning-driven cross-graph mapping edge is introduced between the action node graph and the environment node graph. Based on the causal alignment mechanism of graph homeomorphism constraint, the key actions of the operators and environmental risk events are matched in time sequence to generate a cross-domain causal semantic alignment graph.

[0247] The cross-domain causal semantic alignment graph is input into a causal graph regularized multi-level contrastive learning network to generate semantic behavior labels for workers.

[0248] Preferably, the semantic behavior label determination unit 403, based on the causal alignment mechanism of graph homeomorphism constraints, performs time-series pairing of key actions of operators with environmental risk events to generate a cross-domain causal semantic alignment graph, including:

[0249] The node time information in the operator action semantic graph and the environment semantic graph is standardized and synchronized, and time-series slicing is performed to extract candidate event sequences with time stamps.

[0250] Based on the candidate event sequence, a cross-graph mapping method with graph structure homeomorphism constraint is adopted to establish a cross-domain correspondence between the action node graph and the environment node graph, and to construct an initial cross-domain causal mapping graph.

[0251] A weighted causal edge set is formed by introducing an edge weight calculation mechanism based on the causal strength function for the node pairs in the cross-domain causal mapping graph.

[0252] Based on the weighted causal edge set, node semantic attributes, graph structure topology information, and temporal dynamic features are integrated to generate a cross-domain causal semantic alignment graph.

[0253] Preferably, the causal strength function includes:

[0254]

[0255] Among them, I c (v i ,v j ) represents the worker's action node v i With environment node v j The causal strength value between them; Δt ij This represents the time difference between personnel action nodes and environmental event nodes; f represents the time-series decay scale parameter; ij Indicates the number of times a worker's actions and a specific environmental event co-occur in adjacent frame sequences; F i With F j They represent nodes v respectively i With node v j Total number of occurrences in the global sequence; d ij Represents node v i With node vj Spatial distance between them; σ d Represents the spatial attenuation parameter; γ ij α represents the semantic consistency weight coefficient; α, β1, and β2 represent global adjustment coefficients used to balance the contribution weights of time, frequency, and spatial factors.

[0256] Preferably, the identification unit 404 is used to acquire the trajectory sequence of the operator marked with semantic behavior tags, and determine the process deviation index and abnormal process identification result based on the trajectory sequence.

[0257] Preferably, the identification unit 404 determines the process deviation index and abnormal process identification result based on the trajectory sequence, including:

[0258] The trajectory sequence of workers labeled with semantic behavior tags is constructed into a temporal event graph. The nodes of the temporal event graph include action event nodes, environmental interaction nodes, and safety status nodes. The edges between nodes represent temporal continuity and causal relationship.

[0259] Based on the aforementioned time-series event graph, an asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model the trajectories of multiple personnel. An event intensity function and a background noise term are introduced in the time domain to decouple and cluster the action patterns of different workers.

[0260] In the trajectory clustering results, the process deviation index is calculated based on the temporal distribution and spatial density characteristics of each cluster center;

[0261] Based on the process deviation index and clustering labels, anomaly process identification results are generated using the Bayesian belief propagation mechanism; wherein, the anomaly process identification results include deviation type, deviation magnitude and impact level.

[0262] Preferably, the formula for the asynchronous trajectory clustering algorithm is as follows:

[0263]

[0264] Among them, Λ k (τ) represents the intensity value of the trajectory event of the k-th type of worker at time τ; η k λ represents the normalized intensity coefficient of the k-th type of trajectory; k τ represents the time decay rate parameter of the k-th type of trajectory; 0k δ represents the initial time point of the k-th type of trajectory; k P represents the background noise term of the k-th type trajectory; k ω represents the total number of behavioral events contained in the k-th type of trajectory; kpThe semantic weight coefficient of the p-th behavioral event in the k-th trajectory is represented by r*τ; the spatial coordinate position vector of the worker at time τ is represented by r*kp; the spatial center position of the p-th behavioral event in the k-th trajectory is represented by r*kp. The spatial scale parameter represents the k-th type of trajectory.

[0265] Preferably, the monitoring unit 405 is used to perform hierarchical monitoring based on the process deviation index and the abnormal process identification results.

[0266] Preferably, the monitoring unit 405 performs hierarchical monitoring based on the process deviation index and the abnormal process identification results, including:

[0267] Based on the anomaly identification results, risk values ​​of different dimensions are calculated for the behavioral trajectory of each worker within the current time window;

[0268] The risk values ​​from different dimensions are weighted and summed to determine the real-time risk level label;

[0269] Regulatory decision instructions are determined based on the real-time risk level labels, and supervision is carried out based on the regulatory decision instructions.

[0270] The on-site operation monitoring system 400 based on image intelligent recognition in one embodiment of the present invention corresponds to the on-site operation monitoring method 100 based on image intelligent recognition in another embodiment of the present invention, and will not be described again here.

[0271] According to another aspect of the present invention, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the steps of a field operation monitoring method based on image intelligent recognition.

[0272] According to another aspect of the present invention, the present invention provides an electronic device, comprising:

[0273] The aforementioned computer-readable storage medium; and

[0274] One or more processors for executing a program in the computer-readable storage medium.

[0275] The present invention has been described with reference to a few embodiments. However, it will be apparent to those skilled in the art that other embodiments besides those disclosed above fall equivalently within the scope of the present invention.

[0276] Generally, all terms used in this invention are interpreted according to their ordinary meaning in the art, unless otherwise expressly defined herein. All references to “a / the / the [device, component, etc.]” ​​are openly interpreted as at least one instance of said device, component, etc., unless otherwise expressly stated. The steps of any method disclosed herein need not be performed in the exact order disclosed unless explicitly stated otherwise.

[0277] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0278] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0279] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0280] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0281] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for on-site operation supervision based on image intelligent recognition, characterized in that, The method includes: Image data of on-site operations is acquired using image acquisition equipment that integrates visible and infrared light; Based on the on-site operation image data, a set of foreground target images is generated; Based on the foreground target image set, construct a personnel action semantic map and an environment semantic map, and based on the personnel action semantic map and the environment semantic map, generate semantic behavior labels for the workers; Obtain the trajectory sequence of the workers labeled with semantic behavior tags, and determine the process deviation index and abnormal process identification result based on the trajectory sequence; Based on the process deviation index and the results of abnormal process identification, hierarchical supervision is carried out.

2. The method according to claim 1, characterized in that, The on-site operation image data includes: images of the head and limb postures of the workers, images of the wearing status of reflective vests and safety helmets, images of the tools and materials carried, and images of the operating status of construction machinery and equipment and environmental safety signs in the work area.

3. The method according to claim 1, characterized in that, Based on the on-site operation image data, a set of foreground target images is generated, including: The on-site operation image data is constructed as a multi-layer graph structure represented by nodes; wherein, the nodes of the multi-layer graph structure include: personnel area nodes, mechanical equipment nodes and environmental background nodes, and a topology-preserving transformation operator is introduced into the multi-layer graph structure to maintain the boundary connectivity of the construction area; Define a graph potential field constraint energy function on the multi-layer graph structure; A time-series decay parameter and a perturbation adjustment kernel are introduced into the graphical potential field constraint energy function to form a dynamic graphical potential field constraint model; The graph potential field constraint model is used to execute the graph potential field constraint segmentation algorithm to output a set of foreground target images; wherein, the set of foreground target images includes: image regions of workers, image regions of construction machinery and image regions of safety signs with topological labels.

4. The method according to claim 1, characterized in that, Based on the foreground target image set, construct a person action semantic map and an environment semantic map, including: The worker region in the foreground target image set is constructed as a multi-channel motion node graph; wherein, the nodes of the motion node graph include: skeletal topology nodes, joint mechanics nodes and tool interaction nodes, and dynamic edges driven by the Euler angle change rate are introduced between the nodes; The construction environment area in the foreground target image set is constructed as a multi-dimensional environment node graph; wherein, the nodes of the multi-dimensional environment node graph include: mechanical equipment operation status nodes, safety sign location nodes, and hazard source distribution nodes, and energy edges modeled by thermal field distribution functions are established between the nodes.

5. The method according to claim 1, characterized in that, Based on the personnel action semantic map and the environment semantic map, semantic behavior tags for the workers are generated, including: A causal reasoning-driven cross-graph mapping edge is introduced between the action node graph and the environment node graph. Based on the causal alignment mechanism of graph homeomorphism constraint, the key actions of the operators and environmental risk events are matched in time sequence to generate a cross-domain causal semantic alignment graph. The cross-domain causal semantic alignment graph is input into a causal graph regularized multi-level contrastive learning network to generate semantic behavior labels for workers.

6. The method according to claim 5, characterized in that, Based on the causal alignment mechanism of graph homeomorphism constraints, key actions of workers are temporally paired with environmental risk events to generate a cross-domain causal semantic alignment graph, including: The node time information in the operator action semantic graph and the environment semantic graph is standardized and synchronized, and time-series slicing is performed to extract candidate event sequences with time stamps. Based on the candidate event sequence, a cross-graph mapping method with graph structure homeomorphism constraint is adopted to establish a cross-domain correspondence between the action node graph and the environment node graph, and to construct an initial cross-domain causal mapping graph. A weighted causal edge set is formed by introducing an edge weight calculation mechanism based on the causal strength function for the node pairs in the cross-domain causal mapping graph. Based on the weighted causal edge set, node semantic attributes, graph structure topology information, and temporal dynamic features are integrated to generate a cross-domain causal semantic alignment graph.

7. The method according to claim 6, characterized in that, The causal strength function includes: Among them, I c (v i ,v j ) represents the worker's action node v i With environment node v j The causal strength value between them; Δt ij This represents the time difference between personnel action nodes and environmental event nodes; f represents the time-series decay scale parameter; ij Indicates the number of times a worker's actions and a specific environmental event co-occur in adjacent frame sequences; F i With F j Representing node v respectively i With node v j Total number of occurrences in the global sequence; d ij Represents node v i With node v j Spatial distance between them; σ d Represents the spatial attenuation parameter; γ ij α represents the semantic consistency weight coefficient; α, β1, and β2 represent global adjustment coefficients used to balance the contribution weights of time, frequency, and spatial factors.

8. The method according to claim 1, characterized in that, Based on the trajectory sequence, the process deviation index and abnormal process identification results are determined, including: The trajectory sequence of workers labeled with semantic behavior tags is constructed into a temporal event graph. The nodes of the temporal event graph include action event nodes, environmental interaction nodes, and safety status nodes. The edges between nodes represent temporal continuity and causal relationship. Based on the aforementioned time-series event graph, an asynchronous trajectory clustering algorithm driven by a Poisson hybrid process is used to model the trajectories of multiple personnel. An event intensity function and a background noise term are introduced in the time domain to decouple and cluster the action patterns of different workers. In the trajectory clustering results, the process deviation index is calculated based on the temporal distribution and spatial density characteristics of each cluster center; Based on the process deviation index and clustering labels, anomaly process identification results are generated using the Bayesian belief propagation mechanism; wherein, the anomaly process identification results include deviation type, deviation magnitude and impact level.

9. The method according to claim 8, characterized in that, The formula for the asynchronous trajectory clustering algorithm is as follows: Among them, Λ k (τ) represents the intensity value of the trajectory event of the k-th type of worker at time τ; η k λ represents the normalized intensity coefficient of the k-th type trajectory; k τ represents the time decay rate parameter of the k-th type of trajectory; 0k δ represents the initial time point of the k-th type of trajectory; k P represents the background noise term of the k-th type trajectory; k ω represents the total number of behavioral events contained in the k-th type of trajectory; kp The semantic weight coefficient of the p-th behavioral event in the k-th trajectory is represented by r*τ; the spatial coordinate position vector of the worker at time τ is represented by r*kp; the spatial center position of the p-th behavioral event in the k-th trajectory is represented by r*kp. The spatial scale parameter represents the k-th type of trajectory.

10. The method according to claim 1, characterized in that, Based on the process deviation index and abnormal process identification results, hierarchical supervision is implemented, including: Based on the anomaly identification results, risk values ​​of different dimensions are calculated for the behavioral trajectory of each worker within the current time window; The risk values ​​from different dimensions are weighted and summed to determine the real-time risk level label; Regulatory decision instructions are determined based on the real-time risk level labels, and supervision is carried out based on the regulatory decision instructions.

11. A field operation monitoring system based on image intelligent recognition, characterized in that, The system includes: The data acquisition unit is used to acquire on-site operation image data based on image acquisition equipment that fuses visible light and infrared light; An image processing unit is used to generate a set of foreground target images based on the on-site operation image data; The semantic behavior label determination unit is used to construct a personnel action semantic map and an environment semantic map based on the foreground target image set, and to generate semantic behavior labels for the workers based on the personnel action semantic map and the environment semantic map; The identification unit is used to acquire the trajectory sequence of the workers marked with semantic behavior tags, and to determine the process deviation index and abnormal process identification result based on the trajectory sequence; The monitoring unit is used to conduct hierarchical monitoring based on the process deviation index and the abnormal process identification results.

Images