A cloud-edge collaborative aerial operation violation detection method and device, a terminal device, and a computer readable storage medium

Abstract

This invention discloses a cloud-edge collaborative method, device, terminal equipment, and computer-readable storage medium for detecting violations in high-altitude operations, belonging to the field of high-altitude operation safety monitoring technology. The method includes: acquiring a first identification result at the edge end containing a personnel ID, a first timestamp, a first violation type, and a first confidence level; and a second identification result at the cloud end containing location information, a second timestamp, a second violation type, and a second confidence level; matching the two timestamps, and if successful, mapping and associating the location information with the personnel ID to obtain spatiotemporally aligned data; dynamically assigning weights based on the image quality signal and the status signal, and weighted summing the two confidence levels to obtain a fused confidence level; comparing the two violation types to obtain a matching result, and triggering a violation warning based on the spatiotemporally aligned data when the results are consistent and the fused confidence level is not less than a first preset threshold. This invention solves the problem that existing violation monitoring methods are easily affected by environmental obstruction or abnormal sensor interference, leading to misjudgments of status.

Description

Technical Field

[0001] This invention relates to the field of high-altitude operation safety monitoring technology, and in particular to a cloud-edge collaborative method, device, terminal equipment, and computer-readable storage medium for detecting violations in high-altitude operations. Background Technology

[0002] With the development of IoT and AI vision technology, safety monitoring technology based on edge sensors and cloud video has been widely used in the task of identifying violations in high-altitude operations, becoming an important technical means to prevent falls from heights and improve the efficiency of on-site safety management.

[0003] Existing conventional methods for monitoring violations are highly susceptible to interference from environmental obstructions or sensor aging when dealing with complex high-altitude work scenarios, leading to misjudgments and distorted violation data. This results in a high false alarm rate and uncontrollable warnings for high-altitude work violations. Therefore, reducing the false alarm rate in complex environments and improving the accuracy of high-altitude work violation identification and warning has become a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0004] This invention provides a cloud-edge collaborative method, device, terminal equipment, and computer-readable storage medium for detecting violations in high-altitude operations, which can solve the problem that existing violation monitoring methods are easily affected by environmental obstruction or abnormal sensor interference, resulting in misjudgments of status.

[0005] One embodiment of the present invention provides a cloud-edge collaborative method for detecting violations in high-altitude operations, comprising: acquiring a first identification result generated based on sensor data from an edge-sensing terminal, wherein the first identification result includes a personnel ID, a first timestamp, a first violation type, and a first confidence level; Obtain a second recognition result generated based on a cloud video stream. The second recognition result includes location information, a second timestamp, a second violation type, and a second confidence level. The first timestamp is matched with the second timestamp. When the match is successful, the location information is mapped and associated with the person ID to obtain spatiotemporal aligned data. The image quality signal of the cloud video stream and the status signal of the sensor are obtained. Dynamic weight allocation is performed based on the image quality signal and the status signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. The first confidence score and the second confidence score are weighted and summed according to the first weight and the second weight to obtain the fused confidence score. The first violation type and the second violation type are compared to obtain the type matching result; When the type matching result is consistent and the fusion confidence is not less than the first preset threshold, a violation warning for the personnel ID is triggered based on the spatiotemporal alignment data.

[0006] Furthermore, the step of obtaining the first recognition result generated based on sensor data from the edge-sensing terminal includes: acquiring sensor data and performing moving average filtering and normalization on the sensor data to obtain preprocessed time-series data; inputting the preprocessed time-series data into a pre-constructed one-dimensional convolutional neural network for spatiotemporal feature extraction to obtain a feature vector sequence; inputting the feature vector sequence into a long short-term memory network for continuous action modeling and classification to obtain a preliminary result containing the first timestamp, the first violation type, and the first confidence level; reading a pre-stored personnel ID and combining the personnel ID with the preliminary result to obtain the first recognition result.

[0007] Furthermore, obtaining the second recognition result generated based on the cloud video stream includes: extracting keyframes from the cloud video stream, inputting the keyframes into a pre-trained scene classifier for scene recognition, and obtaining scene type labels; dynamically loading the corresponding target visual recognition model from a preset model repository according to the scene type labels; inputting the cloud video stream into the target visual recognition model for personnel tracking and posture estimation, and obtaining a second recognition result including the location information, the second timestamp, the second violation type, and the second confidence level.

[0008] Furthermore, the first timestamp and the second timestamp are matched. When the match is successful, the location information is mapped and associated with the personnel ID to obtain spatiotemporally aligned data. Specifically, this includes: calculating the time difference between the first timestamp and the second timestamp; determining that the first timestamp and the second timestamp are successfully matched when the time difference is less than a preset synchronization window threshold and the first violation type and the second violation type meet preset association action matching conditions; after successful matching, extracting the location information from the second recognition result and concatenating the location information with the personnel ID to obtain the spatiotemporally aligned data.

[0009] Furthermore, dynamic weight allocation is performed based on the image quality signal and the state signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. Specifically, this includes: at the initial moment, obtaining a preset initial edge weight and an initial cloud weight; using the initial edge weight as the first weight and the initial cloud weight as the second weight; wherein the sum of the preset initial edge weight and the initial cloud weight is always 1. At the current operating moment, acquire the current image quality signal and status signal; wherein, the image quality signal includes the image occlusion ratio value, and the status signal includes the sensor power value; The image occlusion ratio is compared with a preset occlusion ratio threshold, and the sensor power value is compared with a preset power threshold. When the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor power value is not less than the preset power threshold, the initial edge weight is used as the first weight and the initial cloud weight is used as the second weight. When the screen occlusion ratio is greater than the preset occlusion ratio threshold and the sensor power value is not less than the preset power threshold, a preset first weight compensation value is extracted, the initial cloud weight is deducted from the first weight compensation value and used as the second weight, and the initial edge weight is superimposed on the first weight compensation value and used as the first weight. When the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor power value is less than the preset power threshold, a preset second weight compensation value is extracted, the initial edge weight is deducted from the second weight compensation value and used as the first weight, and the initial cloud weight is superimposed on the second weight compensation value and used as the second weight. When the screen occlusion ratio is greater than the preset occlusion ratio threshold and the sensor power value is less than the preset power value threshold, the first weight and the second weight are reset to the preset degradation weight, and a data source degradation alarm signal is output.

[0010] Furthermore, when the type matching result is consistent, the fusion confidence is not less than a preset second preset threshold, and the fusion confidence is less than a preset first threshold, a manual review prompt for suspected violations is triggered; when the type matching result is consistent, and the fusion confidence is less than the second preset threshold, the current identified event is marked as pending observation and the warning is filtered; when the type matching result is inconsistent, a pending review signal for type conflict is output.

[0011] Furthermore, the step of triggering a violation warning for the personnel ID based on the spatiotemporal alignment data includes: extracting the personnel ID and location information from the spatiotemporal alignment data; sending an on-site alarm command to the corresponding edge sensing terminal based on the personnel ID, driving the alarm worn by the corresponding worker to perform on-site reminder; sending a linkage tracking command to the PTZ control terminal based on the location information, driving the camera device to automatically track the worker corresponding to the location information and record video synchronously; and sending an alarm message containing the personnel ID, the location information, and the corresponding video recording to the remote monitoring platform.

[0012] Another embodiment of the present invention provides a cloud-edge collaborative high-altitude operation violation detection device, comprising: a first acquisition module, a second acquisition module, a spatiotemporal alignment module, a weight allocation module, a weighted fusion module, a type comparison module, and an early warning triggering module; the first acquisition module is used to acquire a first identification result generated based on sensor data from an edge sensing terminal, the first identification result including a personnel ID, a first timestamp, a first violation type, and a first confidence level; the second acquisition module is used to acquire a second identification result generated based on a cloud video stream, the second identification result including location information, a second timestamp, a second violation type, and a second confidence level; the spatiotemporal alignment module is used to match the first timestamp with the second timestamp, and when the match is successful, to perform a mapping association between the location information and the personnel ID to obtain spatiotemporal aligned data; the weight allocation module is used to acquire the image quality signal of the cloud video stream and the state signal of the sensor, and to perform dynamic weight allocation based on the image quality signal and the state signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level; The weighted fusion module is used to perform a weighted summation of the first confidence level and the second confidence level based on the first weight and the second weight to obtain a fused confidence level; the type comparison module is used to compare the first violation type and the second violation type to obtain a type matching result; the warning triggering module is used to trigger a violation warning for the person ID based on the spatiotemporal alignment data when the type matching result is consistent and the fused confidence level is not less than a first preset threshold.

[0013] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a cloud-edge collaborative high-altitude operation violation detection method of the present invention.

[0014] Another embodiment of the present invention provides a computer-readable storage medium, including: a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the cloud-edge collaborative high-altitude operation violation detection method of the present invention.

[0015] The embodiments of the present invention have the following beneficial effects: This invention provides a cloud-edge collaborative method for detecting violations in high-altitude operations. It involves acquiring a first recognition result generated from sensor data from an edge-sensing terminal and a second recognition result generated from a cloud-based video stream. The first and second timestamps are matched; upon successful matching, the location information is mapped and associated with the personnel ID to obtain spatiotemporally aligned data. Then, the image quality signal from the cloud-based video stream and the sensor's status signal are acquired. Dynamic weight allocation is performed based on the image quality signal and the status signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. The first and second confidence levels are weighted and summed according to the first and second weights to obtain a fused confidence level. Finally, the first violation type and the second violation type are compared to obtain a type matching result. When the type matching result is consistent and the fused confidence level is not less than a first preset threshold, a violation warning is triggered based on the spatiotemporally aligned data for the personnel ID.

[0016] By adopting the above technical solution, this invention addresses the shortcomings of existing conventional violation monitoring methods, which are easily affected by environmental obstructions or sensor aging when dealing with complex high-altitude operation scenarios, leading to misjudgments of status and resulting in high false alarm rates and uncontrollable violation warnings. By acquiring the image quality signal of the cloud video stream and the status signal of the sensor, a dynamic weight allocation and weighted summation fusion decision mechanism is constructed. At the same time, timestamp matching and violation type comparison are used to eliminate the distortion of violation data caused by local environmental interference, realize the accurate locking of real violations in high-altitude operations, significantly reduce the false alarm rate in complex operation environments, and improve the accuracy and reliability of high-altitude operation violation detection. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a cloud-edge collaborative method for detecting violations in high-altitude operations, provided by an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the device for a cloud-edge collaborative high-altitude operation violation detection method provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0022] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0025] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0026] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0027] Specifically, in the step descriptions of the various method embodiments below, the executing entity involved is collectively referred to as the "system". It should be understood that the "system" does not specifically refer to a single physical computing device, but rather to the overall logical architecture for implementing the solution of this application. In actual physical deployment, the "system" encompasses edge sensing terminals worn by operators, cloud collaboration platforms deployed remotely, and bidirectional heterogeneous data verification units connecting the two, etc.

[0028] Those skilled in the art should understand that the parameter acquisition, spatiotemporal alignment, dynamic weight calculation, logical comparison, and early warning triggering operations involved in the following steps, without departing from the core logic of this application, can be flexibly configured and executed in the microcontrollers of edge devices, cloud server clusters, or the distributed collaborative interaction of both, based on the actual industrial site's computing power distribution, network bandwidth, and communication latency. Therefore, the phrase "system performs a certain operation" used in the following concise description includes both cases where a single hardware entity or underlying software completes the operation independently, and cases where multiple devices in the cloud and edge collaborate in a distributed manner. It should not be narrowly interpreted as a strict limitation on the specific physical execution location of the algorithm or a specific hardware entity.

[0029] To address the problem that existing violation monitoring methods are easily affected by environmental obstruction or sensor malfunctions, leading to misjudgments, one embodiment of the present invention provides a cloud-edge collaborative method for detecting violations in high-altitude operations, comprising: Step S1: Obtain the first identification result generated based on sensor data from the edge sensing terminal. The first identification result includes personnel ID, first timestamp, first violation type, and first confidence level. In a preferred embodiment, obtaining the first recognition result generated from sensor data based on the edge-sensing terminal includes: acquiring sensor data and performing moving average filtering and normalization on the sensor data to obtain preprocessed time-series data; inputting the preprocessed time-series data into a pre-constructed one-dimensional convolutional neural network for spatiotemporal feature extraction to obtain a feature vector sequence; inputting the feature vector sequence into a long short-term memory network for continuous action modeling and classification to obtain a preliminary result containing the first timestamp, the first violation type, and the first confidence level; reading a pre-stored personnel ID and combining the personnel ID with the preliminary result to obtain the first recognition result.

[0030] Preferably, sensor data is acquired through the following steps, and the sensor data is subjected to moving average filtering and normalization to obtain preprocessed time-series data: First, multimodal time-series sensing data is synchronously acquired through multiple heterogeneous sensors built into the edge sensing terminal, specifically including: acquiring motion frequency signals reflecting human micro-movements using a human vibration detection sensor (e.g., a high-sensitivity piezoelectric ceramic vibration sensor TE 820M1); acquiring the buckle closure / opening level status signals using a seat belt buckle status monitoring sensor (e.g., a miniature reed switch or Hall sensor Allegro A1104 installed inside the buckle); acquiring physical contact signals of a person's hand approaching or leaving the hook using a hook hand monitoring sensor (e.g., a capacitive touch sensor or infrared photocell installed on the hook handle); acquiring continuous distance signals between the hook and the attachment point (e.g., a pole crossarm) using a human proximity sensor (e.g., an infrared ranging sensor or an ultrasonic sensor HC-SR04); ​​and acquiring the three-axis acceleration and three-axis angular velocity signals of the worker using an attitude sensor (e.g., a six-axis inertial measurement unit MPU6050). Because the hardware sampling frequencies of the aforementioned sensors are inconsistent (e.g., the attitude sensor has a sampling rate of 50Hz, while the status switch is interrupt-triggered or polled at a low frequency), this embodiment first uses a set reference sampling rate (e.g., =20Hz) Timestamp alignment is performed on all heterogeneous data; for discrete data sampled asynchronously, linear interpolation or zero-order hold method is used to fill in the values ​​under that timestamp.

[0031] Subsequently, to eliminate high-frequency noise spikes caused by motor vibration and electromagnetic interference at the work site, the aligned continuous numerical data (such as acceleration and distance) were smoothed using a moving average filtering algorithm. Let the input sequence be x[k], the window size be M (e.g., M=5), and the filtered output sequence... The mathematical expression is: ; Finally, considering the significant differences in data dimensions between different sensors (e.g., acceleration is...), To address the issue of gradient vanishing or exploding in neural networks (where the distance is in cm), a minimum-maximum normalization method is used to linearly map the data from all sensor channels to the dimensionless interval [0,1]. The calculation formula is as follows: ; in, and These are the physical minimum and maximum values ​​of the channel within a preset calibration period, respectively, where k represents the discrete time of the current sampling sequence. This represents the dimensionless output value after normalization at the k-th sampling time. After the above processing, the normalized data from multiple channels in parallel is concatenated according to the time step to output the preprocessed time series data.

[0032] Preferably, the preprocessed time-series data is input into a pre-constructed one-dimensional convolutional neural network for spatiotemporal feature extraction to obtain a feature vector sequence through the following steps: To extract action segments with complete semantics from continuous preprocessed time-series data, this embodiment introduces a sliding time window mechanism. The sliding window length is set to T (e.g., T = 2 seconds, corresponding to L = T × ...). =40 sampling frames), with a window sliding stride of S (e.g., S=10 frames). Each time, a two-dimensional matrix of size C×L is extracted as the network input, where C is the total number of sensor channels. The one-dimensional convolutional neural network (1D-CNN) pre-constructed in this embodiment is specifically used to extract local spatial linkage features between sensor channels in a very short time (e.g., the coupling relationship between a hand gripping a hook and a sudden acceleration change). The 1D-CNN network structure preferably includes two stacked convolutional layers and one max pooling layer: the first convolutional layer is set with 16 convolutional kernels of size 3 and a stride of 1; the second convolutional layer is set with 32 convolutional kernels of size 3 and a stride of 1; then a max pooling layer of size 2 is connected for feature dimensionality reduction, retaining the most significant action feature peaks. After the data in the sliding window is forward propagated by the 1D-CNN, the redundant information of the spatial dimension is compressed, and a highly condensed one-dimensional feature vector is output; as the window continues to slide, multiple one-dimensional feature vectors output continuously are concatenated on the time axis, thus forming the feature vector sequence.

[0033] Preferably, the feature vector sequence is input into a Long Short-Term Memory (LSTM) network for continuous action modeling and classification through the following steps to obtain preliminary results containing the first timestamp, the first violation type, and the first confidence level: Considering that violations in high-altitude operations (such as climbing without a hook) are often complex long-sequence actions with strict temporal causal relationships, relying solely on local features of a CNN cannot accurately identify them. This embodiment uses a Long Short-Term Memory (LSTM) network to perform temporal modeling on the feature vector sequence. The LSTM network is preferably configured with two hidden layers, each containing 64 hidden units. The forget gate, input gate, and output gate within the LSTM can effectively solve the problem of gradient vanishing in long sequences, thus remembering long-term historical states such as "the seatbelt was unfastened a few minutes ago." After inputting the feature vector sequence corresponding to N consecutive sliding windows into the LSTM network, the hidden state vector at the last moment of the last layer is taken, connected to a fully connected layer, and the Softmax activation function is used to output the probability distribution set corresponding to each preset category at the current moment. Where K is the preset total number of categories, and the category labels are predefined as: normal, no seatbelt, hook not attached, incorrect fastening method, abnormal climbing posture, etc. The system selects the category label corresponding to the maximum probability in the probability distribution set and defines it as the first violation type; at the same time, the specific value of the maximum probability (i.e., the maximum value output by Softmax, ranging from 0 to 1) is extracted and directly defined as the first confidence level of this judgment; finally, the Unix system time corresponding to the last sampling frame of the current sliding window is extracted as the first timestamp, thereby encapsulating the preliminary result.

[0034] Preferably, the first identification result is obtained by reading the pre-stored personnel ID through the following steps: To solve the addressing problem of subsequent cloud-edge data not being spatially aligned due to the edge device's inability to know the wearer's identity, this embodiment embeds an identity binding module (such as a near-field communication (NFC) module or a low-power Bluetooth (BLE) module) and non-volatile memory (such as a Flash chip) in the edge sensing terminal. Before the operator starts work, the terminal and the operator are wirelessly bound and paired by scanning a code, swiping a work badge, or issuing a mobile APP. The string representing the operator's unique identity (such as employee number, full name spelling, or system-assigned UUID) is written into the non-volatile memory and permanently fixed as the personnel ID. When the preliminary result is output in real time on the LSTM network, the microcontroller of the edge sensing terminal (such as an STM32L4 series low-power MCU) will immediately trigger a read-store instruction to retrieve the personnel ID from the non-volatile memory and combine it with the preliminary result in a lightweight data exchange format for serialization. Specifically, it is preferable to encapsulate it in JSON string format, for example: {"Device_ID":"Edge_001","Personnel_ID":"ZhangSan_12345","Timestamp": 1683014400,"Violation_Type":"Hook not attached","Confidence":0.92,"Sensor_State":"Normal"}; This JSON data packet is the first recognition result of the final complete output.

[0035] In this embodiment, the implementation steps based on edge multimodal sensor fusion and a lightweight CNN-LSTM dual-layer network architecture have the following significant effects: First, by using multi-sensor temporal alignment and normalization, the hard false alarms caused by poor contact or mechanical aging of a single sensor (such as relying solely on microswitches) are overcome from the physical source, improving the robustness of the perception dimension. Second, by using the combination of 1D-CNN and LSTM to extract spatiotemporal features, the edge terminal can directly complete high-precision recognition of complex continuous actions (such as climbing and hooking) on ​​the local microcontroller without uploading massive amounts of raw sensor data to the cloud, greatly saving wireless communication bandwidth and reducing network transmission latency. Finally, by reading the pre-bound personnel ID locally and encapsulating it into JSON format, the isolated sensor data detached from the network has a clear identity and time anchor, laying a data foundation of unified format, clear identity, and reliability for subsequent high-order spatiotemporal anchor alignment and dynamic confidence weighted verification without GPS dependence in the cloud-edge collaborative platform.

[0036] Step S2: Obtain the second recognition result generated based on the cloud video stream. The second recognition result includes location information, a second timestamp, a second violation type, and a second confidence level. In a preferred embodiment, obtaining the second recognition result generated based on the cloud video stream includes: extracting keyframes from the cloud video stream; inputting the keyframes into a pre-trained scene classifier for scene recognition to obtain scene type labels; dynamically loading the corresponding target visual recognition model from a preset model repository according to the scene type labels; and inputting the cloud video stream into the target visual recognition model for personnel tracking and posture estimation to obtain a second recognition result including the location information, the second timestamp, the second violation type, and the second confidence level.

[0037] Preferably, keyframes are extracted from the cloud video stream through the following steps, and the keyframes are input into a pre-trained scene classifier for scene recognition to obtain scene type labels: First, the cloud collaboration platform accesses the real-time high-definition video stream transmitted from the on-site monitoring sphere or fixed camera via a standard streaming media protocol (e.g., RTSP or ONVIF). To balance cloud computing power consumption and real-time recognition, the system extracts keyframe images from the continuous video stream at a preset sampling frequency (e.g., 1 frame per second) and performs size scaling and normalization preprocessing on the keyframes. Subsequently, the preprocessed keyframes are input into the pre-trained scene classifier. Here, the pre-trained scene classifier specifically refers to a neural network model that, before deployment, uses a deep residual network (e.g., ResNet-50) as the backbone network, first performs feature pre-training on a general large-scale image dataset (e.g., ImageNet), and then uses transfer learning to fine-tune the neural network model through multi-classification training on a massive power operation image dataset (including at least 20 typical scenarios such as pole tower operation, scaffolding operation, and distribution transformer operation) with specific operation scene annotations. After the keyframe image undergoes multi-layer convolutional feature extraction and global average pooling by the scene classifier, the fully connected layer outputs the probability distribution of the current scene belonging to various preset operation scenes. The system selects the classification label with the highest probability that exceeds the scene determination threshold (e.g., 0.85) and uses it as the scene type label for the current video segment (e.g., the output label is "tower operation").

[0038] Preferably, the corresponding target visual recognition model is dynamically loaded from a preset model repository according to the scene type label through the following steps: The preset model repository is specifically a database deployed on a cloud server hard drive or distributed storage system, which pre-classifies and stores multiple independent large-scale visual recognition model weight files and configuration parameters. Because the background of high-altitude operation scenarios is extremely complex, using a single general detection model is easily interfered with by irrelevant components (such as the huge visual differences between the steel pipes of scaffolding and the angle steel of towers). Therefore, this embodiment pre-trains customized dedicated recognition models for different scene type labels (e.g., a hook detection model for tower scenarios and a fall arrestor detection model for scaffolding scenarios). After obtaining the scene type label of the current image, the model loader of the cloud platform sends a call request to the preset model repository to read the target model weight file bound to that label. Subsequently, relying on the high-performance GPU server and deep learning service framework (e.g., TensorFlow Serving or Triton Inference Server) in the cloud, the computation graph and weight parameters of the target model are directly hot-loaded into the video memory, completing the dynamic instantiation of the target visual recognition model. This on-demand loading mechanism avoids the resource exhaustion problem caused by keeping all scene models resident in video memory at the same time.

[0039] Preferably, the cloud video stream is input to the target visual recognition model for personnel tracking and pose estimation through the following steps to obtain a second recognition result containing the location information, the second timestamp, the second violation type, and the second confidence level: First, the target detection sub-network (e.g., YOLOv5) in the target visual recognition model performs frame-by-frame inference on the continuously input video frames to detect the two-dimensional bounding boxes of all workers in the scene. The center pixel coordinates (X, Y) and width and height dimensions of the bounding box are extracted and converted into the relative preset position coordinates of the control ball, which are used as the location information; at the same time, using a multi-target tracking algorithm (e.g., DeepSORT algorithm), based on the intersection-union ratio (IOU) of the detection boxes and the appearance feature extraction network, a continuous cloud temporary tracking ID is assigned to and maintained for each person in the scene. Next, for each tracked region of interest (ROI), the pose estimator network (e.g., OpenPose or HRNet) in the target visual recognition model is used to extract key points of the human skeleton (such as the coordinates of nodes like the wrist, shoulder, and waist), and image segmentation technology is combined to identify the relative physical positional relationship between the seat belt, hook, and pole attachment point. Subsequently, the sequence of key points of the skeleton and the relative positional features of the hook from the above consecutive frames are input into the temporal action classification head to determine whether there is an anomaly in the currently tracked object. If the probability of the anomaly category output by the classification network exceeds a set threshold, the category with the highest probability value is defined as the second violation type (e.g., hook not securely attached), and the specific value of this probability is extracted as the second confidence score; at the same time, the absolute system time (usually uniformly synchronized across the network by the NTP network time protocol) attached to the video frame at the time when the current judgment takes effect is captured, and its Unix timestamp is extracted as the second timestamp. Finally, the cloud platform encapsulates the above-calculated location information, second timestamp, second violation type, and second confidence score together with the cloud temporary tracking ID (e.g., also in JSON format) and outputs it as the complete second recognition result.

[0040] In this embodiment, the above-mentioned scene-adaptive dynamic model loading and cloud-based large-scale visual recognition steps achieve the following significant effects: First, by introducing a two-level cascaded recognition architecture of scene classification + dynamic hot-loading of models, the problems of high false alarms due to background interference and excessively large model parameters caused by a single model in traditional video surveillance are eliminated. By accurately deploying customized models for the current working environment (such as poles or scaffolding), not only can GPU memory consumption be significantly reduced, but the visual false positives caused by complex steel structure environments can also be greatly suppressed by utilizing prior scene background, thus greatly improving the accuracy of visual recognition. Second, by using YOLOv5 in conjunction with large-scale computationally intensive networks such as DeepSORT and OpenPose, high-precision pixel-level skeletal motion capture and relative position segmentation of hanging points are completed in the cloud. Leveraging the computing power advantage of the cloud platform, structured data containing precise location information and action type (second violation type) is accurately output. Under the harsh conditions of lacking absolute physical positioning methods such as GPS, this provides an indispensable visual verification credential for subsequent steps to align the ID reported by the edge perception terminal with the spatiotemporal anchor point using timestamps and action features.

[0041] Step S3: Match the first timestamp with the second timestamp. If the match is successful, map and associate the location information with the personnel ID to obtain spatiotemporal aligned data. In a preferred embodiment, matching the first timestamp with the second timestamp and mapping and associating the location information with the personnel ID to obtain spatiotemporal aligned data specifically includes: calculating the time difference between the first timestamp and the second timestamp; determining that the first timestamp and the second timestamp are successfully matched when the time difference is less than a preset synchronization window threshold and the first violation type and the second violation type meet preset association action matching conditions; after successful matching, extracting the location information from the second recognition result and concatenating the location information with the personnel ID to obtain the spatiotemporal aligned data.

[0042] Preferably, the time difference between the first timestamp and the second timestamp is calculated through the following steps, and when the time difference is less than a preset synchronization window threshold and the first violation type and the second violation type meet the preset associated action matching conditions, the first timestamp and the second timestamp are determined to be successfully matched: First, the bidirectional heterogeneous data verification unit (or spatiotemporal alignment engine) receives the first identification result from the edge end and the second identification result from the cloud. Since the edge sensing terminal inevitably experiences network transmission delay when uploading data via wireless network (such as 4G or Bluetooth Low Energy), and the large-scale visual model in the cloud also has computational time consumption for video frame target detection and pose inference, the absolute system time of the edge sensing event and the cloud visual event arriving at the verification unit will inevitably be asynchronous and misaligned. Therefore, the system directly extracts the underlying timestamps (i.e., the first timestamp T1 and the second timestamp T2) inherent in the two data packets at the moment the action occurs, and calculates their absolute difference ΔT=|T1−T2|. Furthermore, the preset synchronization window threshold is specifically the maximum time concurrency tolerance range (preferably set to 1.0 second in this embodiment) set by the system to tolerate the aforementioned differences in edge hardware sampling frequency, network transmission jitter, and cloud AI inference latency. When ΔT≤1.0 second, the system initially determines that these two heterogeneous data belong to the same physical spatiotemporal section in the time dimension. Further, in order to complete spatial identity locking in the absence of absolute physical location coordinates (such as the absence of GPS or UWB positioning signals), this embodiment introduces preset associated action matching conditions. Specifically, the system internally pre-builds and maintains an edge-cloud heterogeneous action semantic mapping table. Because edge sensors capture physical contact and microscopic motion states (e.g., the first type of violation manifests as a physical break in the buckle or high-frequency acceleration oscillation), while cloud vision captures macroscopic skeletal posture and spatial position relationships (e.g., the second type of violation manifests as visual features such as not wearing a seatbelt or abnormal climbing posture), the two have completely different dimensions of representation. The associated action matching condition is to determine whether the extracted first violation type and second violation type point to the same type of physical violation event in the mapping table (for example, the hook at the edge not being stressed and the hook and hanging point being separated in the cloud vision are defined as strong association matching in the mapping table).

[0043] If so, the violation types of both strictly meet the associated action matching conditions; at this time, when the calculated time difference is also less than the synchronization window threshold, the system determines that the time concurrency feature and the action semantic feature are both consistent. The system officially confirms that the first timestamp and the second timestamp are successfully matched, that is, in the algorithm logic, it is determined that the temporary tracking target that triggered the abnormal action in the cloud image and the real operator entity that reported the abnormal data at the edge are the same person; If not, the violation types of the two do not meet the matching conditions for associated actions. Logically, the system determines that the abnormal data captured by the cloud and edge ends do not belong to the same physical event (it may be that different people violated the rules concurrently or there are false alarms at the underlying level). Thus, the matching fails. At this time, the system directly intercepts and discards the group of unmatched asynchronous data, blocking the subsequent mapping and association process.

[0044] Preferably, after a successful match, the system extracts the location information from the second recognition result through the following steps, and concatenates the location information with the personnel ID to obtain the spatiotemporal alignment data: After confirming successful identity matching based on time-action dual anchor points, the verification engine will perform a cross-modal dynamic coordinate assignment operation. First, the system extracts the location information specific to the cloud-based visual target from the second recognition result. This location information is specifically represented as the two-dimensional pixel coordinate set (X,Y) of the operator in the current high-definition video frame of the surveillance sphere, and the preset position parameters of the pan-tilt-z (horizontal azimuth, vertical pitch, and zoom) of the camera device at this time. Subsequently, the system uses the extracted location information as a newly added spatial dimension attribute field, directly attaches it unidirectionally and injects it into the JSON structure containing the personnel ID reported by the edge, and performs field concatenation and serialization reconstruction. The concatenated data structure breaks down the barrier between virtual tracking identifiers and real-world physical identities, forming an aggregated structured message containing key fields such as {"Personnel_ID":"ZhangSan_12345","Visual_Position": {"Pixel_X": 850,"Pixel_Y": 420,"PTZ": [120.5, 30.2, 2.0]},"Sync_Timestamp":1683014400}. This reconstructed structured message is the final spatiotemporal aligned data output, which is pushed onto the system memory bus as the sole authoritative data credential for subsequent dynamic confidence-weighted fusion and triggering precise PTZ tracking recording.

[0045] In this embodiment, the spatiotemporal alignment implementation steps based on timestamp difference calculation and associated action semantic matching have the following significant effects: First, this solution overcomes the bottleneck of traditional distributed high-altitude monitoring systems, which heavily rely on high-cost, high-power absolute physical positioning hardware such as RTK, GPS, or UWB to achieve edge-cloud data binding. In complex operating environments such as high-altitude power grid towers and dense construction scaffolding, where GPS multipath effects or metal shielding can easily lead to signal loss, this solution cleverly utilizes the concurrent uniqueness of heterogeneous data when the same physical event occurs, based on a soft spatial topology mapping mechanism using time anchor points and action semantic anchor points, to achieve accurate matching and spatial identity binding of heterogeneous cloud-edge data under conditions without positioning hardware. Second, by setting flexible synchronization window thresholds and cross-modal associated action semantic mapping tables, the monitoring system's fault tolerance to network communication jitter, AI computing power inference latency, and differences in edge-cloud data representation is greatly enhanced. This event-feature-based alignment method successfully and seamlessly stitches together the unique identity determined at the edge with the precise location determined by cloud vision. This not only provides a reliable data source pointing to the same object for subsequent confidence fusion decisions, but also avoids the identity mismatch problem caused by the frequent loss of visual tracking IDs due to people crossing over in the image or entering and exiting blind spots. This significantly improves the robustness and engineering feasibility of the cloud-edge collaborative security monitoring system.

[0046] Step S4: Obtain the image quality signal and sensor status signal of the cloud video stream, and perform dynamic weight allocation based on the image quality signal and the status signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. In a preferred embodiment, performing dynamic weight allocation based on the image quality signal and the status signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level specifically includes: At an initial moment, obtaining a preset initial edge weight and an initial cloud weight; using the initial edge weight as the first weight and the initial cloud weight as the second weight; wherein the sum of the preset initial edge weight and the initial cloud weight is always 1. At the current operating moment, acquire the current image quality signal and status signal; wherein, the image quality signal includes the image occlusion ratio value, and the status signal includes the sensor power value; The image occlusion ratio is compared with a preset occlusion ratio threshold, and the sensor power value is compared with a preset power threshold. When the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor power value is not less than the preset power threshold, the initial edge weight is used as the first weight and the initial cloud weight is used as the second weight. When the screen occlusion ratio is greater than the preset occlusion ratio threshold and the sensor power value is not less than the preset power threshold, a preset first weight compensation value is extracted, the initial cloud weight is deducted from the first weight compensation value and used as the second weight, and the initial edge weight is superimposed on the first weight compensation value and used as the first weight. When the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor power value is less than the preset power threshold, a preset second weight compensation value is extracted, the initial edge weight is deducted from the second weight compensation value and used as the first weight, and the initial cloud weight is superimposed on the second weight compensation value and used as the second weight. When the screen occlusion ratio is greater than the preset occlusion ratio threshold and the sensor power value is less than the preset power value threshold, the first weight and the second weight are reset to the preset degradation weight, and a data source degradation alarm signal is output.

[0047] Preferably, the system obtains a preset initial edge weight and an initial cloud weight at the initial moment through the following steps, and uses the initial edge weight as the first weight and the initial cloud weight as the second weight: During the system's initialization or first registration and deployment phase, the confidence fusion decision-maker reads the preset initial edge weight from the cloud configuration center or local configuration file. ) and initial cloud weight ( In this embodiment, considering that edge sensors (such as microswitches and piezoelectric ceramics) worn by workers in high-altitude operations are close-range sensing devices that directly contact the human body, their accuracy in sensing microscopic physical states such as whether a buckle is closed is theoretically superior to long-distance cloud-based visual perception, which is susceptible to interference from ambient light. Therefore, it is preferable to set initial edge weights. =0.6, initial cloud weight =0.4. Meanwhile, the system introduces strict normalization mathematical constraints into its underlying algorithm logic, meaning that regardless of the initial parameter configuration, the sum of their values ​​must always be 1. + =1). After completing the reading and verification of the above system benchmark parameters, the system directly assigns the initial edge weight to the first weight register in the running memory and assigns the initial cloud weight to the second weight register, which serves as the benchmark for subsequent dynamic adjustments.

[0048] Preferably, the following steps are performed to acquire the current image quality signal and status signal at the current operating moment, and compare the extracted image occlusion ratio value with a preset occlusion ratio threshold, and compare the sensor power value with a preset power threshold: At any time window during continuous system operation (i.e., the current operating moment), the cloud collaboration platform will call an image quality assessment algorithm (such as using a semantic segmentation network to extract the intersection area of ​​the tower steel component mask and the personnel bounding box mask), calculate the pixel ratio of the current worker being occluded, generate the image quality signal, and parse out the specific image occlusion ratio value (e.g., 25%) from it; synchronously, the power management module of the edge sensing terminal will collect the discharge voltage of the lithium battery in real time, calculate the remaining power percentage through analog-to-digital conversion, encapsulate it in a data frame and report it, and the system will generate the status signal accordingly, and parse out the specific sensor power value (e.g., 80%) from it. Furthermore, the preset occlusion ratio threshold is specifically a tolerance upper limit set by the system to determine whether the cloud-based visual results still have high confidence reference value; in this embodiment, it is preferably set to 30%. The preset power threshold is specifically a safety red line set by the system to determine whether the edge sensor hardware circuit can maintain a stable operating voltage (to avoid abnormal output levels and garbled characters in the logic gate circuit due to undervoltage); in this embodiment, it is preferably set to 15%. The system calls a comparator function in memory to perform Boolean logic comparison between the dynamically acquired values ​​and the static threshold in real time to drive the subsequent four-branch decision tree.

[0049] Preferably, the dynamic weight allocation mechanism is executed through the following steps: For the first operating condition (both sources are normal): when the image occlusion ratio is not greater than the preset occlusion ratio threshold (e.g., occlusion 25% ≤ 30%) and the sensor battery level is not less than the preset battery level threshold (e.g., battery level 80% ≥ 15%), the system is determined to be in an ideal observation environment. At this time, no weight compensation is required. The system performs a horizontal transmission operation, directly using the initial edge weight (0.6) as the first weight and the initial cloud weight (0.4) as the second weight.

[0050] For the second working condition (visual limitation): when the image occlusion ratio is greater than the preset occlusion ratio threshold (e.g., a worker is heavily obscured by a tower component by 45%) and the sensor power level is not less than the preset power level threshold, the cloud-based vision is determined to be unreliable. In this case, the system extracts a preset first weight compensation value (this compensation value represents the weight penalty step size during visual degradation; in this embodiment, it is preferably set to 0.2). Mathematically, the system subtracts the first weight compensation value (0.2) from the initial cloud weight (0.4) to obtain 0.2, which is then assigned to the second weight. Simultaneously, the initial edge weight (0.6) is superimposed on the first weight compensation value (0.2) to obtain 0.8, which is then assigned to the first weight. Through this weight redistribution, while maintaining the total weight at 1, the decision-making balance is tilted towards the edge sensor, where the data is more reliable.

[0051] For the third operating condition (sensor limitation): when the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor battery level is less than the preset battery level threshold (e.g., the battery level is only 10%, posing a significant risk of false alarms), edge perception is deemed unreliable. In this case, the system extracts a preset second weight compensation (this compensation value represents the weight penalty step size during sensor degradation; in this embodiment, it is preferably set to 0.3). The system subtracts the second weight compensation value (0.3) from the initial edge weight (0.6) to obtain 0.3, which is then assigned to the first weight; and adds the second weight compensation value (0.3) to the initial cloud weight (0.4) to obtain 0.7, which is then assigned to the second weight. At this point, the decision center adaptively shifts to cloud-based vision.

[0052] For the fourth operating condition (the extreme case where both sources are limited): when the image occlusion ratio is greater than the preset occlusion ratio threshold and the sensor battery level is less than the preset battery level threshold (i.e., the video is severely obstructed and the sensor is on the verge of power failure), it is determined that both data streams have lost their reliable reference value. In this case, the system resets the first and second weights to preset degraded weights (e.g., both are forcibly set to 0, or set to a specific invalid flag). This reset operation fundamentally cuts off the subsequent weighted summation calculation link, preventing the forced fusion of two erroneous data points to generate an alarm. Simultaneously, the system sends a data source degradation alarm signal to the upper-layer application, indicating to the backend security personnel that the system has entered a blind spot and immediate manual intervention or notification to on-site personnel to replace the device battery is required.

[0053] In this embodiment, the dynamic weight allocation steps based on dual-parameter threshold comparison and a four-branch matrix, as described above, achieve the following significant effects: First, it breaks away from the rigid calculation mode of fixed weights used in traditional multi-source fusion systems. In the rapidly changing working environment at high altitudes, strong backlight, dense fog, complex metal scaffolding obstructions, and fluctuations in the power of outdoor equipment can all cause a data source to suddenly fail. This solution introduces two physical probes that best characterize the health of the data source: the proportion of screen obstruction and the sensor power level. Through rigorous compensation calculations (deducting and superimposing the first / second compensation values), the system possesses adaptive adjustment capabilities. When the video image is obstructed, the system shifts its focus to the physical sensors; when low sensor power leads to signal instability, the system automatically trusts the macroscopic video surveillance. Secondly, by constructing four strictly mutually exclusive logical branches (i.e., cross combinations of greater than / not greater than), a completely exhaustive and mutually independent truth table decision tree is formed in the algorithm engineering implementation. This eliminates race conditions or logical dead ends when multiple conditions are concurrent. In particular, the degradation interception design for extreme working conditions where both sources are limited prevents the spread of erroneous data in the fusion formula from a mathematical perspective. This enhances the noise resistance, fault tolerance, robustness, and adaptive capability of the cloud-edge collaborative monitoring system in harsh industrial scenarios, providing a solid algorithmic guarantee for finally obtaining a fusion confidence score with high accuracy and low false alarm rate.

[0054] Step S5: The first confidence level and the second confidence level are weighted and summed according to the first weight and the second weight to obtain the fused confidence level; Preferably, the fused confidence level is obtained by weighting and summing the first confidence level and the second confidence level according to the first weight and the second weight through the following steps: First, the bidirectional heterogeneous data verification unit (or confidence fusion decision-maker) extracts the core quantization parameters generated in the previous step from the memory bus: namely, the first confidence from the edge-aware terminal ( ) and the second confidence level from the cloud-based visual model ( Furthermore, both the classification probability output by the edge LSTM network and the prediction probability output by the cloud-based visual classification head have been mapped to the continuous real number interval [0,1] at the end of their respective networks through the Softmax or Sigmoid activation function, representing the degree of certainty of a single-source physical device that the current violation has occurred.

[0055] Subsequently, the system extracts the first weight dynamically allocated in step S4 at the current runtime. ) and the second weight ( The system calls the floating-point multiply-add unit in its underlying logic to multiply the first weight by the first confidence level, multiply the second weight by the second confidence level, and then linearly sum the products. The mathematical formula for this is: ; in, This refers to the calculated fusion confidence level.

[0056] To more clearly illustrate the physical meaning and corrective effect of this weighted summation operation, this embodiment provides a specific calculation scenario: Suppose a worker violates regulations by not securing a hook properly on a pole, but due to the shooting angle, the cloud video feed is partially obscured by structural components. In this case, the cloud visual model outputs a low second confidence score due to feature loss (e.g., ...). =0.35), relying solely on vision would result in missed detections; however, the micro-switches and attitude sensors of the edge sensing terminal are unaffected by visual obstruction, clearly capturing the high-frequency vibration characteristics of the unattached components, and outputting a very high first confidence level (e.g., 0.35). =0.95). According to the dynamic adjustment mechanism in step S4, due to the detection of video occlusion, the system has automatically downgraded the initial cloud weight (0.4) to the second weight. =0.2, the initial edge weight (0.6) is promoted to the first weight. =0.8. Substituting into the weighted summation formula above: =0.8×0.95+0.2×0.35=0.76+0.07=0.83 After the above algebraic calculation, even if the cloud data gives extremely low confidence interference, the total confidence of the system after fusion still reaches a high score of 0.83, successfully maintaining a high sensitivity to real violations.

[0057] It is important to note that, as part of the program's robustness design, the system performs a degradation status interception check before executing the above formula. If, in step S4, it is determined that both sources are restricted and a data source degradation alarm signal is output (i.e., the first and second weights are reset to specific invalid flag bits or 0), then this step directly skips the multiplication-addition operation and forces the fusion confidence to 0 or NaN (not a number) to block the transmission of dirty data to the downstream decision tree.

[0058] In this embodiment, the implementation steps of weighted summation of the confidence scores based on dynamic weights have the following significant effects: First, it completely overturns the rigid OR and AND logic of traditional high-altitude operation monitoring systems from a mathematical model perspective. Traditional OR logic will lead to false alarms if any sensor is falsely triggered, while AND logic will lead to missed alarms if one end is missed. This solution constructs a linear weighted summation model over continuous intervals, smoothly projecting discrete and conflicting single-source judgment results into a unified joint probability scalar. Second, this weighted summation formula perfectly inherits the dynamic weight allocation mechanism of the previous step, achieving the filtering effect of amplifying reliable signals and suppressing interference noise at the computing power execution level. It can adaptively absorb misjudgment interference from harsh environments. Regardless of visual blind spots or sensor anomalies, as long as one end of the data is reliable and given a high weight, the fusion confidence score can be raised to a reasonable range through mathematical calculation, or the false high scores (false alarms) of a single source can be forcibly diluted with low weights. Finally, the extremely complex process of perceiving multimodal heterogeneous data converges and is reduced in dimensionality to a clean, dimensionless numerical value (i.e., a fusion confidence score between 0 and 1). This standardized single parameter decouples the underlying sensing hardware from the upper-level business logic, providing an intuitive and robust mathematical decision-making benchmark for the next step of implementing unified violation type comparison and graded threshold alarms at the decision-making level.

[0059] Step S6: Compare the first violation type and the second violation type to obtain a type matching result; preferably, when the type matching result is consistent, the fusion confidence is not less than the second preset threshold and the fusion confidence is less than the first preset threshold, trigger a manual review prompt for suspected violations; when the type matching result is consistent and the fusion confidence is less than the second preset threshold, mark the current identified event as a pending observation state and filter the warning; when the type matching result is inconsistent, output a pending review signal for type conflict.

[0060] Preferably, the first violation type and the second violation type are compared through the following steps to obtain a type matching result: First, the type comparison module is invoked. Because the first violation type extracted by the edge-aware terminal based on multimodal physical sensing (e.g., the underlying code might be "Edge_Status_Hook_Open", representing a microswitch opening) and the second violation type extracted by the cloud-based collaborative platform based on large-scale image segmentation (e.g., the underlying code might be "Cloud_Visual_Unattached", representing a hook detaching from a crossarm in the image) have inherent heterogeneity in their underlying data structures and semantic representations, the system cannot directly perform a simple string equality judgment. Therefore, this embodiment pre-configures an edge-cloud heterogeneous violation semantic mapping dictionary table in the database. The system uses the codes of the first and second violation types as a joint primary key to perform a hash query in this dictionary table. If a corresponding associated mapping entry is found in the dictionary table, the physical events identified by both ends are determined to be equivalent in business logic, and a type matching result with a boolean value of true is output (i.e., the result represents consistency); if no match is found, a type matching result with a boolean value of false is output (i.e., the result represents inconsistency).

[0061] To achieve refined, tiered early warning and interception, and to avoid rigid alarms caused by a single threshold, this embodiment introduces a tiered judgment benchmark, namely a preset first preset threshold and a preset second preset threshold. Specifically, the first preset threshold is a high-confidence red line set by the system to confirm that the violation is absolutely established and to trigger the highest level of on-site audible and visual alarm (preferably set to 0.7 in this embodiment); the second preset threshold is a basic observation baseline set by the system to distinguish between suspicious actions and normal sensor-level white noise fluctuations (preferably set to 0.4 in this embodiment).

[0062] Preferably, the following steps are taken to trigger a manual review prompt for suspected violations when the type matching result is consistent, the fusion confidence score is not less than the second preset threshold, and the fusion confidence score is less than the first preset threshold: When the type matching result is determined to be consistent, and the system extracts the fusion confidence score output in step S5. Falling into the intermediate transition range (i.e., 0.4≤ When the probability of visual error is less than 0.7, the system determines that the worker's action is in the gray area (e.g., the worker hesitates while holding the hook, causing the edge sensor to vibrate at high frequency, and the cloud image is partially obscured, resulting in a visual probability of only 0.5). In this case, since the fusion confidence level has not reached the absolute confirmation threshold, directly triggering a high-decibel alarm on-site could easily frighten the worker, constituting excessive intervention. Therefore, the system marks the event's data packet status as a suspected violation and pushes an asynchronous manual review prompt to the remote safety monitoring mini-program's backend or the safety officer's monitoring screen via an interface. This prompt automatically includes the spatiotemporal aligned data (including personnel ID and location information) generated in step S3 and a captured 10-second short video, which is then submitted to a human safety officer for final review, thus introducing human intervention when the algorithm is uncertain.

[0063] Preferably, when the type matching result is consistent and the fusion confidence is less than the second preset threshold, the current identification event is marked as an observation state and a warning is filtered: when the type matching result is consistent, but the fusion confidence is extremely low (i.e., When the threshold is <0.4, the system determines that the identification is most likely caused by a minor disturbance due to ambient noise (e.g., a strong wind blowing a seatbelt causing a very weak level fluctuation at the edge, while the cloud mistakenly identifies a bird as not hooked; although the violation type matches, the probability at both ends is extremely low, and after fusion, the threshold is not crossed). To prevent underlying noise from penetrating to the application layer and causing alarm fatigue, the system triggers a warning filtering mechanism. Instead of issuing any audio, visual, or pop-up commands, it silently writes the data record containing the person's ID to the backend log database and marks the event lifecycle as pending observation. This "cold treatment" approach preserves the traceability of all data while ensuring a clean and uncluttered front-end monitoring interface.

[0064] Preferably, when the type matching result is inconsistent, a type conflict verification signal is output through the following steps: When the underlying hash query determines that the type matching result is inconsistent (i.e., False), it means that the edge device and the cloud vision have reached opposite conclusions (e.g., the edge device reports that the seat belt is not fastened, but the cloud vision firmly believes that the seat belt is fastened, but the fall arrestor is not attached). This cross-modal semantic conflict often means that a serious hardware failure has occurred at one end (such as sensor detachment) or extreme visual deception has been encountered (such as strong backlight causing the model to hallucinate). At this time, no matter how high the previously calculated fusion confidence score is, its mathematical basis has collapsed. The system executes the highest priority circuit breaker logic, directly bypassing the confidence score threshold judgment, forcibly generating a type conflict verification signal containing the original conflict code of both ends, and sending the highest level equipment inspection order to the operation and maintenance personnel, requiring manual intervention to verify the equipment health or the actual on-site condition.

[0065] In this embodiment, the implementation steps of type comparison and multi-level threshold decision tree based on heterogeneous semantic mapping have the following significant effects: First, by constructing a heterogeneous violation semantic mapping dictionary table between the edge and cloud, the problem of incompatibility of underlying data structures of sensing devices from different manufacturers and different modalities in distributed IoT systems is cleverly solved, enabling simple physical level signals and high-dimensional image classification labels to be accurately compared at the same business logic level. Second, this solution abandons the one-size-fits-all binary threshold judgment logic in traditional security monitoring. By introducing a hierarchical alarm model that includes a first preset threshold and a second preset threshold, and by dividing the fusion confidence into high, medium, and low three-segment intervals, supplemented by a type conflict circuit breaker mechanism, the system can accurately warn of confirmed violations, manually review questionable actions, silently filter underlying noise, and trigger circuit breaker alarms for device conflicts. This not only prevents invalid false alarms from interfering with on-site operations from the business process perspective, but also constructs a closed-loop, explainable, and anti-fluctuation intelligent decision evaluation system, improving the reliability and availability of the monitoring system.

[0066] Step S7: When the type matching result is consistent and the fusion confidence is not less than the first preset threshold, trigger a violation warning for the personnel ID based on the spatiotemporal alignment data.

[0067] In a preferred embodiment, triggering a violation warning for the personnel ID based on the spatiotemporal alignment data includes: extracting the personnel ID and location information from the spatiotemporal alignment data; sending an on-site alarm command to the corresponding edge sensing terminal based on the personnel ID, driving an alarm worn by the corresponding worker to perform an on-site reminder; sending a linkage tracking command to the PTZ control terminal based on the location information, driving a camera device to automatically track the worker corresponding to the location information and record video synchronously; and sending an alarm message containing the personnel ID, the location information, and the corresponding video recording to the remote monitoring platform.

[0068] Preferably, the personnel ID and location information in the spatiotemporal alignment data are extracted through the following steps when the type matching result is consistent and the fusion confidence is not less than a first preset threshold: First, when the decision engine determines that the type matching result of the current event is true (i.e., the semantic cognition of the illegal physical action is consistent between the end and the cloud), and the fusion confidence calculated in step S5 is... When the threshold is greater than or equal to the absolute confirmation threshold set by the system (i.e., the first preset threshold, such as 0.7), the system logically classifies the event as a confirmed violation. At this time, the system immediately triggers the highest priority differentiated early warning linkage mechanism. To ensure that the alarm command can be accurately sent to the target object without causing false damage, the system first accesses the memory bus and retrieves the spatiotemporally aligned data structure that has been completed with cross-modal binding in step S3. The system's underlying JSON parser accurately extracts the personnel ID, which serves as the unique identifier for communication addressing, and the location information (including the relative coordinate parameters of video pixel coordinates and camera pan / tilt) as the proof of spatial physical coordinates from this structure, and pushes these two core parameters into the linkage distribution queue as input parameters for subsequently driving various hardware actuators.

[0069] Preferably, the following steps are used to send an on-site alarm command to the corresponding edge sensing terminal based on the personnel ID, driving the alarm worn by the corresponding worker to perform on-site alerts: The downlink communication gateway in the system backend reads the extracted personnel ID, performs address matching in the current online device routing table, and establishes a dedicated low-latency wireless communication link (such as through a 4G / 5G network or a LoRa channel relayed by a base station) between the personnel and the specific edge sensing terminal worn by the violating worker. Subsequently, the system sends an on-site alarm command containing a high-priority interrupt signal to this dedicated link. Upon receiving the command, the microcontroller of the edge sensing terminal immediately drives the peripheral alarm hardware circuitry. The alarm system described here preferably includes a multi-level physical alert device: firstly, it drives a built-in wearable vibration motor to generate strong high-frequency physical vibrations, penetrating the noisy construction environment through tactile feedback to provide workers with the most direct sensory warning; simultaneously, it drives the smart safety helmet headset to deliver directional TTS (Text-to-Speech) voice broadcasts (e.g., repeatedly broadcasting "Please note that your hook is not securely fastened, please correct it immediately"); if an external sound and light alarm is configured, it will simultaneously trigger a high-decibel buzzer and a red strobe light. This on-site alert, precisely routed by personnel ID, ensures that the alarm is triggered only for the violator, avoiding the alarm and interference caused to other compliant workers by broadcasting throughout the site.

[0070] Preferably, the following steps are used to send a linkage tracking command to the PTZ control terminal based on the location information, driving the camera device to automatically track the worker corresponding to the location information and record video synchronously: Within the same millisecond-level time slice that triggers the alarm of the front-end edge device, the system's video linkage service module synchronously reads the extracted location information. Based on the pixel coordinates and spatial parameters in the location information, the system calls the coordinate transformation matrix to inversely calculate the PTZ (horizontal, tilt, zoom) control parameters required by the camera device (such as a field-deployed PTZ camera or a high-speed PTZ camera). Subsequently, the system sends a linkage tracking command to the PTZ control terminal of the target camera device through a standard security equipment control protocol (such as ONVIF or Pelco-D protocol). After receiving the command, the camera device's underlying servo motor quickly rotates, aligning the optical center of the image with the violator, and automatically adjusts the lens focal length to obtain a clear close-up image; simultaneously, the camera's built-in automatic target tracking algorithm is activated, ensuring the lens continuously tracks the worker's movement trajectory. The moment the camera locks onto the screen, the system sends a command to the network hard disk recorder or cloud storage node via the RTSP streaming media protocol, forcibly starting the highest bitrate synchronous recording and snapshot operation, automatically capturing 15 seconds of high-definition video clips before and after the violation, thus completing the solidification of tamper-proof visual evidence.

[0071] Preferably, an alarm message containing the personnel ID, location information, and corresponding video recording is sent to the remote monitoring platform through the following steps: To achieve a closed loop on the management side, the system's data distribution engine packages and encapsulates all the aforementioned multi-dimensional data. Specifically, the system encapsulates the personnel ID, location information (which is further converted into specific pole numbers or latitude and longitude coordinates via a GIS system), violation type, and fusion confidence level, along with the corresponding high-definition video clips and close-up screenshots of the violation moment captured in the previous step, into a standard alarm message. Subsequently, the system asynchronously pushes this alarm message to the remote safety monitoring cloud platform. On the large electronic map in the command center, a prominent red coordinate flashing point and a violation details card will immediately pop up; simultaneously, the system will push the alarm message directly to the mobile phone of the safety officer responsible for the work area through the on-site safety monitoring mini-program, reminding the safety officer to immediately view the video evidence on their mobile phone and to manually intervene and deduct points for violations via walkie-talkie or by going to the site.

[0072] In this embodiment, the multi-level differentiated early warning linkage implementation steps based on spatiotemporal alignment data described above have the following significant effects: First, this solution constructs a complete technical closed loop from algorithm recognition to physical intervention, effectively overcoming the technical shortcomings of traditional monitoring systems that are limited to backend alarms and lack real-time front-end feedback, resulting in delayed intervention and easily lost evidence. Through the coordinated use of on-site vibration and voice alerts based on personnel ID-based directional routing, and automatic tracking by location-based camera equipment, this solution achieves both extremely low-latency, localized, and strong intervention against violators and immediate blocking of safety hazards. Furthermore, through target locking and automatic zoom, it automates the retention and preservation of video evidence of violations, significantly improving the substantive protective effectiveness of high-altitude safety equipment.

[0073] Secondly, this early warning and linkage mechanism fully utilizes and verifies the spatiotemporal alignment data and fusion confidence levels output from previous steps, achieving a strict closed loop for global parameter flow. Relying on precise spatiotemporal alignment without absolute positioning dependence, it enables accurate physical addressing of on-site terminal equipment and spatial vector guidance for camera equipment. The dynamic weighted fusion of bidirectional heterogeneous data effectively eliminates environmental interference, providing highly reliable decision support for issuing physical intervention commands, fundamentally avoiding the proliferation of invalid alarms caused by underlying noise or single-source false alarms.

[0074] like Figure 2 As shown, another embodiment of the present invention also provides a cloud-edge collaborative high-altitude operation violation detection device, including: a first acquisition module, a second acquisition module, a spatiotemporal alignment module, a weight allocation module, a weighted fusion module, a type comparison module, and an early warning triggering module; the first acquisition module is used to acquire a first identification result generated based on sensor data from an edge sensing terminal, the first identification result including a personnel ID, a first timestamp, a first violation type, and a first confidence level; the second acquisition module is used to acquire a second identification result generated based on a cloud video stream, the second identification result including location information, a second timestamp, a second violation type, and a second confidence level; the spatiotemporal alignment module is used to match the first timestamp with the second timestamp, and when the match is successful, to perform a mapping association between the location information and the personnel ID to obtain spatiotemporal aligned data; the weight allocation module is used to acquire the image quality signal of the cloud video stream and the state signal of the sensor, and to perform dynamic weight allocation based on the image quality signal and the state signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level; The weighted fusion module is used to perform a weighted summation of the first confidence level and the second confidence level based on the first weight and the second weight to obtain a fused confidence level; the type comparison module is used to compare the first violation type and the second violation type to obtain a type matching result; the warning triggering module is used to trigger a violation warning for the person ID based on the spatiotemporal alignment data when the type matching result is consistent and the fused confidence level is not less than a first preset threshold.

[0075] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can realize the cloud-edge collaborative high-altitude operation violation detection method provided by any of the above-described method embodiments of the present invention.

[0076] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0077] Based on the above-described embodiment of a cloud-edge collaborative method for detecting violations in high-altitude operations, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements any one of the cloud-edge collaborative methods for detecting violations in high-altitude operations of the present invention.

[0078] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0079] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0080] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0081] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the cloud-edge collaborative high-altitude operation violation detection method described in any of the above-described method embodiments of the present invention.

[0082] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0083] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting aerial work violation, characterized in that, include: A first identification result generated from sensor data based on an edge-sensing terminal is obtained, the first identification result including a person ID, a first timestamp, a first violation type, and a first confidence level; Obtain a second recognition result generated based on a cloud video stream. The second recognition result includes location information, a second timestamp, a second violation type, and a second confidence level. The first timestamp is matched with the second timestamp. When the match is successful, the location information is mapped and associated with the person ID to obtain spatiotemporal aligned data. The image quality signal of the cloud video stream and the status signal of the sensor are obtained. Dynamic weight allocation is performed based on the image quality signal and the status signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. The first confidence score and the second confidence score are weighted and summed according to the first weight and the second weight to obtain the fusion confidence score; The first violation type and the second violation type are compared to obtain the type matching result; When the type matching result is consistent and the fusion confidence is not less than the first preset threshold, a violation warning for the personnel ID is triggered based on the spatiotemporal alignment data.

2. The cloud-edge collaboration based aerial work violation detection method of claim 1, wherein, The step of obtaining the first recognition result generated from sensor data based on the edge-sensing terminal includes: acquiring sensor data and performing moving average filtering and normalization on the sensor data to obtain preprocessed time-series data; inputting the preprocessed time-series data into a pre-constructed one-dimensional convolutional neural network for spatiotemporal feature extraction to obtain a feature vector sequence; inputting the feature vector sequence into a long short-term memory network for continuous action modeling and classification to obtain a preliminary result containing the first timestamp, the first violation type, and the first confidence level; and reading a pre-stored personnel ID and combining the personnel ID with the preliminary result to obtain the first recognition result.

3. The cloud-edge collaboration based aerial work violation detection method of claim 2, wherein, The step of obtaining the second recognition result generated based on the cloud video stream includes: extracting keyframes from the cloud video stream, inputting the keyframes into a pre-trained scene classifier for scene recognition, and obtaining scene type labels; dynamically loading the corresponding target visual recognition model from a preset model repository according to the scene type labels; inputting the cloud video stream into the target visual recognition model for personnel tracking and posture estimation, and obtaining a second recognition result including the location information, the second timestamp, the second violation type, and the second confidence level.

4. The cloud-edge collaboration based aerial violation detection method of claim 3, wherein, The first timestamp and the second timestamp are matched. When the match is successful, the location information is mapped and associated with the personnel ID to obtain spatiotemporal aligned data. Specifically, this includes: calculating the time difference between the first timestamp and the second timestamp; when the time difference is less than a preset synchronization window threshold and the first violation type and the second violation type meet preset association action matching conditions, it is determined that the first timestamp and the second timestamp are successfully matched; after successful matching, the location information in the second recognition result is extracted, and the location information is concatenated with the personnel ID to obtain the spatiotemporal aligned data.

5. The cloud-edge collaboration based aerial violation detection method of claim 4, wherein, Dynamic weight allocation is performed based on the image quality signal and the state signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. Specifically, this includes: at the initial moment, obtaining a preset initial edge weight and an initial cloud weight; using the initial edge weight as the first weight and the initial cloud weight as the second weight; wherein the sum of the preset initial edge weight and the initial cloud weight is always 1. At the current operating moment, acquire the current image quality signal and status signal; wherein, the image quality signal includes the image occlusion ratio value, and the status signal includes the sensor power value; The image occlusion ratio is compared with a preset occlusion ratio threshold, and the sensor power value is compared with a preset power threshold. When the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor power value is not less than the preset power threshold, the initial edge weight is used as the first weight and the initial cloud weight is used as the second weight. When the screen occlusion ratio is greater than the preset occlusion ratio threshold and the sensor power value is not less than the preset power threshold, a preset first weight compensation value is extracted, the initial cloud weight is deducted from the first weight compensation value and used as the second weight, and the initial edge weight is superimposed on the first weight compensation value and used as the first weight. When the image occlusion ratio is not greater than the preset occlusion ratio threshold and the sensor power value is less than the preset power threshold, a preset second weight compensation value is extracted, the initial edge weight is deducted from the second weight compensation value and used as the first weight, and the initial cloud weight is superimposed on the second weight compensation value and used as the second weight. When the screen occlusion ratio is greater than the preset occlusion ratio threshold and the sensor power value is less than the preset power value threshold, the first weight and the second weight are reset to the preset degradation weight, and a data source degradation alarm signal is output.

6. The cloud-edge collaborative high-altitude operation violation detection method as described in claim 5, characterized in that, Also includes: When the type matching result is consistent, the fusion confidence is not less than the second preset threshold and the fusion confidence is less than the first preset threshold, a manual review prompt for suspected violation is triggered. When the type matching result is consistent and the fusion confidence is less than the second preset threshold, the current identification event is marked as an observation state and a warning is filtered out. When the type matching result is inconsistent, a type conflict signal to be reviewed is output.

7. The cloud-edge collaborative high-altitude operation violation detection method as described in claim 6, characterized in that, The step of triggering a violation warning for the personnel ID based on the spatiotemporal alignment data includes: extracting the personnel ID and location information from the spatiotemporal alignment data; sending an on-site alarm command to the corresponding edge sensing terminal based on the personnel ID, driving the alarm worn by the corresponding worker to perform on-site reminder; sending a linkage tracking command to the pan-tilt control terminal based on the location information, driving the camera device to automatically track the worker corresponding to the location information and record video synchronously; and sending an alarm message containing the personnel ID, the location information, and the corresponding video recording to the remote monitoring platform.

8. A cloud-edge collaborative high-altitude operation violation detection device, characterized in that, include: The module comprises a first acquisition module, a second acquisition module, a spatiotemporal alignment module, a weight allocation module, a weighted fusion module, a type comparison module, and an early warning triggering module. The first acquisition module is used to acquire a first identification result generated based on sensor data from an edge-sensing terminal. The first identification result includes a person ID, a first timestamp, a first violation type, and a first confidence level. The second acquisition module is used to acquire a second recognition result generated based on a cloud video stream. The second recognition result includes location information, a second timestamp, a second violation type, and a second confidence level. The spatiotemporal alignment module is used to match the first timestamp with the second timestamp. When the match is successful, the location information is mapped and associated with the personnel ID to obtain spatiotemporal alignment data. The weight allocation module is used to acquire the image quality signal of the cloud video stream and the status signal of the sensor, and to perform dynamic weight allocation based on the image quality signal and the status signal to obtain a first weight corresponding to the first confidence level and a second weight corresponding to the second confidence level. The weighted fusion module is used to perform a weighted summation of the first confidence level and the second confidence level based on the first weight and the second weight to obtain a fused confidence level; the type comparison module is used to compare the first violation type and the second violation type to obtain a type matching result; The warning triggering module is used to trigger a violation warning for the personnel ID based on the spatiotemporal alignment data when the type matching result is consistent and the fusion confidence is not less than a first preset threshold.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the cloud-edge collaborative high-altitude operation violation detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the cloud-edge collaborative high-altitude operation violation detection method as described in any one of claims 1-7.

Images