Hot-line work safety wearing time sequence semantic recognition and alarm method and system based on AR glasses

Abstract

The invention discloses a hot-line work safety wearing time sequence semantic recognition and alarm method and system based on AR glasses, and relates to the technical field of hot-line work safety detection.The method comprises the steps that work scene video streams are continuously collected through an AR glasses camera, and a specified time sequence video clip is extracted through a sliding window; detecting the operating personnel and the protective equipment frame by frame by using the first neural network model, and outputting the equipment category and existence confidence; inputting the time sequence fragment and the detection result into a second neural network model based on a Transform architecture, analyzing continuous frame actions and equipment state changes, and identifying a behavior semantic mode; and outputting a risk level according to the semantic mode and triggering a corresponding early warning signal. According to the invention, the problem of false alarm and missing alarm of single-frame detection is solved, real-time performance and accuracy of protection monitoring are realized, and live working safety is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of live-line work safety detection technology, and in particular to a method and system for live-line work safety wearable timing semantic recognition and alarm based on AR glasses. Background Technology

[0002] In the field of live-line work, workers must wear safety protective equipment such as gloves and masks in a standardized manner to avoid risks such as electric shock and arc burns. Therefore, real-time monitoring of the wearing status of protective equipment is crucial.

[0003] Current mainstream detection solutions mostly rely on single-frame image detection models such as YOLOv8, which analyze single-frame images to determine equipment wearing status. However, in real live-line work, violations are often process-based rather than instantaneous, such as workers briefly removing gloves to wipe sweat, masks slipping and requiring subsequent adjustments, or straps loosening and immediately tightening. Single-frame detection models can only capture instantaneous states and are prone to false alarms, such as alarming or missing alarms for brief equipment removal, or failing to detect equipment smoothly slipping off, thus interfering with normal operations or overlooking real risks.

[0004] Meanwhile, existing solutions mostly rely on fixed equipment to acquire images, making it difficult to capture key areas such as hands and head from the worker's first-person perspective. Furthermore, they lack understanding of the worker's intentions and cannot distinguish between active equipment adjustments and passive equipment slippage. In addition, some solutions fail to consider the computing power limitations and privacy protection of edge devices, resulting in high model latency or data transmission risks that could lead to privacy breaches, making them unsuitable for mobile and high-safety-requirement scenarios in live-line work.

[0005] Therefore, there is a need for a method and system for time-series semantic recognition and alarm of safety wearable devices for live-line work based on Augmented Reality (AR) glasses. Summary of the Invention

[0006] To address this issue, the present invention provides a method and system for timing semantic recognition and alarm of safety wearable devices for live-line work based on AR glasses, in order to solve the above-mentioned technical problems.

[0007] This invention provides a method for time-series semantic recognition and alarm of safety wearable devices for live-line work based on AR glasses, including the following method steps: S1 continuously captures video streams of the work scene through the camera of the AR glasses; S2, extract a time-series video segment of a specified length and frame rate from the video stream using a sliding window method; S3, using the first neural network model to perform target detection on each frame of the time-series video segment, identify the workers and the safety protection equipment worn by the workers in the image, and output the detection result of each frame, the detection result including the type of equipment and the existence confidence level; S4, input the time-series video segment and the detection result into the second neural network model to obtain the behavioral semantic pattern corresponding to the time-series video segment output by the second neural network model. The behavioral semantic pattern is used to indicate the operator's use behavior of the safety protection equipment. S5. Based on the behavioral semantic pattern, output the corresponding risk level and trigger the warning signal corresponding to the risk level.

[0008] Preferably, the second neural network model is a lightweight improved TimeSformer model, and the improvement includes at least one of the following: reducing the model width to 25% to 50% of the original model, reducing the number of model layers to 6 to 8 layers, introducing local windowing processing in the spatial attention mechanism, and using a factorized linear attention mechanism to process the time dimension.

[0009] Preferably, inputting the temporal video segment and the detection result into the second neural network model includes: performing feature encoding on the detection result to generate a feature vector, and concatenating the feature vector as an additional token with the visual token after image block embedding processing, and inputting them together into the second neural network model for fusion analysis.

[0010] Preferably, the feature vector includes at least one of the following: the existence confidence of the security equipment, the category one-hot encoding of the security equipment, and the relative position information of the bounding box of the security equipment relative to the image size.

[0011] Preferably, the time-series video segments are extracted from the video stream through the sliding window, wherein the step size of the sliding window is smaller than the window length, so as to achieve overlap between the time-series video segments.

[0012] Preferably, the frame images in the time-series video segment are downsampled before being input into the second neural network model.

[0013] Preferably, a lightweight human pose estimation model is used to extract the human body key point information of the worker from the frame images of the time-series video segment, and the time sequence of the human body key point information is input into the second neural network model to help determine the worker's action intention.

[0014] Preferably, when the behavioral semantic pattern is identified as a brief removal and re-wearing of the garment and the duration of this behavior is less than a preset first time threshold, a low-risk level flexible reminder is triggered; when the behavioral semantic pattern is identified as a continuous failure to wear the garment or an uncorrected slippage and the duration of this state reaches or exceeds a preset second time threshold, a high-risk level sound alarm is triggered and reported to the cloud platform for review.

[0015] Preferably, after the video stream acquisition step, the image information of non-critical areas is automatically blurred, and the processed video clips are uploaded to the cloud only when high-risk events are identified.

[0016] In another aspect, this application also provides a timing semantic recognition and alarm system for live-line work safety wearable based on AR glasses, including: The video stream acquisition module is used to continuously acquire video streams of the work scene through the camera of the AR glasses; The temporal video segment extraction module is used to extract temporal video segments of a specified length and frame rate from the video stream in a sliding window manner; The detection result acquisition module is used to perform target detection on each frame of the time-series video segment using the first neural network model, identify the workers and the safety protection equipment worn by the workers in the image, and output the detection result of each frame, including the type of equipment and the existence confidence level. The behavior recognition module is used to input the time-series video segment and the detection result into the second neural network model to obtain the behavior semantic pattern corresponding to the time-series video segment output by the second neural network model. The behavior semantic pattern is used to indicate the operator's behavior in using the safety protection equipment. The early warning processing module is used to output the corresponding risk level based on the behavioral semantic pattern and trigger an early warning signal corresponding to the risk level.

[0017] This invention leverages the first-person perspective of AR glasses for data acquisition, combines it with sliding window extraction of time-series video segments, and employs YOLOv8 frame-by-frame detection and lightweight TimeSformer temporal semantic analysis to identify behaviors such as brief removal and re-wearing of protective equipment and continuous non-wearing, thus resolving the issues of false alarms and missed alarms in single-frame detection. It incorporates lightweight human posture estimation to assist in judging action intent, distinguishing between active adjustments and passive slippage; its lightweight design adapts to edge devices to reduce latency; and it combines a privacy protection strategy of blurring non-critical areas and reporting only high-risk items to achieve a balance between real-time performance, accuracy, and compliance in live-line work protection monitoring, ensuring operational safety. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0019] Figure 1 The flowchart illustrates a method for timing semantic recognition and alarm of live-line work safety wearable based on AR glasses, provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the second neural network model architecture provided in an embodiment of the present invention.

[0021] Figure 3 This is a flowchart illustrating the process of determining the operator in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the risk level determination and early warning process provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of a timing semantic recognition and alarm system for live-line work safety wearable based on AR glasses, provided as an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this disclosure are information and data authorized by the user or fully authorized by all parties.

[0026] like Figure 1 As shown, this embodiment of the invention discloses a method 100 for timing semantic recognition and alarm of safety wearable devices for live-line work based on AR glasses, including the following method steps: S1 continuously captures video streams of the work scene through the camera of the AR glasses; S2, extract a time-series video segment of a specified length and frame rate from the video stream using a sliding window method; S3, using the first neural network model to perform target detection on each frame of the time-series video segment, identify the workers and the safety protection equipment worn by the workers in the image, and output the detection result of each frame, the detection result including the type of equipment and the existence confidence level; S4, the temporal video segment and the detection result are input together into the second neural network model; the second neural network model is a temporal behavioral semantic understanding model based on the Transformer architecture, which identifies behavioral semantic patterns related to the use of safety protection equipment by analyzing the actions of the workers and the state changes of the safety protection equipment between consecutive frames. S5. Based on the behavioral semantic pattern, output the corresponding risk level and trigger the warning signal corresponding to the risk level.

[0027] In some embodiments, for step S1, the video stream acquisition hardware is a camera built into the AR glasses. The AR glasses, as a carrier, provide a first-person perspective, naturally capturing the wearer's view—the work scene directly in front of the worker, their hand movements, and facial features—providing the optimal viewing angle for accurately determining the wearing status of safety protective equipment, such as gloves and masks. The camera is typically a color CMOS sensor, supporting autofocus and exposure adjustment to adapt to the complex and variable lighting environment of outdoor live-line work sites.

[0028] Optionally, to ensure the accuracy and real-time performance of subsequent analysis, the parameters of the acquired video stream need to be optimized. The frame rate of the video stream is typically set to 25 to 30 frames per second. This frame rate range can smoothly record the continuous process of key actions such as adjusting a person's arm to put on or remove a mask or gloves, avoiding the loss of motion information due to a low frame rate. The initial resolution of the video is usually 720p or 1080p to achieve a balance between ensuring image clarity and processing load.

[0029] Preferably, after the video stream acquisition step, the image information of non-critical areas is automatically blurred, and the processed video clips are uploaded to the cloud only when high-risk events are identified.

[0030] Specifically, after video stream acquisition but before subsequent processing, preliminary processing can be performed according to preset privacy protection rules. For example, the faces of non-operational personnel, such as pedestrians or other background figures, can be automatically blurred in real time. Alternatively, by setting a region of interest, only parts directly related to the operation, such as the upper body and hands of the operator, can be retained, while other background information is filtered out. Thus, privacy protection design is embedded at the data source, adhering to the principle of data minimization.

[0031] In some embodiments, step S2 involves converting a continuous video stream into short segments with clear temporal context that can be analyzed by a time-series model.

[0032] Specifically, the window length refers to the duration of the sequence segment. Based on the analysis of typical violations during live-line work, such as briefly removing gloves to wipe sweat or adjusting a mask after it slips down, each extracted segment contains 8 to 16 frames of images. A window of 8 to 16 frames can capture a complete start and trend of an action that is semantically understandable, without causing system response delays due to an excessively long window.

[0033] Frame rate refers to the temporal density of time segments. In this embodiment, frames can be extracted from the original video stream as needed. For example, to balance computational efficiency and information integrity, an adaptive frame extraction strategy based on motion amplitude can be adopted. That is, when the system detects small differences between consecutive frames, the extraction frequency is reduced; when large motion is detected, the frame extraction rate is increased to ensure that key action information is not missed. Ultimately, this ensures that each segment input into the subsequent model contains sufficient information density in the temporal dimension to discriminate behavioral semantics.

[0034] In one embodiment, the window sliding operation, for example, starts from the beginning of the video stream and first captures the first temporal segment, such as frames 1 to 16. Subsequently, instead of jumping to frame 17 to capture the next segment, the window moves forward in "steps".

[0035] Preferably, the step size is smaller than the window length. For example, the step size is 4 frames, meaning the first segment contains frames 1-16, the second segment contains frames 5-20, the third segment contains frames 9-24, and so on. This results in a 12-frame overlap between adjacent segments, achieving an overlap rate of 75%. This overlap rate ensures that any brief behavioral event, such as the "putting on or taking off gloves" action that occurs and ends within two seconds, will not be interrupted by falling exactly at the boundary of two windows, but will necessarily be captured and analyzed within a complete window.

[0036] Meanwhile, because the window slides rapidly with small steps, it can respond to the latest developments in near real-time. Whenever a new video frame enters the buffer, it can be combined with previous frames to form a new analysis window, thereby keeping the detection latency within the window duration, approximately one second, thus meeting the real-time requirements of security monitoring.

[0037] Thus, through this step, the continuous video stream is transformed into a series of temporally continuous and highly overlapping time-series video segments of fixed length, which obviously carry short but rich spatiotemporal information.

[0038] In some embodiments, the first neural network model is preferably YOLOv8 (You Only Look Onceversion8), which is a single-stage target detector that balances detection speed and accuracy and is suitable for real-time processing by edge computing devices such as AR glasses.

[0039] The model was pre-trained and fine-tuned on an image dataset containing numerous power operation scenarios. Understandably, the dataset is labeled with the target categories to be identified, primarily including: workers, insulating gloves, and protective face shields. The trained model can specifically identify these key targets and safety equipment such as insulating gloves and / or face shields in complex backgrounds; the specific training and fine-tuning process will not be detailed here.

[0040] Next, from each time-series video segment generated in step S2, each independent image frame is extracted sequentially and input into the first neural network model. The model performs forward inference on the input images, extracts features through its convolutional neural network backbone, and uses the detection head to predict the bounding boxes, categories, and confidence scores of all potential targets in the image. Subsequently, post-processing algorithms such as non-maximum suppression are applied to remove redundant and overlapping detection boxes, retaining the most reliable detection result for each target.

[0041] For each frame of image, the model outputs a structured list of detection results. Each element in this list corresponds to a successfully detected target, and for example, includes the following key information: Target category, for example, clearly identifying whether the target is a worker, insulating gloves, or a protective face shield; Existence confidence, for example, is a value between 0 and 1 that represents how confident the model is about the existence of the object and its category. For example, a confidence score of 0.95 for "gloves" means that the model is very confident that gloves exist in the image; Bounding box information, for example, gives the specific location and extent of the target in the image in pixel coordinates, usually represented as [x_center, y_center, width, height].

[0042] Therefore, the time segment provided in step S2, after being processed by this step, will result in a sequence of detection results arranged in chronological order. This sequence constitutes a basic timeline, reflecting the instantaneous changes in the "appearance" and "disappearance" of each protective device within this short period of time.

[0043] For example, the filtered detection results are structured to generate a format that can be used by TimeSformer. Each frame of the detection results includes "target category", "existence confidence", and "bounding box information". "Target category" is represented by an integer ID, such as 1=worker, 2=gloves, 3=face mask, 4=goggles; "existence confidence" is a floating-point number between 0 and 1, such as 0.92, which means that the probability of the target's existence is 92%; "bounding box information" is the normalized coordinates (x_center, y_center, width, height), which is convenient for subsequent calculation of the target's relative position.

[0044] In one embodiment, the detection result of a single frame is stored, as shown in the following example: { "frame_id":3,#frame number "timestamp":1680000000123,#Frame capture timestamp "detections":[ {"class_id":1,"confidence":0.98,"bbox":[0.5,0.4,0.3,0.6]},#Workers {"class_id":2,"confidence":0.92,"bbox":[0.3,0.7,0.1,0.15]},#Left glove {"class_id":2,"confidence":0.89,"bbox":[0.7,0.72,0.11,0.14]},#Right glove {"class_id":3,"confidence":0.95,"bbox":[0.48,0.25,0.12,0.1]}#face mask ] } In some embodiments, for step S4, for the feature encoding processing of the detection results, based on the frame-by-frame detection results output in step S3, the feature extraction object is clearly defined as the core association information of the security protection equipment, specifically including the existence confidence of the security protection equipment, the category one-hot encoding, and the relative position information of the bounding box relative to the image size, to ensure that the extracted features can accurately reflect the equipment status and spatial distribution, and provide effective data support for subsequent semantic analysis.

[0045] Specifically, for the existence confidence feature encoding, for example, the existence confidence of various safety protective equipment, such as gloves, face masks, and goggles, in each frame of detection results is encoded. First, the existence confidence of each type of equipment is sorted according to a preset equipment category order to form an initial confidence sequence. Then, a normalization process is used to uniformly map the confidence values ​​to the [0,1] interval, eliminating the interference caused by the difference in confidence values ​​of different equipment. For example, if the existence confidence of gloves is 0.92, face mask is 0.85, and goggles is 0.90 in a certain frame, the initial sequence after sorting is [0.92, 0.85, 0.90]. After normalization, the original sequence remains unchanged. Since the original confidence is already in the [0,1] interval, the existence confidence feature vector is finally generated.

[0046] For category-specific one-hot encoding, for example, one-hot encoding is used to convert the category information of each type of safety protection equipment. First, a set of equipment categories is constructed, and a unique index number is assigned to each category, such as index 0 for gloves, index 1 for face masks, and index 2 for goggles. For each frame, a vector with the same dimension as the number of categories is generated, with the position value corresponding to the equipment category index set to 1 and the other positions set to 0. For example, when gloves and face masks are detected in a frame, the one-hot vector corresponding to gloves is [1,0,0], and the one-hot vector corresponding to face masks is [0,1,0]. The two vectors are concatenated in category order to obtain the one-hot encoded feature vector of the equipment category for that frame, [1,0,0,0,1,0].

[0047] For encoding the relative position information of the bounding box, for example, the original coordinates (x1, y1, x2, y2) of the bounding box of the security equipment in each frame are first obtained, where (x1, y1) are the coordinates of the upper left corner of the bounding box, and (x2, y2) are the coordinates of the lower right corner. Next, the width (W) and height (H) of the image are calculated. Then, the bounding box coordinates are converted into relative positions relative to the image size, calculated as follows: relative x1 = x1 / W, relative y1 = y1 / H, relative x2 = x2 / W, relative y2 = y2 / H. The four relative position values ​​are then arranged in order to form the relative position information sequence of the equipment's bounding box. If there are multiple pieces of equipment in a frame, the relative position information sequence of each piece of equipment is concatenated according to the equipment category order to generate the relative position information feature vector of the bounding box. For example, in a certain frame, the original coordinates of the glove bounding box are (50, 100, 150, 200), and the image width and height are (400, 600), then the relative position is (0.125, 0.167, 0.375, 0.333), and the relative position of the mask bounding box is (0.4, 0.2, 0.6, 0.3). After concatenation, the relative position information feature vector of the bounding box of this frame is obtained as [0.125, 0.167, 0.375, 0.333, 0.4, 0.2, 0.6, 0.3].

[0048] In this embodiment, the existence confidence feature vector, the category one-hot encoding feature vector, and the bounding box relative position information feature vector are concatenated in sequence to form a complete feature vector corresponding to each frame's detection result. If some feature information is missing in a frame, such as the absence of a certain type of equipment leading to the loss of related features, the missing feature value is set to 0 to supplement it, ensuring that the feature vector dimension is consistent for each frame, which facilitates subsequent model processing.

[0049] In this embodiment, the image block embedding processing for time-series video segments may include image preprocessing, image block segmentation, and image block embedding conversion.

[0050] Specifically, for image preprocessing, each frame of the temporal video segment extracted in step S2 is preprocessed. Preferably, the frame images in the temporal video segment are downsampled before being input into the second neural network model.

[0051] Specifically, the image is first downsampled to a resolution of 224×224 or 256×256 to reduce computational load and adapt to the edge computing capabilities of AR glasses. The downsampling process employs bilinear interpolation to ensure that the key visual features of the safety equipment are preserved even after the image size is reduced, avoiding loss of equipment details due to downsampling. Subsequently, the downsampled image undergoes pixel normalization, mapping pixel values ​​from the [0,255] interval to the [0,1] interval, eliminating the impact of differences in pixel value magnitudes on model training and inference.

[0052] For image patch partitioning, for example, a fixed-size image patch partitioning method can be used to divide each preprocessed frame of image into several non-overlapping image patches. For instance, when the image resolution is 224×224, if the image patch size is set to 16×16, each frame of image can be divided into (224 / 16)×(224 / 16)=196 image patches; when the resolution is 256×256 and the image patch size is 16×16, each frame can be divided into 256 image patches. The choice of image patch size needs to comprehensively consider computational efficiency and feature preservation effect. A size of 16×16 can effectively extract local features of the image while controlling the number of image patches, avoiding excessive computational load on the model.

[0053] For image patch embedding transformation, for example, each image patch after being divided is embedded to convert a two-dimensional image patch into a one-dimensional vector (token). First, each 16×16 image patch is unfolded into a 16×16×3=768-dimensional vector. Assuming the image has RGB three channels, the vector is then linearly transformed through a fully connected layer to map it to a preset embedding dimension, such as 256-dimensional or 384-dimensional, to obtain the visual token corresponding to each image patch.

[0054] Understandably, the choice of embedding dimension needs to be combined with the model's lightweight requirements. For example, the embedding dimension can be adjusted according to the model width. If the original model's embedding dimension is 768, it can be set to 256 or 384 after lightweighting. The model improvement will be elaborated in detail below and will not be repeated here. Finally, the image block embedding tokens of all frames in the temporal video clip are arranged in chronological order to form a temporal visual token sequence.

[0055] Finally, the feature vectors are concatenated with the visual tokens. Specifically, the feature vector corresponding to the detection result of each frame is used as an additional token and concatenated with the visual token after the image patch embedding process for that frame. During concatenation, for example, the order is "visual token first, additional feature token second," ensuring that the token sequence corresponding to each frame contains both image visual information and detection result feature information. For temporal video segments, the token sequences concatenated from all frames are combined in chronological order to form a complete input token sequence. This sequence contains both visual change information from consecutive frames and incorporates the status and spatial feature information of the security equipment, providing comprehensive data input for the second neural network model to perform temporal behavioral semantic understanding.

[0056] In one embodiment, such as Figure 2The diagram shows the architecture of the second neural network model. The second neural network model is based on the Transformer architecture, with a lightweight and improved TimeSformer as its core. The overall architecture includes an input layer, a PatchEmbedding layer, a spatiotemporal attention layer, a fully connected layer, and an output layer.

[0057] The input layer receives the concatenated input token sequence; the spatiotemporal attention layer is responsible for capturing the spatiotemporal correlation features between consecutive frames and analyzing the changes in the actions of workers and the status of safety protection equipment; the fully connected layer further processes and integrates the features output by the attention layer; and the output layer outputs the behavioral semantic pattern recognition results related to the use of safety protection equipment.

[0058] Preferably, the improvement includes at least one of the following: reducing the model width to 25% to 50% of the original model, reducing the number of model layers to 6 to 8 layers, introducing local windowing processing in the spatial attention mechanism, and using a factorized linear attention mechanism to handle the time dimension.

[0059] Specifically, for model width reduction, for example, the width of the original TimeSformer model, i.e., the embedding dimension and the output feature dimension of each layer, can be reduced to 25% to 50% of the original model. For example, if the original model has an embedding dimension of 768 and an output feature dimension of 768, the lightweighted embedding dimension can be set to 192 (approximately 25%), 256 (approximately 33%), or 384 (approximately 50%), with the output feature dimension of each layer reduced proportionally. By reducing the number of model parameters, the computational load and memory usage of the model are reduced, enabling the model to run efficiently on edge devices such as AR glasses. At the same time, by reasonably selecting the reduction ratio, the model's running speed is maximized while ensuring the semantic recognition accuracy of the model.

[0060] For example, adjusting the number of Transformer layers in the original TimeSformer model can be done by reducing the number of Transformer layers from 12 to 6 or 8. This layer adjustment considers both model performance and computational efficiency. A setting of 6 to 8 layers ensures the model has sufficient depth to capture complex temporal behavioral semantic features while avoiding increased computational time due to too many layers. During the layer reduction process, the functions of each layer are optimized, with early layers primarily responsible for extracting basic spatiotemporal features, and mid-to-late-stage layers focusing on recognizing complex behavioral semantic patterns. This ensures the model maintains good semantic understanding capabilities even after the layer count is reduced.

[0061] For spatial attention mechanisms, local windowing is introduced, for example, into the model's spatial attention mechanism. Specifically, firstly, the visual token sequence corresponding to each frame image is divided into several local windows, each containing a fixed number of tokens, such as 4×4 tokens. When calculating spatial attention, attention weights are calculated only within each local window, rather than globally across all tokens in the entire image, significantly reducing the computational cost of spatial attention. Simultaneously, to avoid information isolation between windows caused by local window partitioning, a window shift operation is performed every certain number of layers, such as every two layers, allowing tokens in different windows to interact and ensuring the model can capture global spatial correlation features.

[0062] For example, a time-dimensional factorized linear attention mechanism can be used to process temporal features. Traditional self-attention mechanisms have computational complexity proportional to the square of the sequence length when calculating temporal attention, resulting in enormous computational costs when the number of frames in a temporal video segment is large. The factorized linear attention mechanism reduces the computational complexity to a level linearly related to the sequence length by factoring the attention matrix.

[0063] Specifically, the attention calculation in the time dimension is decomposed into two linear transformation processes. First, the features of each time step are linearly transformed, and then the attention weights are quickly calculated through matrix multiplication. This significantly reduces the amount of computation while effectively capturing the temporal dependencies between consecutive frames, ensuring that the model can accurately analyze the temporal changes in the actions of operators and the status of equipment.

[0064] The concatenated input token sequence is fed into the lightweight and improved TimeSformer model. The model uses the spatiotemporal attention mechanism of each layer to progressively extract and analyze the action characteristics of workers and the state change characteristics of safety protection equipment between consecutive frames.

[0065] Specifically, the early layers of the model first extract basic spatiotemporal features, such as changes in the position of the worker's hands and changes in the appearance of the equipment. As the model deepens, the middle and later layers integrate and abstract these basic features to capture more complex behavioral semantic features, such as the behavioral feature sequence of brief glove removal: hand raised → glove removed → hand lowered → glove put back on. Finally, the output layer of the model outputs the predicted probabilities of various behavioral semantic patterns. The category with the highest predicted probability is selected as the final behavioral semantic pattern recognition result, such as brief removal and put-back of equipment, equipment slippage without correction, continuous absence of equipment, and normal equipment wearing, providing a basis for subsequent risk level judgment and early warning signal triggering.

[0066] In one embodiment, the training of the model can be specifically implemented as follows: A temporal video dataset containing various safety equipment usage behaviors in live-line working scenarios is constructed, covering multiple behavioral semantic patterns such as workers wearing equipment normally, briefly removing equipment, equipment slipping, and continuous absence of equipment. For each temporal video segment in the dataset, a corresponding behavioral semantic category label is annotated, along with the detection results of the safety equipment in each frame, including presence confidence, category, and bounding box position, for supervised learning during model training. The dataset needs to be large enough, containing more than 100,000 temporal video segments, and covering different working environments, worker postures, and equipment types to ensure the model has good generalization ability.

[0067] Cross-entropy loss was chosen as the primary loss function for model training to measure the difference between the predicted probabilities of behavioral semantic patterns output by the model and the true labels. Simultaneously, to improve the model's ability to recognize behavioral semantic patterns in small samples, a focal loss function was introduced. Higher loss weights were assigned to samples that are difficult to classify, such as boundary samples between briefly removing equipment and continuously not wearing equipment, thereby improving the model's classification accuracy for these samples. During training, a dynamic learning rate adjustment strategy was adopted. The initial learning rate was set to 1e-4, and as the training epochs increased, the learning rate was gradually reduced using a cosine annealing algorithm to avoid overfitting in the later stages of training.

[0068] Preferably, in one embodiment, a lightweight human pose estimation model is used to extract the human body key point information of the worker from the frame images of the time-series video segment, and the time sequence of the human body key point information is input into the second neural network model to assist in judging the worker's action intention.

[0069] Specifically, to adapt to the computing power and latency requirements of edge computing scenarios for AR glasses, lightweight human pose estimation models for mobile devices are preferred, such as MobilePose and Lightweight Open Pose, without limitation in this invention. Preferably, the model parameter size is controlled within 10MB, and the single-frame inference time is ≤15ms to avoid consuming too many system resources and affecting the operation of video acquisition and detection models; at the same time, it should support the extraction of at least 17 core human key points, focusing on key points related to the operation of protective equipment, including the head, neck, shoulders, elbows, wrists, and hands, to ensure that key joint changes in actions such as raising hands to adjust masks and removing or putting on gloves can be captured.

[0070] In one embodiment, human keypoint information extraction is based on a lightweight human pose estimation model, such as... Figure 3 As shown, the flowchart for determining the operator's actions may include: S301, taking the temporal video segment extracted in step S2 as input, performs lightweight preprocessing on each frame of the image to ensure that it meets the input requirements of the pose estimation model. Specifically, this includes size normalization: if the resolution of the temporal video segment is inconsistent with the input size of the pose estimation model, a bilinear interpolation algorithm is used to scale the image to keep the aspect ratio unchanged and avoid key point position shifts. Pixel normalization: Normalizes image pixel values ​​from [0,255] to [-1,1] to match the pixel distribution of the model's pre-training data and reduce the interference of lighting changes on key point detection; Region of Interest (ROI) clipping: Based on the worker bounding boxes detected by YOLOv8 in step S3, the ROI region centered on the worker is clipped to remove background interference and reduce the computational load of the model.

[0071] Next, in step S302, the preprocessed image of each frame is input into the lightweight pose estimation model to perform keypoint detection. The specific process may include... The model extracts image features through a backbone network, generates multi-scale feature maps, and captures features from low to high levels. The prediction head outputs two core results: 1) Key point heatmap: Each key point corresponds to a heatmap. The peak position in the heatmap is the approximate coordinate of the key point, and the peak intensity reflects the confidence of key point detection; 2) Coordinate offset: The peak position of the heatmap is finely adjusted to compensate for the coordinate error caused by the downsampling of the heatmap and obtain the accurate pixel coordinates of the key point. Preferably, a key point confidence threshold is set, and key points with confidence below the threshold are filtered out and determined as not detected. If a key joint, such as the wrist, is not detected, linear interpolation is performed based on the coordinates of adjacent joints to complete the sequence, such as wrist coordinate = (elbow coordinate + hand coordinate) / 2, to ensure the integrity of the key point sequence.

[0072] S303 performs structured processing on the key points of each detected human body in each frame, generating a format that facilitates subsequent time-series analysis, specifically including: The keypoint pixel coordinates are converted into normalized coordinates relative to the ROI region of the operator. For example, x_norm=x_pixel / ROI_width, y_norm=y_pixel / ROI_height, eliminating the influence of ROI size differences between different frames. The normalized coordinate range is [0,1]. Following a preset keypoint order, the normalized coordinates and confidence scores of the head → neck → shoulder → elbow → wrist → hand → torso → leg are combined to form a single-frame keypoint feature vector, as shown in the example below: { "frame_id":5,#frame number "timestamp":1680000000500,#Frame capture timestamp "keypoints":[ {"id":0,"x_norm":0.52,"y_norm":0.18,"confidence":0.92},#Head Key Point 1 {"id":1,"x_norm":0.51,"y_norm":0.22,"confidence":0.90},#Head Key Point 2 {"id":7,"x_norm":0.35,"y_norm":0.68,"confidence":0.88},#Left wrist {"id":8,"x_norm":0.65,"y_norm":0.69,"confidence":0.86},#right wrist #Other key points... ] } Subsequently, in step S304, based on the frame-by-frame keypoint structuring results of the time-series video clips, a time-series sequence of keypoints is constructed in chronological order. The specific process includes: Using time segments as units, the key point feature vectors of each frame are concatenated in order of frame number to form a dimension of: number of frames × number of key points × 3, where 3 represents the time tensor of x_norm, y_norm, and confidence. For example, 16 frame segments correspond to a tensor of 16×17×3. Moving average filtering is used to smooth the coordinates of key points, eliminating coordinate jumps caused by single-frame detection noise and ensuring that the time sequence reflects the continuous change trend of the action.

[0073] Subsequently, in step S305, the keypoint temporal sequence is used as an additional temporal feature and fused with the visual token and detection result feature token from step S4. This fusion is then input into the lightweight TimeSformer model. For example, the specific fusion methods include: Perform a linear transformation on the key point time sequence to convert its dimension from frame number × 17 × 3 to frame number × D, where D is the token dimension of TimeSformer, such as 256 or 384, to ensure consistency with the dimensions of the visual token and the feature token of the detection result. The tokens are concatenated within each frame in the following order: visual token (number of frames × N, where N is the number of image patches), detection result feature token (number of frames × M, where M is the detection feature dimension), and key point feature token (number of frames × D) to form the fused token for each frame. Then, the fused tokens of all frames are arranged in chronological order to form the final model input sequence. In the spatiotemporal attention layer of TimeSformer, differentiated attention weights are assigned to tokens of different modalities. For example, higher weights are assigned to "wrist-hand" keypoint tokens and "head-shoulder" keypoint tokens to ensure that the model prioritizes capturing motion features related to the operation of protective equipment.

[0074] Finally, S306, the lightweight TimeSformer model, by fusing key point time series, enables the judgment of the operator's action intention; the model learns the correspondence between key point time series changes and protective equipment operation intentions through training. For example, typical related scenarios include: The intention to remove or put on gloves can be determined as follows: For example, if the temporal coordinate changes of the key points of the left wrist and the key points of the left hand satisfy the following sequence: moving outward from the body → staying for 0.5-1 seconds → moving towards the hand, and the confidence of the existence of the left glove in step S3 is detected to decrease from 0.9 to 0.2 and then increase to 0.8, then the model determines that the intention of the action is to briefly remove the left glove and then put it back on. The intention to adjust the mask can be determined as follows: for example, if the temporal coordinate changes of the head key point and the right wrist key point satisfy the following: the right wrist moves towards the head → the distance to the head key point is <0.1 → stays for 0.3-0.5 seconds → the right wrist moves downward, and the confidence of the mask's existence does not decrease significantly in step S3, then the model determines that the action intention is to adjust the mask position.

[0075] Therefore, with the assistance of key point time sequence, the TimeSformer model can effectively solve ambiguous scenarios that are difficult to judge by vision and detection results alone. For example, it can distinguish between a mask slipping and active adjustment: if only visual frames and detection results are used, the downward movement of the mask position may be misjudged as "slipping"; but if the key point time sequence is combined, if the hand is not detected to move towards the head, it is judged as "mask slipping"; if the hand is detected to move towards the head, it is judged as "active adjustment"; and it can eliminate interference from non-operational actions: if the temporal change of the key points of the operator's hand is a movement towards the tool, the model can identify the intention of the action as "taking the tool" through the key point features, thus eliminating the misjudgment of the semantic pattern of "glove removal and putting on".

[0076] When the model finally outputs the behavioral semantic pattern, it uses the action intent determined by key points as one of the core bases, and makes decisions together with visual features and detection result features. For example, the output results include: Precise semantic tags, such as briefly removing gloves, mask slipping, actively adjusting goggles, and normal operation; Intent confidence, for example, the confidence of the output action intent determination. If the confidence is > 0.8, it is directly used as the semantic determination result; if the confidence is between 0.5 and 0.8, it is further verified by combining the temporal changes of multiple frames; if the confidence is < 0.5, it is marked as "semantic ambiguity" and needs to be supplemented by subsequent frames.

[0077] In some embodiments, for step S5, such as Figure 4 As shown, the risk level determination and early warning process begins with step S401, where the behavioral semantic pattern result output by the second neural network model is received and parsed. This result includes semantic category labels and semantic judgment confidence levels. The semantic category labels must cover the core behaviors of using safety protection equipment in live-line working scenarios, specifically including but not limited to: Normal wearing: Workers wear safety protective equipment in accordance with regulations throughout the entire process, and there are no abnormal changes in the status of the equipment; Temporary removal and re-wearing: Workers temporarily remove a type of equipment and then put it back on within a short period of time; Equipment slippage: The equipment shifts position without being actively removed and is not adjusted in time; Continuous non-wearing: The equipment was removed and not put back on for an extended period of time; Uncorrected slippage: After the equipment slips off, it is not adjusted or the adjustment fails, and it remains in an abnormal state.

[0078] The received semantic pattern results are parsed. First, it is determined whether the semantic judgment confidence level meets the preset threshold (e.g., confidence level ≥ 0.8). If it does, the semantic category label is directly adopted. If the confidence level < 0.8, the second neural network model is triggered to perform secondary inference verification on the time-series video segment. If the confidence level still does not meet the standard after secondary verification, it is marked as "semantic ambiguity" and the risk level is not determined for the time being. It will be re-determined after subsequent frame supplementary analysis.

[0079] S402, based on the timestamp of the video stream acquired in step S1 and the frame timing information of the sliding window in step S2, calculate the duration corresponding to the current behavior semantic pattern. For example, the specific calculation method is as follows: Extract the corresponding acquisition timestamps from the detection results of the first and last frames of the time sequence video segment, and record them as the behavior start time T_start and the current time T_current, respectively. The behavior duration T_duration = T_current - T_start. If there is overlap in the sliding window, the timestamps of adjacent windows are associated through frame numbers to avoid double counting or time breaks. For example, if the timestamp of the last frame of the previous window is T_prev_end and the timestamp of the first frame of the current window is T_curr_start, and if T_curr_start ≤ T_prev_end, then T_prev_end is used as the starting time reference for the current behavior. If there are frame losses or frame completions in the sequential video segment, such as frame loss and completion in step S2, the duration is interpolated and corrected according to the timestamps of the completed frames to ensure that the calculation result error ≤ 100 ms.

[0080] In one embodiment, in S403, two types of time thresholds are preset in combination with the safety specifications for live working and the actual requirements of on-site operations to distinguish different risk levels. Exemplarily, the specific thresholds are set as follows: The first time threshold T1: For the behavior of briefly removing and re-wearing, T1 is preset to 3 seconds. If the re-wearing has not been completed after more than 3 seconds, it is considered a substantial violation. This threshold can be dynamically adjusted according to the type of operation scenario, such as high-voltage live working and low-voltage power distribution operations. For example, in a high-voltage operation scenario, T1 can be shortened to 2 seconds, and in a low-voltage operation scenario, it can be extended to 4 seconds. The second time threshold T2: For the behavior of continuously not wearing or not correcting the slippage, T2 is preset to 5 seconds. If the operation scenario involves high-risk environments such as high altitude and high voltage, T2 can be shortened to 3 seconds to ensure that violations in high-risk scenarios can be quickly identified.

[0081] In one embodiment, in S404, based on the behavior semantic category label and the corresponding duration, in combination with the preset time thresholds, risk level determination is performed. The risk levels are divided into three categories: "no risk", "low risk", and "high risk". Exemplarily, the specific determination logic is as follows: No-risk determination: If the behavior semantic pattern is "worn normally", regardless of the duration, it is directly determined as no risk; If the behavior semantic pattern is "briefly removed and re-worn", and T_duration < T1, but the re-wearing action has been completed, it is determined as no risk.

[0082] Low-risk determination: Core determination condition: The behavior semantic pattern is "briefly removed and re-worn", and T_duration < T1, but the re-wearing action has not been completed; Supplementary determination condition: The behavior semantic pattern is "equipment slippage", and T_duration < T2, and there is a movement trend of the hand key points of the operator towards the equipment, it is determined as low risk.

[0083] High-risk assessment: Condition 1: The behavioral semantic pattern is "not worn continuously", and T_duration≥T2; Condition 2: The behavioral semantic pattern is "slippage uncorrected", and T_duration ≥ T2; Condition 3: The behavioral semantic pattern is "briefly remove and re-wear", but T_duration≥T1, and it is still not re-weared; If any of the above conditions are met, it is considered a high-risk situation.

[0084] In one embodiment, S405, when the risk level is determined to be low risk, a local flexible reminder is triggered on the AR glasses. The reminder method is designed to not interfere with the operator's operation. For example, the specific implementation is as follows: Visual reminder: In the AR glasses' display interface, a virtual flashing border is generated for equipment areas with abnormalities. For example, if gloves are not being worn, a red dotted border is displayed in the hand area, flashing at a frequency of 1 time per second, with the border transparency set to 50% to avoid obstructing the working field of vision; at the same time, a text prompt is displayed in the corner of the interface, such as "Please put on your gloves in time", with a moderate font size; Tactile alerts: If AR glasses are equipped with a vibration module, they can trigger low-frequency vibrations to enhance the alert effect through tactile feedback, without interfering with on-site communication like sound alerts. Termination conditions for reminders: When the S3 step detection result shows that the equipment is put back on, or the behavioral semantic mode changes to normal wearing, the flexible reminder will automatically terminate to avoid invalid reminders interfering with the operation.

[0085] In one embodiment, S406, when the risk level is determined to be high risk, the AR glasses' local sound alarm is first triggered. For example, the specific settings are as follows: Alarm sound selection: The alarm sound is preset with a high frequency warning tone and adjustable sound intensity. The alarm sound uses a combination mode of "short tone and long tone" to distinguish it from other sounds in the work area. Continuous alarm mechanism: The audible alarm will continue to be triggered until the operator manually confirms it or the risk level is reduced to low risk / no risk; if the operator does not confirm it in time, the alarm sound will increase by 10 decibels every 30 seconds to ensure the warning effect.

[0086] While triggering the sound alarm, preferably, the data reporting process to the cloud security management platform is automatically initiated. Following the principles of data minimization and privacy protection, after receiving the reported data, the cloud platform automatically generates a high-risk violation review work order, which is assigned to security management personnel for manual review. The review results will be fed back to the AR glasses in real time. If the review is a "false judgment", the alarm will be terminated immediately, and the semantic judgment parameters of the second neural network model will be adjusted to reduce the probability of subsequent false judgments.

[0087] Figure 5 A timing semantic recognition and alarm system 500 for live-line work safety wearable based on AR glasses is shown. The system implementation is similar to... Figure 1 Corresponding to the illustrated method embodiments, the specific methods include: The video stream acquisition module 501 is used to continuously acquire video streams of the work scene through the camera of the AR glasses; The temporal video segment extraction module 502 is used to extract temporal video segments of a specified length and frame rate from the video stream in a sliding window manner; The detection result acquisition module 503 is used to perform target detection on each frame of the time-series video segment using the first neural network model, identify the workers and the safety protection equipment worn by the workers in the image, and output the detection result of each frame, including the type of equipment and the existence confidence level. The behavior recognition module 504 is used to input the time-series video segment and the detection result into the second neural network model to obtain the behavior semantic pattern corresponding to the time-series video segment output by the second neural network model. The behavior semantic pattern is used to indicate the operator's behavior in using the safety protection equipment. The early warning processing module 505 is used to output the corresponding risk level according to the behavioral semantic pattern and trigger an early warning signal corresponding to the risk level.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An AR glasses-based live working safety wearing timing semantic recognition and warning method, characterized in that, The method comprises the following steps: continuously collecting a video stream of a work scene through a camera of an AR glasses; extracting a time sequence video segment of a specified length and frame rate from the video stream in a sliding window manner; performing target detection on each frame image in the time sequence video segment by using a first neural network model, identifying a work personnel and safety protection equipment worn by the work personnel in the image, and outputting a detection result of each frame, the detection result comprising a category and existence confidence of the equipment; inputting the time sequence video segment and the detection result into a second neural network model to obtain a behavior semantic mode corresponding to the time sequence video segment output by the second neural network model, the behavior semantic mode being used to indicate a use behavior of the work personnel on the safety protection equipment; outputting a corresponding risk level according to the behavior semantic mode, and triggering a warning signal corresponding to the risk level.

2. The live-line work safety wearing time sequence semantic recognition and warning method based on AR glasses according to claim 1, characterized in that: the second neural network model is a TimeSformer model improved in a light weight manner, and the improvement manner comprises at least one of the following: reducing the model width to 25% to 50% of the original model, reducing the model layer to 6 to 8 layers, introducing local windowing processing in the spatial attention mechanism, and using factorized linear attention mechanism to process the time dimension. 3.The AR glasses-based live working safety wearing timing semantic recognition and warning method according to claim 1, characterized in that, the inputting of the time sequence video segment and the detection result into the second neural network model comprises: performing feature coding on the detection result to generate a feature vector, splicing the feature vector as an additional token with a visual token processed by image block embedding, and inputting the spliced token into the second neural network model for fusion analysis.

4. The method of claim 3, wherein the method further comprises: the feature vector comprises at least one of the following information: existence confidence of the safety protection equipment, category one-hot coding of the safety protection equipment, and relative position information of the safety protection equipment boundary box relative to the image size.

5. The live-line work safety wearing time sequence semantic recognition and warning method based on AR glasses according to claim 1, characterized in that: the time sequence video segment is extracted from the video stream through the sliding window, wherein the step length of the sliding window is less than the window length to realize the overlap between the time sequence video segments.

6. The live-line work safety wearing time sequence semantic recognition and warning method based on AR glasses according to claim 1, characterized in that: the frame image in the time sequence video segment is subjected to down-sampling processing before being input into the second neural network model. further comprising:

7. The method of claim 1, wherein the method further comprises: determining whether the AR glasses are worn on the head of the worker; and if the AR glasses are not worn on the head of the worker, outputting a warning message to the worker. extracting human key point information of the work personnel from the frame image of the time sequence video segment by using a light weight human pose estimation model, and inputting a time sequence sequence of the human key point information into the second neural network model to assist in judging the action intention of the work personnel.

8. The live-line work safety wearing time sequence semantic recognition and warning method based on AR glasses according to claim 1, characterized in that: ​ When the behavior semantic pattern is identified as short-term removal and re-wearing and the duration of the behavior is less than a preset first time threshold, a low-risk level flexible reminder is triggered; when the behavior semantic pattern is identified as continuous non-wearing or slipping non-correction and the duration of the state reaches or exceeds a preset second time threshold, a high-risk level sound alarm is triggered and reported to a cloud platform for review.

9. The AR glasses-based live-line work safety wearing time sequence semantic recognition and warning method according to claim 1, characterized in that, After the video stream acquisition step, the image information of the non-critical area is automatically blurred, and only when a high-risk level event is identified, the processed video clip is uploaded to the cloud.

10. An AR glasses-based live working safety wearing timing semantic recognition and warning system, characterized in that, Comprise: a video stream acquisition module for continuously acquiring a video stream of a work scene through a camera of the AR glasses; a time sequence video clip extraction module for extracting time sequence video clips of a specified length and frame rate from the video stream in a sliding window manner; a detection result acquisition module for performing target detection on each frame of image in the time sequence video clip by using a first neural network model, identifying a work personnel and safety protection equipment worn by the work personnel in the image, and outputting a detection result of each frame, the detection result comprising a category and existence confidence of the equipment; a behavior recognition module for inputting the time sequence video clip and the detection result into a second neural network model to obtain a behavior semantic pattern corresponding to the time sequence video clip output by the second neural network model, the behavior semantic pattern being used to indicate a use behavior of the work personnel on the safety protection equipment; a warning processing module for outputting a corresponding risk level according to the behavior semantic pattern and triggering a warning signal corresponding to the risk level.

Images