An ar glasses and a control ball linkage live working site safety management and control method and system

Abstract

The application discloses a live working site safety management and control method and system linked with AR glasses and a control ball, relates to the technical field of image processing, and comprises the following steps: first, acquiring video streams of a first visual angle of AR glasses and a second visual angle of the control ball, and eliminating time offset through time synchronization processing; second, generating a target trajectory sequence through target detection and key point extraction, and determining a cross-view matching target pair; third, inputting the matching target pair trajectory into a cross-view similarity evaluation network CVRN, fusing spatial similarity, time consistency and semantic consistency to obtain a comprehensive reliability score R; fourth, correcting the original confidence of the double visual angles based on R through a confidence correction function; and finally, outputting a safety detection result according to the corrected confidence value and managing and controlling the site. The application avoids the limitation of a single visual angle, improves detection accuracy, and guarantees the safety of live working.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for safety management of live-line work sites that links AR glasses with a surveillance sphere. Background Technology

[0002] Safety management at live-line work sites is a core requirement in the power operation and maintenance field, and its accuracy directly affects the personal safety of workers and the stable operation of the power grid. Currently, most live-line work sites use single-view AI detection solutions, which are mainly divided into two categories: one is close-range detection based on AR glasses, which can capture high-definition information such as protective gear wearing and operational details, but has a narrow field of view, is easily obstructed, and limits the detection scenarios; the other is wide-angle detection based on a surveillance sphere, which can cover the entire work area, but is limited by the shooting distance, has low resolution, and insufficient accuracy in detail recognition.

[0003] In existing technologies, AI detection algorithms such as YOLOv8 and CenterNet rely solely on the confidence level of single-view detection results. When dual-view detection results conflict, such as when there are contradictions in the judgment of protective gear wearing status and personnel actions, blindly selecting high-confidence single-view results can easily lead to misjudgment, making it difficult to meet the stringent requirements for the accuracy and comprehensiveness of safety detection at live-line work sites. Summary of the Invention

[0004] To address this issue, the present invention provides a method and system for safety management of live-line work sites that links AR glasses with a monitoring ball, in order to solve the aforementioned technical problems.

[0005] This invention provides a method for safety management of live-line work sites through the linkage of AR glasses and a surveillance ball, comprising the following steps: Acquire a first-view video stream and a second-view video stream, wherein the first view has a higher image detail resolution than the second view, and the second view has a wider field of view than the first view. The first-view video stream and the second-view video stream are time-synchronized to eliminate the time offset between them and to align the synchronized video frame sequence formed on a unified time base. Target detection and key point extraction are performed on first-view and second-view video frames to obtain the target detection box, category label and first original confidence score in each view, and to generate a target trajectory sequence containing the target's spatiotemporal information. The target trajectory in the first-view video stream is associated with the target trajectory in the second-view video stream to determine cross-view matching target pairs; The trajectory sequence of the matched target pair is input into the cross-view similarity evaluation network CVRN to calculate the comprehensive reliability score R of the matched target pair within a preset time window. The comprehensive reliability score R is formed by fusing the evaluation results of at least three dimensions: spatial similarity, temporal consistency and semantic consistency. Based on the comprehensive reliability score R, the first original confidence score C1 of the first perspective and the second original confidence score C2 of the second perspective in the matched target pair are corrected using a preset confidence correction function f(R) to obtain the first corrected confidence score. Second modified confidence level The confidence correction function f(R) is configured to enhance the confidence level when the value of R increases. Based on the first modified confidence level and the second modified confidence level The numerical relationship is used to output the safety detection results, and the safety control of the live-line work site is carried out based on the safety detection results.

[0006] Preferably, when and When the difference is less than the difference threshold and both are higher than the high confidence threshold, the consistency judgment of the detection results from the two perspectives is adopted as the final result; when and When the difference is greater than the difference threshold and the maximum value is lower than the low confidence threshold, a low confidence alarm requiring manual review is generated.

[0007] Preferably, the time synchronization processing of the first-view video stream and the second-view video stream specifically includes: Time-series signals representing target motion features are extracted from two video streams respectively; based on maximizing cross-correlation, a time offset Δt that maximizes the correlation between the two time-series signals is calculated within a sliding time window; the time offset Δt sequence is smoothed using a low-pass filter, and re-estimation is triggered when a jump in offset exceeds a set tolerance; based on the finally determined time offset Δt, frame alignment or time resampling is performed on the first-view video stream and / or the second-view video stream to achieve synchronization.

[0008] Preferably, determining the cross-view matching target pair based on the spatiotemporal matching algorithm specifically includes: estimating the approximate distance between the target and the camera in each view using a preset average physical size of the target and its pixel size in the image; mapping the center point of the two-dimensional detection box of the target in each view to a unified approximate three-dimensional coordinate system based on the field plane; calculating the spatial distance between the target in the first view video stream and the target in the second view video stream in the unified approximate three-dimensional coordinate system; and determining the trajectory pair with the closest spatial distance and less than the matching threshold as the cross-view matching target pair.

[0009] Preferably, the calculation process of the cross-view similarity evaluation network CVRN includes: using a temporal encoder to extract temporal features from the keypoint sequences and detection box sequences of the first and second perspectives in the matched target pair, respectively, to obtain a first temporal representation S1 and a second temporal representation S2; inputting S1 and S2 into the cross-view interaction module, exchanging information through a cross-attention mechanism, and outputting cross-correlation features M; fusing the cross-correlation features M with the category probability vectors of the two perspectives, and calculating the semantic consistency degree s_sem through a semantic consistency head; calculating the spatial consistency degree s_spat through a spatial consistency head based on the spatial distance and orientation difference of the matched target pair in a unified approximate three-dimensional coordinate system; calculating the temporal consistency degree s_temp through a temporal consistency head based on the comparison of the start and end times of key actions in the two perspectives; and weighting and summing s_sem, s_spat, and s_temp and mapping them through an activation function to obtain the comprehensive reliability score R.

[0010] Preferably, if the first corrected confidence level and the second modified confidence level If the difference is greater than the difference threshold, but the comprehensive reliability score R is higher than the score threshold, then the detection result from the perspective of the party with the higher score in the spatial consistency assessment is selected as the final result.

[0011] Preferably, the cross-perspective interaction module employs a bidirectional cross-attention mechanism to achieve information interaction, specifically including: inputting the first temporal representation S1 into the query projection layer to generate a query vector Q1, and inputting it into the key-value projection layer to generate a key vector K1 and a value vector V1; inputting the second temporal representation S2 into the query projection layer to generate a query vector Q2, and inputting it into the key-value projection layer to generate a key vector K2 and a value vector V2; calculating the similarity between Q1 and K2 to obtain the attention weight matrix of S1 on S2, and weighting and summing V2 based on the attention weight matrix of S1 on S2 to obtain the interaction feature from S2 to S1; calculating the similarity between Q2 and K1 to obtain the attention weight matrix of S2 on S1, and weighting and summing V1 based on the attention weight matrix of S2 on S1 to obtain the interaction feature from S1 to S2; and performing residual connections and layer normalization on the bidirectional interaction features with S1 and S2 respectively, and fusing them to form a cross-correlation feature M.

[0012] Preferably, the confidence correction function f(R) adopts a nonlinear Sigmoid function, expressed as f(R)=Sigmoid(α(R−β)); where α is used to adjust the correction sensitivity so that the change of R in the interval [0,1] is converted into a significant correction magnitude, and β is used as the threshold inflection point; when R>0.5, f(R)>1, which has an enhancing effect on the original confidence and the closer R is to 1, the more significant the enhancement; when R<0.5, f(R)<1, which has a weakening effect and the closer R is to 0, the more obvious the weakening.

[0013] Preferably, based on the classification of safety detection results, at least one of the following control measures is adopted: high-risk control, using audible and visual warnings, or pushing text alerts through AR glasses; or, pushing alarm information to a manual review terminal and sorting it by priority, and triggering high-risk control if the review confirms the danger; or, updating the monitoring status periodically and generating a log.

[0014] In another aspect, this application also provides a live-line work site safety management system that links AR glasses with a surveillance camera, including: The video stream acquisition module is used to acquire a first-view video stream and a second-view video stream, wherein the first-view has a higher image detail resolution than the second-view, and the second-view has a wider field of view than the first-view. The synchronization processing module is used to perform time synchronization processing on the first-view video stream and the second-view video stream to eliminate the time offset between them and align the synchronized video frame sequence formed on a unified time base. The target trajectory sequence generation module is used to perform target detection and key point extraction on first-view video frames and second-view video frames, obtain the target detection box, category label and first original confidence of the target in each view, and generate a target trajectory sequence containing the target's spatiotemporal information. The target pair determination module is used to associate the target trajectory in the first viewpoint video stream with the target trajectory in the second viewpoint video stream to determine the cross-viewpoint matching target pair; The reliability score determination module is used to input the trajectory sequence of the matched target pair into the cross-view similarity evaluation network CVRN, and calculate the comprehensive reliability score R of the matched target pair within a preset time window. The comprehensive reliability score R is formed by fusing the evaluation results of at least three dimensions: spatial similarity, temporal consistency and semantic consistency. The confidence correction module is used to correct the first original confidence C1 from the first perspective and the second original confidence C2 from the second perspective in the matched target pair based on the comprehensive reliability score R and using a preset confidence correction function f(R), to obtain a first corrected confidence. Second modified confidence level The confidence correction function f(R) is configured to enhance the confidence level when the value of R increases. The control module is used to adjust the confidence level based on the first corrected confidence level. and the second modified confidence level The numerical relationship is used to output the safety detection results, and the safety control of the live-line work site is carried out based on the safety detection results.

[0015] This invention utilizes the dual-view linkage of AR glasses and a surveillance sphere. It leverages the high detail resolution of the AR glasses to capture crucial local information during operations, while the surveillance sphere's wide field of view covers the entire scene, avoiding the limitations of a single perspective. Through time synchronization, cross-view matching, and CVRN network fusion, a multi-dimensional consistency reliability score is generated. Combined with the Sigmoid function to dynamically adjust confidence, this solves the problems of dual-view detection conflicts and single-view misjudgments. Detection results are output according to different scenarios and linked to multiple terminals for execution and control, enabling real-time hazard alarms, low-confidence manual review, and continuous safety monitoring. No additional synchronization or depth equipment is required, improving the accuracy and reliability of live-line work safety detection and reducing control risks. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0017] Figure 1 This is a flowchart of a method for safety management of live-line work sites that links AR glasses with a surveillance ball, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of the target pair matching process provided in an embodiment of the present invention; Figure 3 A schematic diagram of the CVRN network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a live-line work site safety management system that links AR glasses with a surveillance ball, provided as an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] like Figure 1 As shown in the figure, this invention discloses a method 100 for safety management of live-line work sites involving AR glasses and a surveillance sphere, comprising the following steps: S1, acquire a first-view video stream and a second-view video stream, wherein the first view has a higher image detail resolution than the second view, and the second view has a wider field of view than the first view; S2, perform time synchronization processing on the first-view video stream and the second-view video stream to eliminate the time offset between them and align the synchronized video frame sequence formed on a unified time base. S3, perform target detection and key point extraction on the first-view video frames and the second-view video frames, obtain the target detection box, category label and first original confidence of the target in each view, and generate a target trajectory sequence containing the target spatiotemporal information; S4, associate the target trajectory in the first perspective video stream with the target trajectory in the second perspective video stream to determine the cross-view matching target pair; S5, input the trajectory sequence of the matched target pair into the cross-view similarity evaluation network CVRN, and calculate the comprehensive reliability score R of the matched target pair within a preset time window. The comprehensive reliability score R is formed by fusing the evaluation results of at least three dimensions: spatial similarity, temporal consistency and semantic consistency. S6, based on the comprehensive reliability score R, the first original confidence C1 of the first perspective and the second original confidence C2 of the second perspective in the matched target pair are corrected using a preset confidence correction function f(R) to obtain the first corrected confidence. Second modified confidence level The confidence correction function f(R) is configured to enhance the confidence level when the value of R increases. S7, based on the first corrected confidence level and the second modified confidence level The numerical relationship is used to output the safety detection results, and the safety control of the live-line work site is carried out based on the safety detection results.

[0020] In some embodiments, for step S1, by deploying dual-view acquisition devices with differentiated characteristics, video data acquisition of the live-line work site is completed in a coordinated manner, ensuring that the image detail resolution of the first view is higher than that of the second view, and that the field of view of the second view is wider than that of the first view.

[0021] Specifically, AR glasses can be used as the first-view acquisition device. These AR glasses are equipped with a high-definition camera with a resolution of at least 4K pixels and a lens focal length of 8-12mm, enabling macro photography. This allows for clear capture of high-definition images of details such as the wear of protective gear on workers' hands, equipment interface operation details, and wire connections, ensuring that the image detail resolution meets the requirements for refined inspection. A surveillance sphere can be used as the second-view acquisition device. This sphere is equipped with a wide-angle camera with a field of view of at least 120° and a lens focal length of 2.8-4mm, supporting 360° panoramic shooting. It is fixedly deployed on a high bracket in the center of the work site, ensuring that the shooting range covers the entire work area, the surrounding safe distance area, and the main equipment distribution area, achieving wide field-of-view coverage. For example, the surveillance sphere is installed at a height of at least 5 meters.

[0022] The acquisition parameters of the AR glasses and the surveillance sphere were uniformly configured, with both set to a sampling rate of 30fps to ensure consistent temporal continuity of the video stream. The image resolution of the AR glasses was set to 3840×2160, and the image resolution of the surveillance sphere was set to 1920×1080. This differentiated resolution configuration further enhanced the ability to capture details from the first-person perspective while controlling the data transmission volume of the surveillance sphere. Both devices were enabled with autofocus and auto exposure functions. The AR glasses focused on operational details within a range of 1-3 meters, while the surveillance sphere focused on panoramic scenes within a range of 5-50 meters, ensuring image clarity at different shooting distances.

[0023] A high-precision time synchronization module is embedded in both acquisition devices. During the acquisition process, each frame of video data is synchronously recorded with a local timestamp. For example, the timestamp format is "year-month-day hour:minute:second:millisecond", ensuring that the time information of each frame of data is traceable. For example, the clock error between the two acquisition devices does not exceed 1ms.

[0024] Optionally, the two devices package the acquired continuous video frames into a time-series frame stream in H.265 encoding format. The frame stream contains metadata such as frame number, timestamp, image data, and device identifier. The frame stream is then transmitted to the backend processing host in real time via 5G or wireless LAN.

[0025] In some embodiments, for step S2, by extracting motion features, estimating time offset, smoothing optimization and frame alignment operations, the time difference between the first-view video stream and the second-view video stream is eliminated, forming a synchronized video frame sequence with a unified time reference.

[0026] Specifically, motion feature time series extraction includes extracting target motion features frame by frame from both video streams. For each frame, the worker area is first located using a human detection algorithm. For example, three types of core motion features are extracted: first, the projected position of the human centroid, obtained by calculating the pixel coordinates (x, y) of the center point of the human detection box; second, the height of key points, selecting the y-axis pixel coordinates of key joints such as the head, shoulders, hands, and waist to reflect the vertical positional changes of the human body; and third, motion energy, calculated by summing the absolute values ​​of the coordinate changes of all extracted key points between two consecutive frames (sum|Δkeypoint|), representing the intensity of the motion. The above features from the two video streams are arranged in time stamp order to form the first-view motion feature time series and the second-view motion feature time series, respectively.

[0027] The time offset estimation specifically includes setting a sliding time window of 3-5 seconds with a window step of 1 second, and segmenting the two motion feature time series. For the feature sequence within each window, for example, the cross-correlation maximization algorithm is used first to calculate the correlation coefficient. All possible time offset values ​​are traversed to find the time offset Δt that maximizes the correlation coefficient between the two sequences. This Δt is the time difference between the two video streams within the current window.

[0028] Optionally, if the feature sequence within the window exhibits non-linear time warping, the Dynamic Time Warping (DTW) algorithm is switched to find the optimal matching path by stretching or compressing one of the sequences, thereby accurately estimating the time offset Δt and ensuring that the offset estimation adapts to different action scenarios. For example, this non-linear time warping is a local inconsistency in action rhythm.

[0029] Offset smoothing and jump reestimation specifically include smoothing the Δt sequence estimated by each sliding window using an exponential smoothing low-pass filter with α=0.3. For example, the calculation formula is Δt_smoothed=α×Δt_new+(1−α)×Δt_old, where Δt_new is the estimated value of the current window and Δt_old is the smoothed value of the previous window. This process eliminates the offset error caused by instantaneous fluctuations and improves the stability of time synchronization.

[0030] For example, the difference in Δt between two adjacent smoothed values ​​is monitored in real time. When |Δt_smoothed_new−Δt_smoothed_old|>100ms, it is determined that the offset has changed significantly. The offset re-estimation process for the current window and the next two windows is immediately triggered, while historical smoothing data is retained to avoid synchronization interruption caused by the change.

[0031] Frame alignment and time base unification specifically include using the timestamp of the control sphere's viewpoint (i.e., the second viewpoint) as the standard time base, and adjusting the frames of the first viewpoint video stream based on the final smoothed time offset Δt. If Δt is positive, the first viewpoint lags behind the second viewpoint, and the first viewpoint video stream is shifted forward by the number of frames corresponding to Δt, where the number of frames = Δt × sampling rate 30fps. If Δt is negative, the first viewpoint is ahead of the second viewpoint, and blank frames corresponding to the number of frames are filled at the beginning of the first viewpoint video stream. These blank frames are marked with a "time alignment frame padding" label.

[0032] Optionally, if the two video streams have inconsistent frame rates or dropped frames, linear interpolation can be used to supplement the motion feature information of the missing frames, or temporal resampling can be used to uniformly map the two video streams to a standard frame rate of 30fps, ensuring that there is a corresponding synchronized video frame at each time point. The final output is a first-view synchronized video frame sequence and a second-view synchronized video frame sequence on a unified time reference, ensuring the temporal consistency of subsequent detection and matching.

[0033] In some embodiments, for step S3, target detection and key point extraction of dual-view video frames are completed based on the pre-trained model, and target spatiotemporal information is associated through a multi-target tracking algorithm to generate a complete target trajectory sequence.

[0034] Specifically, for example, a pre-trained YOLOv8 model is selected as the core model for target detection. This model has been specially optimized and trained for live-line work safety scenarios and supports accurate identification of target categories such as protective gear wearing status and dangerous actions. Among them, protective gear includes safety helmets, insulating gloves, and insulating shoes, and dangerous actions include touching live parts, bending over in violation of regulations, and throwing objects from heights. For example, the pre-trained CenterNet model is selected as the human keypoint extraction model, which can stably output the coordinate information of 17 core human joints; among them, the core human joints include the head, neck, shoulder, elbow, hand, waist, knee, and foot.

[0035] Optionally, the confidence threshold of YOLOv8 is set to 0.3, and detection results below this value are directly filtered; the IOU threshold is set to 0.5 for non-maximum suppression to eliminate duplicate detection boxes; the keypoint confidence threshold of CenterNet is set to 0.4, and only keypoints with the required confidence are retained for subsequent processing to ensure the reliability of the extraction results.

[0036] The single-view target detection process specifically includes inputting the synchronized first-view and second-view video frame sequences frame by frame into the YOLOv8 model for inference. The model outputs the four-dimensional bounding box coordinates (x1, y1, x2, y2) for each detected target, where (x1, y1) are the top-left pixel coordinates and (x2, y2) are the bottom-right pixel coordinates. It also outputs the corresponding target category label, such as "not wearing insulated gloves," "safe operation," or "touching a live conductor." Finally, it outputs the raw confidence score of the detection result, denoted as C1 for the first view and C2 for the second view, with a confidence score range of [0, 1]. Higher values ​​indicate stronger confidence in the detection result. Simultaneously, the bounding box, category label, raw confidence score of each frame are associated and stored with the corresponding frame's timestamp and viewpoint identifier, forming a single-frame detection result set.

[0037] The human body key point extraction process specifically includes cropping the image region within the corresponding detection box for the detected worker target region in each frame of the image and inputting it into the CenterNet model. The model outputs the two-dimensional pixel coordinates (x, y) of 17 human body key points, along with a confidence score for each key point.

[0038] For example, low-confidence keypoints with a confidence level below 0.4 are filtered out. Missing non-critical keypoints are supplemented by interpolation of keypoint coordinates from two adjacent frames to ensure the continuity of the keypoint sequence for each worker target. The extracted keypoint coordinates are associated with the track_id and frame timestamp of the corresponding detected target to form a single-frame keypoint dataset. For example, the missing non-critical keypoint is an occluded hand keypoint in a single frame.

[0039] The target trajectory sequence generation specifically includes, for example, employing a ByteTrack-style monocular multi-target tracking algorithm to continuously track the detected targets in both video streams. A unique and non-repeating track_id is assigned to the first detected target in each video stream, in the format "viewpoint identifier-serial number", such as "AR-001" or "control ball-001". For detected targets in subsequent frames, the same target in adjacent frames is associated by calculating the IOU similarity of the detection boxes, the consistency of the category labels, and the continuity of the keypoint motion trends, thus achieving continuous inheritance of track_id.

[0040] All information corresponding to the same track_id is integrated in time stamp order to generate a target trajectory sequence. The trajectory sequence includes track_id, timestamps of each frame, detection box coordinate sequence, category label sequence, original confidence score sequence, and key point coordinate sequence, which fully records the spatial position change and detection attribute information of the target in the time dimension, providing data support for subsequent cross-view target matching.

[0041] In some embodiments, for step S4, based on the spatiotemporal matching algorithm, cross-view association between the target trajectories of the first and second perspectives is achieved through target approximate distance estimation, unified three-dimensional coordinate system mapping, spatial distance calculation, and matching filtering, thereby determining matching target pairs, such as... Figure 2 The diagram shown illustrates the target pair matching process, and the specific implementation process is as follows: S201, estimate the approximate distance to the target z_obj.

[0042] For each frame of the target trajectory detection results in each viewpoint, the pixel size of the human detection box is extracted: the height pixel value h_pix, which is the vertical pixel difference, and the width pixel value w_pix, which is the horizontal pixel difference, are extracted from the first viewpoint and the second viewpoint respectively. Based on preset camera intrinsic parameters, a proportional conversion method is used to estimate the approximate distance between the target and the corresponding camera, i.e., the approximate target distance z_obj. For example, the distance is first calculated using the detection box height pixel value h_pix combined with the average height H. The formula is derived based on the imaging principle of "target physical size / target pixel size = distance / focal length" to ensure the accuracy of the distance estimation. If the detection box height cannot be accurately extracted due to occlusion or other reasons, the detection box width pixel value w_pix combined with the average shoulder width W is used to supplement the estimation, ensuring the integrity of the target distance data in each frame.

[0043] S202, Unified approximate three-dimensional coordinate system mapping.

[0044] Establish a unified approximate three-dimensional coordinate system based on the site plane of the live-line work site. The projection point of the installation position from the second perspective on the site plane is taken as the origin O(0,0,0). The direction parallel to the long side of the work area in the site plane is set as the X-axis, the direction perpendicular to the long side is set as the Y-axis, and the direction perpendicular to the site plane and upward is set as the Z-axis. The unit of the coordinate system is unified as meters. Pre-enter the external parameter information of the two acquisition devices: including the installation position coordinates (X_cam, Y_cam, Z_cam) of the AR glasses and the control ball in a unified three-dimensional coordinate system, and attitude parameters such as camera pitch angle, roll angle, and yaw angle; For each frame of detection results for each target in each viewpoint, the pixel coordinates (x_img, y_img) of the center point of the two-dimensional detection box are extracted. Combined with the approximate distance z_obj estimated in step S201, the two-dimensional pixel coordinates are mapped to three-dimensional coordinates (X_obj, Y_obj, Z_obj) in a unified three-dimensional coordinate system using the camera extrinsic parameter transformation formula. Z_obj is the projection component of the target approximate distance z_obj in the direction perpendicular to the camera imaging plane, ensuring that the mapping result closely matches the actual spatial position relationship on site.

[0045] S203, calculates the spatial distance of the target across different viewpoints.

[0046] Trajectory set A is the set of all target trajectories traversed from the first perspective, and trajectory set B is the set of all target trajectories from the second perspective. For any trajectory A_i in trajectory set A and any trajectory B_j in trajectory set B, the three-dimensional coordinates (X_Ai, Y_Ai, Z_Ai) and (X_Bj, Y_Bj, Z_Bj) of the two at the same time node are extracted according to the timestamp alignment principle. For example, the Euclidean distance formula is used to calculate the spatial distance d_ij between two points. This distance intuitively reflects the physical proximity of two targets at the same time point. The average spatial distance d_ij between trajectories A_i and B_j over the entire time series is taken to obtain the average spatial distance D_ij between the two trajectories, thus eliminating the influence of single-frame coordinate errors on distance judgment.

[0047] S204, Filter and determine matching target pairs.

[0048] First, a preset spatial distance matching threshold T is set to retain only trajectory pairs (A_i, B_j) with an average spatial distance D_ij less than T in the candidate matching set, filtering out invalid trajectory pairs with excessively large spatial differences. For example, the default configuration of the spatial distance matching threshold is 0.5m, which can be dynamically adjusted according to the size of the work site. Next, the trajectory pairs in the candidate matching set are sorted in ascending order of their average spatial distance D_ij, and the trajectory pair with the smallest D_ij is selected as the initial matching target pair. If multiple trajectories compete for the same matching object, such as A_1 from the first perspective simultaneously satisfying the distance condition with B_1 and B_2 from the second perspective, the distance differences between the competing trajectory pairs are compared, and only the trajectory pair with the smallest distance to the target trajectory and a distance difference greater than 0.1m from all other competing trajectories is retained as the final matching target pair, thus avoiding mismatches of one-to-many.

[0049] In some embodiments, for step S5, the Cross-View Regularization Network (CVRN) is used to complete temporal feature extraction, cross-view information interaction, three-dimensional consistency evaluation, and comprehensive score fusion in stages, outputting a comprehensive reliability score R for the matched target pair; such as Figure 3 As shown, Figure 3 This is a schematic diagram of the CVRN network structure, which specifically includes: a temporal encoder, a cross-view interaction module, a semantic consistency head, a spatial consistency head, a temporal consistency head, and an aggregation layer.

[0050] Specifically, firstly, a preset time window is selected and data preprocessing is performed. This includes, for example, setting a preset time window of 1.5-3 seconds, which can be dynamically adjusted according to the complexity of the operation. 1.5 seconds is used for intensive operation scenarios and 3 seconds is used for smooth operation scenarios. Trajectory sequence segments of matching target pairs are selected within the window according to the timestamp alignment principle. The trajectory sequence segment includes keypoint sequences, detection box sequences, and category probability vectors from both the first and second perspectives. Missing frame data is supplemented by interpolation between adjacent frames to ensure sequence integrity and provide regular input for subsequent feature extraction.

[0051] Specifically, the keypoint sequence includes the continuous frame coordinates of 17 core keypoints, the detection box sequence includes the detection box coordinates and pixel size of continuous frames, and the class probability vector includes the confidence distribution of each class of target detection in each frame.

[0052] Next, temporal features are extracted. Specifically, for example, a 3-layer 1DTemporalConv network is used as the temporal encoder, with a kernel size of 3 and ReLU activation function. The keypoint sequences and detection box sequences from the first and second perspectives of the matched target pair are concatenated as joint input features and input to the temporal encoder for convolution operations to capture the dependencies and trends of target actions in the temporal dimension. Finally, the first temporal representation S1 corresponding to the first perspective and the second temporal representation S2 corresponding to the second perspective are output. The dimensions of the two types of representations are unified as a fixed-dimensional vector to ensure compatibility with subsequent interactive calculations.

[0053] Subsequently, cross-perspective information interaction is carried out. In one embodiment, the cross-perspective interaction module is constructed using a bidirectional cross-attention mechanism, which includes a query projection layer, a key-value projection layer, and an attention calculation layer.

[0054] Specifically, the temporal representations S1 and S2 are first input into the query projection layer and the key-value projection layer, respectively. Through linear transformation, query vector Q, key vector K, and value vector V are generated. S1 generates three sets of vectors through linear transformation: query vector Q1 is used to actively query features from another perspective, key vector K1 is used to be matched by queries from another perspective, and value vector V1 contains the core feature information of S1. Similarly, S2 generates Q2, K2, and V2 through linear transformation, completing the conversion from temporal representation to the vector format required for attention calculation.

[0055] When calculating attention weights, a similarity operation, such as a dot product, is performed between the query vector Q1 of S1 and the key vector K2 of S2 to obtain the attention weight matrix of S1 on S2. This matrix quantifies the importance of each feature dimension in S2 to S1. Based on this weight matrix, the value vector V2 of S2 is weighted and summed to filter and aggregate key information in S2 that is related to S1, forming the information interaction features from S2 to S1. On the other hand, the same logic is executed in reverse: the similarity between Q2 of S2 and K1 of S1 is calculated to obtain the attention weight matrix of S2 on S1. The V1 of S1 is then weighted and summed to generate the information interaction features from S1 to S2, realizing the bidirectional flow of features from both perspectives.

[0056] The bidirectional interactive features, namely the information interaction features from S2 to S1 and from S1 to S2, are combined with the original time series representations S1 and S2 through residual connection and layer normalization to form a cross-correlation feature M. This achieves deep correlation and information complementarity between the two perspective features and captures the consistency features of cross-perspective targets. For example, the residual connection is a residual addition operation.

[0057] Subsequently, a multi-dimensional consistency assessment is performed. Specifically, the semantic consistency score s_sem is calculated by extracting the category probability vectors from the first and second perspectives and concatenating them with the cross-correlation features M to form a semantic fusion feature. The semantic fusion feature is then input into a 3-layer fully connected network, i.e., the semantic consistency head. The output layer of this network uses the Sigmoid activation function to map the output values ​​to the [0,1] interval, thus obtaining the semantic consistency score s_sem. The closer the value is to 1, the more consistent the two perspectives are in their judgment of the target category. The category probability vector can be output by the aforementioned YOLOv8 model, containing the confidence distribution of various protective gear states and action categories. Spatial consistency s_spat calculation: Based on the continuous frame 3D coordinates of the matched target pair in a unified approximate 3D coordinate system, the real-time spatial Euclidean distance d_t between the two viewpoint targets is calculated, and the average spatial distance d_avg within the time window is taken; at the same time, the human pose orientation vector is calculated through the key point coordinates, the angle θ between the two viewpoint orientation vectors is solved, d_avg is normalized to the interval [0,1] and then inverted, and weighted and fused with the cosine value of θ, and then input into the Sigmoid function for mapping, to obtain the spatial consistency s_spat∈[0,1]. The closer the spatial distance and the smaller the orientation difference, the closer s_spat is to 1; The time consistency s_temp is calculated as follows: The start and end times of key actions are determined by detecting peak velocity at key points. The rate of change of each key point's coordinates within the time window, i.e., its velocity, is calculated. The maximum velocity of all key points is taken as the peak value. The first frame where the peak appears is the start time t_start, and the last frame where the peak disappears is the end time t_end. The start and end times (t_start1, t_end1) and (t_start2, t_end2) of the actions from both perspectives are obtained. The normalized result of the difference between the start and end times is calculated and converted into a time consistency s_temp ∈ [0,1] using a reverse mapping. The smaller the time difference, the closer s_temp is to 1, reflecting the time synchronization of the actions from both perspectives.

[0058] Finally, the aggregation layer uses a combination of weighted summation and activation function mapping to fuse the three-dimensional consistency. Specifically, initial weights are first assigned to s_sem, s_spat, and s_temp, and the weighted sum S = w1 × s_sem + w2 × s_spat + w3 × s_temp is calculated, with a bias term b added. The weighted sum S is then input into the Sigmoid activation function, and the output is mapped to the [0,1] interval to obtain the comprehensive reliability score R. The closer the R value is to 1, the stronger the consistency of the matching target pair in the spatial, temporal, and semantic dimensions, providing a basis for subsequent confidence correction.

[0059] The initial weights w1=w2=w3=1 / 3 corresponding to s_sem, s_spat, and s_temp can be optimized and adjusted through model training. For example, the superimposed bias term b is preset to 0.1 to balance the base score.

[0060] In some embodiments, for step S6, based on the comprehensive reliability score R, the original confidence level of the two perspectives is dynamically adjusted using a preset confidence level correction function f(R) to enhance or weaken the confidence level, thereby obtaining a first corrected confidence level. Second modified confidence level .

[0061] Specifically, the raw confidence scores are first extracted and preprocessed. This includes extracting the raw confidence score data of the matching target pairs within a preset time window from the target trajectory sequences of the first and second perspectives. The raw confidence scores of the first perspective are denoted as C1, and the raw confidence scores of the second perspective are denoted as C2. Both are values ​​in the [0,1] interval output by the YOLOv8 model.

[0062] The average raw confidence score of multiple frames within the time window is taken to obtain the average raw confidence score (C) for a single viewpoint. 1_ avg, C 2_(avg) to eliminate the impact of single-frame detection noise on confidence; if there are outliers in the confidence (such as extreme values ​​below 0.1 or above 0.95) within the window, for example, the 3σ principle is used to remove them before calculating the average value to ensure the stability and representativeness of the original confidence.

[0063] In one embodiment, a nonlinear Sigmoid function is selected as the confidence correction function f(R). This function has the nonlinear characteristic of "weak correction for low R values ​​and strong enhancement for medium and high R values", which can match the influence of cross-perspective consistency on confidence. For example, the function expression is f(R)=Sigmoid(α(R−β)).

[0064] For example, α is set to 6 to adjust the sensitivity of the correction function, ensuring that changes in the R value within the [0,1] interval can be converted into significant confidence correction magnitudes; β is set to 0.5 as the threshold inflection point of the correction function. When R>0.5, f(R)>1, which enhances the original confidence level, and the closer R is to 1, the more significant the enhancement effect; when R<0.5, f(R)<1, which weakens the original confidence level, and the closer R is to 0, the more obvious the weakening effect. The purpose is to enhance confidence level when the consistency is high and reduce confidence level when the consistency is low.

[0065] Finally, the average original confidence level C of the first-person perspective is calculated. 1_ The average original confidence level (avg) and the second-view average original confidence level (C2_avg) are multiplied by the output value of the confidence level correction function f(R) to obtain the corrected confidence level. The specific calculation formula is as follows: =C 1_ avg×f(R), =C 2_ avg×f(R). After calculation, boundary constraints are applied to the modified confidence level: if the result is greater than 1, it is truncated to 1; if it is less than 0, it is truncated to 0, ensuring... and It remains within the [0,1] interval, which conforms to the numerical definition of confidence level and provides standardized data for subsequent result determination.

[0066] In some embodiments, for step S7, based on the first modified confidence level Second modified confidence level The numerical relationship and comprehensive reliability score R are used to output safety test results according to multi-scenario judgment rules, and to link the on-site control system to perform safety control operations.

[0067] Specifically, firstly, three core judgment thresholds are preset, which can be dynamically adjusted according to the risk level of the work scenario. For example, these include: Difference threshold ε: set to 0.1, used to determine the consistency of the confidence level corrected by the two perspectives; High confidence threshold τ_high: set to 0.7, used to define detection results with high confidence; Low confidence threshold τ_low: set to 0.4, used to define detection results with low confidence; The scoring threshold R_th is set to 0.7 to filter confidence conflict scenarios based on high consistency.

[0068] Next, in one embodiment, the decision is made according to the logic of consistency priority, low confidence fallback, and high consistency conflict arbitration, specifically, Scenario 1: Dual-view consistency determination. Calculation. and absolute difference | - |, if| - |<ε and max( , If the value of τ_high is greater than the target value, the detection results from the two perspectives are determined to be highly consistent. The consistent category label from the two perspectives is adopted as the final security detection result. Simultaneously, the maximum value is set to τ_high. , This serves as the final confidence level, enhancing the credibility of the results. Scenario 2: Low-confidence alarm determination. If | - |≥ε and max( , If the confidence level is less than τ_low, the detection result is deemed insufficiently reliable, and a low-confidence alarm result of "requiring manual review" is generated. For example, the alarm information includes dual-view trajectory segments, original confidence level, corrected confidence level, comprehensive reliability score R, and difference value, providing complete data support for manual review; Scenario 3: Highly Consistent Conflict Arbitration Determination. If | - If |≥ε but R>R_th, it means that although the confidence levels of the two perspectives conflict, they have a high degree of consistency in the spatial, temporal, and semantic dimensions. At this point, the conflict arbitration mechanism should be initiated. For example, the spatial consistency s_spat of the cross-view similarity assessment network output is extracted. The category label and modified confidence corresponding to the view with higher s_spat are selected as the final result. Since spatial consistency directly reflects the degree of physical location matching of the target in both views, its priority is higher than that of a single confidence value.

[0069] In one embodiment, for scenario one, if the final detection result is in the "dangerous" category, such as "touch of a live conductor" or "not wearing a safety helmet", an instruction is immediately sent to the audible and visual alarm device at the work site to trigger a high-frequency audible and visual alarm. At the same time, a text warning message is pushed to the AR glasses worn by the worker, such as "Dangerous operation detected, please stop immediately".

[0070] For example, the sound level of a high-frequency audible and visual alarm is ≥85dB, and the flashing frequency of the light is ≥2Hz.

[0071] For Scenario 2, after generating a low-confidence alarm, the alarm information is pushed to the manual review terminal in real time, such as the manager's mobile APP, the back-end monitoring screen, etc., and the review priority is marked; if the review determines it to be dangerous, the high-risk control action of Scenario 1 is triggered again; where the review priority is set according to max( , The difference between τ_low and τ_low is obtained by reverse sorting.

[0072] If the final detection result is "safe", the real-time detection result will be continuously output, and the dual-view correction confidence and consistency status will be updated periodically and synchronized to the back-end management platform to form a safety monitoring log. When the detection result remains in a safe state for 10 consecutive frames, the sensitivity of the audible and visual alarm device can be automatically reduced to reduce interference to the operators, while retaining the real-time monitoring capability to ensure that the entire operation process is controllable.

[0073] Figure 4 An AR glasses-based on-site safety management system 400 for live-line work, linked to a surveillance sphere, is shown. The system implementation is similar to... Figure 1 Corresponding to the illustrated method embodiments, the specific methods include: The video stream acquisition module 401 is used to acquire a first-view video stream and a second-view video stream, wherein the first view has a higher image detail resolution than the second view, and the second view has a wider field of view than the first view. The synchronization processing module 402 is used to perform time synchronization processing on the first viewpoint video stream and the second viewpoint video stream to eliminate the time offset between them and align the synchronized video frame sequence formed on a unified time base. The target trajectory sequence generation module 403 is used to perform target detection and key point extraction on the first-view video frame and the second-view video frame, obtain the target detection box, category label and first original confidence of the target in each view, and generate a target trajectory sequence containing the target spatiotemporal information. The target pair determination module 404 is used to associate the target trajectory in the first viewpoint video stream with the target trajectory in the second viewpoint video stream to determine the cross-viewpoint matching target pair. The reliability score determination module 405 is used to input the trajectory sequence of the matched target pair into the cross-view similarity evaluation network CVRN, and calculate the comprehensive reliability score R of the matched target pair within a preset time window. The comprehensive reliability score R is formed by fusing the evaluation results of at least three dimensions: spatial similarity, temporal consistency and semantic consistency. The confidence correction module 406 is used to correct the first original confidence C1 of the first perspective and the second original confidence C2 of the second perspective in the matched target pair based on the comprehensive reliability score R and through a preset confidence correction function f(R), to obtain a first corrected confidence. Second modified confidence level The confidence correction function f(R) is configured to enhance the confidence level when the value of R increases. Control module 407 is used to adjust the first corrected confidence level. and the second modified confidence level The numerical relationship is used to output the safety detection results, and the safety control of the live-line work site is carried out based on the safety detection results.

[0074] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for safety management and control of live-line work sites through the linkage of AR glasses and a surveillance ball, characterized in that, Includes the following steps: Acquire a first-view video stream and a second-view video stream, wherein the first view has a higher image detail resolution than the second view, and the second view has a wider field of view than the first view. The first-view video stream and the second-view video stream are time-synchronized to eliminate the time offset between them and to align the synchronized video frame sequence formed on a unified time base. Target detection and key point extraction are performed on first-view and second-view video frames to obtain the target detection box, category label and first original confidence score in each view, and to generate a target trajectory sequence containing the target's spatiotemporal information. The target trajectory in the first-view video stream is associated with the target trajectory in the second-view video stream to determine cross-view matching target pairs; The trajectory sequence of the matched target pair is input into the cross-view similarity evaluation network CVRN to calculate the comprehensive reliability score R of the matched target pair within a preset time window. The comprehensive reliability score R is formed by fusing the evaluation results of at least three dimensions: spatial similarity, temporal consistency and semantic consistency. Based on the comprehensive reliability score R, the first original confidence score C1 of the first perspective and the second original confidence score C2 of the second perspective in the matched target pair are corrected using a preset confidence correction function f(R) to obtain the first corrected confidence score. Second modified confidence level The confidence correction function f(R) is configured to enhance the confidence level when the value of R increases. Based on the first modified confidence level and the second modified confidence level The numerical relationship is used to output the safety detection results, and the safety control of the live-line work site is carried out based on the safety detection results.

2. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 1, characterized in that, The first modified confidence level and the second modified confidence level The numerical relationship is used to output the security detection results, including: when and When the difference is less than the difference threshold and both are higher than the high confidence threshold, the consistency judgment of the detection results from the two perspectives is adopted as the final result; when and When the difference is greater than the difference threshold and the maximum value is lower than the low confidence threshold, a low confidence alarm requiring manual review is generated.

3. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 1, characterized in that, The time synchronization processing of the first-view video stream and the second-view video stream specifically includes: Extract time-series signals representing the motion characteristics of the target from the two video streams respectively; Based on maximizing cross-correlation, the time offset Δt that maximizes the correlation between the two time series signals is calculated within the sliding time window; The time offset Δt sequence is smoothed using a low-pass filter, and a re-estimation is triggered when an offset jump exceeds a set tolerance. Based on the final determined time offset Δt, frame alignment or time resampling is performed on the first-view video stream and / or the second-view video stream to achieve synchronization.

4. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 1, characterized in that, Identifying cross-perspective matching target pairs specifically includes: Using the preset average physical size of the target and its pixel size in the image, the approximate distance between the target and the camera in each viewpoint is estimated; Map the center point of the two-dimensional detection box of the target in each viewpoint to a unified approximate three-dimensional coordinate system based on the site plane; Calculate the spatial distance between the target in the first-view video stream and the target in the second-view video stream in the unified approximate three-dimensional coordinate system; The trajectory pairs that are spatially closest and less than the matching threshold are identified as the cross-view matching target pairs.

5. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 1, characterized in that, The calculation process of the cross-view similarity evaluation network CVRN includes: Using a time encoder, temporal features are extracted from the key point sequences and detection box sequences of the first and second perspectives in the matched target pair, respectively, to obtain the first temporal representation S1 and the second temporal representation S2. S1 and S2 are input into the cross-view interaction module, and information is exchanged through the cross-attention mechanism to output cross-correlation features M. The cross-correlation feature M is fused with the category probability vectors of the two perspectives, and the semantic consistency degree s_sem is calculated through the semantic consistency head; Based on the spatial distance and orientation differences of the matching target pair in a unified approximate three-dimensional coordinate system, the spatial consistency degree s_spat is calculated through the spatial consistency head; Based on the comparison of the start and end times of key actions in the two perspectives, the time consistency degree s_temp is calculated through the time consistency header; The comprehensive reliability score R is obtained by weighted summation of s_sem, s_spat, and s_temp and then mapping through an activation function.

6. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 1, characterized in that, The first modified confidence level and the second modified confidence level The numerical relationship is used to output the security detection results, including: If the first corrected confidence level and the second modified confidence level If the difference is greater than the difference threshold and the comprehensive reliability score R is higher than the score threshold, then the detection result from the perspective of the party with the higher score in the spatial consistency assessment is selected as the final result.

7. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 5, characterized in that, The cross-perspective interaction module employs a bidirectional cross-attention mechanism to achieve information interaction, specifically including: The first temporal representation S1 is input into the query projection layer to generate query vector Q1, and input into the key-value projection layer to generate key vector K1 and value vector V1; the second temporal representation S2 is input into the query projection layer to generate query vector Q2, and input into the key-value projection layer to generate key vector K2 and value vector V2. The similarity between Q1 and K2 is calculated to obtain the attention weight matrix of S1 to S2. Based on the weighted sum V2 of the attention weight matrix of S1 to S2, the interaction features from S2 to S1 are obtained. The similarity between Q2 and K1 is calculated to obtain the attention weight matrix of S2 to S1. Based on the weighted sum V1 of the attention weight matrix of S2 to S1, the interaction features from S1 to S2 are obtained. The bidirectional interactive features are residually connected to S1 and S2 respectively and then normalized layer by layer to form cross-correlation features M.

8. The method for safety management of live-line work sites by linking AR glasses with a control ball as described in claim 1, characterized in that, The confidence correction function f(R) adopts a nonlinear Sigmoid function, and its expression is f(R)=Sigmoid(α(R−β)); where α is used to adjust the correction sensitivity so that the change of R in the interval [0,1] is converted into a significant correction magnitude, and β is used as the threshold inflection point; when R>0.5, f(R)>1, which has an enhancing effect on the original confidence and the closer R is to 1, the more significant the enhancement; when R<0.5, f(R)<1, which has a weakening effect and the closer R is to 0, the more obvious the weakening.

9. A method for safety management of live-line work sites in conjunction with AR glasses and a control ball as described in claim 1, characterized in that, The method further includes: Based on the classification of the security test results, at least one of the following control measures shall be adopted: High-risk management employs audio-visual warnings and AR glasses to push text alerts; Alternatively, alarm information can be pushed to a manual review terminal and sorted by priority. If the review confirms a danger, high-risk control measures will be triggered. Alternatively, the monitoring status can be updated periodically and a log generated.

10. A live-line work site safety management system that links AR glasses with a surveillance sphere, characterized in that, include: The video stream acquisition module is used to acquire a first-view video stream and a second-view video stream, wherein the first-view has a higher image detail resolution than the second-view, and the second-view has a wider field of view than the first-view. The synchronization processing module is used to perform time synchronization processing on the first-view video stream and the second-view video stream to eliminate the time offset between them and align the synchronized video frame sequence formed on a unified time base. The target trajectory sequence generation module is used to perform target detection and key point extraction on first-view video frames and second-view video frames, obtain the target detection box, category label and first original confidence of the target in each view, and generate a target trajectory sequence containing the target's spatiotemporal information. The target pair determination module is used to associate the target trajectory in the first viewpoint video stream with the target trajectory in the second viewpoint video stream to determine the cross-viewpoint matching target pair; The reliability score determination module is used to input the trajectory sequence of the matched target pair into the cross-view similarity evaluation network CVRN, and calculate the comprehensive reliability score R of the matched target pair within a preset time window. The comprehensive reliability score R is formed by fusing the evaluation results of at least three dimensions: spatial similarity, temporal consistency and semantic consistency. The confidence correction module is used to correct the first original confidence C1 from the first perspective and the second original confidence C2 from the second perspective in the matched target pair based on the comprehensive reliability score R and using a preset confidence correction function f(R), to obtain a first corrected confidence. Second modified confidence level The confidence correction function f(R) is configured to enhance the confidence level when the value of R increases. The control module is used to adjust the confidence level based on the first corrected confidence level. and the second modified confidence level The numerical relationship is used to output the safety detection results, and the safety control of the live-line work site is carried out based on the safety detection results.

Images