A method and device for guiding and controlling standard operation of an industrial environment of a hydropower plant

Abstract

This invention discloses a method and device for guiding and controlling standard operating procedures (SOPs) in a hydropower plant industrial environment, relating to the field of augmented reality industrial auxiliary operation technology. The method includes: collecting multi-sensor data through a head-mounted device; constructing a semantic 3D dense map through tight-coupled vision-inertial navigation fusion and calculating the grid visual noise index; establishing a multimodal spatial fingerprint database based on the map for coarse and fine positioning; adaptively setting the anchoring verification intensity according to the noise index to obtain accurate anchoring results; rendering SOP guidance information based on the results; statistically analyzing historical verification results of each step to derive an upper bound for the failure probability; adaptively adjusting the verification intensity and implementing mandatory control; and fusing multi-channel interactive events to confirm the control results and push them to a back-end collaborative dashboard. This invention effectively solves the technical problems of easy drift in visual positioning and difficulty in quantifying the reliability of step verification in complex and degraded environments.

Description

Technical Field

[0001] This invention relates to the field of augmented reality industrial assisted operation technology, specifically to a method and device for guiding and controlling standard operating procedures in the industrial environment of a hydropower plant. Background Technology

[0002] In complex industrial settings such as underground powerhouses, corridors, and shafts in hydropower plants, augmented reality technology has been widely applied to equipment inspection and standard operating procedure (SOP) guidance. Technical solutions include visual tag-based positioning schemes, satellite and inertial navigation-based outdoor navigation schemes, and spatial registration methods based on visual simultaneous localization (SLR) and mapping, as well as electronic standard operating procedure (SOP) management systems. These solutions, by constructing environmental maps in real time to estimate poses and present inventory lists, have achieved certain application results in virtual-real fusion and operational guidance in conventional environments.

[0003] However, existing industrial augmented reality-assisted operation solutions generally adopt a fixed-parameter spatial registration and static electronic inventory presentation mode. The system only outputs basic positioning coordinates and a list of steps, lacking the ability to adaptively perceive environmental degradation and operational risks. To improve positioning stability and interaction efficiency, some solutions have introduced multi-sensor fusion mechanisms, such as combining depth cameras and inertial measurement units for pose compensation, or using noise reduction algorithms to optimize speech recognition in high-noise environments.

[0004] Existing industrial augmented reality-assisted operation technologies suffer from several technical shortcomings: First, they only apply a uniform visual feature extraction and matching threshold to the global scene, lacking modeling constraints for spatial heteroscedastic noise. The generated spatial registration results are prone to feature loss and pose drift in degraded environments with dim lighting and repetitive textures, leading to severe misalignment between virtual and real-world anchoring. Second, spatial positioning and SOP (Standard Operating Procedure) step control operate independently, lacking a unified spatial semantic framework for collaborative binding. Furthermore, the step execution uses a single binary judgment, failing to provide reliable mandatory control conclusions when visual comparisons are misjudged in dim environments. In addition, the interaction confirmation mechanism under high noise and protective gear constraints lacks statistically calibrated reliability quantification, making it impossible for operators to judge the confidence level of step verification, and lacking dynamic adaptive credibility guarantees in mandatory control scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned defects of the prior art and provide a method and device for guiding and controlling standard operations in the industrial environment of hydropower plants, which solves the technical problems of easy drift in visual positioning and difficulty in quantifying the reliability of operation steps in the complex and degraded environment of hydropower stations.

[0006] The technical solution of the present invention is as follows:

[0007] This invention provides a method for guiding and controlling standard operating procedures in the industrial environment of a hydropower plant, comprising:

[0008] A multi-sensor data stream from an augmented reality (AR) headset is collected. Through tightly coupled visual-inertial navigation (VIS) fusion localization and mapping, a semantic 3D dense map is constructed, and the visual noise index is calculated for each grid cell of the semantic 3D dense map. Based on the semantic 3D dense map, a multimodal spatial fingerprint database is established at each inspection point. The AR headset is coarsely localized based on the multimodal spatial fingerprint database, and then finely localized by visual feature matching of the multi-sensor data stream. The anchoring verification strength is adaptively set according to the visual noise index to obtain the anchoring result. Standard operation guidance information is rendered in the AR field of view based on the anchoring result. Historical verification results of the operation steps are statistically analyzed to obtain the upper bound of the verification failure probability of the operation steps. The step verification strength is adaptively set according to the upper bound of the verification failure probability, and step-level mandatory control is performed to obtain the step control result. Multi-channel interactive events are fused to confirm the step control result, and the operation process data is encapsulated and pushed to the back-end collaborative dashboard.

[0009] Further, the process of coarsely locating the augmented reality head-mounted device based on the multimodal spatial fingerprint database, then finely locating it by visual feature matching based on the multi-sensor data stream, and adaptively setting the anchoring verification strength according to the visual noise index to obtain the anchoring result includes: when the augmented reality head-mounted device enters the inspection point, querying the visual noise index of the grid cell where the inspection point is located; setting the matching threshold for the visual feature matching fine positioning to be the product of a reference threshold and the coefficient obtained by linear mapping of the visual noise index; and setting the number of consecutive consistent frames required for anchoring to be the number of reference frames and the product of the reference frame number and the coefficient obtained by linear mapping of the visual noise index. The sum of the increments obtained by linear mapping of the visual noise index; the size of the temporal verification window is set to the sum of the number of consecutive consistent frames and the preset window increment; visual feature matching and fine localization are performed on the multi-sensor data stream according to the matching threshold, and the pose obtained by fine localization is anchored within the temporal verification window. When the number of consecutive frames that pass the verification reaches the number of consecutive consistent frames, the anchoring is determined to be successful, and the anchoring result is obtained; when the visual noise index exceeds the preset noise threshold, multimodal assisted verification is triggered, requiring the operator to confirm the anchoring result by voice confirmation or physical button confirmation.

[0010] Further, the visual noise index is obtained and updated in the following manner, including: statistically analyzing the average number of feature points extracted, the average confidence score of visual feature matching, and the average in-between point matching rate of the grid cell in each localization and mapping process; normalizing and complementing the average number of feature points extracted, the average confidence score of visual feature matching, and the average in-between point matching rate to obtain normalized feature missingness, normalized match confidence missingness, and normalized in-between point missingness; and weighting and summing the normalized feature missingness, the normalized match confidence missingness, and the normalized in-between point missingness according to a first weight, a second weight, and a third weight. The visual noise index of the grid cell is obtained; wherein the sum of the first weight, the second weight, and the third weight is one; the visual noise index of each grid cell is stored in the metadata of the semantic 3D dense map in the form of a heatmap; after completing the anchoring verification, the visual feature matching quality index observed during the current anchoring verification process is recorded as the current observation noise index; the visual noise index of the grid cell where the current anchoring verification is located is updated to the weighted sum of the visual noise index before the update and the current observation noise index, and the weight of the visual noise index before the update is greater than the weight of the current observation noise index.

[0011] Further, the statistical analysis of historical verification results for each work step to obtain an upper bound on the verification failure probability of that work step includes: recording a structured verification log tuple containing step identifier, spatial region identifier, ambient lighting and texture conditions, visual comparison score, operator confirmation method, and subsequent anomaly flags after each execution of a work step; if a work step is determined to be completed, but a defect related to the work step appears in the equipment to be inspected within a preset time period, then the subsequent anomaly flag is set to false negative; the structured verification log tuples of each work step are aggregated to obtain the historical verification results of the work step; based on the historical verification results, a binomial distribution Bayesian posterior model is maintained for each work step, where the prior of the Bayesian posterior model is two beta distributions with shape parameters of one; for For each structured verification log tuple of the work step in the historical verification results, if the subsequent anomaly flag is a false negative, the first shape parameter of the beta distribution is incremented by one; otherwise, the second shape parameter of the beta distribution is incremented by one to obtain the updated beta distribution. The estimated failure probability of the work step is determined as the ratio of the first shape parameter to the sum of the first and second shape parameters, and the estimated failure probability is the expected value of the updated beta distribution. The quantile of the updated beta distribution at a preset confidence level is determined as the upper bound of the verification failure probability of the work step. The quantile at the preset confidence level is such that the cumulative distribution function of the updated beta distribution is equal to the failure probability value at the preset confidence level, and the upper bound of the verification failure probability is greater than the expected value of the updated beta distribution.

[0012] Furthermore, the method further includes: when the upper bound of the verification failure probability is less than a first safety threshold, setting the step verification intensity as a baseline verification intensity, and controlling the operation step with the baseline verification intensity; when the upper bound of the verification failure probability is not less than the first safety threshold and not greater than a second safety threshold, setting the step confirmation threshold as a first confirmation threshold less than the baseline confirmation threshold, and setting the number of re-verifications as a first re-verification number greater than the baseline re-verification number, and controlling the operation step with the first confirmation threshold and the first re-verification number, wherein the second safety threshold is greater than the first safety threshold; when the upper bound of the verification failure probability is greater than the second safety threshold, triggering an enhanced verification mode, requiring the operation step to pass through three verifications simultaneously: visual comparison, voice confirmation, and physical button confirmation; multiplying the difference between the estimated failure probability of each operation step and a certain value to obtain a joint safety degree with no omissions throughout the inspection process; determining the top preset number of operation steps with the highest estimated failure probability as safety bottleneck steps, and reserving additional verification time and verification resources for the safety bottleneck steps in the inspection plan.

[0013] Furthermore, the construction of a semantic 3D dense map through tightly coupled vision-inertial navigation fusion localization and mapping includes: deploying three parallel localization and mapping pipelines with different computational accuracies on the augmented reality head-mounted device, the three parallel pipelines sharing the multi-sensor data stream, the three parallel pipelines including a first pipeline, a second pipeline, and a third pipeline; performing inference on the first pipeline with a first computational accuracy, running on sampling frames at preset frame intervals to obtain the pose estimate of the first pipeline, the pose estimate of the first pipeline serving as a pose calibration benchmark; and performing inference on the second pipeline with a second computational accuracy at full frame rate. The system operates frame by frame to obtain the pose estimate of the second pipeline, where the second computational precision is lower than the first computational precision. The third pipeline is then binarized with the visual feature descriptor using a third computational precision to obtain the pose estimate of the third pipeline, where the third computational precision is lower than the second computational precision. The pose estimates of the second and third pipelines are fused to obtain a fused pose, which is then corrected on the sampled frame using the pose calibration benchmark to obtain a corrected pose. Based on the corrected pose, a semantic 3D dense map is constructed through tightly coupled visual-inertial mapping.

[0014] Furthermore, the method further includes: calculating the first moment of the pose estimation of the first pipeline, the pose estimation of the second pipeline, and the pose estimation of the third pipeline according to weights positively correlated with their respective calculation accuracy, to obtain the moment fusion pose, and using the moment fusion pose as the pose output for localization and mapping; calculating the second moment of the pose estimation of the first pipeline, the pose estimation of the second pipeline, and the pose estimation of the third pipeline relative to the moment fusion pose, to obtain the pose covariance matrix, and using the trace of the pose covariance matrix as the fusion uncertainty; and for the... The pose estimations of the first pipeline, the second pipeline, and the third pipeline are used to calculate the fourth moment relative to the moment fusion pose, thus obtaining the fourth moment deviation of each pipeline. Pipelines whose fourth moment deviation exceeds a preset deviation threshold are identified as outliers. When an outlier is detected, the calculation accuracy of the outlier is adjusted to a target calculation accuracy greater than the current calculation accuracy of the outlier, or the weight of the pose estimation of the outlier relative to the moment fusion pose is reset to zero, pausing the contribution of the outlier to the moment fusion pose.

[0015] Furthermore, the method also includes: monitoring the battery level and heat dissipation status of the augmented reality headset to obtain a battery level monitoring value and a heat dissipation temperature monitoring value; when the battery level monitoring value is greater than a first battery level threshold, activating the first pipeline, the second pipeline, and the third pipeline to obtain a current set of operating pipelines; when the battery level monitoring value is not greater than the first battery level threshold and not less than a second battery level threshold, deactivating the third pipeline from the current set of operating pipelines, retaining the first pipeline and the second pipeline, wherein the first battery level threshold is greater than the second battery level threshold; when the battery level monitoring value is less than the second battery level threshold, deactivating the first pipeline and the third pipeline from the current set of operating pipelines, retaining only the second pipeline, and triggering an energy-saving inspection mode to guide the operator to an area where the lighting meets preset conditions; when the heat dissipation temperature monitoring value exceeds a preset temperature threshold, adjusting the operating frame rate of the first pipeline in the current set of operating pipelines to a target operating frame rate that is less than the current operating frame rate of the first pipeline.

[0016] Furthermore, a standard operating procedure guidance and control device for a hydropower plant industrial environment is provided, comprising: a mapping module for collecting multi-sensor data streams from an augmented reality head-mounted device, constructing a semantic 3D dense map through vision-inertial navigation tightly coupled fusion positioning and mapping, and calculating the visual noise index for each grid cell of the semantic 3D dense map; a fingerprint database module for establishing a multimodal spatial fingerprint database at each inspection point based on the semantic 3D dense map; and a virtual-real anchoring module for performing coarse positioning of the augmented reality head-mounted device based on the multimodal spatial fingerprint database, and then performing visual positioning of the multi-sensor data streams. Feature matching is used for precise localization, and the anchoring verification intensity is adaptively set according to the visual noise index to obtain the anchoring result. The operation control module is used to render standard operation guidance information in the augmented reality field of view based on the anchoring result, statistically analyze the historical verification results of the operation steps, obtain the upper bound of the verification failure probability of the operation steps, adaptively set the step verification intensity according to the upper bound of the verification failure probability, and perform step-level mandatory control to obtain the step control result. The interaction and collaboration module is used to integrate multi-channel interaction events to confirm the step control result, and encapsulate the operation process data and push it to the back-end collaboration dashboard.

[0017] The beneficial effects of this invention are as follows:

[0018] First, by calculating the visual noise index for each grid cell of the semantic 3D dense map and persistently storing it in the form of a heatmap, the augmented reality head-mounted device can adaptively set the anchoring verification intensity according to the differences in visual conditions in the area. High-noise areas use low matching thresholds, multi-frame verification, and multimodal assisted verification, while low-noise areas use high matching thresholds and fast confirmation with fewer frames. The anchoring verification resources are allocated spatially as needed, overcoming the shortcomings of traditional uniform parameters in areas with great differences such as the main powerhouse, corridors, and shafts of hydropower stations, which either waste resources or have insufficient capabilities, and improving the robustness of virtual and real anchoring in degraded environments.

[0019] Second, by maintaining a Bayesian posterior model of beta distribution for each operation step and continuously updating it with the occurrence of relevant defects within a preset time as a false negative indicator, the reliability of the step verification mechanism itself is upgraded from binary judgment to a quantifiable upper bound of the verification failure probability. Based on the upper bound, the step verification intensity is adaptively divided into three levels: baseline verification, enhanced re-verification, and triple verification. At the same time, the joint safety degree of no omissions in the inspection process is obtained by multiplying the difference between the estimated failure probability of all operation steps and one. Verification resources are tilted towards safety bottleneck steps, making it possible to quantify and adaptively adjust the reliability of operation step verification.

[0020] Third, by deploying three parallel pipelines on the augmented reality head-mounted device—a full-precision baseline, a first-quantization-precision real-time pipeline, and a second-quantization-precision auxiliary pipeline—and calculating the first-order moments of the three pose estimations with weights positively correlated with the calculation accuracy to obtain the moment fusion pose, the second-order moments to obtain the fusion uncertainty, and the fourth-order moments to detect outliers, and then dynamically adjusting the pipeline switching and running frame rate based on three power levels and an over-temperature frame rate reduction strategy, the power consumption and heat dissipation pressure are significantly reduced while ensuring positioning accuracy, thus ensuring the battery life of the augmented reality inspection device during long task cycles of six to eight hours. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an overall flowchart of a standard operation guidance and control method for industrial environment in hydropower plants according to the present invention;

[0023] Figure 2 This is a flowchart of the spatial heteroscedasticity noise perception adaptive anchoring verification process of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a standard operation guidance and control device for industrial environment in a hydropower plant according to the present invention. Detailed Implementation

[0025] Example 1

[0026] like Figure 1 As shown, this invention provides a method for guiding and controlling standard operating procedures in the industrial environment of a hydropower plant, comprising:

[0027] S1: Collect multi-sensor data streams from the augmented reality head-mounted device, construct a semantic 3D dense map through vision-inertial navigation tight coupling fusion positioning and mapping, and calculate the visual noise index for each grid cell of the semantic 3D dense map.

[0028] The standard operating procedure guidance and control method for the industrial environment of hydropower plants uses a multi-sensor data stream from an augmented reality (AR) head-mounted device as input. This AR head-mounted device is equipped with multiple red-green-blue cameras, a depth camera, and an inertial measurement unit (IMU). The multi-sensor data stream includes synchronously sampled image sequences, depth map sequences, and high-frequency readings from the IMU. Visual-inertial navigation tightly coupled fusion localization and mapping uses this multi-sensor data stream as input. A six-degree-of-freedom pose estimate is obtained through front-end visual feature extraction and inertial pre-integration joint optimization. Accumulated drift is then eliminated through back-end global bundle adjustment optimization, resulting in a sparse point cloud and keyframe pose. Based on the sparse point cloud, dense mapping and semantic segmentation fusion are performed to obtain a semantically labeled 3D dense map, which serves as the semantic 3D dense map. To support subsequent steps in perceiving differences in visual conditions, the semantic 3D dense map is divided into 0.5m × 0.5m grid cells, and a visual noise index is calculated for each grid cell. The visual noise index is obtained by weighting three indicators of the grid cell in each mapping process: the average number of feature points extracted, the average confidence score of visual feature matching, and the average inlier matching rate. The value ranges from 0 to 1, with higher values ​​indicating worse visual conditions for the grid cell. The visual noise index is stored in the metadata of the semantic 3D dense map in the form of a heatmap and is updated synchronously with map incremental updates. Specifically, the calculation method and adaptive update method of the visual noise index will be detailed in subsequent embodiments.

[0029] It should be noted that the visual conditions vary greatly across different areas of the underground powerhouse of the hydropower station. The main powerhouse area is well-lit with rich textures, resulting in a large number of extracted feature points and a high inlier matching rate, corresponding to a low visual noise index. The corridor area is dimly lit with repetitive textures on equipment surfaces, significantly reducing feature matching confidence and corresponding to a medium visual noise index. The shaft and cable layer areas are almost entirely dark with highly repetitive textures, leading to frequent feature point loss and an extremely low inlier matching rate, corresponding to a high visual noise index. Traditional visual simultaneous localization and mapping (VLM) systems use uniform feature extraction and matching parameters across the entire scene, resulting in unnecessary computational redundancy in low-noise areas and severely insufficient verification capabilities in high-noise areas. This invention calculates and persistently stores the visual noise index for each grid cell, enabling the system to perceive the degree of visual degradation in its current area during runtime, thus providing a quantitative basis for setting the subsequent adaptive anchoring verification intensity.

[0030] Furthermore, in low-light scenarios, the tightly coupled vision-inertial navigation (VIS) fusion localization and mapping process can incorporate contrast-limited adaptive histogram equalization and a retina-based joint image enhancement method during the front-end visual feature extraction stage. This involves performing brightness-adaptive preprocessing on the original image before feature extraction, thereby maintaining a sufficient number of reliable feature points under low-light conditions and reducing front-end tracking failures due to insufficient illumination. The image enhancement process is embedded in the front-end feature extraction submodule in a pipelined manner, without increasing the overall end-to-end latency of the system.

[0031] S2: Based on the semantic three-dimensional dense map, establish a multimodal spatial fingerprint database at each inspection point.

[0032] After obtaining the semantic 3D dense map, it is necessary to establish a positioning benchmark for each inspection point that can be quickly compared during runtime. Therefore, a multimodal spatial fingerprint database is established at each inspection point based on the semantic 3D dense map. Each fingerprint entry in the multimodal spatial fingerprint database includes a WiFi signal strength vector, a Bluetooth Low Energy beacon signal strength vector, three components of geomagnetic intensity, and a depth geometric feature vector. The WiFi signal strength vector is composed of the received signal strength indications of a preset number of wireless access points around the inspection point during the calibration phase; the Bluetooth Low Energy beacon signal strength vector is composed of the received signal strength indications of Bluetooth Low Energy beacons deployed near the inspection point during the calibration phase; the three components of geomagnetic intensity are obtained by statically collecting data from a geomagnetic sensor at the inspection point and then taking a moving average; the depth geometric feature vector is obtained by encoding the local point cloud collected by a depth camera at the inspection point using a fast point feature histogram. Each fingerprint entry in the multimodal spatial fingerprint database is associated with the 3D coordinates of the fingerprint entry in the semantic 3D dense map, serving as a benchmark for subsequent coarse positioning. Optionally, when wireless access points near the inspection site are unavailable, only three types of fingerprints—Bluetooth Low Energy Beacon, Geomagnetic Intensity, and Depth Geometry—are retained to adapt to different field deployment conditions.

[0033] S3: Based on the multimodal spatial fingerprint database, perform coarse localization of the augmented reality head-mounted device, then perform visual feature matching for fine localization of the multi-sensor data stream, and adaptively set the anchoring verification strength according to the visual noise index to obtain the anchoring result.

[0034] Specifically, the coarse localization uses the k-nearest neighbor algorithm to match the current real-time fingerprint vector with fingerprint entries in the multimodal spatial fingerprint database to obtain coarse localization coordinates and their accuracy, which does not exceed 2 meters. Within the local map area corresponding to the coarse localization coordinates, depth feature extraction and matching are performed on the current image in the multi-sensor data stream, followed by random sampling consistent geometric verification to eliminate mismatches, resulting in a fine localization pose. The anchoring verification strength is adaptively set based on the visual noise index of the current grid cell. A higher visual noise index results in a more lenient matching threshold, more consecutive consistent frames, and a larger temporal window. If necessary, multimodal assisted verification via voice confirmation or physical button confirmation is triggered. Finally, when the number of consecutively verified frames reaches the required number of consecutive consistent frames, the anchoring is considered successful, and the anchoring result is obtained. The anchoring result includes the six-degree-of-freedom pose of the AR head-mounted device relative to the semantic 3D dense map and its uncertainty estimate.

[0035] S4: Based on the anchoring results, render standard operation guidance information in the augmented reality field of view, statistically analyze the historical verification results of the operation steps, obtain the upper bound of the verification failure probability of the operation steps, adaptively set the step verification intensity based on the upper bound of the verification failure probability, and perform step-level mandatory control to obtain the step control results.

[0036] After obtaining the anchoring results, the SOP needs to be bound to the on-site spatial location and presented in a controllable manner. The standard operation guidance information includes a highlighted outline of the operation target, an operation guidance panel, and navigation guidance arrows. The highlighted outline of the operation target is coded in green, yellow, and red to correspond to three states: pending operation, in operation, and deviation alarm, respectively. The execution status of the operation step is determined by the visual comparison model based on the comparison score between the current image and the standard operation image, and is marked as completed after confirmation by the operator. To continuously quantify the reliability of the operation step verification mechanism, a Bayesian posterior model with a beta distribution is maintained for each operation step, and is continuously updated using the occurrence of relevant defects within a preset time period as a false negative indicator. The upper bound of the verification failure probability of the operation step is obtained from the Bayesian posterior model. Based on the upper bound of the verification failure probability, the step verification intensity is adaptively divided into three levels: baseline verification, enhanced re-verification, and triple verification. More stringent verification requirements are imposed on operation steps with a high upper bound of verification failure probability, and the AR interface is locked when the step is not completed or skipped. The results of the step control include the execution status of each operation step, the verification intensity level, and the verification log.

[0037] When imported into the system, the SOP document is parsed into a structured sequence, with each work step associated with the spatial coordinates of a specific device in the semantic 3D dense map. Based on this anchoring, the augmented reality headset can accurately overlay the highlighted outline of the operation target onto the physical contour of the corresponding device, with navigation arrows pointing to the next operation point. When the operator deviates from the designated operation point or skips the current step, the AR interface automatically locks and displays a red deviation warning highlight, forcing the operator to complete the current step before continuing. This combination of spatial association and step-level mandatory control fundamentally eliminates the risks of omissions and misoperations inherent in traditional electronic SOP lists.

[0038] S5: Confirm the results of the step control by integrating multi-channel interactive events, and encapsulate and push the operation process data to the back-end collaborative dashboard.

[0039] After obtaining the results of the aforementioned step control, it is necessary to provide operators with confirmation methods suitable for high-noise industrial environments and synchronize the operation process to the back-end. Multi-channel interactive events include event streams acquired in parallel from three channels: voice, physical button, and gesture. The voice channel uses beamforming and neural noise reduction to preprocess the audio acquired by the helmet microphone array, and then obtains the command text through speech recognition; the physical button channel consists of physical button events on the side of the helmet; the gesture channel consists of gesture classification results based on hand key point detection. The three event streams are merged into a unified confirmation command after priority arbitration and time window alignment. The priority arbitration order is physical button > voice > gesture, and the time window alignment tolerance is ±500 milliseconds. The operation process data is encapsulated in a protocol buffer format and pushed to the back-end via 5G or WiFi 6 network with a delay of no more than 200 milliseconds. The back-end renders a three-column collaborative dashboard, with the three columns representing a first-person view, an SOP dashboard, and an equipment radar chart, and supports point-to-point remote expert two-way collaboration based on real-time web communication, with the entire process recorded and archived for future reference.

[0040] In high-noise industrial environments with equipment operating noise levels not less than 85 decibels, both voice channels and single touchscreen interaction methods are unreliable. A multimodal fusion mechanism with three-channel parallel acquisition and priority arbitration ensures reliable interaction confirmation by assigning physical buttons the highest priority. This is achieved even when voice recognition is hampered by noise or gesture recognition is difficult due to operators wearing heavy gloves. Simultaneously, the voice channel, after beamforming spatial filtering to directionally enhance the helmet wearer's voice signal and then using a neural noise reduction network to suppress residual environmental noise, maintains high voice recognition accuracy even in high-noise environments, serving as an effective backup channel for the physical buttons.

[0041] Example 2

[0042] like Figure 2 As shown, based on Example 1, the process of performing coarse localization of the augmented reality head-mounted device based on the multimodal spatial fingerprint database, followed by fine localization through visual feature matching of the multi-sensor data stream, and adaptively setting the anchoring verification strength according to the visual noise index to obtain the anchoring result includes:

[0043] S3.1: When the augmented reality head-mounted device enters the inspection site, query the visual noise index of the grid cell where the inspection site is located.

[0044] During the inspection process, the AR headset continuously outputs coarse positioning coordinates. When these coordinates fall within a preset radius of any inspection point, the AR headset is considered to have entered that inspection point. Upon entering the inspection point, the headset determines its location within a specific grid cell in the semantic 3D dense map based on the coarse positioning coordinates, and then queries the metadata of the semantic 3D dense map to obtain the visual noise index (NI) of that grid cell. The NI value ranges from 0 to 1, with higher values ​​indicating worse visual conditions in the grid cell. The NI value serves as the basis for adaptively setting anchoring verification parameters in subsequent steps, ensuring that the anchoring verification strength matches the visual conditions of the grid cell.

[0045] S3.2: Set the matching threshold for precise localization of the visual feature matching to the product of the baseline threshold and the coefficient obtained by linear mapping of the visual noise index.

[0046] Based on the visual noise index NI, set the matching threshold θ for the precise localization of the visual feature matching. match The baseline threshold θ base The product of the linear mapping coefficients, where the linear mapping coefficients are 1 plus the visual noise index NI, is the matching threshold adaptive formula. Where θ match For adaptive matching threshold, θ base θ is the baseline matching threshold, and NI is the visual noise index of the grid cell in which it is located. base Values ​​can be selected within the range of 0.6 to 0.8 (e.g., 0.7) to balance matching speed and accuracy. When NI is 0, the matching threshold is 0.7, and when NI is 1, the matching threshold is 1.4, which requires two independent matches to pass. This relaxes the strictness of a single match in high-noise regions and requires multiple independent verifications, while maintaining a high threshold in low-noise regions for rapid confirmation.

[0047] The matching threshold θ match The descriptor matching distance determination step applied to the output of the deep feature extraction module: matching distance less than θ match Descriptor pairs are considered candidate matches, and then random sampling consensus geometric verification is used to select the inlier set. In high-noise regions, due to the reduced number of reliable feature points, appropriately relaxing the single-match threshold can retain more candidate match pairs, thus providing a sufficient number of inputs for subsequent geometric verification and avoiding the failure of random sampling consensus to converge due to too few candidate pairs. The risk of mismatch caused by relaxing the single-frame matching threshold is compensated by requiring multiple independent frames to meet the inlier number threshold, so that the anchoring reliability in high-noise regions is maintained at the same level as in low-noise regions.

[0048] S3.3: Set the number of consecutive consistent frames required for anchoring to the sum of the base frame number and the increment obtained by linear mapping of the visual noise index.

[0049] Combined with the matching threshold θ match Set the number of consecutive consistent frames m required for anchoring as the base frame number. The sum of the linear mapping increments, where the linear mapping increment is the frame increment coefficient. The product of the visual noise index NI and rounded to the nearest integer is the formula for the adaptive number of consecutive consistent frames. Where m is the number of consecutive consistent frames required for anchoring. As the baseline frame number, is the frame rate increment coefficient, and NI is the visual noise index of the grid cell in which it resides. This is a rounding function. The reference frame number... The frame increment coefficient is set to 3. The value of NI can range from 2 to 6 (e.g., 4). When NI is 0, the number of consecutive consistent frames is 3, and when NI is 1, the number of consecutive consistent frames is 7. Therefore, more consecutive consistent frames are required in high-noise regions to improve anchoring reliability.

[0050] S3.4: Set the timing verification window size to the sum of the number of consecutive consistent frames and the preset window increment.

[0051] Based on the number of consecutive consistent frames m, the timing verification window size k is set to the number of consecutive consistent frames m plus a preset window increment. The sum, i.e., the time window formula is Where k is the temporal verification window size, and m is the number of consecutive consistent frames. This is the preset window increment. The value is set to 2, and those skilled in the art can adjust it within the range of 1 to 4. The timing verification window size k specifies the maximum number of frames allowed to be observed before the anchoring is determined to be successful. If the number of consecutive consistent frames has not been accumulated after the timing verification window is exceeded, the anchoring is determined to have failed and coarse positioning is restarted.

[0052] The temporal verification window mechanism, by observing pose consistency across k consecutive frames, naturally suppresses instantaneous noise disturbances in a single frame. In high-noise regions, the value of k increases synchronously with the value of m, allowing the system more time to wait for a sufficient number of consecutive consistent frames to appear. This avoids unnecessary relocalization triggered by premature timeouts due to anchoring failures during instantaneous illumination fluctuations or motion blur. (Preset window increment) The settings ensure that, at any noise level, the timing window always has a tolerance margin of at least 2 frames more than the required number of consecutive and consistent frames, enabling the system to cross isolated noisy frames without interrupting continuous counting.

[0053] S3.5: Perform visual feature matching for fine localization of the multi-sensor data stream according to the matching threshold, and perform anchoring verification on the pose obtained by fine localization within the time-series verification window. When the number of consecutively verified frames reaches the number of consecutive consistent frames, the anchoring is determined to be successful, and the anchoring result is obtained.

[0054] In the matching threshold θ match Based on the number of consecutive consistent frames m and the temporal verification window size k, and according to the matching threshold θ match Depth feature extraction and matching are performed on the current image in the multi-sensor data stream, and then mismatches are eliminated through random sampling consistent geometric verification to obtain the precise localization pose. Anchoring verification is performed on the precise localization pose of each frame within the temporal verification window k. The anchoring verification includes pose consistency check and interior point count check. If the current frame passes the anchoring verification, the consecutive pass frame count is incremented by one; if the current frame fails the anchoring verification, the consecutive pass frame count is reset to zero. When the number of consecutive pass frames reaches the consecutive consistent frame count m, the anchoring is considered successful, and the anchoring result is obtained. The anchoring result includes the six-degree-of-freedom pose of the AR head-mounted device relative to the semantic 3D dense map and its uncertainty estimate. If the consecutive consistent frame count m has not been accumulated beyond the temporal verification window k, the anchoring is considered failed, and S3.1 is re-executed to reinitialize the anchoring parameters at the current or new inspection point and retry.

[0055] Specifically, pose consistency testing is achieved by comparing the rotation angle deviation and translation distance deviation between the current frame's fine-localized pose and the fine-localized pose of the previous frame: if the rotation deviation exceeds a preset rotation tolerance or the translation deviation exceeds a preset translation tolerance, the current frame is deemed to have failed the pose consistency test. Interior point count testing is achieved by determining whether the number of interior points obtained from random sampling consistent geometric verification is not lower than a preset interior point count threshold. When both tests pass, the current frame is counted in the consecutive verification frame count. Optical flow tracking combined with Kalman filtering is used to maintain the continuity of fine-localization between adjacent frames. When depth feature extraction and matching temporarily fail in a frame due to motion blur, optical flow tracking can extrapolate the pose prediction value of the current frame based on the fine-localized pose of the previous frame, preventing the consecutive verification frame count from prematurely resetting to zero due to a single frame extraction failure.

[0056] S3.6: When the visual noise index exceeds the preset noise threshold, multimodal assisted verification is triggered, requiring the operator to confirm the anchoring result by voice confirmation or physical button confirmation.

[0057] Based on the anchoring result determination, when the visual noise index NI exceeds a preset noise threshold, multimodal assisted verification is triggered, requiring the operator to confirm the anchoring result via voice or physical button. Considering that the visual noise index NI ranges from 0 to 1, the preset noise threshold is set to, for example, 0.8 (the specific range can be 0.7 to 0.9) to define extremely high-noise areas with very poor visual conditions and a high risk of false positives based solely on visual consistency. The multimodal assisted verification broadcasts a prompt "Anchoring mark identified, please confirm" through the voice channel and waits for the operator's voice confirmation command or physical button confirmation event. When the operator provides voice confirmation or physical button confirmation within the preset waiting time, the anchoring result is finally adopted; when the operator does not provide any confirmation within the preset waiting time, the anchoring result is determined to be pending review and triggers remote expert intervention. The introduction of multimodal assisted verification means that the anchoring confirmation in high-noise areas no longer relies solely on visual evidence, thereby avoiding false positives caused by the failure of a single channel.

[0058] The preset waiting time is set to 30 seconds, which can be adjusted within the range of 15 to 60 seconds according to the pace of on-site operations. In the extremely high noise region (NI>0.8), visual features are extremely scarce, and relying solely on the visual consistency of consecutive consistent frames to determine successful anchoring carries a high risk of false positives. By introducing the operator's subjective judgment as an additional verification channel, human spatial perception ability is incorporated into the anchoring confirmation closed loop, forming an effective safety fallback mechanism in the area where visual perception ability is weakest.

[0059] Example 3

[0060] Based on Example 2, the visual noise index is obtained and updated in the following manner:

[0061] S3.1.1: Statistically calculate the average number of feature points extracted, the average confidence score of visual feature matching, and the average inlier rate of the grid cell in each localization and mapping process.

[0062] For each grid cell of the semantic 3D dense map, three visual metrics are statistically analyzed during each localization and mapping process. These three visual metrics include the average number of feature points extracted, F. avg Mean confidence score C for visual feature matching avg and the mean inside-match rate R avg The average number of feature points extracted, F avgThe visual feature matching confidence score C_avg is obtained by taking the arithmetic mean of the number of feature points extracted by the grid cell in each mapping keyframe; the matching confidence score C_avg is obtained by taking the arithmetic mean of the matching confidence scores of the inliers of the grid cell that have been verified by random sampling consistency in each matching; the matching inlier rate R_avg is obtained by taking the arithmetic mean of the ratio of the number of inliers to the total number of matches of the grid cell in each matching. These three visual indicators characterize the visual conditions of the grid cell from three dimensions: feature richness, matching reliability, and geometric consistency.

[0063] The three visual metrics are automatically accumulated and updated each time a keyframe of the mapping passes through the grid cell, without the need for an additional calibration process. During the initial deployment of the system, the three visual metrics of each grid cell are statistically initialized by the keyframes passing through the grid cell during the initial mapping process. Thereafter, they are continuously updated with each incremental mapping, so that the visual noise index increasingly reflects the true visual conditions of the area over time.

[0064] S3.1.2: Normalize the average number of extracted feature points, the mean of the visual feature matching confidence score, and the mean of the matching inlier rate, and then take the complement to obtain the normalized feature missing score, the normalized matching confidence missing score, and the normalized inlier missing score.

[0065] Based on the three visual indicators, the average number of feature points extracted, F avg The mean confidence score C of the visual feature matching avg and the mean of the matched in-point rate R avg Normalize each missing value to the interval 0 to 1 using its respective global maximum value, then subtract the normalization result from one to obtain three missing values. The normalized complement formula is as follows: .in, , , These represent normalized feature missingness, normalized match confidence missingness, and normalized interior missingness, respectively. avg C avg R avg These are respectively the average number of extracted feature points, the mean confidence score of visual feature matching, and the mean inlier rate of matching, F max C max R max These are the average number of extracted feature points, the average confidence score of visual feature matching, and the average in-place matching rate, respectively, the global maximum values ​​of these three missing values ​​across all grid cells in the semantic 3D dense map. The values ​​for these three missing values ​​range from 0 to 1, with higher values ​​indicating worse visual conditions in the corresponding dimension.

[0066] Among them, F max C max R maxThe three global maximum values ​​are recalculated and synchronously updated to the map metadata after each incremental mapping to adapt to feature statistical benchmark drift caused by scene structure changes or device replacements. When the number of keyframes in the previous mapping of a certain grid cell is less than the preset minimum number of frames (preferably 5 frames), the confidence of the three visual indicators of the grid cell is insufficient. The three missing values ​​of the grid cell are set to 1 in the most conservative estimation method, and the corresponding visual noise index is the maximum value of 1. The system is driven to adopt the highest intensity anchoring verification strategy in this area until enough historical frames are accumulated and then automatically updated to the measured values.

[0067] S3.1.3: The normalized feature missing degree, the normalized matching confidence missing degree, and the normalized interior point missing degree are weighted and summed according to the first weight, the second weight, and the third weight to obtain the visual noise index of the grid cell.

[0068] Based on the three missing values, the normalized feature missing value, the normalized match confidence missing value, and the normalized interior missing value are weighted and summed according to the first weight, the second weight, and the third weight to obtain the visual noise index NI of the grid cell. The formula for the visual noise index is: Wherein, NI is the visual noise index of the grid cell, and w f w c w r The weights are the first weight, the second weight, and the third weight, respectively, and they satisfy w. f +w c +w r =1, , , These are, respectively, the normalized feature missing value, the normalized matching confidence missing value, and the normalized interior point missing value. The first weight w f The second weight w c and the third weight w r The values ​​were set to 0.4, 0.3, and 0.3 respectively, and the specific values ​​can be adjusted based on the relative impact of the three indicators—feature richness, matching confidence, and inlier rate—on anchoring stability. The number of extracted feature points has the greatest impact on simultaneous visual localization and mapping in degraded environments; therefore, the first weight is higher than the second and third weights.

[0069] It should be noted that the weighting method for the three missing values ​​mentioned above reflects a prior estimate of the contribution of visual degradation in each dimension to the overall anchoring reliability. In the typical scenario of an underground powerhouse in a hydropower station, the scarcity of feature points is the primary factor restricting anchoring stability; therefore, the first weight w... f=0.4 is assigned the maximum contribution; the matching confidence and inlier rate reflect two complementary dimensions of matching quality, each assigned a weight of 0.3. By using three-dimensional weighted fusion rather than a single index, the visual noise index can more comprehensively characterize the visual degradation degree of each grid cell, avoiding misjudgment of the noise index caused by abnormal fluctuations in a certain dimension.

[0070] S3.1.4: Store the visual noise index of each grid cell in the metadata of the semantic 3D dense map in the form of a heatmap.

[0071] Based on the visual noise index NI, the visual noise index of each grid cell is stored in the metadata of the semantic 3D dense map in the form of a heatmap. The heatmap uses the 2D planar coordinates of the grid cell as an index, the visual noise index NI as the heat value, and is rendered using a blue-green-yellow-red color mapping. The heatmap is updated synchronously with the incremental updates of the semantic 3D dense map, allowing subsequent inspections to directly query the visual noise index NI of any grid cell from the metadata without requiring real-time online calculation.

[0072] The heatmap is stored serially in binary format, with the grid cell number as the key and the single-precision floating-point representation of the visual noise index NI as the value, stored in a dedicated field of the metadata file of the semantic 3D dense map. After each incremental mapping is completed, only the heatmap values ​​corresponding to the grid cells that were passed through by keyframes during the current mapping process are updated. The visual noise index of grid cells not affected by the current mapping remains unchanged, thus ensuring that the incremental update cost of the heatmap is proportional to the coverage area of ​​the current mapping, rather than proportional to the total map area. The heatmap can be presented in a 3D scene overlay manner through the map visualization interface of the back-end management platform, allowing operation and maintenance personnel to intuitively understand the visual degradation distribution of each area and guide the optimization of on-site lighting or calibration strategies.

[0073] S3.1.5: After completing the anchoring verification, record the visual feature matching quality index observed during this anchoring verification process as the observation noise index.

[0074] After the anchoring verification is completed, the visual feature matching quality index observed during this anchoring verification process is recorded as the observation noise index NI. obs Specifically, based on the three indicators actually observed and statistically analyzed during this anchoring verification process—the average number of extracted feature points, the mean visual feature matching confidence score, and the mean in-between matching rate—each indicator was normalized to the 0-1 range according to its global maximum value in all grid cells of the semantic 3D dense map, and then complemented to obtain the normalized feature missingness, normalized matching confidence missingness, and normalized in-between missingness of this observation. These were then weighted and summed according to the first, second, and third weights to obtain the noise index NI of this observation.obs The first weight is preferably 0.4, the second weight is preferably 0.3, and the third weight is preferably 0.3. The observed noise index NI... obs This reflects the actual visual conditions of the grid cell at the moment of this anchoring verification. It may differ from the visual noise index NI accumulated from previous mappings. This difference will be used for subsequent self-calibration updates.

[0075] S3.1.6: Update the visual noise index of the grid cell where the current anchoring verification is located to a weighted sum of the visual noise index before the update and the current observed noise index, and the weight of the visual noise index before the update is greater than the weight of the current observed noise index.

[0076] According to the noise index NI observed in this study obs Update the visual noise index NI of the grid cell where this anchoring verification is performed to the original visual noise index NI. old The noise index NI observed in this study obs The weighted sum, i.e., the noise figure self-calibration update formula, is: .in, This is the updated visual noise index. The visual noise index before the update. The noise index of this observation is... The forgetting factor, λ, ranges from 0 to 1. The forgetting factor λ can be adjusted between 0.8 and 0.95; a larger λ indicates a higher level of confidence in the historical accumulated values. The self-calibration update makes the visual noise index increasingly accurate with the number of anchoring verifications. This is a time-converging self-calibration mechanism that solves the problem of the implicit dependence of traditional simultaneous localization and mapping systems on uniform noise failing in the non-uniform noise scenario of hydropower stations.

[0077] The exponential moving average self-calibration mechanism also possesses the ability to adaptively track changes in visual conditions with seasonal variations and equipment status. When temporary lighting conditions in a certain area improve due to equipment maintenance, consistently below After multiple anchoring verifications The intensity of the light will gradually decrease, and the system will automatically relax the verification intensity requirements for that area; conversely, when the lighting conditions in a certain area continue to deteriorate due to equipment aging, the intensity will gradually decrease. The intensity will gradually increase, and the system will automatically increase the verification strength in that area. Setting the forgetting factor λ=0.9 allows the visual noise index to strongly suppress short-term fluctuations while maintaining sufficient response speed to continuous trend changes, thus balancing estimation stability and adaptability.

[0078] Example 4

[0079] Based on Example 1, the step of statistically analyzing the historical verification results of the work steps to obtain the upper bound of the verification failure probability of the work steps includes:

[0080] S4.1: After each operation step is executed, record a structured verification log tuple containing step identifier, spatial area identifier, ambient lighting and texture conditions, visual comparison score, operator confirmation method, and subsequent anomaly flags.

[0081] After each execution of a task step, a structured verification log tuple is recorded. The step identifier is the unique number of the task step in the SOP; the spatial region identifier is the number of the grid cell where the task step is executed; the ambient lighting and texture conditions are composed of the light sensor reading and the visual noise index NI of the grid cell; the visual comparison score is the output of the visual comparison model comparing the current image with the standard operation image, with a value ranging from 0 to 1; the operator confirmation method is one or more of voice confirmation, physical button confirmation, and gesture confirmation; the subsequent anomaly flag is initialized to a pending tracking state during recording and is used to characterize the actual execution quality of the task step in subsequent monitoring cycles.

[0082] Specifically, the structured verification log tuple is timestamped to mark the recording time, and is linked to the augmented reality head-mounted device pose estimation and fusion uncertainty during the execution of that job step. All data is stored in a verification log database for subsequent analysis of the correlation between verification failures and environmental conditions and positioning accuracy. The verification log database uses a relational database for storage, with a composite primary key of step identifier and timestamp. It supports multi-dimensional retrieval by step identifier, spatial region identifier, time range, and other dimensions, providing an efficient data access interface for updating the Bayesian posterior model and for security audit tracing.

[0083] S4.2: If the operation step is determined to be completed, but a defect related to the operation step appears in the equipment to be inspected within a preset time period, then the subsequent abnormality flag is set to false negative.

[0084] Within a preset time period after the structured verification log tuple is recorded, the system continuously monitors the operating status of the equipment to be inspected corresponding to the work step. If a defect related to the work step occurs in the equipment to be inspected during this period, the system backtracks the structured verification log tuple and updates the subsequent anomaly flag to a false negative. The preset time period ranges from 12 hours to 72 hours (e.g., set to 24 hours) to adapt to the development pattern of equipment defects. The defects related to the work step are determined by defect records in the equipment defect registration system that have a direct causal relationship with the work step. The false negative flag indicates that although the work step was confirmed as completed by the visual comparison model and the operator, it was later proven that the work step was not actually executed correctly; it is a post-hoc truth signal that quantifies the reliability of the work step verification mechanism.

[0085] It should be noted that the determination of the causal relationship between the defects related to the aforementioned work steps and the work steps themselves is achieved either by the equipment operation and maintenance personnel manually marking them in the equipment defect registration system through associated fields, or by the system automatically matching them based on a predefined step-equipment-defect association rule base. This association rule base is pre-built by domain experts based on their experience in hydropower station equipment operation and maintenance, covering the mapping relationship between common work steps and corresponding equipment defect types. When a new defect record is added to the equipment defect registration system, the system automatically retrieves the verification logs of all completed related work steps on the same equipment within a preset time period, and updates the subsequent anomaly flags of verification logs that meet the causal relationship conditions to false negatives.

[0086] S4.3: Collect the structured verification log tuples of each job step to obtain the historical verification results of the job step.

[0087] Based on the record-by-record structured verification log tuples, the structured verification log tuples for each job step are aggregated according to the step identifier to obtain the historical verification results for that job step. The historical verification results are persistently stored in the verification log database, using the step identifier as the key and a list of all structured verification log tuples for that job step arranged in chronological order as the value. These historical verification results provide observation samples for subsequent updates to the Bayesian posterior model and accumulate continuously with the increase in the number of jobs.

[0088] S4.4: Based on the historical verification results, maintain a binomial distribution Bayesian posterior model for each operation step, wherein the prior of the Bayesian posterior model is two beta distributions with a shape parameter of one.

[0089] The historical verification results show that a Bayesian posterior model with a binomial distribution is maintained for each operation step. The prior of the Bayesian posterior model consists of two beta distributions with a shape parameter of one, i.e. The prior is a non-informative prior on a uniform distribution. The first shape parameter is the a priori shape parameter. The beta distribution is the conjugate prior of the binomial likelihood function, therefore the posterior distribution is still a beta distribution, enabling online updates to have an analytical form without numerical integration.

[0090] The lack of information prior The uniform distribution indicates that, without any historical observation data, no prior assumptions are made regarding the failure probability of the operation step, equivalent to assuming that the failure probability is equally likely within the interval of 0 to 1. As the number of structured verification log tuples in the historical verification results increases, the posterior distribution gradually concentrates near the actual failure probability, the influence of the prior decreases relatively, and the system's quantification of the reliability of the operation step verification mechanism becomes increasingly accurate. For newly deployed systems, the absence of prior information allows the system to adopt a conservative high verification strength strategy in the early stages of historical data accumulation, automatically converging towards a verification strength that matches historical experience as data accumulates, without the need for manual parameter initialization.

[0091] S4.5: For each structured verification log tuple of the operation step in the historical verification results, if the subsequent anomaly flag is a false negative, increment the first shape parameter of the beta distribution by one; otherwise, increment the second shape parameter of the beta distribution by one to obtain the updated beta distribution.

[0092] For each structured verification log tuple of the work step in the historical verification results, a posterior update is performed in chronological order. If the subsequent anomaly flag is a false negative, the first shape parameter α of the beta distribution is incremented by one; otherwise, the second shape parameter β of the beta distribution is incremented by one, resulting in the updated beta distribution. Where α is the updated first shape parameter and β is the updated second shape parameter. The first shape parameter α accumulates the number of false negatives of the operation step, and the second shape parameter β accumulates the number of non-false negatives of the operation step. The sum of the two, α + β, is equal to the total number of structured verification log tuples in the historical verification results of the operation step plus one.

[0093] Specifically, the posterior update operation performed in chronological order is implemented in batch processing in the system background. Whenever the status of the subsequent anomaly flag for a certain job step in the verification log database changes (from pending to false negative or normal), the incremental update of the Bayesian posterior model parameters for that job step is immediately triggered. After the update is completed, the Bayesian parameters are synchronously written to the persistent storage. The time complexity of the incremental update is O(1), which is not affected by the number of existing log entries in the historical verification results, ensuring that the system can still respond to new observations in real time after a large amount of historical data has accumulated.

[0094] S4.6: The estimated failure probability of the operation step is determined as the ratio of the first shape parameter to the sum of the first shape parameter and the second shape parameter, wherein the estimated failure probability is the expected value of the updated beta distribution.

[0095] Based on the updated beta distribution The estimated failure probability of the operation step. The ratio of the first shape parameter α to the sum of the first shape parameter α and the second shape parameter β is determined, and the formula for estimating the failure probability is: .in, The estimated failure probability for the stated task step. β is the first shape parameter of the updated beta distribution, and β is the second shape parameter of the updated beta distribution. The estimated failure probability... That is, the updated beta distribution The expected value reflects the posterior average probability of a false negative in the described operation step verification mechanism under the current historical verification results.

[0096] S4.7: The quantiles of the updated beta distribution at a preset confidence level are determined as the upper bound of the verification failure probability of the operation step. The quantiles at the preset confidence level are those that make the cumulative distribution function of the updated beta distribution equal to the failure probability value at the preset confidence level, and the upper bound of the verification failure probability is greater than the expected value of the updated beta distribution.

[0097] In the updated beta distribution Based on this, the quantiles of the updated beta distribution at the preset confidence level δ are determined as the upper bound of the verification failure probability of the operation step. That is, the formula for the upper bound of the verification failure probability is: .in, This is the upper bound for the verification failure probability of the aforementioned task step. The preset confidence level has a value range of 0 to 1. β and α are the first and second shape parameters of the updated beta distribution, respectively. This is the inverse function of the cumulative distribution function of the updated beta distribution. The preset confidence level δ can be adjusted within the range of 0.005 to 0.05 (e.g., a value of 0.01, i.e., determining the upper bound of the verification failure probability with a 99% confidence level) according to the stringency of security requirements. The upper bound of the verification failure probability... Greater than the expected value of the updated beta distribution The difference between the two converges as the number of structured validation log tuples in the historical validation results increases, reflecting a conservative quantification of the estimation uncertainty when the sample is finite.

[0098] In the initial stage, when historical verification data is relatively scarce, and The significant difference between the two reflects the conservatism of Bayesian uncertainty estimation for small samples: when only a few historical observations are available, even if no false negatives are observed, the system still maintains a high level of conservatism. The system alerts users to potential failures not yet captured by historical data, driving the system to maintain a high level of validation strength. As the number of historical validations increases, Gradually towards With convergence and a tighter upper bound, the system can automatically reduce unnecessary verification redundancy based on more accurate risk quantification, achieving a dynamic balance between security and operational efficiency.

[0099] Example 5

[0100] Based on Example 4, the method further includes:

[0101] S4.7.1: When the upper bound of the verification failure probability is less than the first safety threshold, the step verification strength is set as the baseline verification strength, and the operation steps are controlled by the baseline verification strength.

[0102] The upper bound of the verification failure probability Based on this, when the upper bound of the verification failure probability is... Less than the first safety threshold At that time, the step verification strength is set as the baseline verification strength, and the operation steps are controlled based on the baseline verification strength. The first safety threshold. The threshold can be adjusted within the range of 0.005 to 0.02 according to safety requirements. The benchmark verification strength includes a benchmark confirmation threshold and a benchmark re-verification count. The benchmark confirmation threshold is preferably 0.85, and the benchmark re-verification count is preferably 1. When the visual comparison score of the work step is not lower than the benchmark confirmation threshold and the operator provides one confirmation, the work step is determined to be completed. This level is suitable for work steps with sufficient historical verification results and high verification mechanism reliability, avoiding wasting verification resources on high-reliability steps.

[0103] S4.7.2: When the upper bound of the verification failure probability is not less than the first security threshold and not greater than the second security threshold, the step confirmation threshold is set to a first confirmation threshold that is less than the baseline confirmation threshold, and the number of re-verifications is set to a first re-verification number that is greater than the baseline re-verification number. The operation steps are controlled by the first confirmation threshold and the first re-verification number, and the second security threshold is greater than the first security threshold.

[0104] When the upper bound of the verification failure probability Not less than the first safety threshold And not greater than the second safety threshold When this happens, the step confirmation threshold is set to a first confirmation threshold that is less than the baseline confirmation threshold, and the number of re-verifications is set to a first number of re-verifications that is greater than the baseline number of re-verifications. The operation steps are controlled using the first confirmation threshold and the first number of re-verifications. Second security threshold. greater than the first security threshold The threshold can be adjusted within the range of 0.03 to 0.1 (e.g., 0.05). The first confirmation threshold is preferably 0.75, and the first number of re-verifications is preferably 3. When the visual comparison score of the task step is not lower than the first confirmation threshold in three consecutive frames and the operator provides confirmation once, the task step is determined to be completed. This setting is suitable for task steps where the false negative rate in historical verification results has increased, improving sensitivity to missed detections by lowering the confirmation threshold and increasing the number of re-verifications.

[0105] S4.7.3: When the upper bound of the verification failure probability is greater than the second security threshold, the enhanced verification mode is triggered, requiring the operation step to pass the triple verification of visual comparison, voice confirmation and physical button confirmation.

[0106] When the upper bound of the verification failure probability Greater than the second security threshold When the enhanced verification mode is triggered, the operation step requires simultaneous verification through visual comparison, voice confirmation, and physical button confirmation. In this enhanced verification mode, the confirmation threshold for visual comparison is restored to the baseline confirmation threshold of 0.85, but the operator must complete both voice confirmation and physical button confirmation manually; neither can be omitted. If any one of the three verification channels fails, the operation step is deemed incomplete, triggering remote expert intervention. This mode is suitable for operation steps with a significantly high false negative rate in historical verification results, allocating verification resources to the most unreliable critical steps and avoiding missed detections caused by relying solely on visual comparison and single confirmation for low-reliability steps.

[0107] The adaptive adjustment mechanism of the three levels of verification intensity organically combines the reliability quantification of the step verification mechanism with resource allocation. For multiple operation steps with significant differences in verification reliability in the same inspection task, the system automatically concentrates more manual confirmation and time costs on high-risk steps, while low-risk steps are quickly passed with the fewest verification actions, so that the overall inspection efficiency and safety are improved simultaneously. This overcomes the contradiction of resource waste and risk omission caused by the traditional static verification method's "one-size-fits-all" treatment of all steps.

[0108] S4.7.4: Multiply the difference between the estimated failure probability of each of the above-mentioned work steps and one to obtain the joint safety degree of no omissions in the entire inspection process.

[0109] Based on the estimated failure probability The estimated failure probability of each of the aforementioned work steps is calculated and multiplied by the difference between the two values ​​to obtain the joint safety level of no missed inspections throughout the entire inspection process. That is, the joint safety formula is .in, To ensure a comprehensive and uninterrupted safety assessment throughout the inspection process, This represents the total number of operational steps throughout the entire inspection process. Let be the estimated failure probability of the i-th job step, where i is the job step number. The joint safety factor... This reflects the joint probability that no missed checks will occur throughout the entire inspection process, given the reliability of the verification mechanisms at each operational step. It elevates the independent verification of a single step to a joint safety guarantee for the entire process. When the aforementioned joint safety level... When the overall safety level falls below the preset safety target, the system automatically identifies the operational steps that contribute the most to the decrease in the joint safety level and prompts for enhanced verification.

[0110] It is worth noting that the calculation of the joint safety factor relies on the assumption that the verification results of each operational step are independent. In actual hydropower station inspection scenarios, adjacent steps are usually performed consecutively by the same operator under similar time and space conditions, and the correlation between steps cannot be ignored. For sets of adjacent steps with strong correlation, a step correlation coefficient matrix can be introduced in the system backend to replace the simple multiplication under the independence assumption with a modified joint safety factor formula, so as to more accurately reflect the actual distribution of the risk of missed detection throughout the process. When the correlation coefficient matrix cannot be obtained accurately, the simple multiplication formula provides a conservative lower bound estimate of the joint safety factor, which has conservative significance for safety decision-making.

[0111] S4.7.5: The first preset number of operation steps with the highest estimated failure probability are identified as safety bottleneck steps, and additional verification time and verification resources are reserved for the safety bottleneck steps in the inspection plan.

[0112] Based on the above estimates of failure probability With the aforementioned joint security Filter out the estimated failure probability The maximum number of pre-set operation steps are identified as safety bottleneck steps, and additional verification time and resources are reserved for these steps in the inspection plan. This pre-set number can be adjusted from 2 to 5 depending on the scale of the inspection task. The additional verification time includes extending the number of visual comparison observation frames and the operator confirmation waiting time for the operation step; the additional verification resources include configuring a remote expert pre-review process and highlighting key video recordings for the safety bottleneck steps. The safety bottleneck steps and their verification intensity adjustment history are persistently stored as part of the security audit trail, enabling subsequent inspections to re-determine safety bottleneck steps based on updated historical verification results.

[0113] The identification of safety bottleneck steps is automatically calculated by the system and pushed to the back-end collaborative dashboard before the start of each inspection task. This allows operation and maintenance personnel to be aware of high-risk aspects of the current inspection task in advance and to allocate remote expert resources to be on standby online in real time during the execution of the safety bottleneck step. The verification time and verification resources for the safety bottleneck step are reserved during the inspection planning and scheduling stage, ensuring that on-site operators have sufficient verification margin when executing the safety bottleneck step without affecting the overall inspection task schedule. As historical verification data continues to accumulate, the estimated failure probability of each operation step tends to stabilize, the composition of safety bottleneck steps also tends to be determined, and the allocation of safety resources in the system becomes increasingly accurate, gradually forming a data-driven closed loop for operation step risk quantification and resource allocation.

[0114] Example 6

[0115] Based on Example 1, the construction of a semantic 3D dense map through tightly coupled visual-inertial navigation fusion localization and mapping includes:

[0116] S1.1: Deploy three parallel pipelines for positioning and mapping with different computational accuracies on the augmented reality head-mounted device. The three parallel pipelines share the multi-sensor data stream. The three parallel pipelines include a first pipeline, a second pipeline, and a third pipeline.

[0117] Specifically, to balance positioning accuracy and power consumption constraints, the computational accuracy and operating power consumption of the first, second, and third pipelines decrease sequentially, forming a multi-precision heterogeneous pipeline set with complementary accuracy and power consumption. The three parallel pipelines run in parallel on the edge processor of the AR headset as independent threads or processes, each outputting independent pose estimates, providing diverse information sources for subsequent fusion and adaptive scheduling.

[0118] The power consumption ratio of the three parallel pipelines is approximately 1:0.35:0.12. With all three pipelines fully open, the total power consumption increase compared to a single full-precision pipeline is approximately 47%. However, this comes at the cost of the dual advantages of real-time continuous pose estimation and accuracy benchmark correction, as well as the potential for subsequent moment fusion and outlier detection. When the battery is sufficiently high, all three pipelines are fully open. As the battery level decreases, high-power pipelines are sequentially shut down according to their power contribution, allowing the device's battery life to adaptively extend with the task progress, supporting a full-cycle inspection of six to eight hours.

[0119] S1.2: The first pipeline is inferred with a first computational precision and run on sampling frames at a preset interval to obtain the pose estimate of the first pipeline. The pose estimate of the first pipeline is used as the pose calibration reference.

[0120] Based on the three parallel pipelines, the first pipeline is inferred with a first computational precision, running on sampling frames at preset intervals to obtain the pose estimate of the first pipeline. The first computational precision is 32-bit floating-point full precision, and the preset number of frames can be adjusted within the range of 3 to 10 according to the computing power margin. The first pipeline runs in a standard configuration of full-precision vision-inertial navigation tightly coupled fusion localization and mapping, including all modules of front-end feature extraction, inertial pre-integration, back-end bundle adjustment optimization, and loop closure detection. The pose estimate of the first pipeline has the highest precision but the lowest operating frequency, and is used as a pose calibration benchmark for subsequent correction of the output of the low-precision pipeline, avoiding the cumulative drift caused by the long-term operation of the low-precision pipeline.

[0121] S1.3: Perform inference on the second pipeline with a second computational precision, running frame by frame at full frame rate to obtain the pose estimate of the second pipeline. The second computational precision is lower than the first computational precision.

[0122] Based on the first pipeline, the second pipeline is inferred with a second computational precision, running frame-by-frame at full frame rate to obtain the pose estimate of the second pipeline. The second computational precision is lower than the first computational precision. The second computational precision is 8-bit integer quantization precision, and the full frame rate is preferably 30 frames per second, consistent with the image sampling frequency of the multi-sensor data stream. The second pipeline uses a tensor inference acceleration toolchain to train and quantize the depth feature extraction subnetwork and matching subnetwork in the first pipeline, reducing inference power consumption to approximately 35% of the full-precision pipeline while maintaining feature discriminative power. The second pipeline, as a real-time main pipeline, outputs continuous pose estimates, providing low-latency pose feedback for AR rendering.

[0123] S1.4: The third pipeline is binarized with the visual feature descriptor and then run to obtain the pose estimate of the third pipeline. The third operational precision is lower than the second operational precision.

[0124] Based on the second pipeline, the third pipeline is run with a third computational precision after binarizing the visual feature descriptor, resulting in the pose estimation of the third pipeline. This third computational precision is lower than the second. The third computational precision is 4-bit binarization precision, meaning the visual feature descriptor is binarized into a 4-bit representation using a sign function. The third pipeline operates with lower power consumption every 3rd frame, serving as an auxiliary pipeline to provide redundant pose estimation. The power consumption of the third pipeline is approximately 12% of that of the full-precision pipeline. Although the third pipeline suffers significant accuracy loss in low-texture regions, it can still provide effective pose estimation in well-lit regions, making it the lowest-power member of the multi-precision heterogeneous pipeline set.

[0125] S1.5: The pose estimation of the second pipeline and the pose estimation of the third pipeline are fused to obtain a fused pose, and the fused pose is corrected on the sampling frame using the pose calibration reference to obtain a corrected pose.

[0126] Based on the pose estimates of the first, second, and third pipelines, the pose estimates of the second and third pipelines are fused to obtain a fused pose. This fused pose is then corrected on the sampled frame using the pose calibration benchmark to obtain the corrected pose. The fusion employs a weighted average method positively correlated with computational accuracy, with the second pipeline having a higher weight than the third pipeline. On each sampled frame, using the pose estimate of the first pipeline as a benchmark, the offset between the fused pose and the benchmark is calculated, and this offset is exponentially decayed to compensate for the fused pose in subsequent non-sampled frames, resulting in the corrected pose. The corrected pose combines the advantages of the real-time performance of the second pipeline, the redundancy of the third pipeline, and the accuracy benchmark of the first pipeline.

[0127] The time constant for the exponential decay compensation is preferably half the time length of the preset number of frames, so that the offset compensation is smoothly distributed between two adjacent sampling frames, avoiding pose jumps at the sampling frames. When the second pipeline or the third pipeline fails to track within the sampling frame interval, the pose continuity is maintained by pre-integration extrapolation of the inertial measurement unit until the next sampling frame is corrected again with the pose calibration reference of the first pipeline.

[0128] S1.6: Based on the corrected pose, construct the semantic 3D dense map through visual-inertial navigation tightly coupled fusion mapping.

[0129] The vision-inertial navigation tightly coupled fusion mapping includes five stages: keyframe selection, local bundle adjustment optimization, loop closure detection and global pose map optimization, dense point cloud reconstruction, and semantic segmentation fusion. Keyframe selection is based on the change amplitude of the corrected pose; local bundle adjustment optimization uses the keyframe and its co-view relationship as input; loop closure detection is based on depth global descriptor matching; and global pose map optimization constructs the pose map using loop closure constraints as additional edges. Dense point cloud reconstruction uses the corrected pose and depth map as input, and is fused using a truncated signed distance function to obtain a dense surface. Semantic segmentation fusion projects the image-level semantic segmentation result onto the dense surface to complete semantic annotation, ultimately obtaining the semantically annotated 3D dense map.

[0130] The keyframe selection strategy is based on the condition that the rotation angle of the corrected pose relative to the previous keyframe exceeds a preset rotation threshold or the translation distance exceeds a preset translation threshold, ensuring that the keyframe set uniformly covers the inspection area in space. The semantic segmentation fusion uses a lightweight semantic segmentation network to infer keyframe images, projecting pixel-level semantic labels along the depth direction to corresponding 3D points on the dense surface. Stable semantic annotations are obtained through multi-keyframe voting fusion, ensuring that each dense point in the semantic 3D dense map simultaneously carries geometric coordinates and semantic category information, providing a semantic retrieval basis for spatial association binding in subsequent operation steps. Loop closure detection employs a global image descriptor based on deep network coding, performing efficient nearest neighbor retrieval in the keyframe database of the semantic 3D dense map to achieve cross-scene loop closure candidate pair identification. After random sampling and consistent geometric verification, loop closure constraints are added to the global pose graph optimization to eliminate long-distance cumulative drift, ensuring that the global consistency accuracy of the semantic 3D dense map does not exceed 0.3 meters.

[0131] Example 7

[0132] Based on Example 6, the method further includes:

[0133] S1.7: Calculate the first moment of the pose estimation of the first pipeline, the pose estimation of the second pipeline, and the pose estimation of the third pipeline according to the weights positively correlated with their respective calculation accuracy, obtain the moment fusion pose, and output the moment fusion pose as the pose for localization and mapping.

[0134] Based on the pose estimates of the first pipeline, the second pipeline, and the third pipeline, first-order moments are calculated for the pose estimates T1 of the first pipeline, T2 of the second pipeline, and T3 of the third pipeline according to weights positively correlated with their respective calculation accuracy, thus obtaining the moment fusion pose. That is, the first-order moment fusion formula, the calculation formula is:

[0135]

[0136] in, For the moment fusion pose, T1, T2, and T3 are the pose estimates of the first pipeline, the second pipeline, and the third pipeline, respectively. , , The fusion weights of the first pipeline, the second pipeline, and the third pipeline are respectively, and satisfy the following conditions: The fusion weights are positively correlated with their respective calculation accuracy. The fusion weights of the first pipeline... The preferred value is 0.6, which is the fusion weight of the second pipeline. The preferred value is 0.3, which is the fusion weight of the third pipeline. The preferred value is 0.1, and those skilled in the art can adaptively adjust it based on the measured accuracy of each pipeline under the constraint that the sum of the weights is one. The moment fusion pose... As the pose output for localization and mapping.

[0137] The first-order moment fusion is performed on the Lie group SE(3). The weighted average of the pose is achieved through a combination of logarithmic mapping, linear weighting, and exponential mapping to ensure that the fusion result is still a valid rotation matrix plus translation vector, rather than the non-orthogonal result that may be generated by directly weighting the elements of the rotation matrix in Euclidean space. In the specific implementation, taking the pose estimation T1 of the first pipeline as the reference point, T2 and T3 are mapped to the tangent space of T1 by left multiplying the inverse of T1, and then weighted averaging is performed in the tangent space. Finally, the moment fused pose is obtained by synthesizing it with T1 through exponential mapping. This operation exhibits good numerical stability when pose estimations are similar.

[0138] S1.8: Calculate the second moment of the pose estimation of the first pipeline, the pose estimation of the second pipeline, and the pose estimation of the third pipeline relative to the fused pose to obtain the pose covariance matrix, and use the trace of the pose covariance matrix as the fusion uncertainty.

[0139] In the moment fusion pose Based on the pose estimation T1 of the first pipeline, the pose estimation T2 of the second pipeline, and the pose estimation T3 of the third pipeline, the pose relative to the moment fusion is calculated. The second moment is used to obtain the pose covariance matrix. and the pose covariance matrix The trace as a fusion uncertainty That is, the second-order moment uncertainty formula, the calculation formula is:

[0140] , .in, To integrate uncertainty, Let be the pose covariance matrix. For the trace operation of a matrix, Let i be the fusion weight of the i-th pipeline. For the pose estimation of the i-th pipeline, Let i be the moment fusion pose, and i be the pipeline index. The fusion uncertainty... This reflects the degree of dispersion among the pose estimates of the three parallel pipelines. A larger dispersion indicates a greater discrepancy in the pose estimates of each pipeline, resulting in lower reliability of the moment fusion pose. The fusion uncertainty... pose fused with the moment Output them together for subsequent anchoring verification and operation control.

[0141] Specifically, the fusion uncertainty The status indicator area is pushed to the AR field of view in real time, using color coding to indicate the operator's current location reliability. When the value is below the preset low uncertainty threshold, it will be displayed in green; when it is between the low uncertainty threshold and the high uncertainty threshold, it will be displayed in yellow; when it exceeds the high uncertainty threshold, it will be displayed in red and a warning text "Low positioning reliability, please operate with caution" will be superimposed on the operation guidance panel. Simultaneously serving as a reference input for pose consistency verification during the anchoring verification phase, when When the position is large, the rotation and translation tolerances of the pose consistency test should be appropriately relaxed to coordinate the anchoring verification and the reliability of real-time positioning.

[0142] S1.9: Calculate the fourth moment relative to the moment fusion pose for the pose estimation of the first pipeline, the second pipeline, and the third pipeline to obtain the fourth moment deviation of each pipeline.

[0143] Combined with the aforementioned moment fusion pose With the pose covariance matrix The pose estimation of the first pipeline, the pose estimation of the second pipeline, and the pose estimation of the third pipeline are calculated relative to the moment fusion pose. The fourth moment of each pipeline is obtained by calculating the deviation of the fourth moment. That is, the formula for the deviation of the fourth moment is: .in, Let be the fourth moment deviation of the i-th pipeline. For the pose estimation of the i-th pipeline, Let i be the moment fusion pose, and i be the pipeline number. For the expected operation, It is the Euclidean norm. The fourth-order moment deviation... It is more sensitive to outliers than the second moment, and can detect outliers that are masked by the mean at the second moment level, such as a low-precision pipeline that continuously deviates from the other two pipelines in a specific region.

[0144] In actual calculations, since each of the three pipelines outputs a pose estimate at every moment, the expected computation... A sliding time averaging approximation is employed within a preset time window (preferably 10 consecutive frames) to capture the temporal persistence characteristics of pipeline deviation. Compared to single-frame fourth moments, the time sliding average effectively suppresses false triggering of fourth moments caused by instantaneous noise disturbances, only triggering a pipeline when it continuously deviates over multiple consecutive frames. The value is increased to above the preset deviation threshold, thereby achieving robust detection of truly outlier pipelines.

[0145] S1.10: The pipeline whose fourth-order moment deviation exceeds the preset deviation threshold is identified as an outlier pipeline. When the outlier pipeline is detected, the calculation accuracy of the outlier pipeline is adjusted to a target calculation accuracy greater than the current calculation accuracy of the outlier pipeline, or the weight of the pose estimation of the outlier pipeline to the moment fusion pose is reset to zero, and the contribution of the outlier pipeline to the moment fusion pose is paused.

[0146] The deviation of the fourth moment Based on this, pipelines whose fourth-order moment deviation exceeds a preset deviation threshold are identified as outliers. The preset deviation threshold is determined by the pose covariance matrix. The value is determined by four times the square of the second moment. When the outlier pipeline is detected, one of two approaches is taken: the first is to adjust the calculation accuracy of the outlier pipeline to a target calculation accuracy greater than the current calculation accuracy of the outlier pipeline, for example, to increase the 4-bit binarization accuracy of the third pipeline to 8-bit integer quantization accuracy; the second is to adjust the pose estimation of the outlier pipeline relative to the moment fusion pose. The weights are reset to zero, pausing the outlier pipeline's position on the moment fusion pose. The choice of handling method is based on the out-of-line duration and the remaining power of the device: if the out-of-line duration is less than the preset duration threshold and the power is sufficient, the first handling method is used to restore the effective contribution of the out-of-line; if the out-of-line duration exceeds the preset duration threshold or the power is low, the second handling method is used to reduce invalid calculation overhead.

[0147] Example 8

[0148] Based on Example 6, the method further includes:

[0149] S1.11: Monitor the power and heat dissipation status of the augmented reality head-mounted device to obtain power monitoring values ​​and heat dissipation temperature monitoring values.

[0150] Specifically, the battery power monitoring value is read by the battery management chip at a frequency of 1 Hz and output after being filtered by a moving average, and the value is taken as the remaining battery percentage; the heat dissipation temperature monitoring value is read by a temperature sensor located on the edge processor housing at a frequency of 1 Hz and output after being filtered by a moving average, and the value is taken as the Celsius temperature. The battery power monitoring value and the heat dissipation temperature monitoring value together constitute the operating status input of the AR headset, providing a basis for the dynamic scheduling of subsequent pipelines.

[0151] S1.12: When the power monitoring value is greater than the first power threshold, the first pipeline, the second pipeline and the third pipeline are turned on to obtain the current set of operating pipelines.

[0152] The first battery threshold can be adjusted within the range of 50% to 70% based on the remaining task time. At this level, all three parallel pipelines are running, and the moment fusion pose is... Simultaneously utilizing three-way pose estimation information, it achieves the highest positioning accuracy but also consumes the most power, making it suitable for inspection phases where power is sufficient and task priority is high.

[0153] S1.13: When the power monitoring value is not greater than the first power threshold and not less than the second power threshold, the third pipeline is shut down from the current operating pipeline set, while the first pipeline and the second pipeline are retained, and the first power threshold is greater than the second power threshold.

[0154] The second battery threshold can be adjusted within the range of 20% to 40% according to the battery discharge curve. At this level, the third pipeline has the lowest power consumption but also the lowest accuracy; turning it off has minimal impact on overall positioning accuracy and saves approximately 12% of total power consumption, thus extending the battery life of the AR headset during the medium battery phase. The moment fusion pose... It is obtained solely by fusing the pose estimates of the first pipeline and the second pipeline with normalized weights.

[0155] S1.14: When the power monitoring value is less than the second power threshold, the first pipeline and the third pipeline are shut down from the current operating pipeline set, leaving only the second pipeline, and the energy-saving inspection mode is triggered to guide the operator to an area where the lighting conditions meet the preset conditions.

[0156] The preset condition is an ambient light intensity of no less than 200 lux, determined by the ambient light sensor reading of the AR head-mounted device. In the energy-saving inspection mode, a navigation guide arrow is rendered in the AR field of view to lead to the nearest high-light area, and after the operator arrives at the high-light area, a prompt is given to prioritize the work steps within that high-light area. The high-light area has rich visual features, and the 8-bit integer quantization accuracy of the second pipeline is sufficient to maintain stable positioning, thus allowing the inspection task to continue in a single-pipeline mode during low-battery phases.

[0157] S1.15: When the heat dissipation temperature monitoring value exceeds the preset temperature threshold, the running frame rate of the first pipeline in the current running pipeline set is adjusted to a target running frame rate that is lower than the current running frame rate of the first pipeline.

[0158] The preset temperature threshold can be adjusted within the range of 55 to 65 degrees Celsius according to the rated operating temperature of the edge processor. The target frame rate is preferably increased by raising the preset frame count of the first pipeline from 5 to 10, that is, reducing the first pipeline from running once every 5 frames to running once every 10 frames, thereby reducing the full-precision computational load of the first pipeline. If the heat dissipation temperature monitoring value continues to rise above the preset shutdown threshold after reducing the frame rate, the forced sleep protection of the AR headset is triggered, and the operator is prompted to suspend inspection. The overheating frame rate reduction strategy prioritizes ensuring that the device does not reduce its frequency or shut down due to overheating, translating the heat dissipation pressure into an acceptable loss of accuracy.

[0159] The scheduling judgment based on the power monitoring value and the heat dissipation temperature monitoring value is implemented using a hysteresis comparison method to avoid frequent switching of pipeline states near the threshold. Specifically, the lower boundary of the power threshold triggered by pipeline shutdown is 5 percentage points lower than the upper boundary triggered by pipeline startup. For example, the third pipeline shuts down when the power drops below 60% and can only be restarted when the power recovers to above 65%. The over-temperature frame rate reduction trigger threshold is 60 degrees Celsius, and the threshold for restoring the original frame rate is 55 degrees Celsius, forming a hysteresis interval of 5 degrees Celsius. This hysteresis comparison mechanism effectively suppresses scheduling jitter caused by small fluctuations in sensor readings, ensuring the stable execution of the pipeline scheduling strategy.

[0160] Example 9

[0161] like Figure 3 As shown, this invention provides a standard operating procedure guidance and control device for industrial environments in hydropower plants. This device can be integrated into edge computing devices or backend servers, and specifically includes:

[0162] The mapping module 301 is used to collect multi-sensor data streams from the AR head-mounted device, construct a semantic 3D dense map through visual-inertial navigation tight coupling fusion positioning and mapping, and calculate the visual noise index for each grid cell of the semantic 3D dense map.

[0163] The mapping module 301 deploys three parallel pipelines for positioning and mapping with different computational precisions on the AR head-mounted device, and the three parallel pipelines share the multi-sensor data stream. Specifically, the first pipeline performs 32-bit floating-point full-precision floating-point operations at preset keyframe intervals (e.g., every 5th frame) as a precision benchmark; the second pipeline performs inference with 8-bit integer first quantization precision at full frame rate (e.g., 30 frames per second) as a real-time pipeline; and the third pipeline performs binarization of the visual feature descriptors with 4-bit second quantization precision as an auxiliary pipeline, with the power consumption of the first, second, and third pipelines decreasing sequentially. The mapping module 301 is also used to perform multi-precision moment fusion on the pose estimates of the three parallel pipelines. It calculates the first moment with weights positively correlated with the calculation accuracy (e.g., 0.6, 0.3, 0.1) to obtain the fused pose and output it as the final pose. It also calculates the second moment between the pose estimates of the three pipelines, using the trace of the corresponding covariance matrix as the fusion uncertainty. Finally, it calculates the fourth moment to detect outliers that continuously deviate from the other pipelines. When an outlier is detected, it automatically increases the calculation accuracy of that outlier or pauses its contribution to the fused pose. Simultaneously, it is configured with an accuracy-power consumption adaptive scheduling strategy, including monitoring device power and heat dissipation status. When the power is greater than 60%, all three pipelines are activated; when the power is between 30% and 60%, the third pipeline is deactivated; when the power is less than 30%, only the second pipeline is retained, and an energy-saving inspection mode is triggered to guide the user to a well-lit area; when the heat dissipation temperature exceeds a threshold, the operating frame rate of the first pipeline is reduced to prevent overheating and frequency reduction.

[0164] The fingerprint database module 302 is used to establish a multimodal spatial fingerprint database at each inspection point based on the semantic three-dimensional dense map.

[0165] The fingerprint database construction module 302 is used to collect and fuse WiFi signal strength, Bluetooth Low Energy beacon signals, geomagnetic anomaly data, and depth geometric features in the area surrounding each key inspection point during the persistent semantic map construction process, thereby constructing a multimodal spatial fingerprint database. When the AR device is running, it first uses the K-nearest neighbor algorithm to perform coarse localization of the device using the multimodal spatial fingerprint database, achieving spatial area locking with an accuracy of no more than 2 meters, providing initial prior poses for subsequent visual fine localization.

[0166] The virtual-real anchoring module 303 is used to perform coarse positioning of the AR head-mounted device based on the multimodal spatial fingerprint database, then perform visual feature matching for fine positioning of the multi-sensor data stream, and adaptively set the anchoring verification strength according to the visual noise index to obtain the anchoring result.

[0167] The virtual-real anchoring module 303 is used to perform precise matching based on coarse positioning, employing SuperPoint deep feature extraction combined with RANSAC geometric verification. To overcome the problem of significant differences in visual conditions in different areas of the hydropower station (such as the well-lit main powerhouse and the dimly lit, repetitive corridors), the virtual-real anchoring module 303 is configured with an adaptive anchoring verification mechanism based on spatial noise perception: when entering an inspection point, the visual noise index NI of the corresponding grid cell is queried. The visual noise index NI is calculated and updated as follows: the average number of feature points extracted, the average matching confidence, and the average matching inlier rate of the grid in each mapping iteration are statistically analyzed, normalized, and then weighted and summed according to preset weights (such as 0.4, 0.3, and 0.3) to obtain NI, which is then stored in the form of a heatmap. After each anchoring verification, the NI is self-calibrated and updated using an exponential moving average method (such as 0.9 times the historical value + 0.1 times the current observation value). During precise positioning, the virtual-real anchoring module 303 dynamically sets verification parameters based on NI: the matching threshold is set as the product of the baseline threshold and the NI linear mapping coefficient (the higher the NI, the stricter the threshold); the required number of consecutive consistent frames is set as the sum of the baseline number of frames and the NI linear mapping increment (3 frames when NI=0, 7 frames when NI=1); and the timing verification window size is set as the number of consecutive consistent frames plus the preset window increment. When NI exceeds the preset high noise threshold (e.g., 0.8), multimodal auxiliary verification is additionally triggered, forcing the operator to confirm the anchoring mark via voice or physical button, thereby ensuring the perception reliability of high-noise areas.

[0168] The operation control module 304 is used to render standard operation guidance information in the AR field of view based on the anchoring result, statistically analyze the historical verification results of the operation steps, obtain the upper bound of the verification failure probability of the operation steps, adaptively set the step verification intensity based on the upper bound of the verification failure probability, and perform step-level mandatory control to obtain the step control result.

[0169] The job control module 304 renders highlighted wireframes of the operation target (green for pending operation, yellow for operation, and red for deviation alarm) and navigation arrows in the AR field of view based on precise anchoring, and performs step-level control using YOLOv8 visual comparison. To quantify the reliability of the verification mechanism itself, the job control module 304 introduces a statistical model verification mechanism, that is, after each step is executed, a structured verification log is recorded, including step ID, region ID, environmental conditions, visual score, confirmation method, and subsequent anomaly markers. If a step is judged to be completed but the device has a related defect within 24 hours, it is marked as a "false negative". A binomial distribution Bayesian posterior model with a prior of Beta(1,1) is maintained for each step. After each verification is completed, if a false negative occurs, the first shape parameter is incremented by 1, otherwise the second shape parameter is incremented by 1. Based on this, the failure probability is estimated, and the quantile of this Beta distribution at the 99% confidence level is determined as the upper bound of the verification failure probability. The job control module 304 adaptively adjusts the verification intensity according to the upper bound of the verification failure probability. If the upper bound of the verification failure probability is less than the first safety threshold (e.g., 0.01), the verification strength of the current step is maintained; if the upper bound of the verification failure probability is between the first and second safety thresholds, the step confirmation threshold is reduced and the number of re-verifications is increased; if the upper bound of the verification failure probability is greater than the second safety threshold, an enhanced verification mode is triggered, requiring simultaneous verification through visual comparison, voice confirmation, and physical button confirmation. Furthermore, the operation control module 304 calculates the joint safety level of no missed checks throughout the inspection process as the product of the difference between the estimated failure probability and the probability of failure for each of all operation steps, and automatically identifies the top preset number of operation steps with the highest estimated failure probability as safety bottleneck steps, reserving additional verification time and resources for these safety bottleneck steps in the inspection plan.

[0170] The interactive collaboration module 305 is used to integrate multi-channel interactive events to confirm the results of the step control and to encapsulate and push the operation process data to the back-end collaborative dashboard.

[0171] The interactive collaboration module 305 is designed to address the high noise levels (≥85dB) of hydropower stations and the industrial constraints of wearing gloves. It simultaneously acquires three-channel interactive event streams: voice (beamformed and RNNoise-denoised), physical button input, and gesture recognition based on MediaPipe. A priority arbitration mechanism (physical button input > voice > gesture) combined with a ±500ms time window alignment enables multimodal fusion confirmation. Simultaneously, the interactive collaboration module 305 encapsulates multi-dimensional data, including the AR terminal's first-person view, SOP dashboard, and equipment radar chart, in a protocol buffer and pushes it to the backend via a 5G / WiFi 6 network with a latency of no more than 200ms. The backend renders a three-column collaborative dashboard and supports bidirectional remote expert collaboration with a latency of no more than 100ms based on the WebRTC P2P protocol. The entire process is recorded and archived, achieving a real-time collaborative system between the terminal and the backend management platform.

[0172] Through the description of the above embodiments, those skilled in the art can clearly realize the technical solution of the present invention. The present invention calculates the visual noise index for each grid cell of a semantically dense 3D map and persistently stores it in the form of a heatmap, enabling AR head-mounted devices to adaptively set the anchoring verification strength based on the differences in visual conditions in the area. By maintaining a Bayesian posterior model with a beta distribution for each operation step and continuously updating it with the occurrence of relevant defects within a preset time period as a false negative indicator, the reliability of the step verification mechanism itself is upgraded from binary judgment to a quantifiable upper bound of the verification failure probability, and the step verification strength is adaptively adjusted. By deploying three parallel pipelines with different computational accuracies on the AR head-mounted device and dynamically adjusting them according to a moment fusion and power dissipation scheduling strategy, power consumption and heat dissipation pressure are significantly reduced while ensuring positioning accuracy. The present invention is not limited to the above embodiments; any modifications, equivalent substitutions, improvements, etc., made within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A water power plant industrial environment standard operation guide management method, characterized by, include: Collect multi-sensor data streams from augmented reality head-mounted devices, construct semantic 3D dense maps through vision-inertial navigation tight coupling fusion localization and mapping, and calculate the visual noise index for each grid cell of the semantic 3D dense map; Based on the semantic 3D dense map, a multimodal spatial fingerprint database is established at each inspection point; The augmented reality head-mounted device is coarsely located based on the multimodal spatial fingerprint database, and then finely located by visual feature matching of the multi-sensor data stream. The anchoring verification strength is adaptively set according to the visual noise index to obtain the anchoring result. Based on the anchoring results, standard operation guidance information is rendered in the augmented reality field of view. The historical verification results of the operation steps are statistically analyzed to obtain the upper bound of the verification failure probability of the operation steps. Based on the upper bound of the verification failure probability, the step verification intensity is adaptively set and step-level mandatory control is performed to obtain the step control results. The results of the step control are confirmed by integrating multi-channel interactive events, and the operation process data is encapsulated and pushed to the back-end collaborative dashboard.

2. The method according to claim 1, characterized in that, The process of coarsely locating the augmented reality head-mounted device based on the multimodal spatial fingerprint database, then finely locating it by visual feature matching of the multi-sensor data stream, and adaptively setting the anchoring verification strength according to the visual noise index to obtain the anchoring result includes: When the augmented reality head-mounted device enters the inspection point, the visual noise index of the grid cell where the inspection point is located is queried; The matching threshold for precise localization of the visual feature matching is set as the product of the baseline threshold and the coefficient obtained by linear mapping of the visual noise index. The number of consecutive consistent frames required for anchoring is set as the sum of the base number of frames and the increment obtained by linear mapping of the visual noise index; The timing verification window size is set to the sum of the number of consecutive consistent frames and the preset window increment; The multi-sensor data stream is precisely localized by visual feature matching according to the matching threshold, and the pose obtained by the precise localization is anchored and verified within the time verification window. When the number of consecutive frames that pass the verification reaches the number of consecutive consistent frames, the anchoring is determined to be successful, and the anchoring result is obtained. When the visual noise index exceeds a preset noise threshold, multimodal assisted verification is triggered, requiring the operator to confirm the anchoring result via voice confirmation or physical button press.

3. The method according to claim 2, characterized in that, The visual noise index is obtained and updated in the following ways: The average number of feature points extracted, the average confidence score of visual feature matching, and the average in-point matching rate of the grid cell were statistically analyzed in each localization and mapping process. The average number of extracted feature points, the mean confidence score of visual feature matching, and the mean in-point matching rate are normalized and then complemented to obtain the normalized feature missing degree, the normalized matching confidence missing degree, and the normalized in-point missing degree. The normalized feature missing value, the normalized matching confidence missing value, and the normalized interior point missing value are weighted and summed according to the first weight, the second weight, and the third weight to obtain the visual noise index of the grid cell; wherein, the sum of the first weight, the second weight, and the third weight is one. The visual noise index of each grid cell is stored in the metadata of the semantic 3D dense map in the form of a heatmap; After completing the anchoring verification, the visual feature matching quality index observed during this anchoring verification process is recorded as the observation noise index. The visual noise index of the grid cell where the current anchoring verification is located is updated to a weighted sum of the visual noise index before the update and the current observed noise index, and the weight of the visual noise index before the update is greater than the weight of the current observed noise index.

4. The method according to claim 1, characterized in that, The statistical analysis of historical verification results for each work step to obtain an upper bound on the verification failure probability of that work step includes: After each operation step is executed, a structured verification log tuple is recorded, which includes step identifier, spatial area identifier, ambient lighting and texture conditions, visual comparison score, operator confirmation method and subsequent anomaly flags. If the operation step is determined to be completed, but a defect related to the operation step appears in the equipment to be inspected within a preset time period, then the subsequent abnormality flag will be set to false negative. The structured verification log tuples of each operation step are aggregated to obtain the historical verification results of the operation step; Based on the historical verification results, a binomial Bayesian posterior model is maintained for each operation step. The prior of the Bayesian posterior model is two beta distributions with a shape parameter of one. For each structured verification log tuple of the operation step in the historical verification results, if the subsequent anomaly flag is a false negative, the first shape parameter of the beta distribution is incremented by one; otherwise, the second shape parameter of the beta distribution is incremented by one to obtain the updated beta distribution. The estimated failure probability of the operation step is determined as the ratio of the first shape parameter to the sum of the first shape parameter and the second shape parameter, and the estimated failure probability is the expected value of the updated beta distribution. The quantiles of the updated beta distribution at a preset confidence level are determined as the upper bound of the verification failure probability of the operation step. The preset confidence level quantiles are those that make the cumulative distribution function of the updated beta distribution equal to the failure probability value at the preset confidence level, and the upper bound of the verification failure probability is greater than the expected value of the updated beta distribution.

5. The method according to claim 4, characterized in that, The method further includes: When the upper bound of the verification failure probability is less than the first safety threshold, the step verification intensity is set as the benchmark verification intensity, and the operation steps are controlled by the benchmark verification intensity. When the upper bound of the verification failure probability is not less than the first security threshold and not greater than the second security threshold, the step confirmation threshold is set to the first confirmation threshold which is less than the baseline confirmation threshold, and the number of re-verifications is set to the first number of re-verifications which is greater than the baseline number of re-verifications. The operation steps are controlled by the first confirmation threshold and the first number of re-verifications, and the second security threshold is greater than the first security threshold. When the upper bound of the verification failure probability is greater than the second security threshold, the enhanced verification mode is triggered, requiring the operation step to pass the triple verification of visual comparison, voice confirmation and physical button confirmation. Multiply the difference between the estimated failure probability of each of the above-mentioned work steps and one to obtain the joint safety degree of no omissions in the entire inspection process; The first preset number of operation steps with the highest estimated failure probability are identified as safety bottleneck steps, and additional verification time and resources are reserved for these safety bottleneck steps in the inspection plan.

6. The method according to claim 1, characterized in that, The method of constructing a semantically dense 3D map through tightly coupled vision-inertial navigation fusion localization and mapping includes: Three parallel pipelines for positioning and mapping with different computational accuracies are deployed on the augmented reality head-mounted device. The three parallel pipelines share the multi-sensor data stream. The three parallel pipelines include a first pipeline, a second pipeline, and a third pipeline. The first pipeline is inferred with a first computational precision and run on sampling frames at a preset frame interval to obtain the pose estimate of the first pipeline. The pose estimate of the first pipeline is used as the pose calibration reference. The second pipeline is inferred with a second computational precision and run frame by frame at full frame rate to obtain the pose estimate of the second pipeline. The second computational precision is lower than the first computational precision. The third pipeline is run after binarizing the visual feature descriptor with a third computational precision to obtain the pose estimate of the third pipeline. The third computational precision is lower than the second computational precision. The pose estimation of the second pipeline and the pose estimation of the third pipeline are fused to obtain a fused pose, and the fused pose is corrected on the sampling frame using the pose calibration reference to obtain a corrected pose. Based on the corrected pose, the semantic 3D dense map is constructed through visual-inertial navigation tightly coupled fusion mapping.

7. The method according to claim 6, characterized in that, The method further includes: The pose estimates of the first pipeline, the second pipeline, and the third pipeline are calculated with weights positively correlated with their respective calculation accuracy to obtain the moment fusion pose, and the moment fusion pose is used as the pose output for localization and mapping. The pose estimations of the first pipeline, the second pipeline, and the third pipeline are calculated to obtain the second moment relative to the fused pose, and the pose covariance matrix is ​​obtained. The trace of the pose covariance matrix is ​​used as the fusion uncertainty. The fourth moment relative to the moment fusion pose is calculated for the pose estimation of the first pipeline, the pose estimation of the second pipeline, and the pose estimation of the third pipeline, and the deviation of the fourth moment of each pipeline is obtained. Pipelines whose fourth-order moment deviation exceeds a preset deviation threshold are identified as outlier pipelines; When the outlier pipeline is detected, the calculation accuracy of the outlier pipeline is adjusted to a target calculation accuracy that is greater than the current calculation accuracy of the outlier pipeline, or the weight of the pose estimation of the outlier pipeline to the moment fusion pose is reset to zero, and the contribution of the outlier pipeline to the moment fusion pose is paused.

8. The method according to claim 6, characterized in that, The method further includes: Monitor the battery level and heat dissipation status of the augmented reality head-mounted device to obtain battery monitoring values ​​and heat dissipation temperature monitoring values; When the power monitoring value is greater than the first power threshold, the first pipeline, the second pipeline and the third pipeline are opened to obtain the current set of operating pipelines; When the power monitoring value is not greater than the first power threshold and not less than the second power threshold, the third pipeline is shut down from the current operating pipeline set, while the first pipeline and the second pipeline are retained, and the first power threshold is greater than the second power threshold. When the power monitoring value is less than the second power threshold, the first pipeline and the third pipeline are shut down from the current operating pipeline set, leaving only the second pipeline, and the energy-saving inspection mode is triggered to guide the operator to an area where the light meets the preset conditions. When the heat dissipation temperature monitoring value exceeds the preset temperature threshold, the running frame rate of the first pipeline in the current running pipeline set is adjusted to a target running frame rate that is lower than the current running frame rate of the first pipeline.

9. A standard operating procedure guidance and control device for industrial environment in a hydropower plant, characterized in that, include: The mapping module is used to collect multi-sensor data streams from augmented reality head-mounted devices, and construct a semantic 3D dense map by combining vision-inertial navigation tight coupling fusion positioning and mapping, and calculate the visual noise index for each grid cell of the semantic 3D dense map; The fingerprint database construction module is used to establish a multimodal spatial fingerprint database at each inspection point based on the semantic 3D dense map. The virtual-real anchoring module is used to perform coarse positioning of the augmented reality head-mounted device based on the multimodal spatial fingerprint database, then perform visual feature matching for fine positioning of the multi-sensor data stream, and adaptively set the anchoring verification strength according to the visual noise index to obtain the anchoring result. The task management module is used to render standard task guidance information in the augmented reality field of view based on the anchoring result, to statistically analyze the historical verification results of the task steps, to obtain the upper bound of the verification failure probability of the task steps, to adaptively set the step verification intensity based on the upper bound of the verification failure probability, and to perform step-level mandatory control to obtain the step control result. The interactive collaboration module is used to integrate multi-channel interactive events to confirm the results of the step control and to encapsulate and push the operation process data to the back-end collaborative dashboard.

Images