Patrol auxiliary system and method based on AR technology

Abstract

The invention relates to the technical field of electric power inspection, and discloses an inspection auxiliary system and method based on the AR technology, the system is arranged in an AR terminal worn by an operator and is in communication connection with a centralized control center, and the system comprises a visual angle correction module, a track determination module, a standardization correction module, a task updating module and an edge calculation module. The method corresponds to the system. According to the invention, the visual angle correction module ensures that the centralized control center obtains a real first visual angle picture, and solves the problem of remote cooperation visual deviation; the trajectory determination module accurately restores the operation path and fuses the environment features to realize safety labeling; the standardization correction module strengthens standardization detection of the operation process through double analysis of action compliance and time sequence compliance; the task updating module realizes intelligent optimization of the patrol task; the edge calculation module reduces cloud transmission delay; and the efficiency, the safety and the process adaptability of the electric power inspection operation are obviously improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of electric power inspection, and particularly relates to an inspection assisting system and method based on AR technology. BACKGROUND

[0002] In the electric power equipment inspection, maintenance and other operation scenarios, the existing AR assistance has many technical defects:

[0003] Firstly, the traditional visual angle correction technology relies on fixed coordinate system conversion, resulting in deviation between the audio and video pictures obtained by the centralized control center and the first visual angle of the operator, and visual misdirection is easy to occur during remote collaboration.

[0004] Secondly, the inspection track is determined based on two-dimensional GPS coordinates, lacks spatial fusion with audio and video data, and cannot construct a three-dimensional track containing height dimension and environmental characteristics, so it is difficult to accurately restore the operation path.

[0005] Thirdly, the standardized operation detection only analyzes the action compliance, does not involve the compliance judgment of the operation time sequence, and cannot find hidden problems such as process sequence errors.

[0006] Fourthly, the initial inspection task is mostly a preset fixed process, lacks a dynamic updating mechanism based on real-time detection results, and is difficult to adapt to the demand changes in complex operation scenarios.

[0007] Fifthly, a cloud centralized computing mode is mostly used, resulting in high data processing delay and failing to meet the real-time requirements of electric power operation.

[0008] A Chinese patent for invention with application publication number CN115512289A discloses an electric power operation safety monitoring method and system based on AR glasses, but the invention performs poorly in solving inspection optimization.

[0009] In summary, a new inspection assisting system based on AR technology is needed to solve the problems of low remote collaboration efficiency, insufficient operation standardization and lagging risk early warning. SUMMARY

[0010] The purpose of the application is to provide an inspection assisting system and method based on AR technology to solve the technical problems proposed in the background.

[0011] To achieve the above purpose, the application discloses the following technical solutions:

[0012] In a first aspect, this application discloses an AR-based inspection assistance system. The system is installed in an AR terminal worn by the operator and communicates with a centralized control center for the inspection operation via the AR terminal to transmit data generated by the AR terminal to the centralized control center. The centralized control center stores operational knowledge required for the inspection operation. The system includes:

[0013] The viewing angle correction module is used to continuously correct the collected audio and video data using the wearing position positioning data of the AR terminal, so that the audio and video data transmitted to the display device of the central control center always maintains the first-person perspective of the operator.

[0014] The trajectory determination module is used to determine the patrol trajectory of the workers based on the audio and video data and the work location data;

[0015] The standardization correction module is used to provide standardized assistance and standardized detection for the real-time operation of the operator based on the standardized process data of the inspection operation and the audio and video data. The standardized detection is used for compliance analysis of the real-time operation and updating of the initial inspection task. The compliance analysis includes action compliance analysis for detecting the compliance of the operation actions and time sequence compliance analysis for detecting the compliance of the operation actions. The initial inspection task stores each inspection node that the operator needs to inspect and the inspection actions that need to be performed at the inspection node.

[0016] The task update module is used to update the initial patrol task based on the results of the standardized detection, and replace the initial patrol task after verifying that the updated patrol task has passed.

[0017] An edge computing module, built into the AR terminal, is used to perform computational processing for viewpoint correction, trajectory determination, standardization correction, and task updates.

[0018] Preferably, the viewpoint correction module specifically includes:

[0019] The wearable data acquisition unit is used to acquire the three-dimensional coordinates and attitude angle data of the wearing position through the inertial measurement and positioning measurement of the AR terminal;

[0020] The viewing angle correction unit is used to correct the viewing angle of the audio and video data in real time based on the three-dimensional coordinates and attitude angle data, so that the display screen of the central control center is in the same direction as the line of sight of the operator.

[0021] Preferably, the trajectory determination module includes:

[0022] A two-dimensional trajectory generation unit is used to generate a continuous trajectory line in a horizontal plane based on the two-dimensional coordinates and timestamps of the work location data.

[0023] The three-dimensional trajectory generation unit is used to combine the two-dimensional coordinates and timestamps of the operation location data and the visual feature points in the audio and video data to construct a three-dimensional inspection trajectory including the height dimension.

[0024] Preferably, the three-dimensional trajectory generation unit is further used for:

[0025] The AR terminal acquires environmental data of the work location using its depth camera, and then fuses this environmental data with the 3D inspection trajectory to generate a 3D trajectory with environmental features. The central control center uses this 3D trajectory with environmental features to perform environmental safety annotation.

[0026] Preferably, the standardization correction module includes:

[0027] The process distribution unit is used to push standardized process data, including text descriptions, action diagrams, and time thresholds, to the AR terminal according to the order of the inspection nodes of the initial inspection task.

[0028] The real-time comparison unit is used to identify the work actions in the audio and video data, match the work actions with the standard action templates in the standardized process data by key points, and analyze the compliance of the actions.

[0029] Preferably, the real-time comparison unit is further used for:

[0030] A time-action state model is established, and the operation sequence in the audio and video data is synchronously compared with the time action in the time-action state model to detect whether there is a situation where the operation action is delayed or advanced, and to analyze the compliance of the timing.

[0031] Preferably, the task update module includes:

[0032] The association analysis unit is used to construct an association graph of the inspection nodes based on the operational knowledge. The association graph stores the functional correlation and spatial proximity weights of each inspection node.

[0033] The node adjustment unit is used to automatically add associated inspection nodes or delete redundant inspection nodes based on the results of the standardized inspection and the relationship diagram. The results of the standardized inspection are whether the compliance analysis results meet the compliance threshold. If yes, redundant inspection nodes are deleted based on the relationship diagram; otherwise, associated inspection nodes are added based on the relationship diagram. The compliance threshold is determined based on the upper limit of the compliance threshold, the lower limit of the compliance threshold, and the results of the compliance analysis.

[0034] Preferably, the task update module further includes:

[0035] The coverage verification unit is used to extract the set of inspection nodes, action requirements and process logic of the initial inspection task and the updated inspection task. If the updated task completely covers all the technical elements of the initial task, it is directly replaced; if there are uncovered parts, the uncovered inspection nodes and their standardized process data are automatically added before replacement.

[0036] Preferably, the compliance threshold is determined based on an upper limit, a lower limit, and the results of compliance analysis, specifically as follows:

[0037] The compliance analysis result is a compliance evaluation obtained by fusing action compliance analysis and time sequence compliance analysis; the compliance evaluation is subtracted from the upper limit of the compliance threshold to obtain the evaluation difference, and the smaller value between the evaluation difference and zero is taken as the output compliance. When the average value of the output compliance is less than a preset average value threshold, the upper limit of the compliance threshold is corrected using the average value to obtain the compliance threshold; and the correction of the upper limit of the compliance threshold stops when the compliance threshold is equal to the lower limit of the compliance threshold.

[0038] Secondly, this application discloses an AR-based patrol assistance method, which is applicable to the AR-based patrol assistance system described above. The method includes:

[0039] The AR terminal's wearing position data is used to continuously correct the collected audio and video data, so that the audio and video data transmitted to the display device in the central control center always maintains the operator's first-person perspective.

[0040] The patrol trajectory of the workers is determined based on the audio and video data and the work location data;

[0041] Based on the standardized process data of the inspection operation and the audio and video data, the real-time operation of the operator is assisted and detected in a standardized manner. The standardized detection is used for the compliance analysis of the real-time operation and the update of the initial inspection task. The compliance analysis includes at least action compliance and timing compliance. The initial inspection task stores each inspection node that the operator needs to inspect and the inspection actions that need to be performed at the inspection node.

[0042] The initial patrol task is updated based on the results of the standardized inspection, and the updated patrol task is replaced after verification.

[0043] The edge computing built into the AR terminal performs computational processing for viewpoint correction, trajectory determination, standardization correction, and task updates.

[0044] Beneficial Effects: The AR-based inspection assistance system and method of this application continuously calibrates the audio and video perspective using the wearer's location data through a perspective correction module, ensuring that the control center obtains a true first-person perspective view and solving the problem of visual bias in remote collaboration; the trajectory determination module combines audio and video with work location data to generate two-dimensional and three-dimensional inspection trajectories, accurately reconstructing the work path and integrating environmental features to achieve safety marking; the standardization correction module strengthens the standardization detection of the work process through dual analysis of action compliance and timing compliance; the task update module dynamically adjusts inspection nodes and verifies task coverage based on the compliance analysis results using a correlation graph, realizing intelligent optimization of inspection tasks; the edge computing module processes data in real time on the terminal, reducing cloud transmission latency; through the collaboration of multiple modules, it effectively solves the problems of perspective bias, single trajectory, insufficient standardization, rigid tasks, and poor real-time performance in existing technologies, significantly improving the efficiency, safety, and process adaptability of power inspection operations. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A structural block diagram of an AR-based patrol assistance system provided in an embodiment of this application;

[0047] Figure 2 A flowchart illustrating the AR-based patrol assistance method provided in this application embodiment. Detailed Implementation

[0048] The technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0049] In this document, the term "comprising" is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] The following provides a detailed description of the specific implementation of the AR-based inspection assistance system and method. The AR-based inspection assistance system is integrated into existing AR terminals and communicates with the power inspection control center. Through various modules, it achieves functions such as perspective correction, trajectory determination, standardized detection, and task updating.

[0051] The first aspect of this embodiment discloses as follows: Figure 1 The system illustrates an AR-based patrol assistance system. The system is installed in an AR terminal worn by the operator and communicates with a centralized control center for patrol operations via the AR terminal to transmit data generated by the AR terminal to the control center. The control center stores operational knowledge required for the patrol operations. The system includes:

[0052] The perspective correction module is used to continuously correct the collected audio and video data using the wearing position positioning data of the AR terminal, so that the audio and video data transmitted to the display device of the central control center always maintains the first-person perspective of the operator. In this embodiment, the display device of the central control center can be a large display screen, etc. Through the display device of the central control center, experts located in the central control center can provide remote guidance from the first-person perspective to guide the operator. In addition, this embodiment realizes the release of the operator's hands based on the voice interaction function of the AR terminal.

[0053] The trajectory determination module is used to determine the patrol trajectory of the operator based on the audio and video data and the work location data; in this embodiment, the work location data may be, but is not limited to, GPS data of the operator's location.

[0054] The standardization correction module is used to provide standardized assistance and standardized detection for the real-time operations of the operators based on the standardized process data of the inspection operation and the audio and video data. The standardized detection is used for compliance analysis of the real-time operation and updating of the initial inspection task. The compliance analysis includes action compliance analysis for detecting the compliance of operation actions and time sequence compliance analysis for detecting the compliance of operation actions. The initial inspection task stores each inspection node that the initial operator needs to inspect and the inspection actions to be performed at the inspection nodes. It should be noted that the standardized process data in this embodiment is the operation knowledge stored in the centralized control data. This standardized process data is reference data for standardized operations set based on common sense known to those skilled in the art. Furthermore, the initial inspection task in this embodiment can be a preset periodic inspection plan for the operators.

[0055] The task update module is used to update the initial patrol task based on the results of the standardized detection, and replace the initial patrol task after verifying that the updated patrol task has passed.

[0056] An edge computing module, built into the AR terminal, is used to perform computational processing for viewpoint correction, trajectory determination, standardization correction, and task update. It should be noted that this embodiment implements its setting in the AR terminal based on existing edge computing technology.

[0057] By means of the above, real-time perspective correction ensures visual consistency in remote collaboration, multi-dimensional patrol trajectories are constructed using audio, video and location data, and standardized operations are enhanced by combining action and time-series dual compliance analysis. Dynamic task updates adapt to complex scenarios, and edge computing improves real-time performance, significantly improving patrol efficiency and safety.

[0058] The specific structure of the perspective correction module is as follows: through the collaboration of the wearable data acquisition unit and the perspective correction unit, real-time correction of audio and video perspective is achieved.

[0059] Specifically, the viewpoint correction module includes:

[0060] The wearable data acquisition unit is used to acquire the three-dimensional coordinates and attitude angle data of the wearing position through the inertial measurement and positioning measurement of the AR terminal. In this embodiment, both inertial measurement and positioning measurement adopt existing measurement methods. For example, inertial measurement is performed using an existing inertial measurement unit, and positioning measurement is performed using GPS positioning.

[0061] The viewing angle correction unit is used to correct the acquisition viewing angle of the audio and video data in real time based on the three-dimensional coordinates and attitude angle data, so that the display screen of the central control center is in the same direction as the line of sight of the operator. In this embodiment, an existing matrix transformation algorithm can be used to correct the acquisition viewing angle of the audio and video data in real time.

[0062] By acquiring wear data through inertial and positioning measurements, and correcting audio and video perspectives in real time, the view from the control center is aligned with the line of sight of the operators, thus solving the problem of perspective deviation in remote collaboration and improving communication efficiency and accuracy.

[0063] Existing technologies often generate patrol trajectories based on two-dimensional coordinates, which cannot reflect height dimension information. The trajectory determination module, through two-dimensional and three-dimensional trajectory generation units, achieves multi-dimensional trajectory construction.

[0064] Specifically, the trajectory determination module includes:

[0065] A two-dimensional trajectory generation unit is used to generate a continuous trajectory line in a horizontal plane based on the two-dimensional coordinates and timestamps of the work location data.

[0066] The three-dimensional trajectory generation unit is used to combine the two-dimensional coordinates and timestamps of the operation location data and the visual feature points in the audio and video data to construct a three-dimensional inspection trajectory including the height dimension. In this embodiment, the construction of the three-dimensional inspection trajectory is achieved by using existing algorithms for processing visual feature points, such as real-time positioning and mapping algorithms.

[0067] Based on the above, a planar trajectory is generated based on two-dimensional coordinates and time sequence, and a three-dimensional trajectory is constructed by combining visual feature points to accurately restore the operation path, meet the spatial positioning needs in complex scenarios, and provide multi-dimensional data support for environmental safety labeling.

[0068] The existing trajectory does not integrate environmental data, making it difficult to intuitively present operational risks. The 3D trajectory generation unit further integrates and utilizes existing environmental data collected by depth cameras to generate feature-rich 3D trajectories for safety annotation at the control center.

[0069] Specifically, the three-dimensional trajectory generation unit is also used for:

[0070] The AR terminal acquires environmental data of the work location via its depth camera, and fuses this environmental data with the 3D inspection trajectory to generate a 3D trajectory with environmental features. The control center then uses this 3D trajectory with environmental features for environmental safety labeling. In this embodiment, the environmental data may include, but is not limited to, visibility data and meteorological data of the work location. Existing AR technology is used to achieve environmental safety labeling, which may include, but is not limited to, reminding or warning workers to pay attention to environmental changes predicted by the control center.

[0071] By using the above, environmental data is acquired through depth cameras and fused with the trajectory to generate a three-dimensional trajectory with environmental features, enabling the central control center to accurately mark safety hazards and improve the intuitiveness and accuracy of risk warnings for the working environment.

[0072] Existing technologies only perform simple action comparisons and lack standardized process delivery and key point matching. The standardized correction module achieves process delivery and action compliance analysis through a process distribution unit and a real-time comparison unit.

[0073] Specifically, the standardization correction module includes:

[0074] The process distribution unit is used to push standardized process data, including text descriptions, action diagrams, and time thresholds, to the AR terminal according to the order of the inspection nodes of the initial inspection task. In this embodiment, the standardized process data provides work reference for the operators, and the time thresholds are used to prompt the operators to reasonably arrange the work progress to avoid the impact on work quality caused by being too fast or too slow.

[0075] A real-time comparison unit is used to identify the work actions in the audio and video data, match the work actions with the standard action templates in the standardized process data for key points, and analyze the compliance of the actions. It should be noted that this embodiment utilizes existing image recognition technology to achieve key point matching and analysis between work actions and the standard action templates in the standardized process data.

[0076] Based on the above, standardized process data containing time thresholds are pushed out in the order of inspection nodes. By analyzing the compliance of actions through key point matching, the standardization of operation actions is ensured, and the standardization of inspection operations and the accuracy of process execution are improved.

[0077] Existing technologies typically do not involve operational sequence compliance analysis, making it difficult to detect process sequence issues. The real-time comparison unit uses existing machine learning techniques, such as state machines, to build a time-action state model, enabling synchronous comparison of operational sequences.

[0078] Specifically, the real-time comparison unit is also used for:

[0079] A time-action state model is established, and the operation sequence in the audio and video data is synchronously compared with the time actions in the time-action state model to detect whether there are any delays or advances in operation actions and to analyze the compliance of the timing. In this embodiment, the time-action state model is based on the time threshold in the standardized process data for synchronous comparison with the operation sequence.

[0080] Based on the above, a time-action state model is constructed and the operation sequence is compared synchronously to detect situations where actions are delayed or advanced, filling the gap in the existing technology for time sequence compliance analysis and improving the logic and security of the operation process.

[0081] The existing patrol task update mechanism is rigid and cannot dynamically adjust nodes based on compliance. The task update module, through the association analysis unit, utilizes existing knowledge graph technology and combines it with the operational knowledge of the central control center to construct an association graph, and combines it with the node adjustment unit to realize the intelligent addition and removal of patrol nodes.

[0082] Specifically, the task update module includes:

[0083] The association analysis unit is used to construct an association graph of the inspection nodes based on the operational knowledge. The association graph stores the functional correlation and spatial proximity weights of each inspection node.

[0084] The node adjustment unit is used to automatically add associated inspection nodes or delete redundant inspection nodes based on the results of the standardized inspection and according to the correlation diagram. The results of the standardized inspection determine whether the compliance analysis results meet the compliance threshold. If yes, redundant inspection nodes are deleted based on the correlation diagram; otherwise, associated inspection nodes are added based on the correlation diagram. The compliance threshold is determined based on the upper limit, lower limit, and compliance analysis results. In this embodiment, the influence of the current inspection node's compliance status on uninspected nodes or those not in inspection tasks is clarified based on the correlation diagram, thus enabling the addition or deletion of nodes.

[0085] Based on the above, a relational graph with functional relevance and spatial proximity weights is constructed based on operational knowledge. According to the compliance analysis results and dynamic threshold comparison, associated nodes are automatically added or redundant nodes are deleted, realizing adaptive optimization of patrol tasks and improving operational efficiency and scenario adaptability.

[0086] Furthermore, if coverage is not verified during task updates, task elements may be missing. The task update module extracts and compares task elements through a coverage verification unit to ensure that the updated task is completely replaced.

[0087] Specifically, the task update module also includes:

[0088] The coverage verification unit is used to extract the set of inspection nodes, action requirements, and process logic of the initial inspection task and the updated inspection task. If the updated task completely covers all the technical elements of the initial task, it is directly replaced; if there are uncovered parts, the uncovered inspection nodes and their standardized process data are automatically added before replacement. In this embodiment, existing semantic analysis technology is used to extract the set of inspection nodes, action requirements, and process logic of the initial inspection task and the updated inspection task, and to perform corresponding technical element analysis.

[0089] Based on the above, by extracting the set of inspection nodes, action requirements and process logic for coverage verification, and automatically adding uncovered elements, we can avoid missing key nodes in the update task, ensure the integrity of the inspection operation and the continuity of the standardized process, and improve the reliability of the task update.

[0090] Furthermore, using fixed compliance thresholds cannot adapt to dynamic changes in operations. In practical applications, those skilled in the art often biasedly use only the lower limit of compliance as a constraint, aiming to achieve a minimum guarantee of inspection quality. However, in existing power line inspections, manual inspections generally mean that a long time has passed since the last manual inspection, or that a problem has arisen requiring manual intervention. Therefore, using only the lower limit of compliance as a constraint cannot fully realize the value of manual inspections, nor can it improve the work attitude of operators from a subjective perspective. Additionally, in actual inspections, power equipment may be located in areas with harsh natural conditions. This embodiment, as mentioned earlier, provides safety guarantees for operators through environmental safety labeling. Here, it appropriately lowers the upper limit of compliance through actual compliance evaluation, thereby seeking a balance between expected quality and actual operation. This invention constructs a dynamic threshold adjustment mechanism by integrating action and time-series compliance evaluations. By dynamically adjusting the compliance threshold, it realizes the value of manual inspections, improves the work attitude of operators, and balances expected quality with actual operation.

[0091] Specifically, 9. The AR-based patrol assistance system according to claim 7, characterized in that the compliance threshold is determined based on an upper limit of the compliance threshold, a lower limit of the compliance threshold, and the results of compliance analysis, specifically as follows:

[0092] The compliance analysis result is a compliance evaluation obtained by fusing action compliance analysis and time sequence compliance analysis; the compliance evaluation is subtracted from the upper limit of the compliance threshold to obtain the evaluation difference, and the smaller value between the evaluation difference and zero is taken as the output compliance. When the average value of the output compliance is less than a preset average value threshold, the upper limit of the compliance threshold is corrected using the average value to obtain the compliance threshold; and the correction of the upper limit of the compliance threshold stops when the compliance threshold is equal to the lower limit of the compliance threshold.

[0093] In one specific application of this embodiment, compliance evaluation can be achieved using existing weighted fusion, wherein the compliance evaluation H for the k-th time... k =f(D k ,S k ), where D k S represents the numerical result of the k-th action compliance analysis. k Let f() be the numerical result of the k-th time-series compliance analysis, and f() be the existing weighted fusion function; further, in this embodiment, the upper limit of the compliance threshold H is... u Compliance threshold lower limit H d and average threshold All values ​​are empirical values ​​set based on common sense known to those skilled in the art, and the compliance of the output H in the k-th iteration is... out_k The formula for calculating H isout_k =min[0,(H k -H u The formula for calculating the average value of the output compliance is: When the average value of the output compliance is less than the preset average threshold, it indicates that the compliance upper limit should be appropriately lowered based on the work location. In this embodiment, the compliance threshold H is always greater than the lower limit of the compliance threshold H. d .

[0094] Based on the above, dynamic thresholds are generated by integrating action and timing compliance evaluations. The upper limit of the threshold is dynamically adjusted based on the comparison between the evaluation difference and the preset threshold, forming an adaptive threshold system. This avoids the limitations of a single fixed threshold and improves the system's adaptability to complex operation scenarios and the accuracy of risk identification.

[0095] The second aspect of this embodiment discloses as follows: Figure 2 The method shown is an AR-based patrol assistance method, applicable to the AR-based patrol assistance system described above. The method includes:

[0096] S1: Utilize the location data of the AR terminal to continuously correct the collected audio and video data, so that the audio and video data transmitted to the display device in the central control center always maintains the first-person perspective of the operator;

[0097] S2: Determine the patrol trajectory of the workers based on the audio and video data and the work location data;

[0098] S3: Based on the standardized process data of the inspection operation and the audio and video data, the real-time operation of the operator is assisted and inspected in a standardized manner. The standardized inspection is used for the compliance analysis of the real-time operation and the update of the initial inspection task. The compliance analysis includes at least action compliance and timing compliance. The initial inspection task stores each inspection node that the operator needs to inspect and the inspection actions that need to be performed at the inspection node.

[0099] S4: Update the initial inspection task based on the results of the standardized inspection, and replace the initial inspection task after verifying that the updated inspection task has passed;

[0100] S5: Utilizes edge computing built into the AR terminal to perform computational processing for viewpoint correction, trajectory determination, standardization correction, and task updates.

[0101] It should be noted that the AR-based patrol assistance method in this embodiment corresponds to the aforementioned AR-based patrol assistance system. Therefore, any content not specifically described in the AR-based patrol assistance method of this embodiment, including but not limited to functional definitions, working principles, and technical effects, can be referred to the description in the aforementioned AR-based patrol assistance system, and will not be repeated here.

[0102] In summary, the AR-based inspection assistance system and method of this embodiment continuously calibrates the audio and video perspective using the wearer's location data through the perspective correction module, ensuring that the control center obtains a true first-person perspective view and solving the problem of visual bias in remote collaboration; the trajectory determination module combines audio and video with work location data to generate two-dimensional and three-dimensional inspection trajectories, accurately reconstructing the work path and integrating environmental features to achieve safety marking; the standardization correction module strengthens the standardization detection of the work process through dual analysis of action compliance and timing compliance; the task update module dynamically adjusts inspection nodes and verifies task coverage based on the compliance analysis results using a correlation graph, realizing intelligent optimization of inspection tasks; the edge computing module processes data in real time on the terminal, reducing cloud transmission latency; through the collaboration of multiple modules, it effectively solves the problems of perspective bias, single trajectory, insufficient standardization, rigid tasks, and poor real-time performance in existing technologies, significantly improving the efficiency, safety, and process adaptability of power inspection operations.

[0103] In the embodiments provided in this application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor may be implemented in one or more of the following: application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments may be performed by a computer program instructing the associated hardware. During implementation, the program may be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available medium accessible to a computer. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0104] Finally, it should be noted that the above description is only a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An AR-based inspection assistance system, wherein the system is installed in an AR terminal worn by the operator, and communicates with a centralized control center for inspection operations via the AR terminal to transmit data generated by the AR terminal to the centralized control center, wherein the centralized control center stores operational knowledge required for the inspection operations, characterized in that... The system includes: The viewing angle correction module is used to continuously correct the collected audio and video data using the wearing position positioning data of the AR terminal, so that the audio and video data transmitted to the display device of the central control center always maintains the first-person perspective of the operator. The trajectory determination module is used to determine the patrol trajectory of the workers based on the audio and video data and the work location data; The standardization correction module is used to provide standardized assistance and standardized detection for the real-time operation of the operator based on the standardized process data of the inspection operation and the audio and video data. The standardized detection is used for compliance analysis of the real-time operation and updating of the initial inspection task. The compliance analysis includes action compliance analysis for detecting the compliance of the operation actions and time sequence compliance analysis for detecting the compliance of the operation actions. The initial inspection task stores each inspection node that the operator needs to inspect and the inspection actions that need to be performed at the inspection node. The task update module is used to update the initial patrol task based on the results of the standardized detection, and replace the initial patrol task after verifying that the updated patrol task has passed. An edge computing module, built into the AR terminal, is used to perform computational processing for viewpoint correction, trajectory determination, standardization correction, and task updates.

2. The AR-based patrol assistance system according to claim 1, characterized in that, The viewpoint correction module specifically includes: The wearable data acquisition unit is used to acquire the three-dimensional coordinates and attitude angle data of the wearing position through the inertial measurement and positioning measurement of the AR terminal; The viewing angle correction unit is used to correct the viewing angle of the audio and video data in real time based on the three-dimensional coordinates and attitude angle data, so that the display screen of the central control center is in the same direction as the line of sight of the operator.

3. The AR-based patrol assistance system according to claim 1, characterized in that, The trajectory determination module includes: A two-dimensional trajectory generation unit is used to generate a continuous trajectory line in a horizontal plane based on the two-dimensional coordinates and timestamps of the work location data. The three-dimensional trajectory generation unit is used to combine the two-dimensional coordinates and timestamps of the operation location data and the visual feature points in the audio and video data to construct a three-dimensional inspection trajectory including the height dimension.

4. The AR-based patrol assistance system according to claim 3, characterized in that, The three-dimensional trajectory generation unit is also used for: The AR terminal acquires environmental data of the work location using its depth camera, and then fuses this environmental data with the 3D inspection trajectory to generate a 3D trajectory with environmental features. The central control center uses this 3D trajectory with environmental features to perform environmental safety annotation.

5. The AR-based patrol assistance system according to claim 1, characterized in that, The standardization correction module includes: The process distribution unit is used to push standardized process data, including text descriptions, action diagrams, and time thresholds, to the AR terminal according to the order of the inspection nodes of the initial inspection task. The real-time comparison unit is used to identify the work actions in the audio and video data, match the work actions with the standard action templates in the standardized process data by key points, and analyze the compliance of the actions.

6. The AR-based patrol assistance system according to claim 5, characterized in that, The real-time comparison unit is also used for: A time-action state model is established, and the operation sequence in the audio and video data is synchronously compared with the time action in the time-action state model to detect whether there is a situation where the operation action is delayed or advanced, and to analyze the compliance of the timing.

7. The AR-based patrol assistance system according to claim 1, characterized in that, The task update module includes: The association analysis unit is used to construct an association graph of the inspection nodes based on the operational knowledge. The association graph stores the functional correlation and spatial proximity weights of each inspection node. The node adjustment unit is used to automatically add associated inspection nodes or delete redundant inspection nodes based on the results of the standardized inspection and the relationship diagram. The results of the standardized inspection are whether the compliance analysis results meet the compliance threshold. If yes, redundant inspection nodes are deleted based on the relationship diagram; otherwise, associated inspection nodes are added based on the relationship diagram. The compliance threshold is determined based on the upper limit of the compliance threshold, the lower limit of the compliance threshold, and the results of the compliance analysis.

8. The AR-based patrol assistance system according to claim 7, characterized in that, The task update module also includes: The coverage verification unit is used to extract the set of inspection nodes, action requirements and process logic of the initial inspection task and the updated inspection task. If the updated task completely covers all the technical elements of the initial task, it is directly replaced; if there are uncovered parts, the uncovered inspection nodes and their standardized process data are automatically added before replacement.

9. The AR-based patrol assistance system according to claim 7, characterized in that, The compliance threshold mentioned above is determined based on the upper limit of the compliance threshold, the lower limit of the compliance threshold, and the results of compliance analysis, specifically as follows: The compliance analysis result is a compliance evaluation obtained by fusing action compliance analysis and time sequence compliance analysis; the compliance evaluation is subtracted from the upper limit of the compliance threshold to obtain the evaluation difference, and the smaller value between the evaluation difference and zero is taken as the output compliance. When the average value of the output compliance is less than a preset average value threshold, the upper limit of the compliance threshold is corrected using the average value to obtain the compliance threshold; and the correction of the upper limit of the compliance threshold stops when the compliance threshold is equal to the lower limit of the compliance threshold.

10. A patrol assistance method based on AR technology, the method being applicable to the patrol assistance system based on AR technology as described in any one of claims 1-9, characterized in that, The method includes: S1: Utilize the location data of the AR terminal to continuously correct the collected audio and video data, so that the audio and video data transmitted to the display device in the central control center always maintains the first-person perspective of the operator; S2: Determine the patrol trajectory of the workers based on the audio and video data and the work location data; S3: Based on the standardized process data of the inspection operation and the audio and video data, the real-time operation of the operator is assisted and inspected in a standardized manner. The standardized inspection is used for the compliance analysis of the real-time operation and the update of the initial inspection task. The compliance analysis includes at least action compliance and timing compliance. The initial inspection task stores each inspection node that the operator needs to inspect and the inspection actions that need to be performed at the inspection node. S4: Update the initial inspection task based on the results of the standardized inspection, and replace the initial inspection task after verifying that the updated inspection task has passed; S5: Utilizes edge computing built into the AR terminal to perform computational processing for viewpoint correction, trajectory determination, standardization correction, and task updates.

Images