Electric power field operation auxiliary method based on AR glasses and mobile large model

Abstract

The invention belongs to the technical field of electric power augmented reality and mobile edge computing, and particularly relates to an electric power field operation auxiliary method based on AR glasses and a mobile large model and a use method. Comprising the following steps: by taking AR glasses as a hardware carrier, carrying out voice wake-up and streaming acquisition, multimedia acquisition and recording and multi-modal information output; uniformly packaging the collected multivariate heterogeneous data; carrying out multi-modal lightweight reasoning and logic decision making; business logic execution is carried out, wherein the business logic execution comprises context sensing and task synchronization logic, AR live-action addressing logic based on sight tracking, voice-driven interactive feedback and AI quality inspection logic and asynchronous and concurrent rapid general survey logic; through a three-stage collaborative architecture of a glasses end, a mobile phone end and a service system, scene-adaptive, voice-driven and AR-enhanced immersive working experience is provided for operators, and the industrial pain points of lagging interaction, split flow and insufficient intelligent degree in traditional industrial field operation are comprehensively solved.

Description

Technical Field

[0001] This invention belongs to the field of power augmented reality and mobile edge computing technology, specifically involving a power field operation assistance method and its usage based on AR glasses and a mobile large model. Background Technology

[0002] With the deepening of the digital and intelligent transformation of the power grid, the efficiency, safety, and standardization of on-site operations have become key factors restricting the high-quality development of enterprises. However, the current power on-site operation mode has serious limitations, hindering further improvement in its efficiency. The primary problem lies in the outdated human-machine interaction methods: existing technologies generally rely on handheld terminals for data recording and querying, requiring operators to frequently look down during operations, greatly increasing safety risks in dangerous operating scenarios such as high-altitude, live, and mobile work, and resulting in low operational efficiency. Secondly, the information acquisition and processing processes are fragmented: equipment identification, file query, and data entry are independent of each other, relying on manual switching between different applications or systems, leading to workflow interruptions, high error rates, and the inability to provide real-time, intelligent guidance to operators. Although traditional AR technology solutions can achieve information overlay display, they mostly rely on preset data or large cloud models, resulting in poor real-time performance, strong network dependence, and shallow integration with business processes, failing to deeply understand the on-site context and provide adaptive workflow support. These shortcomings collectively make it difficult for existing technologies to support the refined, safe, and intelligent management of power on-site operations, becoming a bottleneck restricting the implementation of industrial digital transformation. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a power field operation assistance method and usage method based on AR glasses and mobile large model. Through a three-level collaborative architecture of glasses terminal-mobile terminal-business system, it provides operators with an immersive work experience with scene adaptation, voice drive and AR enhancement, and comprehensively solves the industry pain points of backward interaction, process fragmentation and insufficient intelligence in traditional industrial field operations.

[0004] The objective of this invention is achieved as follows: a power field operation assistance method based on AR glasses and a mobile large-scale model, comprising the following steps: Using AR glasses as the hardware platform, it performs voice wake-up and streaming acquisition, multimedia recording, and multimodal information output; The collected diverse and heterogeneous data (including audio streams, image frames, and video data packets) are uniformly encapsulated using a standard data interface based on a structured serialization protocol. Using mobile computing terminals as edge computing nodes, multimodal lightweight reasoning and logical decision-making are performed; Perform business logic execution, including context-aware and task synchronization logic, eye-tracking-based AR real-scene addressing logic, voice-driven interactive feedback and AI quality inspection logic, and asynchronous concurrent quick survey logic.

[0005] Furthermore, the AR glasses integrate a camera, a display screen, a microphone, and a Bluetooth communication module; The voice wake-up and streaming acquisition are achieved by activating the system with a custom wake-up word and using endpoint detection technology to acquire the user's voice stream in real time. The multimedia recording includes customized photo taking, QR code scanning, video recording, and voice recording according to instructions; The multimodal information output includes receiving and processing results, and displaying them via voice broadcast or on the glasses display screen in the form of custom pages such as text and AR tags.

[0006] Furthermore, in the process of uniformly encapsulating the collected diverse heterogeneous data using a standard data interface based on a structured serialization protocol, the end-to-end transmission adopts a heterogeneous network fusion scheme of BLE and Wi-Fi Direct. Through link quality awareness and dynamic load balancing algorithms, the optimal transmission channel is intelligently selected to ensure millisecond-level low latency of control commands and high throughput and high reliability of multimedia data transmission.

[0007] Furthermore, the multimodal lightweight reasoning and logical decision-making content includes: offline streaming voice interaction based on the Sherpa-onnx voice processing framework, cascaded visual perception based on the PaddlePaddle deep learning framework, and edge-side reasoning optimization and deployment.

[0008] Furthermore, the offline streaming voice interaction based on the Sherpa-onnx voice processing framework includes: • An end-to-end speech recognition model based on the Transducer architecture is adopted to establish an audio circular buffer and receive the PCM audio stream from the AR glasses in real time; The lightweight wake-up model runs in parallel, and performs fine-tuning and low-power instruction listening for power-specific terms such as "Ruyi", "find the meter", and "record". The trained speech model is exported to the ONNX universal format, and the high-performance inference library of Sherpa-onnx is used to directly call the mobile CPU / NPU instruction set for acceleration.

[0009] Furthermore, the cascaded visual perception based on the PaddlePaddle deep learning framework includes first-level scene perception and filtering, second-level object detection and localization, and third-level refined semantic segmentation.

[0010] Furthermore, the context-aware and task synchronization logic includes: Logic triggering refers to the mobile app responding to user commands, such as clicking "AR real-world processing" or similar actions. Data migration refers to the system encapsulating the key structured data of the currently selected work order into a JSON message and pushing it to the AR glasses cache via a wireless channel such as Bluetooth or WiFi; the key structured data includes the target asset number Asset_ID, account name, fault type, and work SOP template; State switching means that when the mobile device enters background running mode, such as screen off or suspended, the AR glasses take over the main control, initialize the voice wake-up engine and IMU posture detection, free the user's hands, and can be woken up by keywords.

[0011] Furthermore, the eye-tracking-based AR real-world addressing logic includes: Visual stream analysis refers to AR glasses responding to voice commands, turning on the camera to capture real-time video streams, and extracting key frames at a preset frame rate. The feature matching algorithm is invoked by transmitting keyframes to the computing unit, mobile device or cloud, and calling the barcode or QR code recognition algorithm; the recognized barcode serial number is compared with the Asset_ID in the cached work order in real time; if they do not match, the system does not respond or only displays general information; if they match successfully, the pixel coordinates (u, v) of the target object in the current field of view are calculated. AR rendering, which is based on coordinate data, draws highlighted anchor points on the display layer of AR glasses and overlays corresponding labels, such as "Target found", to achieve real-world alignment between physical devices and digital work orders.

[0012] Furthermore, the voice-driven interactive feedback and AI quality inspection logic includes: The system parses voice commands, loads feedback question and answer templates, uses TTS speech synthesis to broadcast questions and recognize user voice, and automatically fills in work order form fields. Upon feedback, the camera is triggered to capture the current static image; before the image is uploaded, a lightweight CV model is run locally or at the edge to perform binary classification of the image.

[0013] Furthermore, the asynchronous concurrent rapid census logic includes switching between scenario-based modes, fusion of panoramic data collection and metadata, and asynchronous processing queues.

[0014] The beneficial effects of this invention are as follows: The power field operation assistance method based on AR glasses and mobile large model of this invention includes: using AR glasses as a hardware carrier for voice wake-up and streaming acquisition, multimedia recording and multimodal information output; using a standard data interface based on a structured serialization protocol to uniformly encapsulate the acquired diverse and heterogeneous data; using a mobile computing terminal as an edge computing node for multimodal lightweight reasoning and logical decision-making; and executing business logic, including context awareness and task synchronization logic, AR real-scene addressing logic based on eye tracking, voice-driven interactive feedback and AI quality inspection logic, and asynchronous concurrent quick survey logic. Through the above steps, the power field operation assistance method based on AR glasses and mobile large model of this invention seamlessly embeds advanced AR display, voice interaction and large model intelligent recognition capabilities into the core process of power field operation through a three-level collaborative architecture of glasses-mobile phone-business system. It effectively solves the safety hazards and efficiency bottlenecks caused by traditional handheld operation, enabling hands-free operation; through intelligent understanding of large models and proactive guidance of business processes, it integrates fragmented information processing links into a continuous and natural interactive flow, significantly reducing the error rate and standardizing operation standards; ultimately, the system provides operators with an immersive work experience that is scene-adaptive, voice-driven, and AR-enhanced, comprehensively solving the industry pain points of outdated interaction, fragmented processes, and insufficient intelligence in traditional industrial field operations, and providing a reliable field-level solution for the digital transformation of the power grid. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a method for assisting on-site power operations based on AR glasses and a mobile large-scale model. Detailed Implementation

[0017] The present invention will now be further described with reference to the accompanying drawings.

[0018] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that in the embodiments of the present invention, all directional indications (such as up-down-left-right-forward-backward...) are only used to explain the relative positional relationship and movement between the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly. The connection can be a direct connection or an indirect connection.

[0021] like Figure 1 As shown, the power field operation assistance method based on AR glasses and a mobile large model of the present invention includes the following steps: Using AR glasses as the hardware platform, it performs voice wake-up and streaming acquisition, multimedia recording, and multimodal information output; The collected diverse and heterogeneous data (including audio streams, image frames, and video data packets) are uniformly encapsulated using a standard data interface based on a structured serialization protocol. Using mobile computing terminals as edge computing nodes, multimodal lightweight reasoning and logical decision-making are performed; Perform business logic execution, including context-aware and task synchronization logic, eye-tracking-based AR real-scene addressing logic, voice-driven interactive feedback and AI quality inspection logic, and asynchronous concurrent quick survey logic.

[0022] Furthermore, in one embodiment, the AR glasses integrate a camera, a display screen, a microphone, and a Bluetooth communication module; The voice wake-up and streaming acquisition are achieved by activating the system with a custom wake-up word and using endpoint detection technology to acquire the user's voice stream in real time. The endpoint detection technology uses the FSMN-VAD deep learning model as the core detection algorithm, combined with the engineering optimization of sliding window + state machine decision, to form a hybrid detection strategy of model prediction + rule verification.

[0023] The multimedia recording includes custom photo taking, QR code scanning, video and voice recording according to instructions; QR code scanning can scan barcodes or QR codes. The customized photo-taking process is as follows: The control terminal sends JSON data containing customized photo-taking interface parameters and images via Bluetooth, and can also send voice commands to play the photo-taking audio; after the glasses parse the data, they convert the interface parameters into Android system rendering commands to generate a customized page, display the images synchronously, and play the audio in response to commands, thus solving the problems of fixed and monotonous interaction in traditional glasses photo-taking interfaces.

[0024] The multimodal information output includes receiving and processing results, and displaying them via voice broadcast or on the glasses display screen in the form of custom pages such as text and AR tags.

[0025] Furthermore, in one embodiment, during the process of uniformly encapsulating the collected diverse heterogeneous data using a standard data interface based on a structured serialization protocol, the end-to-end transmission adopts a heterogeneous network fusion scheme of BLE and Wi-Fi Direct. Through link quality awareness and dynamic load balancing algorithms, the optimal transmission channel is intelligently selected to ensure millisecond-level low latency of control commands and high throughput and high reliability of multimedia data transmission.

[0026] Standardized data interfaces are used to encapsulate data uniformly, define a common data header structure, and assemble business units according to data type differences; compatibility design is also carried out to support dynamic negotiation of protocol versions, ensure backward compatibility when adding new data fields, and avoid parsing failures caused by cross-device upgrades.

[0027] The following is an example of a data interface: Input parameter format: { "mainCode": "string", / / AIDL binding identifier "methodName": "string", / / Function name, format: module name.method name "paramJson": "string", / / Method parameter, a JSON string; if no parameter is specified, it will be "". "packageName": "string", / / The package name of the calling application, used for permission verification. "tag": "string" / / Request a unique identifier } Output parameter format: { "retCode": int, / / Result code: 0 - failure, 1 - success, 2 - callback process "msgJson": "string", / / Returns data, a JSON string; if no data is returned, it will be "". "tag": "string" / / Same as the input parameter tag, used to match requests. } The heterogeneous network convergence adopts a dual-link parallel + intelligent switching architecture. BLE is responsible for low-latency control command transmission, while Wi-Fi Direct is responsible for high-throughput multimedia data transmission. The two work together.

[0028] Furthermore, in one embodiment, the multimodal lightweight reasoning and logical decision-making content includes: offline streaming voice interaction based on the Sherpa-onnx voice processing framework, cascaded visual perception based on the PaddlePaddle deep learning framework, and edge-side reasoning optimization and deployment.

[0029] To address the issue of command input in high-noise, off-grid environments at power sites, the Sherpa-onnx open-source speech recognition framework, based on the next-generation Kaldi architecture, was deployed locally.

[0030] Furthermore, in one embodiment, the offline streaming voice interaction based on the Sherpa-onnx voice processing framework includes: • An end-to-end speech recognition model based on the Transducer architecture is adopted to establish an audio circular buffer, receive the PCM audio stream from AR glasses in real time, support speech recognition, and control the latency to within 200ms; The lightweight wake-up model runs in parallel, and performs fine-tuning and low-power instruction listening for power-specific terms such as "Ruyi", "find the meter", and "record". The trained speech model is exported to the ONNX universal format. It can be accelerated by directly calling the mobile CPU / NPU instruction set using the Sherpa-onnx high-performance inference library. It can accurately parse operation instructions such as "query assets" and "take a photo feedback" without the need for an internet connection.

[0031] The visual analysis module adopts a cascaded strategy of coarse screening, localization, and fine segmentation, and is based on the Model Zoo supported by the PaddlePaddle framework for targeted selection and deployment.

[0032] Furthermore, in one embodiment, the cascaded visual perception based on the PaddlePaddle deep learning framework includes first-level scene perception and filtering, second-level object detection and localization, and third-level refined semantic segmentation.

[0033] Level 1 Scene Perception and Filtering: The lightweight CPU network PP-LCNet is used as the front-end classifier to quickly classify the video frames input from the AR glasses, such as determining whether the current field of view contains a meter box, transformer or invalid background; the extremely low parameter count of LCNet (<5M) is used to quickly remove invalid frames, reducing the computational consumption of subsequent detection models.

[0034] Secondary target detection and localization: Once the primary classification confirms the existence of the target, the target detection model DET based on the YOLOv8 or YOLOv5 architecture is activated; It can identify the specific bounding box of the device in the field of view, and locate asset barcodes, dial areas, and switch handle positions; YOLOv8's Anchor-free mechanism can effectively improve the detection accuracy of small electrical targets such as nameplates at a distance.

[0035] Level 3 refined semantic segmentation: For refined tasks such as instrument reading and insulator damage determination, the YOLOV8_SEG instance segmentation model is activated; Pixel-level masks are further generated within the detection box to precisely isolate pointers, digital wheels, or crack areas from the background. Compared to traditional rectangular boxes, the segmentation technique can effectively eliminate complex background interference, providing a clean input for subsequent reading algorithms.

[0036] Edge-side inference optimization and deployment include: • Paddle Lite / FastDeploy integration: The aforementioned LCNET, YOLO, and SEG series models have all undergone model compression (pruning and INT8 quantization) and are deployed on Android / HarmonyOS systems via the Paddle Lite inference engine; Heterogeneous computing scheduling: The inference engine dynamically schedules the GPU or DSP resources of the mobile SoC through NNAPI to ensure that the frame rate can still be maintained above 15FPS when running complex models such as YOLOv8-Seg, meeting the smoothness requirements of AR real-world overlay.

[0037] Furthermore, in one embodiment, the context-aware and task synchronization logic includes: Logic triggering refers to the mobile app responding to user commands, such as clicking "AR real-world processing" or similar actions. Data migration refers to the system encapsulating the key structured data of the currently selected work order into a JSON message and pushing it to the AR glasses cache via a wireless channel such as Bluetooth or WiFi; the key structured data includes the target asset number Asset_ID, account name, fault type, and work SOP template; State switching means that when the mobile device enters background running mode, such as screen off or suspended, the AR glasses take over the main control, initialize the voice wake-up engine and IMU posture detection, free the user's hands, and can be woken up by keywords.

[0038] To address the challenge of locating target devices in densely packed enclosure environments, an AR real-world addressing logic based on eye tracking can be implemented.

[0039] Furthermore, in one embodiment, the eye-tracking-based AR real-world addressing logic includes: Visual stream analysis refers to AR glasses responding to voice commands, turning on the camera to capture real-time video streams, and extracting key frames at a preset frame rate. The feature matching algorithm is invoked by transmitting keyframes to the computing unit, mobile device or cloud, and calling the barcode or QR code recognition algorithm; the recognized barcode serial number is compared with the Asset_ID in the cached work order in real time; if they do not match, the system does not respond or only displays general information; if they match successfully, the pixel coordinates (u, v) of the target object in the current field of view are calculated. AR rendering, which is based on coordinate data, draws highlighted anchor points on the display layer of AR glasses and overlays corresponding labels, such as "Target found", to achieve real-world alignment between physical devices and digital work orders.

[0040] Furthermore, in one embodiment, the voice-driven interactive feedback and AI quality inspection logic includes: The system parses voice commands, loads feedback question and answer templates, uses TTS speech synthesis to broadcast questions and recognize user voice, and automatically fills in work order form fields. Feedback triggers the camera to capture the current static image; before the image is uploaded, a lightweight CV model is run locally or at the edge to perform binary classification of the image. The criteria for judgment include: image clarity (whether it is blurry); integrity of the target subject (whether it includes the meter dial); and whether the brightness meets the standard. If the score is greater than the threshold, the decision is made and the voice feedback is "Feedback is qualified". The image is automatically uploaded and the work order status is updated, and the current process is unlocked. If the score is below the threshold, the photo is deemed unacceptable, and a voice message is sent stating "Photo is unacceptable" along with the specific reason, such as "Please keep the camera stable." The system then forces a rollback to the shooting point to prevent invalid data from being uploaded, ensuring that the on-site operation is done correctly the first time.

[0041] Furthermore, in one embodiment, the asynchronous concurrent rapid census logic includes switching between scenario-based modes, fusion of panoramic data collection and metadata, and asynchronous processing queues; Scene-based mode switching means responding to corresponding voice commands, such as "quick census", and the system switches to wide-angle shooting mode; Panoramic data acquisition and metadata fusion means guiding users to capture a panoramic view of the watch case, automatically obtaining the mobile device's GPS latitude and longitude and timestamp, and synthesizing a panoramic image data package with geographic information. The asynchronous processing queue means that when data packets are added to the upload queue, the background service asynchronously calls the OCR large model and image segmentation algorithm to automatically crop the images of each segment from the panoramic image and recognize the readings. Users do not need to wait for the processing results on site, enabling efficient work that can be done immediately after taking the picture.

[0042] In summary, the power field operation assistance method based on AR glasses and mobile large-scale models of the present invention includes: using AR glasses as a hardware carrier for voice wake-up and streaming data acquisition, multimedia recording, and multimodal information output; using a standard data interface based on a structured serialization protocol to uniformly encapsulate the acquired diverse and heterogeneous data; using a mobile computing terminal as an edge computing node for multimodal lightweight reasoning and logical decision-making; and executing business logic, including context awareness and task synchronization logic, AR real-scene addressing logic based on eye tracking, voice-driven interactive feedback and AI quality inspection logic, and asynchronous concurrent rapid survey logic. Through the above steps, the power field operation assistance method based on AR glasses and mobile large-scale models of the present invention seamlessly embeds advanced AR display, voice interaction, and large-scale model intelligent recognition capabilities into the core process of power field operations through a three-level collaborative architecture of glasses-mobile phone-business system. It effectively solves the safety hazards and efficiency bottlenecks caused by traditional handheld operation, enabling hands-free operation; through intelligent understanding of large models and proactive guidance of business processes, it integrates fragmented information processing links into a continuous and natural interactive flow, significantly reducing the error rate and standardizing operation standards; ultimately, the system provides operators with an immersive work experience that is scene-adaptive, voice-driven, and AR-enhanced, comprehensively solving the industry pain points of outdated interaction, fragmented processes, and insufficient intelligence in traditional industrial field operations, and providing a reliable field-level solution for the digital transformation of the power grid.

[0043] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0044] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for assisting power field operations based on AR glasses and a mobile large-scale model, characterized in that, Includes the following steps: Using AR glasses as the hardware platform, it performs voice wake-up and streaming acquisition, multimedia recording, and multimodal information output; The collected diverse and heterogeneous data is uniformly encapsulated using a standard data interface based on a structured serialization protocol; Using mobile computing terminals as edge computing nodes, multimodal lightweight reasoning and logical decision-making are performed; Perform business logic execution, including context-aware and task synchronization logic, eye-tracking-based AR real-scene addressing logic, voice-driven interactive feedback and AI quality inspection logic, and asynchronous concurrent quick survey logic.

2. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, The AR glasses integrate a camera, display screen, microphone and Bluetooth communication module; The voice wake-up and streaming acquisition are achieved by activating the system with a custom wake-up word and using endpoint detection technology to acquire the user's voice stream in real time. The multimedia recording includes customized photo taking, QR code scanning, video recording, and voice recording according to instructions; The multimodal information output includes receiving and processing results, and displaying them via voice broadcast or on the glasses display screen.

3. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, In the process of uniformly encapsulating the collected diverse and heterogeneous data using a standard data interface based on a structured serialization protocol, the end-to-end transmission adopts a heterogeneous network fusion scheme of BLE and Wi-Fi Direct. Through link quality awareness and dynamic load balancing algorithms, the optimal transmission channel is intelligently selected to ensure millisecond-level low latency of control commands and high throughput and high reliability of multimedia data transmission.

4. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, The multimodal lightweight reasoning and logical decision-making content includes: offline streaming voice interaction based on the Sherpa-onnx voice processing framework, cascaded visual perception based on the PaddlePaddle deep learning framework, and edge-side reasoning optimization and deployment.

5. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 4, characterized in that, The offline streaming voice interaction based on the Sherpa-onnx voice processing framework includes: • An end-to-end speech recognition model based on the Transducer architecture is adopted to establish an audio circular buffer and receive the PCM audio stream from the AR glasses in real time; The lightweight wake-up model is run in parallel, with fine-tuning and command listening for power-specific terminology. The trained speech model is exported to the ONNX universal format, and the Sherpa-ONNX inference library is used to directly call the mobile CPU / NPU instruction set for acceleration.

6. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 4, characterized in that, The cascaded visual perception based on the PaddlePaddle deep learning framework includes first-level scene perception and filtering, second-level object detection and localization, and third-level refined semantic segmentation.

7. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, The context-aware and task synchronization logic includes: Logical triggering, i.e., the mobile app responding to user commands; Data migration refers to the system encapsulating the key structured data of the currently selected work order into a JSON message and pushing it to the AR glasses cache via a wireless channel; the key structured data includes the target asset number (Asset_ID), account name, fault type, and work SOP template; State switching means that when the mobile device enters background running mode, the AR glasses take over the main control and initialize the voice wake-up engine and IMU posture detection.

8. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, The eye-tracking-based AR real-world addressing logic includes: Visual stream analysis refers to AR glasses responding to voice commands, turning on the camera to capture real-time video streams, and extracting key frames at a preset frame rate. The feature matching algorithm is invoked, which involves transmitting the keyframe to the computing unit and calling the barcode or QR code recognition algorithm; the recognized barcode serial number is compared with the Asset_ID in the cached work order in real time; if they do not match, the system does not respond or only displays general information; if they match successfully, the pixel coordinates (u, v) of the target object in the current field of view are calculated. AR rendering, which involves drawing highlighted anchor points on the display layer of AR glasses based on coordinate data and overlaying corresponding labels.

9. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, The voice-driven interactive feedback and AI quality inspection logic includes: The system parses voice commands, loads feedback question and answer templates, uses TTS speech synthesis to broadcast questions and recognize user voice, and automatically fills in work order form fields. Upon feedback, the camera is triggered to capture the current static image; before the image is uploaded, a lightweight CV model is run locally or at the edge to perform binary classification of the image.

10. The power field operation assistance method based on AR glasses and a mobile large model as described in claim 1, characterized in that, The asynchronous concurrent rapid census logic includes switching between scenario-based modes, fusion of panoramic data collection and metadata, and asynchronous processing queues.

Images