Methods, devices, systems, and media for inspection and supervision of power production sites

Abstract

Embodiments of the present application provide a kind of power production site inspection supervision method, equipment, system and medium.The method comprises: the real-time acquisition and upload of AR terminal field video stream input hidden danger identification agent, so that hidden danger identification agent analyzes field video stream, according to the type of identification object, working condition and hidden danger category obtained by analysis to retrieve the standard procedure stored in knowledge base, determine out to standard procedure;Knowledge enhancement is carried out on the hidden danger description information obtained by analysis using to standard procedure, to obtain hidden danger identification information and disposal suggestion;Hidden danger positioning information, hidden danger identification information and disposal suggestion are sent to AR terminal, so that AR terminal is overlaid and displayed in the form of virtual information in the augmented reality picture based on field video stream according to hidden danger positioning information with hidden danger identification information and disposal suggestion by augmented reality technology.The present application can improve the inspection efficiency, improve the timeliness and accuracy of equipment maintenance.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of industrial equipment operation and maintenance technology, and in particular to a method, equipment, system and medium for on-site inspection and supervision of power production. Background Technology

[0002] With the rapid development of the new energy industry, on-site inspection and supervision of new energy power plants has become a crucial link in ensuring the safe and stable operation of the power system. However, traditional inspection and technical supervision of new energy power plants mainly rely on manual experience, which leads to problems such as low inspection efficiency and the tendency to miss or misdiagnose. Summary of the Invention

[0003] The embodiments of the present invention provide a method, equipment, system and medium for on-site inspection and supervision of power production, which can improve inspection efficiency and enhance the timeliness and accuracy of equipment maintenance.

[0004] In a first aspect, the on-site inspection and supervision method for power production provided by embodiments of the present invention includes: The system acquires real-time video streams from the augmented reality (AR) terminal; inputs these video streams into a hazard identification agent, which analyzes the streams to obtain hazard information. This information includes the type of the identified object, operating conditions, hazard category, hazard description, and hazard location. Based on the object type, operating conditions, and hazard category, the system retrieves standard procedures stored in the knowledge base to determine the benchmark procedure. The benchmark procedure is then used to augment the hazard description information, resulting in hazard identification information and handling suggestions. Finally, the system sends the hazard location information, hazard identification information, and handling suggestions to the AR terminal. The AR terminal then uses augmented reality technology to overlay these information as virtual information onto the augmented reality screen based on the video stream, ensuring that the virtual information's display position matches the corresponding real-world identified object or hazard location in the video stream.

[0005] Secondly, the on-site inspection and supervision method for power production provided in this embodiment of the invention includes: The system collects real-time on-site video streams and uploads them to edge computing devices. It receives hazard location information, hazard identification information, and handling suggestions from the edge computing devices. These hazard location information, hazard identification information, and handling suggestions are obtained by the hazard identification agent deployed on the edge computing devices, which uses benchmarking procedures to augment the hazard description information. The benchmarking procedures are obtained by the hazard identification agent by searching the knowledge base for standard procedures based on the type of the identified object, operating conditions, and hazard category. The type of identified object, operating conditions, hazard category, hazard description information, and hazard location information are obtained by the hazard identification agent through analysis of the on-site video stream. Using augmented reality technology, the hazard identification information and handling suggestions are overlaid as virtual information in the augmented reality image based on the on-site video stream, and the display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream.

[0006] Thirdly, the power production site inspection and monitoring device provided in the embodiments of the present invention includes: The acquisition module is used to acquire the live video stream collected and uploaded in real time by the augmented reality (AR) terminal; The identification module inputs the on-site video stream into the hazard identification intelligent agent, enabling the agent to analyze the video stream and obtain on-site hazard information. This information includes the type of the identified object, operating conditions, hazard category, hazard description, and hazard location. Based on the object type, operating conditions, and hazard category, the module retrieves standard procedures stored in the knowledge base to determine the benchmarking procedure. The benchmarking procedure is then used to enhance the hazard description information, resulting in hazard identification information and handling suggestions. The sending module is used to send hazard location information, hazard identification information, and handling suggestions to the AR terminal. The AR terminal can then use augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream, and the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

[0007] Fourthly, the power production site inspection and monitoring device provided in the embodiments of the present invention includes: The acquisition module is used to acquire live video streams in real time and upload them to edge computing devices. The receiving module receives hazard location information, hazard identification information, and handling suggestions returned by the edge computing device. These hazard location information, hazard identification information, and handling suggestions are obtained by the hazard identification intelligent agent deployed on the edge computing device, which uses benchmarking procedures to enhance the hazard description information. The benchmarking procedures are obtained by the hazard identification intelligent agent by retrieving standard procedures stored in the knowledge base based on the type of the identified object, operating conditions, and hazard category. The type of identified object, operating conditions, hazard category, hazard description information, and hazard location information are obtained by the hazard identification intelligent agent through analysis of the on-site video stream. The display module is used to overlay hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream using augmented reality technology, based on the hazard location information. The display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

[0008] Fifthly, the edge computing device provided in the embodiments of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the power production site inspection and supervision method as in any embodiment of the present invention.

[0009] In a sixth aspect, the AR terminal provided in the embodiments of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the power production site inspection and supervision method as in any embodiment of the present invention.

[0010] In a seventh aspect, the inspection and supervision system provided in the embodiments of the present invention includes at least one edge computing device as in any embodiment of the present invention and at least one AR terminal as in any embodiment of the present invention.

[0011] In this embodiment of the invention, acquiring and uploading real-time on-site video streams from augmented reality (AR) terminals overcomes the limitations of traditional manual operations, enabling real-time and efficient on-site data acquisition to effectively avoid missed detections and errors. The on-site video stream is then input into a hazard identification intelligent agent, which analyzes the stream to obtain on-site hazard information. This information includes the type of identified object, operating conditions, hazard category, hazard description, and hazard location information. By retrieving standard procedures stored in the knowledge base based on the type of identified object, operating conditions, and hazard category, benchmarking procedures are determined. This enhances the accuracy of benchmarking procedures and provides high-quality reference information for subsequent knowledge enhancement. Using benchmarking procedures to enhance hazard description information yields hazard identification information and handling suggestions. This combines domain knowledge and standard procedures to improve the credibility and interpretability of on-site hazard identification information, providing compliant and actionable handling suggestions, thereby improving the troubleshooting efficiency of inspection personnel. Hazard location information, hazard identification information, and handling suggestions are sent to an AR terminal. The AR terminal uses augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream. The display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream, achieving precise mapping between virtual hazard locations and hazard locations in real equipment. This improves hazard identification efficiency, enabling on-site operators to promptly eliminate on-site hazards based on accurate and timely technical guidance. Furthermore, post-hazard elimination verification can be performed, forming a closed loop, thereby enhancing the intelligence level of power production sites and ensuring the safe and stable operation of power production. Attached Figure Description

[0012] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a method for on-site inspection and supervision of power production provided in an embodiment of the present invention. Figure 2 This is another flowchart illustrating the on-site inspection and supervision method for power production provided in this embodiment of the invention; Figure 3 This is another flowchart illustrating the on-site inspection and supervision method for power production provided in this embodiment of the invention. Figure 4 This is a schematic diagram of an augmented reality image provided in an embodiment of the present invention; Figure 5 This is another schematic diagram of the augmented reality image provided in the embodiment of the present invention; Figure 6 This is another flowchart illustrating the augmented reality scene provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a power production site inspection and monitoring device provided in an embodiment of the present invention; Figure 8 This is another structural schematic diagram of the power production site inspection and monitoring device provided in this embodiment of the invention; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Figure 1This is a flowchart illustrating a power production site inspection and supervision method provided in an embodiment of the present invention. This method is applicable to scenarios involving the inspection of equipment and facilities at power production sites. The power production site inspection and supervision method can be executed by a power production site inspection and supervision device provided in this embodiment, which can be implemented using software and / or hardware. In one specific embodiment, the device can be integrated into an edge computing device, such as an industrial computer or an industrial smart gateway; alternatively, the device can be integrated into an augmented reality (AR) terminal, such as AR glasses or a tablet computer.

[0017] See Figure 1 This embodiment uses the integration of a power production site inspection and monitoring device into an edge computing device as an example for illustration. The power production site inspection and monitoring method of this embodiment may include the following steps: Step 101: Obtain the live video stream captured and uploaded in real time by the augmented reality (AR) terminal.

[0018] The on-site video stream is continuous video data collected in real time by AR terminals equipped by inspection personnel at the power production site. The on-site video stream typically includes dynamic images of power production equipment and facilities, their surrounding environment, and on-site operators, and may also include written technical supervision documents. For example, the power production site may be a new energy power plant site; the equipment could be key equipment such as wind turbine main shafts and inverters, or auxiliary equipment such as inverter cooling fans; and the written technical supervision documents could include technical reports, design drawings, experimental reports, and maintenance work orders.

[0019] On-site video streams are typically color videos captured by high-definition cameras, but can also be special types of video data such as infrared thermal imaging videos. The specific video type can be selected according to the requirements of the hazard identification task.

[0020] Specifically, AR terminals equipped by inspection personnel can be used to collect on-site video streams from a first-person perspective without physical contact. Data transmission between the AR terminal and the edge computing device can take multiple forms, including wired and wireless transmission. Specifically, the collected on-site video stream is transmitted to the edge computing device in real-time via wireless transmission by default, while wired transmission serves as a backup method when the wireless signal is unstable, ensuring the reliability and stability of data transmission. The AR terminal can also be configured with various sensors, such as light sensors, humidity sensors, and dust sensors, to collect on-site data streams in real time. These data streams are uploaded to the edge computing device simultaneously with the on-site video stream, serving as auxiliary information for subsequent hazard identification.

[0021] Step 102: Input the on-site video stream into the hazard identification intelligent agent so that the hazard identification intelligent agent can analyze the on-site video stream and obtain on-site hazard information.

[0022] The on-site hazard information includes the type of identified object, operating conditions, hazard category, hazard description information, and hazard location information.

[0023] The hazard identification intelligent agent is an intelligent software system used for automated analysis of video streams from power production sites. It integrates multiple hazard identification models. These models are computational models used to identify on-site hazards and can include various computer vision and machine learning models. For example, a hazard identification model could be a computer vision model with a temporal attention mechanism.

[0024] The objects of identification are identifiable entities in the power production site that may pose potential hazards. These objects can include power production equipment, facilities, and written technical supervision documents.

[0025] On-site hazards are identified from on-site video streams as abnormal conditions that may affect the normal function of the identified object. Specifically, these can be physical damage to equipment and facilities, such as cracks, wear, deformation, and corrosion on the surface of equipment structures; abnormal vibration, displacement, or jamming of moving parts of equipment; or non-compliance items in technical supervision documents.

[0026] On-site hazard information is the result of on-site hazard identification obtained directly by the hazard identification agent based on the on-site video stream. Specifically, on-site hazard information can be represented in the form of structured data.

[0027] The object type to be identified is the type of object that poses a potential hazard on site. Specifically, it can be the type of equipment, the type of facility, the type of technical report, etc.

[0028] Operating conditions are a set of parameters representing the operating environment and working status of the identified object at the time of video capture. Operating conditions are used to assist in judging the reasonableness of potential hazards on-site and determining applicable standard procedure thresholds. Specifically, operating conditions may include equipment operating status (e.g., running, stopped); environmental conditions (e.g., ambient temperature, humidity, wind speed, light intensity); and time information (e.g., season, day / night).

[0029] Hazard categories are labels used to classify on-site hazards according to their physical properties, manifestations, or causes. For example, hazard categories include, but are not limited to: structural hazards (cracks, deformation, fractures), abrasion hazards (erosion, scratches, pitting), and electrical hazards (hot spots, discharge, insulation damage), etc.

[0030] Hazard description information is data that quantifies or qualitatively describes the specific characteristics of a hazard. Specifically, hazard description information includes, but is not limited to: the hazard's geometric dimensions, such as length, width, area, and depth, as well as physical properties, such as color, temperature, and texture features.

[0031] Hazard location information refers to the position of a hazard in the on-site video stream. This information can take the form of a two-dimensional bounding box selecting the hazard location, a segmentation mask defining the hazard area, or key point markers of the identified object, which correspond to preset anchor points within the identified object.

[0032] Specifically, the AR terminal can be configured with a voice interaction module, enabling intelligent voice interaction with inspection personnel. It can also perform voice recognition on the voice commands input by the personnel and upload the recognized voice command information to the edge computing device. The edge computing device combines the voice command information with keyframes from the on-site video stream for joint semantic understanding to determine the inspection task. For example, the voice command information could be "Identify whether the blades of these wind turbines are damaged." The inspection task can be represented as {Object type: Wind turbine; Number of devices: 3; Task type: Visual feature recognition}. After determining the inspection task, the hazard identification intelligent agent integrated on the edge computing device can select a matching computational model from various machine learning models and computer recognition models to execute the inspection task. Alternatively, an agreement can be reached between the AR terminal and the hazard identification intelligent agent to automatically identify on-site hazards based on the received on-site video stream without needing to receive voice commands.

[0033] In addition to identifying potential hazards on-site, edge computing devices can also support the processing of various other inspection tasks such as equipment status query, equipment fault diagnosis, and inspection path planning, so as to comprehensively cover various application scenarios of power production site inspection and supervision.

[0034] Edge computing devices can first preprocess the on-site video stream and extract key identification frame sequences from the preprocessed on-site video stream. The key identification frame sequences are then input into the hazard identification agent, enabling the hazard identification agent to call the hazard identification model corresponding to the inspection task and output structured on-site hazard information.

[0035] Step 103: Based on the type of object being identified, the operating conditions, and the category of potential hazards, retrieve the standard procedures stored in the knowledge base and determine the benchmarking procedures.

[0036] The knowledge base is a pre-built knowledge database for the power production field stored on edge computing devices, containing standard procedures. Specifically, the knowledge base can be stored in a vectorized manner to quickly match standard procedures that match the object type, operating conditions, and hazard categories by calculating the similarity of the embedded vectors. In addition, the knowledge base can also store other data such as historical inspection cases and industrial equipment files, which can also serve as reference information for benchmarking analysis.

[0037] Benchmarking procedures are one or more standard procedures that match the currently identified on-site hazards in terms of the type of identified object, operating conditions, and hazard category. Specifically, the type of identified object, operating conditions, and hazard category can be embedded using a pre-trained embedding model, and cosine similarity calculation can be performed between these parameters and the vector representations of all standard procedures in the knowledge base. The procedures with the highest similarity are then recalled as benchmarking procedures.

[0038] Step 104: Use the benchmarking procedures to enhance the knowledge of the hazard description information to obtain hazard identification information and handling suggestions.

[0039] Hazard identification information is a set of information formed by combining on-site hazard information identified by the hazard identification model with knowledge enhancement through benchmarking procedures. Specifically, it may include hazard category, hazard severity, and relevant standards and procedures.

[0040] For example, hazard identification information may include inspection specialty, reference standard or procedure, inspection content or items, hazard category or existing problem, and problem classification. Inspection specialty identifies the specialty to which the current hazard belongs, such as buildings / structures, energy storage battery systems, primary electrical equipment, secondary electrical equipment, wind turbine mechanical systems, etc.; reference standard or procedure identifies the standard clause number corresponding to the identified hazard; inspection content or items identifies the inspection requirements corresponding to the standard clause; hazard category or existing problem description characterizes the identified actual defect, abnormal state, or non-compliance; and problem classification indicates the risk level or severity of the hazard.

[0041] The handling recommendations are generated based on equipment hazards and standard procedures. They are operational instructions or corrective measures used to guide on-site personnel to take specific actions. Specifically, they may include handling recommendations for the current hazards and re-inspection recommendations after rectification.

[0042] Knowledge enhancement refers to the process of semantically fusing domain knowledge and equipment hazards from benchmarking procedures to generate hazard identification information related to domain knowledge. Specifically, the hazard identification agent can combine benchmarking procedures and hazard description information according to a preset prompt template to construct a structured input context, which is then input into a preset large language model. The preset large language model, based on retrieval enhancement generation technology, extracts key judgment rules from the input context, compares them with the hazard description information, and outputs structured hazard identification information and handling suggestions according to a predefined format.

[0043] After identifying potential hazards and providing recommendations for mitigation, on-site inspectors and operators can take appropriate measures to eliminate these hazards. The inspection and supervision method provided in this embodiment is then used to verify and confirm that the hazards have been completely eliminated. The edge computing device can also automatically generate structured inspection reports and automatically classify and archive them, providing data support for the subsequent improvement of the knowledge base and the training of the hazard identification model. The inspection report may include a list of inspectors, hazard identification information, mitigation recommendations, the actual mitigation measures taken by the inspectors, and the results.

[0044] Step 105: Send the hazard location information, hazard identification information, and handling suggestions to the AR terminal so that the AR terminal can use augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information on the augmented reality screen based on the on-site video stream, and the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

[0045] Augmented reality (AR) visuals are composite views created in real time by fusing images of the real physical environment with virtual information generated by the AR terminal. For example, when the AR terminal is AR glasses worn by inspection personnel, the AR visuals are images that the user sees directly through the lenses, consisting of a real-world scene superimposed with virtual information projected at specific locations in their field of vision.

[0046] Virtual information is non-physical data content generated by AR terminals and presented in the form of graphics, text, animation, etc. The content of virtual information includes at least hazard identification information and handling suggestions. Specific forms of virtual information can include highlighted boxes, arrows, lines, text labels, icons, or dynamic indicator animations.

[0047] For example, in the augmented reality view, a semi-transparent, colored highlight frame can be drawn around the identified equipment hazard or the area containing the hazard. The hazard level can be indicated with color and text labels. The size and perspective of the highlight frame can be dynamically adjusted according to the distance and viewing angle of the inspector, always maintaining a close alignment with the actual equipment edge. When the hazard area is in a location difficult to observe directly, an animated arrow can be drawn from the edge of the highlight frame to attract the inspector's attention. A semi-transparent text label is displayed floating near the highlight frame, containing information such as the hazard type, hazard level, and recommended actions.

[0048] In this embodiment, acquiring and uploading real-time on-site video streams from augmented reality (AR) terminals overcomes the limitations of traditional manual operations, enabling real-time and efficient on-site data acquisition to effectively avoid missed detections and errors. The on-site video stream is then input into a hazard identification intelligent agent, which analyzes the stream to obtain on-site hazard information. This information includes the type of identified object, operating conditions, hazard category, hazard description, and hazard location information. By retrieving standard procedures stored in the knowledge base based on the type of identified object, operating conditions, and hazard category, benchmarking procedures are determined. This enhances the accuracy of benchmarking procedures and provides high-quality reference information for subsequent knowledge enhancement. Using benchmarking procedures to enhance hazard description information yields hazard identification information and handling suggestions. This combines domain knowledge and standard procedures to improve the credibility and interpretability of on-site hazard identification information, providing compliant and actionable handling suggestions, thereby improving the troubleshooting efficiency of inspection personnel. Hazard location information, hazard identification information, and handling suggestions are sent to an AR terminal. The AR terminal uses augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream. The display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream, achieving precise mapping between virtual hazard locations and hazard locations in real equipment. This improves hazard identification efficiency, enabling on-site operators to promptly eliminate on-site hazards based on accurate and timely technical guidance. Furthermore, post-hazard elimination verification can be performed, forming a closed loop, thereby enhancing the intelligence level of power production sites and ensuring the safe and stable operation of power production.

[0049] Figure 2 This is another flowchart illustrating the on-site inspection and supervision method for power production provided in this embodiment of the invention. The following is a combined description of... Figure 2 Taking the integration of power production site inspection and supervision devices into edge computing devices as an example, the power production site inspection and supervision method provided in this embodiment of the invention is further illustrated. The power production site inspection and supervision method in this embodiment may include the following steps: Step 201: Obtain the live video stream captured and uploaded in real time by the AR terminal.

[0050] Step 202: Determine the task processing device based on the task complexity, data sensitivity, and real-time requirements of the current inspection task. If the task processing device is a cloud management cockpit, proceed to step 203. If the task processing device is an edge computing device, proceed to step 204.

[0051] Among them, when the task complexity is lower than the preset complexity, the data sensitivity is higher than the preset sensitivity, and the real-time requirement is higher than the preset real-time requirement, the task processing device is determined to be an edge computing device.

[0052] Inspection tasks are hazard identification tasks at power production sites. In this embodiment, it specifically refers to identifying on-site hazards based on on-site video streams, and generating hazard identification information and handling suggestions based on the on-site hazards and benchmarking procedures. Task processing equipment is a specific type of equipment that performs hazard identification tasks.

[0053] Task processing devices include edge computing devices and cloud management dashboards. A cloud management dashboard can be understood as an intelligent analysis platform deployed in a cloud data center. Specifically, this platform possesses higher computing power, larger memory capacity, and a more complete knowledge base than edge computing devices. For example, a lightweight computing model with fewer than 50M parameters can be deployed on an edge computing device, while a complete computing model with over 500M parameters can be deployed in the cloud management dashboard, supporting higher-precision hazard identification. Edge computing devices only store standard procedures and industrial equipment files related to the current new energy power station, while the cloud management dashboard stores a complete standard procedure library, historical inspection cases from multiple other new energy power stations, equipment files, etc. However, due to the greater physical distance between the cloud management dashboard and the industrial production site, the data exchange latency with the AR terminal is greater than that with the AR terminal. For example, because the edge computing device is close to the site, the latency when the AR terminal uploads the on-site video stream to the edge computing device is typically 10-50ms, and the latency when the edge computing device sends the hazard location information, hazard identification information, and handling suggestions to the AR terminal is 5 to 20ms. Due to the influence of network bandwidth, the latency when the AR terminal uploads the on-site video stream to the cloud management dashboard is 50-200ms, and the latency when the cloud management dashboard sends the hazard location information, hazard identification information, and handling suggestions to the AR terminal is 30 to 100ms.

[0054] The task complexity can be calculated based on factors such as the amount of CPU / Graphics Processing Unit (GPU) computation required for the inspection task, the required memory capacity, the amount of knowledge base data to be referenced, the number and scale of the computing models to be called, and the amount of input data.

[0055] Specifically, because cloud-based management dashboards have stronger task processing capabilities, they are suitable for inspection tasks with higher complexity. If the inspection task is a complex reasoning and global optimization task that requires access to a large amount of data in the knowledge base and multiple computational models, the cloud-based management dashboard can be designated as the task management device. For example, when the inspection task involves the collaborative work of multiple devices, such as joint analysis of potential hazards on multiple devices, data correlation analysis of multiple new energy power stations, or prediction of device failure trends based on historical fault information, the task processing device can be designated as the cloud-based management dashboard. When the inspection task is the identification of potential hazards on a single device, the task processing device can be designated as an edge computing device.

[0056] Data sensitivity refers to the degree of sensitivity of on-site video streams and other production-related data in terms of confidentiality, privacy, or compliance. Highly sensitive data is not transmitted over the public internet to avoid the risk of leakage. Real-time requirements are the maximum tolerable delay between the upload of the on-site video stream and the output of the results for an inspection task.

[0057] Specifically, the AR terminal can be configured with a voice interaction module, enabling intelligent voice interaction with inspection personnel. It can recognize the voice commands input by the personnel and upload the recognized voice command information to an edge computing device. The edge computing device combines the voice command information with keyframes from the on-site video stream for joint semantic understanding to determine the inspection task. Once the inspection task is determined, its complexity, data sensitivity, and real-time requirements can be determined according to preset rules. When the task complexity is higher than the preset complexity, the task processing device can be designated as an edge computing device; otherwise, it will be designated as a cloud management dashboard. Similarly, when the data sensitivity is higher than the preset sensitivity, the task processing device can be designated as an edge computing device; otherwise, it will be designated as a cloud management dashboard. The priority relationship between task complexity, data sensitivity, and real-time requirements can also be set as needed. For example, the priority of data sensitivity can be set higher than the priority of real-time requirements, and the priority of real-time requirements can be set higher than the priority of task complexity. Therefore, when the complexity of the inspection task is higher than the preset complexity, but the data sensitivity is higher than the preset sensitivity, the task processing equipment is determined to be the cloud management cockpit.

[0058] In this embodiment, the task processing equipment is determined according to the task complexity, data sensitivity and real-time requirements of the current inspection task, which can realize the efficient allocation of computing resources and improve the real-time response of the system while meeting the quality requirements of the inspection task processing.

[0059] Step 203: Forward the current inspection task to the cloud management cockpit so that the cloud management cockpit can process the current inspection task.

[0060] Specifically, once the task processing device is identified as the cloud management dashboard, the edge computing device first preprocesses the on-site video stream uploaded by the AR terminal, extracting key identification frames and encrypting them. Then, the preprocessed on-site video stream is transmitted to the cloud management dashboard, where the integrated hazard identification intelligence analyzes the video stream to obtain on-site hazard information. This information includes the type of identified object, operating conditions, hazard category, hazard description, and hazard location. Based on the object type, operating conditions, and hazard category, standard procedures stored in the knowledge base are retrieved to determine the benchmark procedure. The benchmark procedure is then used to augment the hazard description information, resulting in hazard identification information and handling suggestions. Finally, the hazard location information, hazard identification information, and handling suggestions are sent to the edge computing device, which then forwards this information to the AR terminal; alternatively, the hazard location information, hazard identification information, and handling suggestions are directly sent to the AR terminal.

[0061] Step 204: Input the on-site video stream into the device's hazard identification intelligent agent so that the hazard identification intelligent agent can analyze the on-site video stream to obtain the identification object type, operating conditions, hazard category, hazard description information, and hazard location information.

[0062] Step 205: Based on the type of object being identified, the operating conditions, and the category of potential hazards, retrieve the standard procedures stored in the knowledge base and determine the benchmarking procedures.

[0063] Step 206: Use the benchmarking procedures to enhance the knowledge of the hazard description information to obtain hazard identification information and handling suggestions.

[0064] Step 207: Extract the image features of the identified object based on the on-site video stream, obtain the standard object image features corresponding to the type of the identified object and the defective object image features corresponding to the category of hidden danger stored in the knowledge base, and determine the visual similarity score based on the first similarity between the image features of the identified object and the standard object image features, and the second similarity between the image features of the identified object and the defective object image features.

[0065] The image features of the identified object are the image features of the identified object contained in the current on-site video stream; the standard object image features are the standard image features of the identified object contained in the on-site video stream when there are no hidden dangers; the defective equipment image features are the typical image features of the identified object contained in the on-site video stream when there are hidden dangers corresponding to the hidden danger category. The standard object image features can be pre-stored in the knowledge base; the defective equipment image features can be collected during each inspection, and then stored in the knowledge base after being screened and confirmed by technicians.

[0066] The first similarity score is the similarity between the features of the identified object image and the features of the standard identified image; the second similarity score is the similarity between the features of the identified object image and the features of the defective object image. Both the first and second similarities can be measured using the cosine similarity of the feature vectors. The visual similarity score quantifies the impact of the feature similarity between the features of the identified object image and the features of reference images in the knowledge base on the confidence level of the hazard identification results. A higher first similarity score results in a lower visual similarity score, and a higher second similarity score results in a higher score.

[0067] Specifically, key recognition frames can be extracted from the on-site video stream. These frames are then input into a visual feature extraction model for feature extraction to obtain the image features of the object to be identified. The similarity between the image features of the object to be identified and each standard object image stored in the knowledge base is calculated. A first similarity is determined based on these similarities; for example, the maximum value among the similarities can be used as the first similarity. Similarly, the similarity between the image features of the object to be identified and the defect object image features corresponding to the hazard category is calculated. A second similarity is determined based on these similarities; for example, the average value among the similarities can be used as the second similarity. The second and first similarities are then weighted, and the visual similarity score is determined based on the difference between the weighted second and first similarities. Other methods for determining the visual similarity score based on the first and second similarities are also acceptable and are not limited in this embodiment.

[0068] Step 208: Determine the contextual compliance score based on the matching degree between historical hazard identification information and on-site hazard information.

[0069] Historical hazard identification information is a collection of hazard identification information for objects identified during previous inspections, contained in the on-site video stream. Contextual compliance scoring quantifies the impact of the matching degree between historical hazard identification information and current on-site hazard information on the confidence level of the hazard identification results. The higher the matching degree between historical hazard identification information and on-site hazard information, the larger the score.

[0070] Specifically, this is due to the correlation between the historical hazards of an identified object and its current hazards. For example, if a hazard has frequently occurred in the past or matches the typical hazard pattern of the identified object, the contextual compliance score is higher; conversely, if a hazard has never occurred in the past and contradicts known hazard patterns, the score is lower. For instance, if a wind turbine has experienced rotational imbalance multiple times, the contextual compliance score will be appropriately increased when the equipment hazard is "rotational imbalance".

[0071] In addition, a contextual compliance score can be calculated based on historical hazard identification information, the type of identified object, and the working condition to further determine the rationality of the occurrence of on-site hazards under the current working conditions.

[0072] Step 209: Extract the key identification frame sequence from the on-site video stream, and determine the timing consistency score based on the equipment hidden dangers corresponding to each frame in the key identification frame sequence.

[0073] The temporal consistency score is used to quantify the impact of the consistency of on-site hazards corresponding to each frame in the keyframe sequence on the confidence level of hazard identification results. The more consistent the on-site hazards corresponding to each frame in the keyframe sequence, the larger the score.

[0074] Specifically, the hazard identification model can output the equipment hazard identification result for each frame in the key identification frame sequence, and determine the temporal consistency score based on the number of frames in the key frame sequence in which the on-site hazard exists. This can avoid false alarms caused by transient interference. For example, if the specific calculation method is to divide the number of frames in 30 consecutive key identification frames that identify the on-site hazard by 30, such as 27 frames in which the on-site hazard is identified, then the temporal consistency score is 27 / 30 = 0.9.

[0075] Step 210: Obtain the on-site data stream uploaded by the AR terminal, and determine the environmental interference intensity score based on the on-site data stream.

[0076] The environmental interference intensity score is used to quantify the impact of environmental interference intensity at the power production site on the confidence level of hazard identification results. The stronger the environmental interference intensity, the lower the score value. For example, multiple sensors integrated into the AR terminal, such as light sensors, humidity sensors, and dust sensors, can be used to acquire on-site data streams, and the environmental interference intensity can be determined based on the on-site data streams. For instance, the environmental interference intensity is set to 1.0 in a sunny, interference-free environment, and 0.6 in severe environments such as heavy rain or strong light.

[0077] Step 211: Determine the data quality score based on the video quality parameters of the live video stream.

[0078] The data quality score quantifies the impact of video quality from the on-site video stream on the confidence level of hazard identification results. The worse the data quality, the higher the score.

[0079] Specifically, the data quality score can be calculated based on the frame rate, image clarity, and device occlusion rate of the live video stream. For example, a frame rate ≥ 30 frames / second earns 0.3 points, a resolution ≥ 1080 pixels earns 0.3 points, and a device occlusion rate ≤ 5% earns 0.4 points. The sum of these three scores is the data quality score: 0.3 + 0.3 + 0.4 = 1 point.

[0080] Step 212: Perform a weighted summation of the visual similarity score, contextual compliance score, temporal consistency score, environmental interference intensity score, and data quality score to obtain the confidence score.

[0081] Confidence level is a quantitative indicator of the reliability of on-site hazard information. Specifically, the confidence level can range from [0, 1.0], with a value closer to 1 indicating a more reliable on-site hazard information.

[0082] For example, the formula for calculating confidence level can be: .in, The confidence level of on-site hazard information. Visual similarity is scored, with values ​​ranging from [0, 1.0]. This is the weighting coefficient for visual similarity scoring. The default value is 0.35, but it can be dynamically adjusted based on the visual recognizability of the object being identified. For example, this weight can be appropriately increased for devices with many appearance defects. The contextual compliance score is given, with a value range of (0, 10). This is the weighting coefficient for the context compliance score. The default value is 0.25, and it can be dynamically adjusted according to the importance of the hazard identification task, such as when inspecting the main shaft of a critical piece of equipment, the fan. Adjust appropriately upwards. The time series consistency score is set, with a value range of (0, 1). It is the weighting coefficient for the time series consistency score. The default value is 0.2, and it can be dynamically adjusted according to the intensity of environmental interference. For example, it can be increased to 0.25 in outdoor multi-interference operation scenarios. The environmental disturbance intensity is scored, with a value range of (0, 1). The importance score for the identified objects ranges from [1.0, 1.5]. It is set based on the degree of impact of the equipment on the safe operation of the new energy power station and is used to increase the identification priority of key identified objects. For example, key equipment such as wind turbine main shafts and inverters are scored from 1.3 to 1.5, and auxiliary equipment such as cooling fans are scored from 1.0 to 1.1. It is the weighting coefficient of the combination of environmental interference intensity score and equipment importance score, used to reflect the combined effect of environmental impact and the priority of the identified object. The default value can be 0.1, and it can be increased to 0.15 in harsh environment operation or critical equipment inspection scenarios. The data quality score is given, with a value range of [0, 1.0]. This is the weighting coefficient for the data quality score. The default value is 0.1, but it can be increased to 0.15 for scenarios where the video is easily blurred or obscured.

[0083] Step 213: Determine whether the confidence level is lower than the preset confidence threshold; if it is lower, proceed to step 214; otherwise, proceed to step 215.

[0084] Specifically, if the confidence level is low, the on-site video stream can be forwarded to the cloud management dashboard for verification and identification to obtain more accurate identification results, or professional technical support can be requested from the technical personnel in the cloud management dashboard; if the confidence level reaches the preset confidence threshold, the confidence level, hazard location information, hazard identification information and handling suggestions can be sent to the AR terminal for inspection personnel to refer to.

[0085] Step 214: Forward the on-site video stream to the cloud management dashboard so that the cloud management dashboard can review and identify on-site potential hazards.

[0086] Specifically, the cloud-based management dashboard inputs the on-site video stream into its deployed hazard identification intelligent agent to identify on-site hazards, obtaining the type of the identified object, operating conditions, hazard category, hazard description information, and hazard location information. Based on the identified object type, operating conditions, and hazard category, it retrieves the standard procedures stored in the knowledge base, determines the benchmarking procedure, and uses the benchmarking procedure to enhance the hazard description information, obtaining hazard identification information and handling suggestions. Finally, it sends the hazard location information, hazard identification information, and handling suggestions to the edge computing device, which then forwards the information to the AR terminal; or, it directly sends the hazard location information, hazard identification information, and handling suggestions to the AR terminal.

[0087] In one feasible implementation, the on-site video stream can also be displayed to technical experts via the cloud-managed cockpit's display unit to obtain expert guidance, which can then be sent to the AR terminal. Specifically, the cloud-managed cockpit and the AR terminal can communicate in real time. Technical experts can view the on-site video stream in real time through the cloud-managed cockpit's display unit and provide technical support to inspection personnel or on-site workers via voice or text, achieving full monitoring and remote guidance of the inspection operation.

[0088] Step 215: Send the confidence level, hazard identification information, and handling suggestions to the AR terminal.

[0089] Specifically, the confidence level, hazard location information, hazard identification information, and handling suggestions are sent to the AR terminal. The AR terminal then uses augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream, and the display position of the virtual information matches the corresponding real equipment or hazard location in the on-site video stream.

[0090] After the inspection is completed, the cloud-based management dashboard can record every detail of the inspection personnel's work, including the inspection route, dwell time, and operational actions. By analyzing the inspection operation from multiple dimensions such as time, business category, and space, a database of inspection operation behavior is established. Using big data intelligent analysis technology, non-standard behaviors and potential risks during the inspection process are identified, and the operation process is optimized accordingly. The dashboard can also provide intelligent evaluation of the inspection personnel and offer improvement suggestions.

[0091] In this embodiment, the on-site video stream collected and uploaded in real time by the AR terminal is acquired; the task processing device is determined according to the task complexity, data sensitivity, and real-time requirements of the current inspection task; when the task processing device is a cloud management cockpit, the current inspection task is forwarded to the cloud management cockpit so that the cloud management cockpit can process the current inspection task, dynamically allocate computing resources to meet the personalized needs of different inspection tasks, ensure that sensitive data does not leave the factory area and tasks with high real-time requirements are processed nearby, while making full use of the computing power of the cloud management cockpit to ensure the accuracy of complex tasks; the on-site video stream is input into the device's hazard identification intelligent agent so that the hazard identification intelligent agent can analyze the on-site video stream to obtain the type of identification object, working conditions, etc. The system identifies hazard categories, hazard descriptions, and hazard locations. Based on the object type, operating conditions, and hazard category, it retrieves standard procedures stored in the knowledge base to determine benchmarking procedures. These benchmarking procedures are then used to enhance the hazard description information, yielding hazard identification information and handling suggestions. This approach combines domain knowledge to improve the interpretability and authority of hazard identification results. A visual similarity score is determined based on the first similarity between the identified object's image features and the standard object's image features stored in the knowledge base, and the second similarity between the visual features of the defective object corresponding to the hazard category. A contextual compliance score is determined based on the matching degree between historical hazard identification information and on-site hazard information. Key identification frame sequences are extracted from the on-site video stream, and the corresponding frames in each key identification frame sequence are analyzed. The system determines equipment hazard identification based on temporal consistency; it acquires on-site data streams uploaded from AR terminals and determines environmental interference intensity based on these data streams; it determines data quality based on video quality parameters of the on-site video stream; and it performs a weighted summation of visual similarity, contextual compliance, temporal consistency, environmental interference intensity, and data quality scores to obtain a confidence score. This allows for multi-dimensional evaluation of the reliability of hazard identification results, considering visual feature similarity, historical equipment operating patterns, environmental interference, and the video quality of the on-site video stream. When the confidence score is below a preset confidence threshold, the on-site video stream is forwarded to a cloud management dashboard for further verification and identification of on-site hazards, leveraging the higher computing power of the cloud management dashboard. This system ensures the accuracy and reliability of hazard identification information obtained by inspection personnel, preventing misjudgments caused by low-confidence hazard identification results. It sends confidence levels, hazard location information, hazard identification information, and handling suggestions to an AR terminal. The AR terminal then uses augmented reality technology to overlay these hazard identification and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream. The virtual information's display position matches the corresponding real equipment or hazard location in the on-site video stream. This allows the AR terminal to effectively integrate virtual information such as hazard identification and handling suggestions with the real on-site video stream, improving the intelligence level of industrial on-site inspection operations based on precise and timely technical guidance from inspection personnel.

[0092] Figure 3 This is another flowchart illustrating the on-site inspection and supervision method for power production provided in this embodiment of the invention. (See attached diagram.) Figure 3 This embodiment uses the integration of a power production site inspection and supervision device into an AR terminal as an example for illustration. The power production site inspection and supervision method of this embodiment may include the following steps: Step 301: Acquire the live video stream in real time and upload it to the edge computing device.

[0093] Specifically, by utilizing the video acquisition unit of the AR terminal device equipped by inspection personnel, such as the high-definition camera on AR glasses, the on-site video stream containing the target object to be inspected can be acquired in real time from a first-person perspective during the inspection process, and the on-site video stream can be uploaded to the edge computing device in real time.

[0094] Step 302: Receive the hazard location information, hazard identification information and handling suggestions returned by the edge computing device.

[0095] Among them, the hazard location information, hazard identification information, and handling suggestions are obtained by the hazard identification intelligent agent deployed on the edge computing device using benchmarking procedures to enhance the hazard description information; the benchmarking procedures are obtained by the hazard identification intelligent agent by searching the standard procedures stored in the knowledge base according to the identification object type, operating conditions, and hazard category; the identification object type, operating conditions, hazard category, hazard description information, and hazard location information are obtained by the hazard identification intelligent agent by analyzing the on-site video stream.

[0096] Specifically, after receiving the on-site video stream from the AR terminal, the edge computing device can determine the task processing device to execute the current inspection task based on its complexity, real-time performance, and sensitivity. This task processing device can be either an edge computing device or a cloud-based management dashboard. The on-site video stream is then input into the integrated hazard identification intelligence agent on the task processing device for analysis, yielding on-site hazard information. This information includes the type of identified object, operating conditions, hazard category, hazard description, and hazard location. Based on the identified object type, operating conditions, and hazard category, the device retrieves standard procedures stored in the knowledge base to determine the benchmarking procedure. This benchmarking procedure is then used to augment the hazard description information, resulting in hazard identification information and handling suggestions. Finally, the hazard location information, hazard description, and handling suggestions are sent to the AR terminal.

[0097] Step 303: Using augmented reality technology, based on the hazard location information, the hazard identification information and handling suggestions are overlaid and displayed as virtual information in the augmented reality screen based on the on-site video stream, and the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

[0098] Optionally, the local coordinates of the identified object can be determined based on the hazard location information and the preset 3D model of the identified object; the current pose of the identified object can be determined based on the on-site video stream and the preset 3D model of the identified object; the current pose of the terminal can be determined based on the on-site video stream, terminal posture data, and visual depth information; the target display position can be determined based on the local coordinates of the identified object, the current pose of the identified object, and the current pose of the terminal; the hazard identification information and handling suggestions can be overlaid and displayed as virtual information at the target display position of the augmented reality screen so that the display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream.

[0099] The preset 3D model of the object to be identified is a digital 3D model that precisely corresponds to the real object in terms of geometric structure and visual features. The preset 3D model of the object to be identified can be pre-built and stored in edge computing devices or AR terminals.

[0100] The local coordinates of the identified object are the three-dimensional spatial location information of the potential hazard location within the object's local coordinate system. The object's local coordinate system is a local three-dimensional coordinate system established with a preset point in the object's preset three-dimensional model as its origin. The object's local coordinate system moves synchronously with the movement or rotation of the identified object, while its origin and coordinate axis directions remain fixed relative to the object itself.

[0101] The current pose of the identified object refers to its spatial position and orientation relative to the three-dimensional world coordinate system, which can be represented by a pose transformation matrix from the local coordinate system to the three-dimensional world coordinate system. The current pose of the identified object is used to convert the coordinates of the potential hazard location in the local coordinate system to its coordinates in the three-dimensional world coordinate system.

[0102] Terminal attitude data refers to the rotational orientation information of the AR terminal relative to the local world coordinate system or the three-dimensional world coordinate system at the current moment, used to describe the pointing direction of the camera integrated into the AR terminal. Specifically, terminal attitude data is usually represented by the AR terminal's three-axis acceleration and three-axis angular velocity data, which can be acquired using the inertial measurement unit integrated into the AR terminal.

[0103] Visual depth information is a set of data used to describe the distances of various points on the surface of an object in a real-world environment relative to an AR terminal. Specifically, visual depth information is usually stored in the form of a depth map and can be acquired using devices with distance perception capabilities, such as binocular cameras or structured light sensors integrated into AR devices.

[0104] The current pose of the terminal refers to the spatial position and orientation information of the AR terminal relative to the three-dimensional world coordinate system, which can be represented by a pose transformation matrix from the three-dimensional world coordinate system to the terminal coordinate system. The terminal coordinate system is a local three-dimensional coordinate system established with the optical center of the AR terminal camera as the origin. The terminal coordinate system moves synchronously with the movement or rotation of the terminal camera, and its origin and coordinate axis directions remain fixed relative to the terminal camera body.

[0105] The target display location is the projection position of the potential hazard or the actual object in the augmented reality image, which can be represented by two-dimensional coordinates in the pixel coordinate system.

[0106] Specifically, the AR terminal first establishes a three-dimensional world coordinate system aligned with the physical space. This three-dimensional world coordinate system is a global reference system that serves as the benchmark for all recognized objects and the AR terminal's location. Geographic coordinate systems such as GPS are commonly used in this system. The AR terminal can use monocular or binocular cameras to capture live video streams; use an inertial measurement unit to collect angular velocity and acceleration data as terminal attitude data; and use binocular cameras or structured light sensors to acquire visual depth information. Employing real-time localization and mapping (RTD) technology, the terminal determines its current pose using the live video stream, attitude data, and visual depth information.

[0107] Determining the current pose of the object to be identified based on the live video stream and a preset 3D model of the object can be implemented in two ways: In one feasible implementation, a preset 3D model of the object to be identified can be stored in advance; the outline and feature points of the object to be identified are extracted from key recognition frames contained in the live video stream; the outline and feature points of the object to be identified are matched with the preset 3D model; and the current pose of the object to be identified is calculated. In another feasible implementation, preset 2D / 3D key point templates of key parts of the object to be identified can be stored in advance; the real key points corresponding to the key point templates are detected from the key recognition frames contained in the live video stream; and the current pose of the object to be identified is determined based on the real key points.

[0108] Hazard location information can include the two-dimensional bounding box coordinates of the hazard location, hazard segmentation masks, etc. Based on the current depth map and the preset three-dimensional model of the object to be identified, the AR terminal determines the three-dimensional coordinates of the hazard location in the equipment coordinate system, i.e., the local coordinates of the object to be identified. If the hazard location information is a key point identifier of the object to be identified, the AR terminal can directly extract the preset annotation anchor point corresponding to the key point from the preset 3D model of the object to be identified as the local coordinates of the object.

[0109] Known local coordinates of the object to be identified Identify the current pose of the object Current position of the terminal Then the coordinates of the hidden danger location or the actual identified object in the terminal coordinate system satisfy: Where K is the intrinsic parameter matrix of the AR terminal camera, used to... The target display position is obtained by performing homogeneous coordinate normalization and projection matrix transformation.

[0110] Figure 4 This is a schematic diagram of an augmented reality scene provided in an embodiment of the present invention. (See attached image.) Figure 5 During the inspection of wind turbine units at a new energy power station, if the image of the wind turbine tower foundation area extracted from the on-site video stream reveals potential steel structure corrosion, and if corrosion is detected at the base of the steel column anchor bolts on the steel platform at the wind turbine tower entrance, the target location is indicated by the red rectangle in the image, with coordinates of the upper left corner (92, 223) and the lower right corner (147, 298). The image also displays a confidence level of 0.92 for the on-site hazard information. Hazard identification information can include: ① Inspection specialty: Buildings (structures). ② Reference regulations and standards: Wind farm technical supervision and inspection evaluation standard 2.7.3. ③ Based on content (inspection items): Regular inspection content includes: 1) Wind turbine foundation settlement and horizontal displacement; 2) Integrity of building (structure) appearance; 3) Corrosion, deformation, damage, cracking, exposed reinforcement, and tilting of the main structure of buildings (structures); 4) Integrity of component surfaces. This includes checking for damage, aging, discoloration, cracking, peeling, flaking, and rust. ④ Hazard Category / Problem: The base bolts of the steel column on the steel platform at the wind turbine tower entrance are corroded and not encased in concrete. ⑤ Problem Classification: General. Recommendation: Protect the base bolts as soon as possible.

[0111] Figure 5 This is another schematic diagram of the augmented reality screen provided in an embodiment of the present invention. (See also...) Figure 5 When inspecting wind turbine units at a new energy power station, if condensation and corrosion hazards are identified in the energy storage battery compartment image in the on-site video stream, the target display location is the red rectangle in the image, with coordinates of upper left (870, 350) and lower right (520, 320). The image also displays a confidence level of 0.96 for the on-site hazard information. Hazard identification information can be: ① Inspection specialty: Energy storage battery system. ② Reference regulations and standards: Energy storage battery system technical supervision and inspection scoring standard 2.5.4. ③ Based on content (inspection items): Check the working status of the heating and dehumidification device installed in the compartment, and whether there is condensation affecting the insulation performance of electrical components and corrosion of metal materials. ④ Hazard category / existing problem: The energy storage battery compartment is not equipped with a dehumidification system, condensation exists in the compartment, and the wiring terminals are severely corroded. ⑤ Problem level: General. The handling suggestion can be: It is recommended to include the dehumidification system in the technical renovation plan, add a dehumidification system to the energy storage battery compartment, and at the same time strengthen daily inspections and replace the existing severely corroded wiring terminals.

[0112] Figure 6This is another schematic diagram of the augmented reality screen provided in the embodiment of the present invention. (See attached diagram.) Figure 6 If non-compliance is found in the preventive test report of the main transformer in the on-site video stream, the target display location is the red rectangle in the image. The rectangle contains the erroneous items in the test report, and the image also displays a confidence level of 0.86 for the on-site hazard information. Hazard identification information can be: ① Inspection specialty: Insulation. ② Reference standard: Articles 7 and 13 in Table 5 of DLT 596-2021 Preventive Test Procedure for Power Equipment. ③ Basis content (inspection item): Preventive test of main transformer. ④ Hazard category / problem: In the preventive test report of the main transformer, temperature conversion was not performed when measuring the insulation resistance of the windings and bushings, and the results were not compared with the previous test results; the insulation resistance of the core and clamps was not compared with the previous test results. ⑤ Problem classification: General. The handling suggestion can be: Temperature conversion should be performed when measuring the insulation resistance of the windings and bushings of the main transformer during the preventive test, and there should be no significant change compared with the previous test results; the insulation resistance of the core and clamps should be no significant difference compared with the previous test results.

[0113] In this embodiment, an AR terminal is used to collect real-time on-site video streams and upload them to an edge computing device, enabling seamless data collection from a first-person perspective. The device receives hazard identification information and handling suggestions from the edge computing device, determines the local coordinates of the identified object based on the hazard location information and a preset 3D model of the identified object, determines the current pose of the identified object based on the on-site video stream and the preset 3D model of the identified object, determines the current pose of the terminal based on the on-site video stream, terminal posture data, and visual depth information, and determines the target display position based on the local coordinates of the identified object, the current pose of the identified object, and the current pose of the terminal. The hazard identification information and handling suggestions are then overlaid on the target display position in the augmented reality screen as virtual information, ensuring that the display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream. This achieves accurate mapping between virtual hazard locations and hazard locations in real equipment, improving hazard identification efficiency. On-site operators can then promptly eliminate on-site hazards based on precise and timely technical guidance. Furthermore, after hazard elimination, verification and identification can be performed, forming a closed loop, thereby improving the intelligence level of the power production site and ensuring the safe and stable operation of power production.

[0114] Figure 7 This is a schematic diagram of a power production site inspection and monitoring device provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the device includes: The acquisition module 401 is used to acquire the live video stream that is collected and uploaded in real time by the augmented reality (AR) terminal; The identification module 402 is used to input the on-site video stream into the hazard identification intelligent agent, so that the hazard identification intelligent agent can analyze the on-site video stream to obtain on-site hazard information. The on-site hazard information includes the type of the identified object, working conditions, hazard category, hazard description information, and hazard location information. Based on the type of the identified object, working conditions, and hazard category, the module retrieves the standard procedures stored in the knowledge base to determine the benchmarking procedures. The benchmarking procedures are used to enhance the hazard description information to obtain hazard identification information and handling suggestions. The sending module 403 is used to send hazard location information, hazard identification information and handling suggestions to the AR terminal, so that the AR terminal can use augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information in the augmented reality screen based on the on-site video stream, and the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

[0115] In one embodiment, the apparatus further includes a device determination module for: The task processing equipment is determined based on the complexity, data sensitivity, and real-time requirements of the current inspection task. When the task processing device is a cloud management cockpit, the current inspection task is forwarded to the cloud management cockpit so that the cloud management cockpit can process the current inspection task; When the task processing device is an edge computing device, the execution of "inputting the on-site video stream into the hazard identification intelligent agent" is triggered; Among them, when the task complexity is lower than the preset complexity, the data sensitivity is higher than the preset sensitivity, and the real-time requirement is higher than the preset real-time requirement, the task processing device is determined to be an edge computing device.

[0116] In one embodiment, the device further includes a confidence determination module, used for: Determine the confidence level of on-site hazard information; Hazard identification information and handling suggestions are sent to the AR terminal, including: The confidence level, hazard identification information, and handling suggestions are sent to the AR terminal.

[0117] In one embodiment, determining the confidence level of on-site hazard information includes: Based on the on-site video stream, extract the image features of the identified object, obtain the standard object image features corresponding to the type of the identified object and the defective object image features corresponding to the category of the hidden danger stored in the knowledge base, and determine the visual similarity score based on the first similarity between the image features of the identified object and the standard object image features, and the second similarity between the image features of the identified object and the defective object image features. The contextual compliance score is determined based on the matching degree between historical hazard identification information and on-site hazard information. Extract key identification frame sequences from the on-site video stream, and determine the temporal consistency score based on the on-site hidden danger information corresponding to each frame in the key identification frame sequence; Acquire the on-site data stream uploaded by the AR terminal, and determine the environmental interference intensity score based on the on-site data stream; The data quality score is determined based on the video quality parameters of the live video stream. The confidence score is obtained by weighting and summing the visual similarity score, contextual compliance score, temporal consistency score, environmental interference intensity score, and data quality score.

[0118] In one embodiment, the device further includes a verification and identification module for: Determine whether the confidence level is lower than the preset confidence threshold; If the value is lower, the on-site video stream will be forwarded to the cloud management dashboard so that the cloud management dashboard can review and identify the on-site hazard information. If the value is not lower than the threshold, the action "send confidence level, hazard identification information and handling suggestions to the AR terminal" will be triggered.

[0119] The apparatus of this invention embodiment.

[0120] Figure 8 This is another structural schematic diagram of the power production site inspection and monitoring device provided in this embodiment of the invention, as shown below. Figure 8 As shown, the device includes: The acquisition module 501 is used to acquire on-site video streams in real time and upload the on-site video streams to edge computing devices; The receiving module 502 is used to receive hazard location information, hazard identification information, and handling suggestions returned by the edge computing device. The hazard location information, hazard identification information, and handling suggestions are obtained by the hazard identification intelligent agent deployed on the edge computing device using benchmarking procedures to enhance the hazard description information. The benchmarking procedures are obtained by the hazard identification intelligent agent by searching the standard procedures stored in the knowledge base according to the type of identification object, operating conditions, and hazard category. The type of identification object, operating conditions, hazard category, hazard description information, and hazard location information are obtained by the hazard identification intelligent agent by analyzing the on-site video stream. Display module 503 is used to overlay hazard identification information and handling suggestions as virtual information on an augmented reality screen based on on-site video stream, using augmented reality technology and based on hazard location information. The display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

[0121] In one embodiment, the display module 503 is specifically used for: The local coordinates of the identified object are determined based on the hazard location information and the preset 3D model of the identified object. The current pose of the object to be identified is determined based on the live video stream and the preset 3D model of the object. The current pose of the terminal is determined based on the on-site video stream, terminal posture data, and visual depth information. The target display position is determined based on the local coordinates of the identified object, the current pose of the identified object, and the current pose of the terminal. Hazard identification information and handling suggestions are displayed as virtual information overlaid on the target display position of the augmented reality screen, so that the display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream.

[0122] The device of this invention acquires and uploads real-time on-site video streams from augmented reality (AR) terminals, overcoming the limitations of traditional manual operation and achieving real-time, efficient on-site data acquisition to effectively avoid missed detections and errors. The on-site video stream is input into a hazard identification intelligent agent, which analyzes the video stream to obtain on-site hazard information. This information includes the type of identified object, operating conditions, hazard category, hazard description, and hazard location information. By retrieving standard procedures stored in the knowledge base based on the type of identified object, operating conditions, and hazard category, benchmarking procedures are determined. This enhances the accuracy of benchmarking procedures and provides high-quality reference information for subsequent knowledge enhancement. Using benchmarking procedures to enhance hazard description information yields hazard identification information and handling suggestions. This combines domain knowledge and standard procedures to improve the credibility and interpretability of on-site hazard identification information, providing compliant and actionable handling suggestions, thereby improving the troubleshooting efficiency of inspection personnel. Hazard location information, hazard identification information, and handling suggestions are sent to an AR terminal. The AR terminal uses augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream. The display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream, achieving precise mapping between virtual hazard locations and hazard locations in real equipment. This improves hazard identification efficiency, enabling on-site operators to promptly eliminate on-site hazards based on accurate and timely technical guidance. Furthermore, post-hazard elimination verification can be performed, forming a closed loop, thereby enhancing the intelligence level of power production sites and ensuring the safe and stable operation of power production.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0124] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device according to embodiments of the present invention. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0125] like Figure 9 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from storage section 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0126] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube, liquid crystal display, etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a modem, etc. Communication section 609 performs communication processing via a network such as the Internet. Drive 610 is also connected to I / O interface 605 as needed. Removable media 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 610 as needed so that computer programs read from them can be installed into storage section 608 as needed.

[0127] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0128] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, etc., or any suitable combination thereof.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The modules and / or units described in the embodiments of the present invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor may include an acquisition module, an identification module, and a transmission module; or a processor may include a collection module, a receiving module, and a display module. The names of these modules do not necessarily limit the module itself.

[0131] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: The system acquires real-time video streams from the augmented reality (AR) terminal; inputs these video streams into a hazard identification agent, which analyzes the streams to obtain hazard information. This information includes the type of the identified object, operating conditions, hazard category, hazard description, and hazard location. Based on the object type, operating conditions, and hazard category, the system retrieves standard procedures stored in the knowledge base to determine the benchmark procedure. The benchmark procedure is then used to augment the hazard description information, resulting in hazard identification information and handling suggestions. Finally, the system sends the hazard location information, hazard identification information, and handling suggestions to the AR terminal. The AR terminal then uses augmented reality technology to overlay these information as virtual information onto the augmented reality screen based on the video stream, ensuring that the virtual information's display position matches the corresponding real-world identified object or hazard location in the video stream.

[0132] Or the device may include: The system collects real-time on-site video streams and uploads them to edge computing devices. It receives hazard location information, hazard identification information, and handling suggestions from the edge computing devices. These hazard location information, hazard identification information, and handling suggestions are obtained by the hazard identification agent deployed on the edge computing devices, which uses benchmarking procedures to augment the hazard description information. The benchmarking procedures are obtained by the hazard identification agent by searching the knowledge base for standard procedures based on the type of the identified object, operating conditions, and hazard category. The type of identified object, operating conditions, hazard category, hazard description information, and hazard location information are obtained by the hazard identification agent through analysis of the on-site video stream. Using augmented reality technology, the hazard identification information and handling suggestions are overlaid as virtual information in the augmented reality image based on the on-site video stream, and the display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream.

[0133] The technical solution of this invention acquires and uploads real-time on-site video streams from augmented reality (AR) terminals, overcoming the limitations of traditional manual operation and achieving real-time, efficient on-site data acquisition to effectively avoid missed detections and errors. The on-site video stream is then input into a hazard identification intelligent agent, which analyzes the video stream to obtain on-site hazard information. This information includes the type of identified object, operating conditions, hazard category, hazard description, and hazard location information. By retrieving standard procedures stored in the knowledge base based on the type of identified object, operating conditions, and hazard category, benchmarking procedures are determined. This enhances the accuracy of benchmarking procedures and provides high-quality reference information for subsequent knowledge enhancement. Using benchmarking procedures to enhance hazard description information yields hazard identification information and handling suggestions. This combines domain knowledge and standard procedures to improve the credibility and interpretability of on-site hazard identification information, providing compliant and actionable handling suggestions, thereby improving the troubleshooting efficiency of inspection personnel. Hazard location information, hazard identification information, and handling suggestions are sent to an AR terminal. The AR terminal uses augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream. The display position of the virtual information matches the corresponding real identified object or hazard location in the on-site video stream, achieving precise mapping between virtual hazard locations and hazard locations in real equipment. This improves hazard identification efficiency, enabling on-site operators to promptly eliminate on-site hazards based on accurate and timely technical guidance. Furthermore, post-hazard elimination verification can be performed, forming a closed loop, thereby enhancing the intelligence level of power production sites and ensuring the safe and stable operation of power production.

[0134] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the power production site inspection and supervision method provided in any embodiment of this invention.

[0135] In the implementation of a computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for on-site inspection and supervision of power production, characterized in that, Applied to edge computing devices, the method includes: Acquire real-time video streams captured and uploaded by augmented reality (AR) terminals; The on-site video stream is input into a hazard identification intelligent agent, which analyzes the video stream to obtain on-site hazard information. This information includes the type of the identified object, operating conditions, hazard category, hazard description, and hazard location. Based on the identified object type, operating conditions, and hazard category, standard procedures stored in the knowledge base are retrieved to determine the benchmarking procedure. The benchmarking procedure is then used to enhance the hazard description information, resulting in hazard identification information and handling suggestions. The hazard location information, hazard identification information, and handling suggestions are sent to the AR terminal, so that the AR terminal can use augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information in the augmented reality image based on the on-site video stream, and the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

2. The method according to claim 1, characterized in that, After acquiring the live video stream collected and uploaded in real time by the AR terminal, the method further includes: The task processing equipment is determined based on the complexity, data sensitivity, and real-time requirements of the current inspection task. When the task processing device is a cloud management cockpit, the current inspection task is forwarded to the cloud management cockpit so that the cloud management cockpit can process the current inspection task; When the task processing device is an edge computing device, the execution of "inputting the on-site video stream into the hazard identification intelligent agent" is triggered; Specifically, if the task complexity is lower than a preset complexity, the data sensitivity is higher than a preset sensitivity, and the real-time requirement is higher than a preset real-time requirement, then the task processing device is determined to be the edge computing device.

3. The method according to claim 1, characterized in that, After analyzing the on-site video stream to obtain on-site hazard information, the process also includes: Determine the confidence level of the on-site hazard information; Sending the hazard identification information and handling suggestions to the AR terminal includes: The confidence level, the hazard identification information, and the handling suggestions are sent to the AR terminal.

4. The method according to claim 3, characterized in that, Determining the confidence level of the on-site hazard information includes: Based on the on-site video stream, extract the image features of the object to be identified, obtain the standard object image features corresponding to the type of the object to be identified and the defective object image features corresponding to the category of the hidden danger stored in the knowledge base, and determine the visual similarity score based on the first similarity between the image features of the object to be identified and the standard object image features, and the second similarity between the image features of the object to be identified and the defective object image features. The contextual compliance score is determined based on the matching degree between historical hazard identification information and the on-site hazard information. Extract key identification frame sequences from the on-site video stream, and determine the temporal consistency score based on the on-site hidden danger information corresponding to each frame in the key identification frame sequence; Acquire the on-site data stream uploaded by the AR terminal, and determine the environmental interference intensity score based on the on-site data stream; A data quality score is determined based on the video quality parameters of the live video stream. The confidence score is obtained by weighted summation of the visual similarity score, the contextual compliance score, the temporal consistency score, the environmental interference intensity score, and the data quality score.

5. The method according to claim 3, characterized in that, After determining the confidence level of the on-site hazard information, the process also includes: Determine whether the confidence level is lower than a preset confidence threshold; If the value is lower, the on-site video stream will be forwarded to the cloud management dashboard so that the cloud management dashboard can review and identify the on-site hazard information. If it is not lower than the threshold, then the action of "sending the confidence level, the hazard identification information and the handling suggestions to the AR terminal" will be triggered.

6. A method for on-site inspection and supervision of power production, characterized in that, Applied to AR terminals, the method includes: Real-time acquisition of on-site video streams, and uploading of the on-site video streams to edge computing devices; The system receives hazard location information, hazard identification information, and handling suggestions returned by the edge computing device. The hazard location information, hazard identification information, and handling suggestions are obtained by the hazard identification intelligent agent deployed on the edge computing device using benchmarking procedures to enhance the hazard description information. The benchmarking procedures are obtained by the hazard identification intelligent agent by retrieving standard procedures stored in the knowledge base based on the identified object type, operating conditions, and hazard category. The identified object type, operating conditions, hazard category, hazard description information, and hazard location information are obtained by the hazard identification intelligent agent through analysis of the on-site video stream. Using augmented reality technology, the hazard identification information and handling suggestions are superimposed on the augmented reality screen based on the on-site video stream in the form of virtual information according to the hazard location information, and the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

7. The method according to claim 6, characterized in that, The step of using augmented reality technology to overlay the hazard identification information and handling suggestions as virtual information onto the augmented reality screen based on the on-site video stream, based on the hazard location information, and having the display position of the virtual information match the corresponding real-world identified object or hazard location in the on-site video stream, includes: The local coordinates of the identified object are determined based on the hazard location information and the preset three-dimensional model of the identified object. The current pose of the object to be identified is determined based on the on-site video stream and the preset 3D model of the object to be identified. The current pose of the terminal is determined based on the on-site video stream, terminal posture data, and visual depth information. The target display position is determined based on the local coordinates of the identified object, the current pose of the identified object, and the current pose of the terminal; The hazard identification information and handling suggestions are displayed as virtual information overlaid at the target display position of the augmented reality screen, so that the display position of the virtual information matches the corresponding real identification object or hazard location in the on-site video stream.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the power production site inspection and supervision method as described in any one of claims 1 to 5, or the power production site inspection and supervision method as described in claim 6 or 7.

9. An inspection and supervision system, characterized in that, It includes at least one edge computing device as described in claims 1 to 5 and at least one AR terminal as described in claim 6 or 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the inspection and supervision method for power production sites as described in any one of claims 1 to 5, or the inspection and supervision method for power production sites as described in claim 6 or 7.

Images