Intelligent inspection method and device for energy equipment based on multi-mode perception

Abstract

The invention relates to the technical field of multi-modal sensing, and discloses an intelligent inspection method and device for energy equipment based on multi-modal sensing, and the method comprises the following steps: firstly, obtaining the sensing data of the working state information of the energy equipment through a plurality of sensors; organizing the obtained perception data into a perception graph; processing the perception graph by using a graph neural network, and extracting a nonlinear relationship between nodes; generating an optimal control strategy by combining an optimal control algorithm and a game model according to the state estimation result; through the optimal control strategy, the robot is controlled to execute an inspection task; and finally, verifying the stability of the system according to the Lyapunov stability theory. According to the invention, through integration and processing of the multi-modal sensing data and in combination with modeling and reasoning of the sensing graph by the graph neural network, an efficient and intelligent inspection process is obtained, and equipment faults can be found in time and potential equipment problems can be predicted, so that the inspection accuracy is improved and human intervention is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal perception, and in particular to an intelligent inspection method and device for energy equipment based on multi-modal perception. BACKGROUND

[0002] With the increasing complexity of modern energy equipment, traditional manual inspection methods have been unable to meet the requirements of efficiency, accuracy, and timeliness. In the daily operation of energy equipment, equipment failures often need to be identified and addressed as soon as possible to avoid equipment downtime, improve safety, and extend the service life of the equipment. However, manual inspection often fails to discover potential problems in a timely manner due to limitations in human resources, experience, and environmental complexity, resulting in equipment failures that have entered an irreparable stage. Therefore, there is an urgent need for a more efficient and intelligent inspection method to improve the accuracy and efficiency of equipment management.

[0003] Multi-modal perception technology, as an intelligent perception method that integrates multiple sensor data, can obtain real-time information on different working states of equipment (such as vision, temperature, vibration, sound, etc.), providing a basis for accurate state assessment and fault diagnosis. This technology combines multiple data collected by sensors to consider multi-dimensional information of equipment, thereby improving comprehensive understanding and accurate judgment of equipment state. SUMMARY

[0004] To address the above shortcomings, the present application provides an intelligent inspection method and device for energy equipment based on multi-modal perception, aiming to improve the traditional manual inspection which relies on human experience and visual judgment, but due to the complexity of the working environment of the equipment, the variety of equipment, and the fact that some faults are not easily detected in the early stage, manual inspection has great limitations in detection accuracy and efficiency, resulting in the problem that the reliability and timeliness of manual inspection often cannot meet the requirements of modern production and management in complex environments or with a large number of equipment.

[0005] In a first aspect, the present application provides an intelligent inspection method for energy equipment based on multi-modal perception, comprising the following steps: First, obtaining working state information perception data of energy equipment through multiple sensors; Organizing the obtained perception data into a perception graph, including nodes of different perception data and their relationships; Then, processing the perception graph using a graph neural network to extract the non-linear relationship between nodes, estimating the posterior distribution of the system state based on a variational Bayesian inference method, and obtaining a state estimation result; According to the state estimation result, combining an optimal control algorithm and a game model to generate an optimal control strategy; By the optimal control strategy, the robot inspection path is planned, and the robot is controlled to perform the inspection task; In the inspection task, the robot execution is monitored in real time, and the control strategy is adjusted through new perception data and state estimation results; Finally, according to Lyapunov stability theory, the stability of the system is verified to determine the running state of the system.

[0006] Preferably, the working state information includes vision, temperature, vibration, sound and current, and the perception data acquisition includes collecting image data through a camera, acquiring temperature data through an infrared sensor, and monitoring the vibration of the device through a vibration sensor.

[0007] Preferably, the perception graph includes visual, temperature, vibration multi-modal data nodes, and the relationship between nodes is learned through a graph neural network model to extract high-dimensional graph embedding representation.

[0008] Preferably, the state estimation and reasoning step includes estimating the posterior distribution of the system state through a variational Bayesian inference algorithm, which includes the position, velocity of the robot and the working state of the device.

[0009] Preferably, the control strategy generation step includes generating an optimal inspection path and action plan based on an optimal control objective function, considering energy consumption, inspection time and task accuracy.

[0010] Preferably, the path planning and execution step includes training the optimal control strategy through a reinforcement learning algorithm to adjust the robot inspection path in real time.

[0011] In the second aspect, the present application provides the following technical solutions: an intelligent inspection device for energy equipment based on multi-modal perception, which includes the following modules: The state modeling module is used to acquire and process the state information of the robot and the device, and to build a system state model; The perception acquisition module is used to acquire visual, sound and temperature data through multi-modal sensors and to build a perception graph; The state estimation module is used to process the perception graph based on a graph neural network and to estimate the posterior distribution of the system state; The control optimization module is used to generate an optimal control strategy based on the estimated system state and to optimize control decisions through a game model; The path planning module is used to solve the optimal path and control the robot to perform the inspection task; The feedback adjustment module is used to monitor and feedback the control strategy in real time and to dynamically adjust the inspection path; The stability analysis module is used to verify the system stability through Lyapunov method and to record the inspection results.

[0012] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above-mentioned intelligent inspection method and device for multi-modal perception energy equipment based on the computer program.

[0013] In a fourth aspect, the present application provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned intelligent inspection method and device for multi-modal perception energy equipment.

[0014] The present application has the following advantages: 1. In the present application, through the integration and processing of multi-modal perception data, combined with the modeling and reasoning of the perception graph by the graph neural network, accurate system state estimation and fault identification are realized, an efficient and intelligent inspection process is obtained, equipment failures can be found in time and potential equipment problems can be predicted, thereby improving the inspection accuracy and reducing human intervention.

[0015] 2. In the present application, by introducing an optimal control strategy generation module and combining a game model and a reinforcement learning method, dynamic inspection path planning and control strategy optimization are realized, an adaptively adjusted inspection path is obtained, which can cope with complex and uncertain environmental changes and ensure the optimal execution effect of the inspection task.

[0016] 3. In the present application, through the design of a real-time feedback adjustment module, combined with the monitoring of real-time perception data and robot state, a closed-loop feedback mechanism is realized, a flexible and stable control system is obtained, which can dynamically adjust the strategy according to real-time data during the inspection process to ensure the continuous and stable operation of the system. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A method flowchart of the intelligent inspection method for multi-modal perception energy equipment proposed by the present application; Figure 2 A source equipment intelligent inspection closed-loop control flowchart of the intelligent inspection method for multi-modal perception energy equipment proposed by the present application; Figure 3 A sensor data parallel collection and fusion flowchart of the intelligent inspection method for multi-modal perception energy equipment proposed by the present application; Figure 4 A perception graph construction and graph embedding learning flowchart of the intelligent inspection method for multi-modal perception energy equipment proposed by the present application; Figure 5 A variational Bayesian state estimation and posterior reasoning flowchart of the intelligent inspection method for multi-modal perception energy equipment proposed by the present application; Figure 6 A multi-objective game optimization control strategy generation flowchart for the intelligent inspection method for the multi-modal perception energy equipment according to the present application is provided. Figure 7 A reinforcement learning driven dynamic path planning flowchart for the intelligent inspection method for the multi-modal perception energy equipment according to the present application is provided. Figure 8 A random Lyapunov stability verification and policy iteration flowchart for the intelligent inspection method for the multi-modal perception energy equipment according to the present application is provided. Figure 9 An energy equipment inspection system multi-module collaborative interaction diagram for the intelligent inspection device for the multi-modal perception energy equipment according to the present application is provided. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] Embodiment One Reference Figures 1-8 In the first embodiment of the present application, the intelligent inspection method for the multi-modal perception energy equipment is provided, which includes the following steps: First, the working state information perception data of the energy equipment is obtained through multiple sensors; The obtained perception data is organized into a perception graph, including nodes of different perception data and their relationships; Then, the perception graph is processed by using a graph neural network to extract the nonlinear relationship between the nodes, estimate the posterior distribution of the system state based on the variational Bayesian inference method, and obtain the state estimation result; According to the state estimation result, the optimal control algorithm and the game model are combined to generate an optimal control strategy; Through the optimal control strategy, the robot inspection path is planned, and the robot is controlled to perform the inspection task; In the inspection task, the robot execution is monitored in real time, and the control strategy is adjusted through new perception data and state estimation results; Finally, according to the Lyapunov stability theory, the stability of the system is verified to determine the running state of the system.

[0020] The working state information includes vision, temperature, vibration, sound, and current. The perception data acquisition includes collecting image data through a camera, obtaining temperature data through an infrared sensor, and monitoring the vibration of the device through a vibration sensor.

[0021] Specifically, image data is collected through a camera, temperature data is obtained through an infrared sensor, and the vibration of the device is monitored through a vibration sensor. A perception graph is established by combining these perception data, and deep learning inference is performed using methods such as graph neural networks (GNN) to accurately assess the device state. The working state information of the device is obtained through multiple sensors. These information includes visual information, temperature data, vibration data, sound data, and current data. The data collected by each sensor is considered as independent modal information, and after fusion, a complete perception graph is formed; The camera is used to collect image data of the device. The image data includes appearance information of the device surface, such as cracks, wear, corrosion, and other signs of failure. The image data captured by the camera is usually a two-dimensional matrix, and the pixel points reflect the details of the device surface. The processing of image data involves image preprocessing, feature extraction (such as edge detection, object recognition, etc.), and analysis of the device state through machine vision algorithms to identify potential failures. In actual use, when the robot patrols, if the camera captures cracks or obvious wear on the device surface, the image recognition algorithm can mark the abnormal area and further trigger the corresponding failure warning; The infrared sensor is used to monitor the temperature change of the device. Overheating or temperature anomalies of the device often indicate device failure or improper operation, especially for devices such as motors and transformers. High temperature may cause serious damage or failure of the device. The infrared sensor captures the infrared radiation of the device surface and converts it into temperature data. By analyzing these temperature data, it can be determined whether the device is within the safe working temperature range. The infrared sensor usually provides accurate temperature measurement data, which can be used to construct a temperature curve of the device operating state to monitor any fluctuations outside the preset range. For example, if the temperature is too high, it may indicate that the device is overloaded or has heat dissipation problems, and the system will issue a warning; The vibration sensor is used to monitor the vibration of the device, especially the vibration of mechanical devices. These devices will produce specific vibration frequencies during operation. If the vibration frequency is abnormal, it often means that the device has mechanical faults such as bearing damage, gear wear, etc. The vibration sensor converts mechanical vibration into electrical signals, and by analyzing signal characteristics such as frequency spectrum and amplitude, it can identify abnormal vibration patterns of the device. Vibration data is processed through frequency spectrum analysis. Abnormal frequency and amplitude values will trigger device maintenance or shutdown warnings to prevent further damage to the device; Current sensors are primarily used to monitor changes in the current of equipment. Fluctuations in current often reflect the load condition of the equipment or the risk of electrical faults. For example, in motor equipment, excessive load may cause a sharp increase in current, while motor overload or short circuit will cause abnormal current changes. Current sensors monitor the current in real time and generate corresponding data. By comparing this data with the equipment's preset operating state, the operating status of the equipment can be determined. Current data acquisition is achieved through sensors, and the data is filtered and processed using certain algorithms to ensure the accuracy and real-time nature of the data.

[0022] The perception graph includes multimodal data nodes such as vision, temperature, and vibration. The relationships between the nodes are learned through a graph neural network model to extract a high-dimensional graph embedding representation.

[0023] Specifically, the system learns the relationships between nodes using a Graph Neural Network (GNN) model, extracts a high-dimensional graph embedding representation of the equipment status, and constructs a multimodal perception graph. By integrating various sensing data, the accuracy and intelligence of equipment inspection are improved. The application of the GNN enables efficient correlation and fusion between various sensing data, helping to achieve in-depth analysis of equipment operating status and fault prediction. The system acquires data from different modalities through multiple sensors, including visual, temperature, and vibration data. During processing, this data is transformed into nodes in the perception graph, and the relationships between nodes are modeled and learned using the GNN. Through deep learning, the system can extract a high-dimensional embedding representation of the equipment status and further analyze and infer the health status and potential faults of the equipment. The nodes in the perception graph represent multimodal data from different sensors, and the edges between nodes reflect the relationships between these data. Through the GNN, the system models the relationships between nodes and gradually extracts the feature representations of each node through multiple rounds of graph convolution operations, ultimately obtaining a high-dimensional embedding representation. This embedding representation integrates the information of each node and their interrelationships, providing rich contextual information for equipment status assessment and fault detection. The system collects and processes sensor data. Images of the equipment surface are captured by a camera, temperature data is obtained using an infrared sensor, and vibration sensors monitor the equipment's vibration status. The data collected by these sensors is transformed into different nodes in a perception graph. The application of graph neural networks (GNNs) offers significant advantages for integrating multimodal data. First, it learns the complex relationships between different sensor data through message passing mechanisms between nodes. In a graph convolution operation, node information is updated through interactions with its neighboring nodes and iteratively optimized, ultimately generating a high-dimensional embedding representation for each node. These high-dimensional embedding representations can more accurately capture the details of the equipment's operating status, aiding in equipment health assessment and fault prediction. The output of the GNN is the embedding vector for each node, which can be used for further equipment status prediction and fault diagnosis. By analyzing these embedding representations, the system can determine whether the equipment is in normal operating condition and predict whether there is a risk of failure. When the embedding representation of a node is similar to historical fault data, the system can issue a warning, indicating a potential equipment failure. For example, when the embedding representation of a vibration node is similar to historical overload or wear data, the system will issue an alarm.

[0024] The state estimation and inference steps include estimating the posterior distribution of the system state using a variational Bayesian inference algorithm. This posterior distribution includes the robot's position, velocity, and the device's operating state.

[0025] Specifically, the state estimation and inference step introduces a variational Bayesian inference algorithm to achieve accurate inference of the system state, including the estimation of the robot's position and velocity, as well as the operating state of the equipment. This step uses the variational Bayesian inference method to estimate the posterior distribution of the system, and further optimizes and adjusts robot navigation, equipment state assessment, and fault prediction based on this posterior distribution. First, the core objective of state estimation and inference is to accurately infer the robot's position and velocity during the inspection process, as well as the current operating state of the equipment (such as temperature and vibration). To achieve this objective, this embodiment employs the variational Bayesian inference algorithm. This algorithm estimates the optimal state of the system by estimating the posterior distribution. Variational Bayesian inference is a probabilistic inference method based on Bayes' theorem, used to infer latent variables or potential states from observed data. In this invention, variational Bayesian inference is used to estimate the robot's position, velocity, and the operating state of the equipment in the system. Its core idea is to approximate the posterior distribution by maximizing the lower bound of evidence (ELBO), thereby inferring the true state of the system, given some observed data. and the model's latent variables Variational inference seeks approximate distributions. To approximate the true posterior distribution , such that the KL divergence between the two is minimized, the goal of variational inference is to find an approximate distribution by optimizing the following objective function:

[0026] where: is the joint probability distribution, representing the joint distribution of the robot state and the equipment state; is the approximate variational distribution, used to approximate the true posterior distribution ; represents the expectation over the approximate distribution, through variational Bayesian inference, the system can efficiently estimate the robot's position, velocity, and equipment's working state from the observed data. These inference results can be further used to dynamically adjust the robot's inspection path, assess the equipment's health status, and predict potential equipment failures; During the inspection process, the robot needs to know its position and velocity in real-time in order to navigate and inspect efficiently. Through variational Bayesian inference, the system can estimate the posterior distribution of the robot's position and velocity. Assuming that the robot's position at time is , and the velocity is , and there is some noise or uncertainty between the robot's state at each time and the observed data, the goal of variational inference is to infer the posterior distribution of and from the observed data Under the framework of variational inference, the robot's position and velocity are modeled as latent variables, while the observed data (such as sensor inputs, camera images, etc.) are treated as observations; In addition to the robot's position and velocity, variational Bayesian inference can also be used to estimate the equipment's working state. These states include the equipment's temperature, vibration, failure mode, etc., which usually change with the equipment's working environment. By introducing the variational Bayesian inference algorithm, the system can estimate the posterior distribution of the equipment's state from real-time sensor data, such as temperature sensors, vibration sensors, etc. For example, assume that the equipment's temperature is , the vibration data is , and the true working state of the equipment is inferred from the observed data . Variational inference calculates the posterior distribution of the equipment's state by maximizing the lower bound of the evidence. This posterior distribution can help the system determine whether the equipment is running normally or whether there is a potential risk of failure.

[0027] The control strategy generation step includes generating the optimal inspection path and action plan based on the optimal control objective function, taking into account energy consumption, inspection time, and task accuracy.

[0028] Specifically, based on the above objective function, the system first constructs a patrol path planning model, considering each site of the patrol task as a node in the path, and the robot or inspection device moves between sites through the optimal path. The selection of the path is not only subject to time and energy consumption constraints, but also needs to consider factors such as the working state of the device and the risk of failure. The construction of the optimal control objective function, the objective function is defined as the weighted sum of energy consumption, time and accuracy, represented as: ; Where: represents the energy consumption on the path ; represents the inspection time on the path . represents the task accuracy on the path . is the weight coefficient, used to balance the importance of different objectives. In this model, the system needs to find the optimal path by optimizing the objective function, that is, to complete the inspection task in the minimum energy consumption and time, while ensuring the high accuracy of the task. The robot needs to consider energy consumption during the inspection process, especially for long-running inspection tasks. The calculation of energy consumption is based on the distance and moving speed of the robot on the path. According to the dynamics model of the robot, energy consumption is closely related to speed, path length and load. By selecting the appropriate inspection path, the system can reduce unnecessary path detours and reduce energy consumption; inspection time is the time required for the robot to complete the entire inspection task. Time optimization needs to consider factors such as the distribution of the inspection area, the distance between sites, the state of the device and the complexity of the path. The optimal path planning is not just the shortest path, the system also needs to consider the health status of the device to ensure that each important device is given sufficient inspection time; task accuracy is the completion degree of the robot inspection task, ensuring that each important part of the equipment is checked and the inspection result is accurate. The selection of the inspection path needs to ensure that each part of the equipment can be comprehensively detected. While considering time and energy consumption, components with higher accuracy priority should be checked first. The optimal path planning algorithm can use classical optimization methods such as dynamic programming, genetic algorithm, ant colony algorithm, etc. Through these methods, the system can consider the above goals in the path planning process and optimize according to the preset objective function. According to the planning of the optimal path, the system generates a plan for the robot or inspection equipment. The plan includes the detailed inspection task sequence, the robot's travel path, the sequence of each equipment inspection, and the depth of the inspection. The core of the control strategy is to ensure that the robot executes the task according to the optimal path, while dynamically adjusting according to the real-time changes of the equipment state.

[0029] The path planning and execution step includes training the optimal control strategy through a reinforcement learning algorithm to adjust the robot inspection path in real time.

[0030] Specifically, in the path planning and execution phase, the system needs to generate and optimize the inspection path through a reinforcement learning algorithm to ensure that the robot can adaptively adjust the path in a real-time changing environment. Through reinforcement learning, the system can continuously obtain feedback information from the environment and adjust the inspection strategy in real time to cope with changes in equipment state and environmental uncertainty. Reinforcement learning is an algorithm that learns the optimal strategy through trial and error. In reinforcement learning, the agent (in this invention, the robot) interacts with the environment, selects different actions, and adjusts its strategy according to the rewards or punishments obtained. First, in path planning, the state is usually the current position of the robot, the working state of the equipment (such as temperature, vibration, etc.), the progress of the inspection task, etc. Each state can be described by perception data (such as vision, temperature, vibration, etc.). The action is the action the robot can take in each state. In the inspection task, the action can be the movement of the robot, adjusting the inspection path, selecting the equipment to be inspected, etc. The reward is the feedback to the robot's behavior, usually calculated based on task completion, inspection time, energy consumption, and task accuracy. For example, if the robot chooses a shorter path that can reduce energy consumption and time, it will get a higher reward; if the robot chooses an inappropriate path that leads to excessive energy consumption or missed equipment, it will be punished. The strategy is the rule or model that the robot uses to select actions in different states. The goal of reinforcement learning is to gradually optimize the strategy through interaction with the environment, so that the robot can take the optimal action in different states, thereby achieving optimal execution of the inspection task. In the path planning process, the robot will continuously optimize its inspection path through reinforcement learning algorithm. Specifically, the system models each inspection task and path node as a state space in reinforcement learning, and the robot learns how to choose the optimal inspection path by exploring different paths and task orders, including the robot's position, device status, inspection task progress, and other multi-dimensional data. For example, the coordinates of the robot's current position, the type of equipment currently being inspected, the working status of the equipment (such as temperature, vibration, etc.), and other factors are considered as components of the state space. The action space includes all actions that the robot can choose, such as moving to the next inspection site, selecting a device for inspection, adjusting the inspection path, etc. The robot performs the inspection task by selecting different actions, and the design of the reward function may include the following aspects: energy efficiency reward: if the robot chooses a path that reduces energy consumption, it will be rewarded, time reward: if the robot can complete the inspection task in the shortest time, it will be rewarded, accuracy reward: if the robot accurately inspects all devices and generates high-quality fault detection results, it will be rewarded; To achieve optimal path planning, the system first optimizes the robot's strategy through extensive simulation training. The initial device state, the initial position of the robot, and the distribution of inspection tasks in the simulation environment are initialized, and the inspection area and device state are set. The robot starts from the initial state and selects actions according to the current strategy to perform the inspection task. After each action is executed, the system rewards or punishes the robot based on its performance (such as energy consumption, time, task accuracy, etc.). Reinforcement learning continuously optimizes the robot's strategy through multiple rounds of training and feedback, gradually improving the efficiency and accuracy of path planning and task execution.

[0031] The system stability verification is performed by calculating the decay rate of the value function, using the stochastic Lyapunov method to verify the convergence of the system, and adjusting the inspection strategy according to the stability requirements. The system's convergence is verified, and the system's stability is evaluated by calculating the decay rate of the value function.

[0032] Specifically, in reinforcement learning and optimal control problems, the value function is used to evaluate the long-term benefits of executing a certain action in a given state. The value function not only helps to evaluate the pros and cons of the strategy, but also can be used to analyze the convergence of the system. The stability of the system is usually judged by observing the decay rate of the value function, which reflects the rate at which the error gradually decreases from the initial state to the optimal strategy execution process. For a stable system, the decay rate of the value function should be asymptotic, meaning that the state of the system gradually approaches the target state over time without large fluctuations. Specifically, the verification of system stability can be performed by analyzing the following formula: ; wherein: represents the state of the optimal value function.

[0033] is the immediate reward obtained by the robot for taking action at time is the discount factor, which determines the weight of future rewards.

[0034] is the state of the robot at time During the operation of the system, as the robot performs actions during the inspection process, the state of the system should gradually converge to the optimal policy. If the decay rate of the value function is too slow, it means that the convergence of the system is poor, and the policy needs to be adjusted; In a stochastic system, the application of the Lyapunov method is more complex because the change of the system state is affected by random factors. The stochastic Lyapunov method helps to evaluate the stability of the system by introducing a noise term to construct a Lyapunov function suitable for a stochastic environment. Given the state transition probability and reward function of the system, a Lyapunov function of the following form can be constructed : ; wherein, is the target equilibrium point, representing the optimal state of the robot. By calculating the time decay rate of the Lyapunov function, it can be determined whether the system will stably converge. The stochastic form of the Lyapunov equation can be expressed as: ; wherein: represents the derivative of the Lyapunov function with respect to time. is the decay coefficient, representing the speed of convergence of the system. is the Lyapunov function, describing the change of the system state. When , the system is stable, indicating that the robot will converge to the target state; when , the system is unstable, which may cause the robot to fail to reach the target position.

[0035] When verifying the stability of the system, by analyzing the decay rate of the value function and the Lyapunov function, the system can evaluate its convergence in real time. If the convergence of the system does not meet the requirements, by using the decay rate of the value function and the stochastic Lyapunov method, the system can effectively verify its stability, ensuring that the robot is always in a convergent state during the inspection process.

[0036] Embodiment Two: Referring to Figure 9In the second embodiment of the present application, the present application provides an intelligent inspection device for a multi-modal perception energy equipment, which comprises the following modules: a state modeling module for acquiring and processing state information of the robot and the equipment, and constructing a system state model; a perception acquisition module for acquiring visual, sound, and temperature data through multi-modal sensors, and constructing a perception graph; a state estimation module for processing the perception graph based on a graph neural network, and estimating a posterior distribution of the system state; a control optimization module for generating an optimal control strategy based on the estimated system state, and optimizing control decisions through a game model; a path planning module for solving an optimal path and controlling the robot to perform an inspection task; a feedback adjustment module for monitoring and feeding back the control strategy in real time, and dynamically adjusting the inspection path; a stability analysis module for verifying system stability through a Lyapunov method and recording inspection results.

[0037] Specifically, the state modeling module is used to acquire and process state information of the robot and the equipment, and then construct a state model of the system. The main task of this module is to acquire the working state of the robot and the equipment in real time, including the position, speed, and running state of the robot, and the temperature and vibration of the equipment. By processing these data, the state modeling module can construct a comprehensive system state model, which is used for subsequent state estimation, path planning, and control decision-making. This module provides basic data for subsequent perception, control, and path planning, so that the system can dynamically adjust according to the real device and robot state; The perception acquisition module is responsible for acquiring data including vision, sound, and temperature through multi-modal sensors, and constructing a perception graph. Through comprehensive perception of the environment by sensors, the construction of the perception graph can provide necessary information for subsequent state estimation. These multi-modal data can provide strong support for device fault detection and environmental changes. The data collected by these sensors are converted into nodes, and the perception graph is constructed in the form of a graph, reflecting the relationship between various types of perception data. The perception acquisition module provides multi-modal data input for subsequent state estimation and control decision-making, supporting more accurate fault detection and system monitoring; The state estimation module processes the perception graph based on a graph neural network, and estimates the posterior distribution of the system state. Through the graph neural network (GNN), the module can efficiently fuse data from multiple sensors and infer the current state of the system, including the robot position and device health state. This module fuses multi-modal data through deep learning technology, thereby providing support for accurate evaluation of the system state, ensuring that subsequent decision-making and path planning are based on real and accurate system state; The control optimization module generates an optimal control strategy based on the estimated system state and optimizes the control decision through a game model. The goal of this module is to make the best control decision based on the real-time state of the system, ensuring that the robot can efficiently complete the inspection task. This module optimizes the control strategy so that the robot can balance multiple goals (such as time, energy efficiency, and accuracy) during the inspection process, ensuring that the task is completed efficiently and accurately. The path planning module is used to solve the optimal path and guide the robot to perform the inspection task according to the control strategy generated by the control optimization module. The core task of this module is to calculate and adjust the inspection path in real time according to the task requirements and device state. The path planning module ensures that the robot can efficiently and accurately perform the inspection task, while adjusting the path according to the task progress and environmental changes to avoid unnecessary energy consumption and time waste. The feedback adjustment module monitors the state of the robot and the device in real time, collects sensor data, and dynamically adjusts the inspection path based on real-time feedback. This module can adjust the control strategy according to the state changes of the device and the progress of the robot inspection. This module ensures that the system can quickly respond to environmental changes and adjust the inspection path and control strategy in real time, improving the adaptability and flexibility of the system. The stability analysis module verifies the stability of the system through the Lyapunov method and records the inspection results. The purpose of this module is to ensure that the robot is always in a stable state during the inspection process and can continuously perform the task until it is completed. This module ensures that the system can operate stably when facing dynamic environments and uncertain factors, avoiding abnormal behavior or loss of control, and ensuring the smooth completion of the task.

[0038] Embodiment Three The third embodiment of the present application is based on the same inventive concept. The present application proposes a computer readable storage medium storing a computer program, which is executed by a processor to implement the steps of the above-mentioned embodiment of the intelligent inspection method and device for multi-modal perception energy equipment.

[0039] Embodiment Four The fourth embodiment of the present application is based on the same inventive concept. The present application proposes a computer device, which includes a processor, a memory, and a communication between the processor and the memory. The memory is used to store instructions, and the processor is used to execute the instructions in the memory to implement the intelligent inspection method and device for multi-modal perception energy equipment as described in the above embodiments.

[0040] It should be understood that portions of the present application can be implemented with hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, implementation can be with any or a combination of the following technologies, which are all well-known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0041] Finally, it should be noted that the above-mentioned only is the preferred embodiment of the present application, and is not used to limit the present application, although the present application is described in detail with reference to the foregoing embodiments, for those skilled in the art, it still can be modified to the technical scheme recorded in the foregoing embodiments, or equivalent replacement of part of the technical features, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. A method for intelligent inspection of multi-modal perception energy devices, characterized in that, The method comprises the following steps: First, acquire perception data of working state information of the energy equipment through multiple sensors; Organize the acquired perception data into a perception graph, including nodes of different perception data and their relationships; Then, process the perception graph using a graph neural network, extract the nonlinear relationships between nodes, estimate the posterior distribution of the system state based on variational Bayesian inference method, and obtain the state estimation result; Generate an optimal control strategy based on the state estimation result, combined with an optimal control algorithm and a game model; Plan a robot inspection path and control the robot to perform an inspection task through the optimal control strategy; In the inspection task, monitor the robot execution in real time, and adjust the control strategy through new perception data and state estimation results; Finally, verify the stability of the system according to Lyapunov stability theory to determine the running state of the system.

2. The method for intelligent inspection of multi-modal perception based energy devices as claimed in claim 1, wherein, The working state information includes vision, temperature, vibration, sound and current, and the perception data acquisition includes collecting image data through a camera, acquiring temperature data through an infrared sensor, and monitoring the vibration of the equipment through a vibration sensor. 3.The intelligent inspection method based on multi-modal perception for energy equipment according to claim 1, wherein, The perception graph includes visual, temperature, vibration multi-modal data nodes, and the relationships between nodes are learned through a graph neural network model to extract high-dimensional graph embedding representation.

4. The method for intelligent inspection of multi-modal perception based energy devices as claimed in claim 1, wherein, The state estimation and reasoning step includes estimating the posterior distribution of the system state through a variational Bayesian inference algorithm, which contains the position, speed of the robot and the working state of the equipment. 5.The intelligent inspection method based on multi-modal perception for energy equipment according to claim 1, wherein, The control strategy generation step includes generating an optimal inspection path and action plan based on an optimal control objective function, considering energy consumption, inspection time and task accuracy. 6.The intelligent inspection method based on multi-modal perception for energy equipment according to claim 1, wherein, The path planning and execution step includes training the optimal control strategy through a reinforcement learning algorithm to adjust the robot inspection path in real time.

7. The method for intelligent inspection of multi-modal perception based energy devices as claimed in claim 1 wherein, The system stability verification is verified by calculating the decay rate of the value function, using a stochastic Lyapunov method to verify the convergence of the system, and adjusting the inspection strategy according to the stability requirement to verify the convergence of the system, and evaluating the stability of the system by calculating the decay rate of the value function.

8. An intelligent inspection device for multi-modal perception based energy equipment, characterized in that, The device for the intelligent inspection method of the multi-modal perception energy equipment according to any one of claims 1-7 comprises the following modules: A state modeling module for acquiring and processing state information of the robot and the equipment, and constructing a system state model; A perception acquisition module for acquiring visual, sound and temperature data through multi-modal sensors and constructing a perception graph; A state estimation module for processing the perception graph based on a graph neural network and estimating the posterior distribution of the system state; A control optimization module for generating an optimal control strategy based on the estimated system state and optimizing control decisions through a game model; A path planning module for solving an optimal path and controlling the robot to perform an inspection task; A feedback adjustment module for real-time monitoring and feedback control strategy and dynamically adjusting the inspection path; A stability analysis module for verifying the stability of the system through a Lyapunov method and recording the inspection results.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the intelligent inspection method for the multi-modal perception energy equipment based on the smart inspection method according to any one of claims 1-7 when the computer program is executed.

10. A readable storage medium, characterized by, The readable storage medium stores the computer program, and the computer program is executed by the processor to implement the intelligent inspection method for the multi-modal perception energy equipment based on the smart inspection method according to any one of claims 1-7.

Images