Multi-mode full-autonomous inspection method and system for electric unmanned aerial vehicle

Abstract

The invention belongs to the field of electric power unmanned aerial vehicle inspection, and provides an electric power unmanned aerial vehicle multi-mode full-autonomous inspection method and system in order to solve the problem that traditional electric power unmanned aerial vehicle inspection is not suitable for an unknown environment. The multi-mode full-autonomous inspection method for the electric unmanned aerial vehicle comprises the following steps: cooperatively processing an electric inspection task instruction in a regional grid range by using multiple agents; guiding the unmanned aerial vehicle to autonomously inspect an unknown environment; performing cross-modal calibration on the multi-modal inspection data of the interested target, weighting the calibrated spatio-temporal joint features of each modal in combination with each modal weight matched with the dynamic environment to obtain fusion features, and evaluating the health condition of the interested target by using a multi-modal large model; and updating a preset knowledge base, and obtaining an interest target active prevention decision jointly driven by multi-modal fusion and association degree in combination with the health condition evaluation result of the interest target. The unmanned aerial vehicle can perform autonomous inspection in an unknown environment, and a corresponding decision is actively provided for stable operation of an inspection target.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of electric power unmanned aerial vehicle inspection, and particularly relates to an electric power unmanned aerial vehicle multi-modal full autonomous inspection method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the expansion of the scale of power grid equipment year by year and the increasing complexity of the operating environment, huge economic losses are caused by power outage accidents every year, and it is urgent to accurately perceive the operating state of power grid equipment and actively provide risk warning. Unmanned aerial vehicle inspection is an indispensable technical means for fine operation and maintenance of power grid equipment. In the traditional unmanned aerial vehicle inspection mode, the unmanned aerial vehicle is carried to the scene by manual operation for inspection, and the data is collected, which is labor-intensive and the operation quality is affected by the experience of personnel, and is uneven. The contradiction between fine inspection of power grid equipment and the traditional unmanned aerial vehicle inspection mode is becoming more and more prominent.

[0004] The traditional unmanned aerial vehicle inspection mode needs to manually re-map and plan the route every time the environment changes. In the inspection process, the static obstacles are generally stopped, and the processing capacity for dynamic obstacles encountered in the flight process is insufficient. Over-reliance on a single main sensor such as vision or laser radar can only inspect specific areas or specific targets, and is easily invalid in vision degradation or laser radar degradation scenarios. The flight process relies on the route and signal, and when the positioning signal in the flight process is poor, it will affect the flight safety, cannot fly in an environment without prior information or can rely on external positioning signals, and is not suitable for power inspection in unknown environments.

[0005] In addition, the traditional unmanned aerial vehicle inspection mode has poor data intelligent recognition effect and needs a large amount of manual checking. There is a lack of grid-based multi-specialty collaborative scheduling means, and the resource utilization rate is low. The detection scheme based on visible light, infrared and other multi-modal data fusion in the prior art usually uses multi-modal fusion methods such as simple splicing and fixed weight summation, and combines target detection technology to identify power target faults. Due to the influence of various factors on the reliability of multi-modal data, the fixed weight fusion method will introduce invalid information to limit the feature quality. Moreover, only image and text are fused, and device operating parameters and environmental data are not integrated, resulting in one-sided information and insufficient use of multi-modal data, which ultimately affects the accuracy of the inspection recognition result and cannot actively provide corresponding decisions for the stable operation of the inspection target. SUMMARY

[0006] In order to solve the above technical problems, the present application provides an electric power unmanned aerial vehicle multi-modal full autonomous inspection method and system, which can enable the unmanned aerial vehicle to perform autonomous inspection in an unknown environment and actively provide corresponding decisions for the stable operation of the inspection target.

[0007] To achieve the above object, the present application adopts the following technical solutions: The first aspect of the present application provides a power unmanned aerial vehicle multi-modal full autonomous inspection method.

[0008] In one or more embodiments, a power unmanned aerial vehicle multi-modal full autonomous inspection method comprises: A multi-agent is used to cooperatively process power inspection task instructions within a regional grid range and issue them to unmanned aerial vehicles of corresponding airports. The unmanned aerial vehicles are controlled to fly and autonomously perceive the dynamic environment of the inspection area to autonomously query interesting targets and guide the unmanned aerial vehicles to autonomously inspect unknown environments and perceive multi-modal inspection data of the interesting targets. The multi-modal inspection data of the interesting targets are cross-modally calibrated, spatio-temporal joint features of each modality after calibration are extracted, and each modality weight matched with the dynamic environment is combined to obtain a fusion feature through weighting. Based on the fusion feature, a multi-modal large model is used to evaluate the health status of the interesting targets. According to the relevance of each modality, a correlation matrix is constructed and secondly revised to update a preset knowledge base, and combined with the health status evaluation result of the interesting targets, an active prevention decision of the interesting targets driven by multi-modal fusion and correlation degree is obtained.

[0009] The second aspect of the present application provides a power unmanned aerial vehicle multi-modal full autonomous inspection system.

[0010] In one or more embodiments, a power unmanned aerial vehicle multi-modal full autonomous inspection system comprises: An inspection task issuing module is used to cooperatively process power inspection task instructions within a regional grid range by a multi-agent and issue them to unmanned aerial vehicles of corresponding airports. A modality inspection data perception module is used to control the unmanned aerial vehicles to fly and autonomously perceive the dynamic environment of the inspection area to autonomously query interesting targets, guide the unmanned aerial vehicles to autonomously inspect unknown environments, and perceive multi-modal inspection data of the interesting targets. A multi-modal inspection data fusion module is used to cross-modally calibrate the multi-modal inspection data of the interesting targets, extract spatio-temporal joint features of each modality after calibration, combine each modality weight matched with the dynamic environment, and obtain a fusion feature through weighting. A health status evaluation module is used to evaluate the health status of the interesting targets based on the fusion feature by using a multi-modal large model. An active prevention decision module is used to construct a correlation matrix according to the relevance of each modality, secondly revise it to update a preset knowledge base, and combined with the health status evaluation result of the interesting targets, obtain an active prevention decision of the interesting targets driven by multi-modal fusion and correlation degree.

[0011] In one or more embodiments, an unmanned aerial vehicle multi-modal full autonomous inspection system for power supply comprises: a data sensing unit carried on the unmanned aerial vehicle, for autonomously sensing multi-modal inspection data of a dynamic environment and an interesting target; a data processing unit comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps of the method when executing the program.

[0012] The third aspect of the application provides a computer readable storage medium.

[0013] A computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method.

[0014] Compared with the prior art, the application has the following beneficial effects: (1) The application innovatively proposes an unmanned aerial vehicle multi-modal full autonomous inspection technology, designs a real-time sensing and path planning method for unmanned aerial vehicle flight operation environment in unknown environment, develops a grid-based power supply airport, constructs a multi-target multi-scale multi-modal fusion feature model of power equipment, proposes a causal reasoning hazard decision analysis and early warning method, designs a multi-agent collaboration and dynamic resource scheduling model, realizes grid-based collaborative operation regulation of multi-nest multi-machine multi-task in a large area, and the unmanned aerial vehicle adaptively carries out inspection operation, intelligently completes image defect diagnosis and decision analysis after inspection, and constructs an unmanned aerial vehicle "autonomous inspection-intelligent analysis-collaborative regulation" full autonomous operation system, solves the contradiction between personnel shortage and increasing inspection demand, and improves the intelligent and unmanned inspection quality and efficiency on site.

[0015] (2) The application innovatively proposes a power equipment multi-modal data dynamic weighted fusion and correlation degree joint driving prevention decision method, constructs a multi-source data dynamic calibration deep fusion and equipment prevention decision generation model, realizes multi-target multi-scale defect accurate diagnosis and early warning decision analysis through a meta-learning framework and a spatio-temporal joint feature model, reduces the artificial auditing workload of post-inspection data analysis, and improves the abnormal defect detection accuracy and equipment prevention decision reliability.

[0016] (3) The application innovatively proposes a large-scale multi-nest multi-machine multi-task grid regulation technology, constructs an airport-equipment relationship graph model, takes the maximum inspection benefit of single flight operation as the goal, realizes collaborative regulation of inspection task and operation unmanned aerial vehicle, and improves the unmanned aerial vehicle inspection operation efficiency and safety.

[0017] (4) The application innovatively proposes an unmanned aerial vehicle intelligent inspection technology in an unknown environment, through multi-modal data fuzzy matching and trajectory prediction technology, adaptively and dynamically adjusts the multi-source perception data fusion weight, generates a global inspection path and dynamically predicts an inspection trajectory, and realizes autonomous exploration and safe flight of the unmanned aerial vehicle in the unknown environment.

[0018] (5) The application innovatively proposes an unmanned aerial vehicle airport power exchange method and system, designs an unmanned aerial vehicle power exchange control model based on a grid model, realizes rapid energy supply of the unmanned aerial vehicle after inspection and uninterrupted unmanned inspection. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated herein by reference. The illustrations are shown for the purpose of enabling those skilled in the art to understand the application better, and do not constitute an improper limitation on the application.

[0020] Figure 1 is a flowchart of the power unmanned aerial vehicle multi-modal full autonomous inspection method in embodiment one of the application; Figure 2 is a process of guiding the unmanned aerial vehicle to autonomously inspect the unknown environment in embodiment one of the application; Figure 3 is a process of updating the preset knowledge base in embodiment one of the application; Figure 4 is a process of finding the airport closest to the position where the inspection task is interrupted to perform the power exchange operation in embodiment one of the application; Figure 5 is a process of managing the battery in embodiment one of the application; Figure 6 is a structural schematic diagram of the power unmanned aerial vehicle multi-modal full autonomous inspection system in embodiment two of the application; Figure 7 is a structural schematic diagram of the power unmanned aerial vehicle multi-modal full autonomous inspection system in embodiment three of the application. DETAILED DESCRIPTION

[0021] The application will be further described below in conjunction with the drawings and embodiments.

[0022] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0023] It is to be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise, and it is further understood that the terms "comprising" and / or "including" when used in this specification intend the presence of the features, steps, operations, devices, components and / or combinations thereof.

[0024] Embodiment one Figure 1 is a flowchart of a power unmanned aerial vehicle multi-modal full autonomous inspection method according to an embodiment of the present application, as Figure 1 The power unmanned aerial vehicle multi-modal full autonomous inspection method according to the present embodiment, as shown in the figure, specifically includes the following steps: Step S101: Use multi-agent to cooperatively process the power inspection task instruction within the regional grid range and issue it to the corresponding airport unmanned aerial vehicle.

[0025] In the specific implementation process of step S101, the process of using multi-agent to cooperatively process the power inspection task instruction is as follows: The user's inspection task instruction is parsed into an inspection area determination subtask, an airport and unmanned aerial vehicle query subtask, and an inspection task list generation subtask; The inspection resource management agent is called from the agent library to execute the inspection area determination subtask, and the inspection area table is queried to obtain the inspection area; The device management agent is called from the agent library to execute the airport and unmanned aerial vehicle query subtask, and the airport table and unmanned aerial vehicle table are queried to obtain the airport and the unmanned aerial vehicle therein for the to-be-executed inspection task; The visualization agent is called from the agent library to execute the inspection task list generation subtask, and the information of the inspection area, the airport for the to-be-executed inspection task, and the unmanned aerial vehicle therein is displayed in the form of a task list to form an inspection task list to receive a confirmation instruction for the task list.

[0026] It should be noted here that the inspection resource management agent, the device management agent, and the visualization agent can all be implemented using existing large models.

[0027] The present embodiment uses multi-agent division of labor and cooperation to decompose the complex inspection task into specialized subtasks, and each agent focuses on its own field of expertise, thereby improving the overall response speed and processing accuracy.

[0028] Step S102: Control the unmanned aerial vehicle to fly and autonomously perceive the dynamic environment of the inspection area to autonomously query the target of interest, guide the unmanned aerial vehicle to perform autonomous inspection of the unknown environment, and perceive multi-modal inspection data of the target of interest.

[0029] Specifically, according to the inspection task sheet, the unmanned aerial vehicle of the corresponding airport is controlled to fly to a set position in the inspection area and autonomously perceive a dynamic environment of the position and autonomously query an interesting target.

[0030] In the specific implementation process, as shown in Figure 2 , the process of guiding the unmanned aerial vehicle to autonomously inspect the unknown environment is as follows: Step S201: Obtain unmanned aerial vehicle multi-source perception data of an unknown environment and extract obstacle information therefrom to plan a preliminary global path from a current point to a target point.

[0031] In an unknown environment without prior environment information, unmanned aerial vehicle multi-source perception data can be obtained by using a heterogeneous sensor array (solid-state laser radar, thermal imaging camera, depth camera, millimeter wave radar, ultrasonic sensor). The sensor array is shown in Table 1: Table 1 Sensor array

[0032] Among them, a D*Lite algorithm is used to plan a preliminary global path from the current point to the target point.

[0033] The D*Lite algorithm is a backward search algorithm that searches from the target point to the current position of the unmanned aerial vehicle. The core of the algorithm is to assume that the unknown area is all free space, and on this basis, incrementally implement path planning to find the shortest distance from the target point to each node by minimizing the rhs value.

[0034] D*Lite determines the priority of node update by a key value, the key value is composed of two parts: and ; Among them, is the current estimated cost from node s to the target point; is a more forward-looking cost estimate value based on the successor node of node s; is a heuristic function from the current position of the unmanned aerial vehicle to node s, which ensures that the search priority updates the most important node for the current position of the unmanned aerial vehicle; is an accumulator that records the sum of the heuristic cost added by all start points since the algorithm starts running; initially , whenever a new start point is moved from the last start point , the heuristic cost of this step is calculated and added to .​

[0035] where the expression for a more forward looking cost estimate of the successor nodes of node s is: where Succ(s) is the set of successor nodes of node s, is the cost of moving from s to is the cost of moving from s to If then node s is consistent, meaning that the value of g(s) is up to date and reflects the best cost to reach the goal, if then a new path from s to the goal has been discovered with a lower cost than the previous g(s) and is set to update; if

[0036] Step S202: dynamically adjust the weight of the multi-source perception data of the unmanned aerial vehicle according to the current environment information of the unknown environment and in combination with the preset fuzzy rule base, and perform weighted fusion on the multi-source perception data of the unmanned aerial vehicle, and then combine the current IMU data to jointly optimize and calculate the current pose of the unmanned aerial vehicle.

[0037] In the specific implementation process, the weight of the multi-source perception data of the unmanned aerial vehicle is dynamically adjusted based on the preset fuzzy rule base; wherein the fuzzy rule base input is the fuzzy value of the current environment information, and the output is the adjustment direction of the weight of the multi-source perception data of the unmanned aerial vehicle.

[0038] The calculation of the fused data is based on a preset fuzzy rule base, which defines the mapping relationship of the reliability of each sensor under different environmental conditions.

[0039] For example, the input variable set , wherein : environmental conditions, such as illumination conditions, weather conditions, space conditions, etc. measured by sensors. The output variable set , wherein : solid-state laser radar weight, : thermal imaging camera weight, : depth camera weight, : millimeter wave radar weight, : ultrasonic sensor weight; for each input variable , define its fuzzy set , : the kth fuzzy set of the ith input variable; for each output variable , define its fuzzy set , : the fuzzy set of the jth output variable. ​​

[0040] Input variable fuzzification: light intensity membership function (trapezoidal function): ; where x: light intensity, a, b, c, d: parameters of the trapezoidal function, : membership degree of light intensity in the fuzzy set.

[0041] Rain and snow weather condition membership function (linear approximation-trapezoidal function): ; where y: precipitation, i: lower limit of precipitation, j: upper limit of precipitation.

[0042] Fog condition membership function (Z-shaped function): ; where z: visibility, m: upper limit of visibility, p: lower limit of visibility, n: midpoint of the function: .

[0043] Specifically, the process of dynamically adjusting the weight of the multi-source perception data of the unmanned aerial vehicle based on the preset fuzzy rule base is: Calculate the matching degree of the current input to each rule antecedent condition; Using the Takagi-Sugeno type fuzzy model, for each rule, the total activation strength is the aggregation of the matching degrees of all conditions in the antecedent, and the activation rule strength is calculated using the product operator; The weight of the multi-source perception data of the unmanned aerial vehicle is calculated by using the ratio of the product of the aggregation of all condition matching degrees in the consequent and the activation rule strength to the cumulative sum of all activation rule strengths.

[0044] For example, the fuzzy rule base: Each rule is in the form of: if is and is and...then is , is ,...; For the current input , calculate the matching degree of each fuzzy set in the rule antecedent to the current input, where : the matching degree of the input value to the fuzzy set , : the input to the jth condition in the rule . a degree of match of the antecedent.

[0045] Using Takagi-Sugeno type (TS type) fuzzy model, for each rule its total activation strength is an aggregation of the degree of match of all conditions of the antecedent, the activation rule strength is calculated using the product operator, .

[0046] The consequent of the TS type rule is a function, the rule form: if is and...and is then , using the first-order TS model: where : constant term of the t-th rule, : weight of the corresponding input variable in the t-th rule, : aggregation of the degree of match of all conditions of the consequent.

[0047] The weight of the unmanned aerial vehicle multi-source perception data is: ; According to the environmental conditions, the sensor weight is dynamically adjusted, the multi-source data after fusion and the IMU data are fused according to the adjusted unmanned aerial vehicle multi-source perception data, and a most reliable unmanned aerial vehicle pose is calculated by common optimization.

[0048] Step S203: based on the current pose of the unmanned aerial vehicle calculated by optimization, when flying along the preliminary global path, whether there is a new obstacle is judged according to the real-time unmanned aerial vehicle multi-source perception data, and the preliminary global path is optimized to realize the unmanned aerial vehicle autonomous inspection in unknown environment.

[0049] During the flight of the unmanned aerial vehicle according to the global planning path, the motion state of each dynamic object is estimated and predicted through real-time sensor data. The state vector of the moving object usually contains position and velocity: where is the three-dimensional position, is the three-dimensional velocity.

[0050] For each tracked dynamic obstacle, an interacting multiple model filter (IMM) is established; The filtering results of all interacting multiple model filters are fused to obtain the overall optimal estimation and uncertainty measure of the obstacle state at the current time, and based on the optimal estimation and model probability at the current time, the future motion trajectory of the dynamic obstacle is predicted.

[0051] IMM includes uniform model, uniform acceleration model, coordinated turning model. The core of IMM is to calculate the posterior probability of each model , and take it as its weight.

[0052] First, for the current model m, calculate the mixing probability of the model n at the last time: ; Wherein represents the contribution weight of the estimation of the model n at the last time to the current model m after considering the inter-model switching relationship, k: current time, Q: total number of models; : probability of model n at the last time; : probability of transition from model n to model m; is a normalization constant, representing the prediction prior probability of model m.

[0053] Calculate the mixed initial state and covariance for each model m, the initial state: , the covariance: , is the estimation uncertainty of model n at the last time, representing the error within each model. is the outer product of the deviation of the estimation value of model n and the mixed mean , representing the divergence between different models. Each model m takes its mixed initial condition as the starting point, independently carries out the standard Kalman filtering process, and obtains the posterior probability at the current time: , m and n are model indexes, is the likelihood of model m, is the prediction mixing probability of model m. Fuse the filtering results of all models to obtain the overall optimal estimation and uncertainty measure of the obstacle state at the current time, and predict the future motion trajectory of the obstacle based on the optimal estimation at the current time and the model probability.

[0054] The predicted dynamic obstacle trajectory is used as temporary obstacle information and fed back to the D* Lite algorithm. After receiving the information, the D* Lite algorithm triggers the replanning process to generate a new path that can avoid both static obstacles and predicted dynamic obstacles.

[0055] The embodiment proposes a UAV inspection method in an unknown environment, which uses a multi-source perception data fusion scheme of the UAV, uses dynamic weight distribution, and adjusts the weight factor according to the environmental information in real time. The most reliable UAV pose is calculated by optimizing the fused multi-source data and IMU data together, solving the problem that the UAV cannot fly in an environment without prior information or external positioning signals, and enabling preliminary detection of unknown environments and improving the environmental adaptability of UAV inspection.

[0056] Step S103: The multi-modal inspection data of the target of interest is cross-modality calibrated, the spatio-temporal joint features of each modality after calibration are extracted, and then each modality weight matched with the dynamic environment is combined to obtain the fusion features through weighting.

[0057] In step S103, the multi-modal inspection data of the target of interest is cross-modality calibrated by a cross-modality calibration model, and then time sequence feature and spatial topology feature extraction and splicing operations are sequentially performed to obtain the spatio-temporal joint features of each modality.

[0058] The training of the cross-modality calibration model adopts a weighted sum of local feature alignment loss and global context matching loss, and the cross-modality calibration model learns model parameters and adjustment factors for weighting through meta-learning training; the weighted sum is: ; wherein, is an adjustment factor; the local feature alignment loss , represents a local feature set extracted by modality A, represents a local feature set extracted by modality B, N represents the dimension of the local feature extracted by modality A, and M represents the dimension of the local feature extracted by modality B, is a cosine similarity, is a constant coefficient; the global context matching loss , is a global feature of modality A, is a global feature of modality B; represents a global feature of modality B of a negative sample, and K represents the number of negative samples.

[0059] After the cross-modality calibration model extracts the local features and global features of different modality data by using different branch encoders, the similarity matrix of the local features between different modality data is calculated to obtain a local similarity matrix, the similarity matrix of the global features between different modality data is calculated to obtain a global similarity matrix, based on the local similarity matrix and the global similarity matrix, a classifier is used to output the matching probability between different modality data, and based on the matching probability, an attention-based decoder is used to obtain the cross-modality calibrated modality data.

[0060] It should be noted that the time sequence feature and the spatial topology feature are extracted by the spatio-temporal graph neural network model ST-GNN in this embodiment, and other existing networks can also be used to extract the time sequence feature and the spatial topology feature.

[0061] Specifically, the correlation between modalities is calculated based on the spatio-temporal joint features, and each modality weight matched with the dynamic environment is calculated by combining the noise level of the dynamic environment and the data acquisition sensor of each modality.

[0062] wherein the modal weight matching the dynamic environment is: ; represents the temperature coefficient; the comprehensive score ; the noise level of the sensor , is the noise standard deviation, is the sensitivity coefficient; the inter-modal correlation score , m represents the total number of modes, is the Pearson correlation coefficient of modal i and j; the scene prior factor , is an indicator function, is the scene gain coefficient.

[0063] The embodiment proposes a meta-learning driven cross-modal feature calibration, spatio-temporal joint feature extraction and dynamic weighting fusion method. Through double loss constraint, the cross-modal consistency is improved. On this basis, by dynamically adjusting the weight, the features are fused, which can automatically filter noise, weaken interference and focus on reliable modal, avoiding the interference of invalid information on the subsequent multi-modal large model, improving the feature quality, and further improving the anomaly detection precision of the interest target.

[0064] Step S104: Based on the fused features, a multi-modal large model is used to evaluate the health status of the interest target.

[0065] The multi-modal large model here can be realized by using the multi-task prediction head of the designed classification and regression task, which is used to obtain the health type and health score.

[0066] (1) Global average pooling followed by Softmax, realizing fault type prediction: ; is the fused feature of multi-modal data; is the average pooling layer; is the weight matrix of the classification task; is the bias term of the classification task.

[0067] (2) The output scalar of the fully connected layer realizes the prediction of the health score s: ; wherein, represents the weight matrix of the regression task; represents the bias term of the regression task; s∈(0, 1). When the output health degree is >0.8, the device state is healthy; when the health degree is in [0.4, 0.8), the device needs attention; when the health degree is less than 0.4, the device has high risk of failure.

[0068] Step S105: According to the correlation of each modality, an association degree matrix is constructed and secondly revised to update the preset knowledge base, and then combined with the health status evaluation result of the interest target, the multi-modal fusion and association degree joint driven interest target active prevention decision is obtained.

[0069] Specifically, the attention weight between the spatio-temporal joint features of each modality is calculated based on the cross-modal attention mechanism, which is used as the association degree to construct the initial association degree matrix, and the initial association degree matrix is secondly revised to update the preset knowledge base.

[0070] In this embodiment, the attention weight between the spatio-temporal joint features of each modality is calculated based on the cross-modal attention mechanism, which is used as the association degree to construct the initial association degree matrix, wherein the elements are: ; wherein, denotes the attention weight between the spatio-temporal joint features of modality and the spatio-temporal joint features of modality ; ; wherein, S denotes the total number of modalities, d is the feature dimension; W Q and W K is a projection matrix.

[0071] In this embodiment, as shown in Figure 3 , the process of updating the preset knowledge base is: Step S301: Using a causal discovery algorithm, the association degree matrix and the historical failure data of the interest target are combined to generate a causal graph.

[0072] The historical failure data of the interest target here includes but is not limited to visible light images, infrared images, vibration data, three-dimensional point cloud data, flight path files, weather data, defect data, defect elimination records, expert experience, etc.

[0073] Step S302: The initial association degree matrix is filtered by pseudo-association to obtain a first revised association degree matrix.

[0074] According to the rules of physical completeness, statistical consistency, knowledge base coverage, etc., pseudo-association filtering is performed, which specifically considers dimensional consistency test, time shift independence test, confidence interval test and fault mode constraint to obtain the purified association relationship: (a) Dimensional consistency test; If the dimensions of modalities i and j and cannot be associated by a physical equation, they are forced to be zero: ; ​ denotes the association weight between modal data i and modal data j.

[0075] (b) Time-shift independence test; True association should satisfy time-shift mutual information peak uniqueness: ; Typical threshold . denotes the modal i data at time t; denotes the modal j data at time t; denotes the modal j data at time t; I denotes the mutual information between and .

[0076] (c) Bootstrap confidence interval; The 95% confidence interval of is calculated by resampling , and if it contains zero, it is filtered: ; denotes the lower limit of the 95% confidence interval of calculated by resampling denotes the upper limit of the 95% confidence interval of calculated by resampling .

[0077] (d) Fault mode constraint; Construct a whitelist matrix that allows association: ; Based on the purified association, estimate the causal effect, estimate the causal effect strength through the SEM structural equation model, and finally output the causal diagram and the revised association matrix R* (i.e., the first revised association).

[0078] Step S303: Strengthen the consistency between modalities by contrast learning on the first revised association matrix, and remove the preset low-confidence association in contrast learning using the causal diagram to obtain the second revised association matrix; wherein the contrast loss function used in the contrast learning process is: ; wherein, is the cosine similarity, is a constant coefficient; is the feature of modal i and j mapped to the shared space; denotes an element in the once-revised correlation matrix; H is the number of modes after once revision.

[0079] The input is a pair of multi-modal positive samples (Fi, Fj+) and negative samples (Fi, Fj-). The positive sample pair is multi-modal data at the same time on the same device (such as image-vibration pairing), and the negative sample pair is data of different devices or different states. First, map the multi-modal features to a shared space: ; wherein, denotes the feature of the n-th modal data; denotes the multi-modal data mapped to the shared space; denotes a d-dimensional real number space.

[0080] Then, through the contrast loss function, minimize the positive sample distance and maximize the negative sample distance.

[0081] In this embodiment, the second revision adopts the way of contrast learning. The correlation relationship after the first revision participates in the generation of the similarity matrix through the contrast loss function, and the main participation way is to embed the correlation relationship after the first revision as a weighted controller into the contrast loss function to directly act on the loss calculation, realizing two-level control.

[0082] Step S304: updating the pre-knowledge base by using the correlation matrix after the second revision.

[0083] For example, based on the data such as power equipment design drawings, nameplate parameters, and material properties, the equipment ontology library is constructed; based on the power industry standards (DL / T 664, IEC 60567, etc.) and manufacturer technical manuals, the defect rule library is constructed, and new defect modes automatically trigger rule generation to realize dynamic updating of the rule library; knowledge is extracted from historical maintenance reports to construct an expert experience library; historical case data is structured to construct a historical case library; and finally, the above-mentioned equipment ontology library, defect rule library, expert experience library, and historical case library are combined to construct a structured knowledge base.

[0084] The correlation matrix after the second revision and the causal diagram are input into the knowledge base, the device parameters are fed back according to the modal correlation, and the equipment ontology library is supplemented; if a new correlation mode is found in the correlation matrix after the second revision, a new rule is automatically generated and added to the defect rule library, and the confidence is labeled to realize dynamic updating of the rule library; the correlation matrix after the second revision is compared with the existing expert experience, if the data-driven rule conflicts with the expert experience, manual review is triggered, the weight of the rule is increased after consistent determination, and the expert experience library is enhanced; the high-confidence correlation in the correlation matrix after the second revision generates a new case template to realize expansion of the historical case library. The knowledge base provides physical interpretability constraints for the correlation matrix after the second revision, and the correlation matrix after the second revision provides data-driven updating materials for the knowledge base.

[0085] When the health of the device is predicted to be low, a similar historical case is retrieved using the Faiss vector database, the root cause affecting the health of the device is analyzed, a warning is issued, and a maintenance decision suggestion is given.

[0086] Every week, the missed cases are analyzed, new defect patterns are extracted, the knowledge base is dynamically updated, and continuous learning is formed to form a closed loop.

[0087] The embodiment utilizes the cross-modal attention mechanism to calculate the attention weight between multi-modalities to construct an initial correlation degree matrix, and the initial correlation degree matrix is modified to update the preset knowledge base, and the health status of the interest target is combined to actively provide a prevention decision for the interest target, thereby ensuring the stable operation of the inspection.

[0088] In some optional embodiments, when the unmanned aerial vehicle encounters an abnormal power limit situation, the inspection task is not completed, and the current power of the unmanned aerial vehicle cannot return to the original airport, the airport closest to the position where the inspection task is interrupted is searched, and the unmanned aerial vehicle continues to perform the inspection task after performing the battery replacement operation. As shown in Figure 4 , the airport closest to the position where the inspection task is interrupted is searched and the battery replacement operation is performed, and the process is as follows: Step S401: A grid model is established according to the actual environment inside the battery replacement airport; wherein the unmanned aerial vehicle position and the battery compartment position inside the battery replacement airport are arranged on the same plane, so that the movement form of the battery replacement mechanical arm is decomposed into two-dimensional plane movement; Step S402: Based on the grid model, the position of the unmanned aerial vehicle position, the position of the empty battery compartment position, and the position of the battery compartment position of the replaceable battery, an obstacle avoidance path of the battery replacement mechanical arm is planned; the obstacle avoidance path of the battery replacement mechanical arm includes an obstacle avoidance path from the starting unmanned aerial vehicle position to the empty battery compartment position, an obstacle avoidance path from the battery compartment position to the battery compartment position of the replaceable battery, and an obstacle avoidance path from the battery compartment position to the starting unmanned aerial vehicle position.

[0089] In the embodiment, the A* algorithm is used for path planning, specifically, the nearby grid position of the set starting position is searched, the nearby grid with the minimum estimated cost function value is selected as the next starting point, and the selected grid is called the current node, then the nearby grid with the minimum estimated cost function value is searched again as the new current node, and the cycle calculation is repeated until the current node reaches the set ending position, and the obstacle avoidance path of the battery replacement mechanical arm is planned.

[0090] Since the A* algorithm only considers the node information of the obstacle when planning the path, the planned route intersects with the corners of the obstacle, which may cause friction and collision between the mechanical arm and the obstacle of the fuselage or the obstacle of the unmanned aerial vehicle landing platform, and needs to be avoided. Here, the traditional A* algorithm is improved, and a corner contact penalty function is added to the estimated cost formula.

[0091] In this embodiment, the estimated cost function is the sum of the actual cost function from the start point to the current node, the heuristic function from the current node to the end point, and the corner contact penalty function, i.e., ; n is the current node, g(n) is the actual cost function from the start point to the current node, and h(n) is the heuristic function from the current node to the end point.

[0092] The corner contact penalty function is obtained by multiplying the penalty intensity coefficient by the exponential power of e; the power exponent of e is obtained by multiplying the difference between the minimum safety distance and the safety distance threshold by the steepness factor.

[0093] The specific expression of the corner contact penalty function is: ; wherein k is the penalty intensity coefficient (the constant value is 100), is the steepness factor (the constant value is 10), is the minimum safety distance, wherein is the set of all obstacle corner points, because each obstacle grid in the grid environment is a square with a side length of d, the four corners of the obstacle grid are: ; Therefore wherein is the set of obstacle grids.

[0094] The actual cost function is: , wherein here, the grid is set as a square with a side length of d.

[0095] P K-1 is the start point of the motion step, and P K is the end point of the motion step.

[0096] The heuristic function uses the Euclidean distance: , wherein is the current node coordinate, is the end point coordinate.

[0097] Minimum safety distance monitoring: , is the path node position vector; is the obstacle corner point position vector.

[0098] is the safety distance threshold: , Safety margin.

[0099] Then , Path segment The path segment safety constraint must be met, that is

[0100] In the established 5-line 13-column grid map, taking the lower left corner of the coordinate point (3, 0) as the starting position of the unmanned aerial vehicle, and the coordinate point (11, 4) as the battery compartment position as the end position, the path planning of the battery conveying mechanical arm is simulated. According to the above A* algorithm, the planning and calculation process is as follows: Define the node data structure, including position, Value, Value, Value, Value and parent node; initialize the open list and add the starting point to the open list; loop until the end point is found or the open list is empty: take the node with the smallest Value as the current node; if the current node is the end point, backtrack the path; otherwise, mark it as closed; generate 8-direction neighbor nodes.

[0101] For each direction: Calculate the neighbor node coordinates; Check if the coordinates are within [0, 4] x [0, 12] and are free grids (non-obstacles); If it is a straight direction (four directions: up, down, left, right), that is , directly enter the next step; If it is a diagonal direction, check if the two adjacent grids of the current node (the neighbor grids in the two straight directions corresponding to the diagonal moving direction) are obstacles at the same time: if not, enter the next step; if so, compare And , if , enter the next step, if , skip this adjacent grid; Calculate the Value, Value (Euclidean distance) and Value of the neighbor node, and get the Value; If the neighbor node is in the open list and the new Value is smaller, update; otherwise, add the neighbor to the open list.

[0102] The above algorithm takes four UAVs as an example. If the battery replacement airport needs to adapt to a larger number of UAVs, more than four UAVs can be placed in a set direction (such as upwards or to the right) based on the above layout, and the travel of the corresponding module can be lengthened according to the actual layout scheme. Then, a grid model is re-established based on the layout environment, and the A* algorithm-based mechanical arm obstacle avoidance path planning scheme is also applicable to the layout environment after increasing the number of UAVs.

[0103] It should be noted that in other embodiments, in addition to the A* algorithm (grid search), APF algorithm (artificial potential field), PRM algorithm (probabilistic roadmap), RRT (rapidly-exploring random tree), etc. can also be used for robot path planning.

[0104] In one or more embodiments, when an obstacle appears in the obstacle avoidance running path of the battery replacement mechanical arm, the control of the battery replacement mechanical arm stops running. In this way, mechanical damage to the battery replacement mechanical arm can be prevented, and operation safety can be ensured. In this way, the process of pulling out the battery from the UAV can be avoided, and the risk of the battery falling without support can be avoided.

[0105] The process of replacing the battery of the UAV is as follows: after one UAV lands, the battery replacement mechanical arm runs to the UAV position, the battery replacement mechanical arm turns off the UAV, the battery gripper pulls out the battery in the UAV, the system detects the position of the existing empty battery compartment, and plans an obstacle avoidance path from the current UAV position to the empty battery compartment position based on the A* algorithm. The battery replacement mechanical arm moves to the empty battery compartment position according to the planned path, the battery gripper inserts the UAV battery into the battery compartment for charging, and after confirming that the inserted battery has started charging, the battery compartment position where the battery can be replaced (generally, the battery charging capacity in the battery compartment reaches more than 95%) is detected. An obstacle avoidance path from the current battery compartment position to the battery compartment position where the battery can be replaced is planned based on the A* algorithm, the battery replacement mechanical arm moves to the battery compartment position where the battery can be replaced according to the planned path, and the battery gripper pulls out the battery in the battery compartment. An obstacle avoidance path from the current battery compartment position to the starting UAV position is planned based on the A* algorithm, the battery replacement mechanical arm moves to the starting UAV position according to the planned path, and the battery gripper inserts the UAV battery into the UAV.

[0106] If the UAV does not have a task to continue to perform, the UAV battery replacement process ends; if the UAV has a flight task to continue to perform, the battery gripper turns on the UAV, the UAV flies out to continue to perform the task, and the UAV battery replacement process ends.

[0107] The above scheme solves the problems of the obstacle avoidance path planning of the battery replacement robot arm when multiple unmanned aerial vehicles are replaced with batteries, the safety hazards of collision between the parked unmanned aerial vehicles and the airport structure, and the like in the battery replacement airport, and plans an obstacle avoidance path of the battery replacement robot arm based on a grid model, positions of the unmanned aerial vehicle sites, positions of the empty battery sites, and positions of the battery sites with replaceable batteries, thereby improving the adaptability of the battery replacement robot arm to complex battery replacement environments and achieving safety control of the battery replacement process in the battery replacement airport with multiple unmanned aerial vehicles.

[0108] In one or more embodiments, as shown in Figure 5 In the process of finding the airport closest to the inspection task interruption position and performing the battery replacement operation, the battery is also managed and scheduled, and the process is as follows: Step S501: Obtain the real-time temperature of the battery cabin of the battery replacement robot, the state of the batteries in the cabin, and the working mode of the battery replacement robot; wherein the working mode includes a storage mode and a standby mode, in the storage mode, the unique standby battery power is kept above the set storage threshold power, and in the standby mode, the power of the batteries other than the unique standby battery is kept within the set standby threshold.

[0109] Step S502: Temperature regulation of the overall temperature in the cabin and the temperature abnormal batteries according to the state of the batteries in the cabin and the real-time temperature; Specifically, the specific steps of temperature regulation of the overall temperature in the cabin and the temperature abnormal batteries according to the state of the batteries in the cabin and the real-time temperature are as follows: Step S5021: Adjust the real-time temperature in the cabin according to the total temperature threshold range; In the management of the batteries in the battery cabin, the temperature information in the cabin of the robot is obtained in real time, and the temperature in the cabin is adjusted in real time according to the temperature data by controlling the air conditioner, so that the temperature in the cabin of the robot is kept at a temperature suitable for battery storage for a long time. The system dynamically controls the temperature according to the real-time temperature in the cabin and the temperature of each battery, so that the temperature in the cabin is kept as much as possible within the total temperature threshold range of 15-35℃. When the temperature in the cabin is greater than 35℃, the cold power is controlled and adjusted, when the temperature drops to 25℃, it is kept, when the temperature is lower than 15℃, the hot power is controlled and adjusted, and when the temperature rises to 25℃, it is kept.

[0110] Step S5022: Detect the temperature abnormal battery, start the single battery temperature control channel, and use the adaptive PID control algorithm to adjust the temperature of the temperature abnormal battery position.

[0111] When the cabin temperature is between 15-35℃, some batteries are put into the battery replacement machine nest because they have just completed a task, and the battery temperature is too high. Therefore, when a battery with a temperature that is too high is detected, a single battery temperature control channel is opened for it, and the temperature control power is adjusted according to the cabin temperature, and a large amount of cold air is preferentially sent to the overheated battery. The system uses an adaptive PID control algorithm to adjust the temperature of the battery detected to be abnormal: .

[0112] wherein, represents the detection temperature of the battery, integrates on [0, t], is the temperature deviation, , is the set temperature is the actual temperature; is the output control quantity (temperature control power); is the proportional term, which provides linear amplification according to the current error, and if there is still a large deviation between the system and the set value, the proportional term can quickly make adjustments, ; is the integral term, which accumulates historical errors to eliminate static errors and enable the system to accurately reach the set value when stable, ; is the derivative term, which predicts the error trend and has a certain anti-disturbance ability, and can suppress system overshoot and oscillation, .

[0113] Step S503: According to different working modes, the battery in the battery replacement machine nest is managed, and the state of the battery in the cabin is updated in real time; In this embodiment, the working modes of the battery replacement machine nest include a storage mode and a standby mode. In the storage mode, only one battery needs to ensure that the power is executable for a task, which can protect the service life of the battery while not affecting the execution of the task. In the storage mode, the power of the only standby battery is kept above the set storage threshold power. In the standby mode, the power of the other batteries is kept within the set standby threshold.

[0114] The battery charging management steps in the storage mode are as follows: setting the storage threshold, when there is no battery in the battery cabin that meets the storage threshold power, finding the battery with the minimum cycle number to set as the only standby battery, and charging the only standby battery to the storage threshold power or above, when the only standby battery is used, finding the battery with the minimum cycle number in the battery cabin again to charge the only standby battery to the threshold power, and using the battery with the minimum cycle number in turn, which is conducive to the equalization management of the battery pack.

[0115] In this embodiment, the storage threshold SOC is set to 80%. In the storage mode, all batteries except the unique standby battery are kept at an appropriate storage power (40%-60%) for a long time. When the battery power is lower than 40%, the battery is charged, and the charging is stopped when the battery power reaches 60%, so that the battery is at an appropriate storage power, which is beneficial to prolong the service life of the battery.

[0116] The charging management steps of the battery in the standby mode are as follows: setting a standby threshold, charging all batteries to the standby threshold power and above, first traversing all batteries to find all batteries below the standby threshold, traversing the batteries below the standby threshold to find the battery with the highest power, first suspending the charging of other batteries, and preferentially charging the battery with the highest power to the standby threshold power and above; such a cycle is repeated until all batteries are charged to the standby threshold power and above, and then all batteries meeting the standby threshold power are traversed for selection and use.

[0117] Step S504: performing battery replacement operation on the unmanned aerial vehicle with the to-be-replaced battery according to the updated in-cabin battery state by using a battery cyclic use strategy, wherein the battery cyclic use strategy is that when the difference between the cycle times of the batteries is within a cycle threshold range, the battery closest to the gripper and meeting the power requirement is selected for replacement, and when the difference between the cycle times of the batteries exceeds the cycle threshold range, the batteries meeting the power requirement are evenly scheduled for use according to the cycle times.

[0118] When the battery is replaced, the batteries are evenly scheduled for use according to the cycle times and the distances of the batteries from the gripper. According to the cycle time balancing scenario, the minimum cycle time priority algorithm is used: , wherein, represents the selected battery to be replaced, i is the battery index, i = 1, 2, 3,..., n; represents the cycle time of the battery with index i, and n is the total number of batteries.

[0119] At the same time, the distance of the battery from the gripper is comprehensively considered: , wherein, represents the distance of the battery with index i from the gripper.

[0120] In this embodiment, the influence of the cycle time of the battery is considered. The fewer the cycle times of the battery, the higher the score. When selecting a battery, the battery with fewer cycle times is preferentially selected. Therefore, the comprehensive battery cycle time use optimization term is designed as: , wherein, is the minimum cycle time of the in-cabin battery.

[0121] In addition, considering the influence between the distance from the clamping jaw, the closer the battery is to the clamping jaw, the higher the score is, when selecting the battery, the battery close to the clamping jaw is preferentially selected, so the distance optimization term of the comprehensive battery from the clamping jaw is designed: ; wherein, is the farthest distance of the battery in the cabin from the clamping jaw; is the closest distance of the battery in the cabin from the clamping jaw.

[0122] In order to balance the priority relationship between the battery cycle number and the distance from the clamping jaw, the comprehensive battery cycle number use optimization term and the comprehensive battery distance from the clamping jaw optimization term are combined to obtain a comprehensive calculation formula: ; wherein, is a weight coefficient, and the maximum battery is selected as the battery to be replaced this time.

[0123] The battery cycle use strategy is used to perform battery replacement operation on the unmanned aerial vehicle to be replaced according to the updated battery state in the cabin.

[0124] The battery cycle use strategy is used to perform battery replacement operation on the unmanned aerial vehicle to be replaced according to the updated battery state in the cabin.

[0125] The battery cycle use strategy is used to perform battery replacement operation on the unmanned aerial vehicle to be replaced according to the updated battery state in the cabin.

[0126] The voltage and capacity of the lithium battery of the unmanned aerial vehicle in the static storage decay nonlinearly with time. The initial decay rate is fast, and then gradually slows down. This phenomenon is related to electrolyte decomposition, electrode side reaction and ion migration rate change. The self-discharge amount is affected by many factors such as state of charge, temperature, humidity, open circuit storage time and the like. The self-discharge compensation equation is used to quantitatively describe the lithium battery static power decay law, and the battery management is performed according to the power decay law: ; wherein, represents the remaining power of the battery, represents the initial state of charge; represents the temperature-dependent self-discharge coefficient; p represents the nonlinear characteristic of the self-discharge rate changing with time (typical value of lithium iron phosphate 0.5, ternary lithium 0.55); t represents the cumulative time of the battery after being separated from the charging and discharging state. ​

[0127] Introducing a temperature correction term : ; in, This represents the baseline value, which is the inherent decay rate when left to stand at 25°C (0.0003 for ternary lithium batteries). This corresponds to approximately 2.2% capacity loss per month, while lithium iron phosphate has a loss of 0.0001%. This corresponds to a capacity loss of approximately 0.7% per month; k is the material sensitivity coefficient (0.08 / ℃ for ternary lithium and 0.06 / ℃ for lithium iron phosphate).

[0128] Introducing aging correction factors: ; Where N represents the number of iterations; Indicates the cumulative storage time (in years).

[0129] By calculating and estimating the remaining battery power, the system comprehensively assesses whether the battery power is greater than the threshold set in the background to meet the task execution requirements. If the threshold is met, the battery will not be replaced and the system will directly enter the corresponding task process, which helps to shorten task preparation time and improve inspection efficiency.

[0130] Self-discharge compensation equation: .

[0131] Based on the battery degradation pattern, during battery monitoring, batteries with charge levels below a threshold are replaced when performing tasks. During prolonged periods of inactivity, batteries are automatically replaced when their charge is too low. In this embodiment, the cycle count of all batteries in the battery compartment is monitored. When the cycle count exceeds a warning threshold (e.g., 200 times), an alarm is triggered, prompting the system to perform maintenance, inspection, or replacement of the battery.

[0132] This embodiment can adaptively adjust the temperature inside the battery compartment to maintain the battery at a suitable storage temperature. It can also manage the battery pack's charge level based on battery mode, ensuring the batteries are stored at appropriate charge levels, which helps extend battery life. Furthermore, it manages battery usage evenly based on the number of battery cycles during missions, which helps reduce maintenance costs.

[0133] In the process of electric power unmanned aerial vehicle inspection, the current multi-unmanned aerial vehicle inspection scheduling scheme only focuses on task allocation and path planning under normal circumstances. Once an exception occurs, the task may be interrupted or cannot be completed on time due to the lack of effective task re-allocation and path re-planning mechanism, which seriously affects the inspection efficiency and quality. In order to solve this problem, in one or more embodiments, all airports in a set region are abstracted as nodes of a graph, an airport correlation network is constructed according to a backup relationship threshold corresponding to the airport density, and a graph connectivity vector is calculated according to the airport correlation network, and then the set region is divided into sub-regions; the airports in the same sub-region are backup airports for each other.

[0134] In the embodiment of the application, the density of large-scale airports is calculated by KNN algorithm, and hierarchical processing is implemented according to the density difference. For example, for a high-density region, a range is delimited with 10km as a limit, so that the airports within 10km apart form a backup relationship network; for a low-density region, the range is expanded to 15km to ensure that the airports within 15km are backup for each other. At the same time, the graph connectivity vector method is used to divide the airports into sub-regions, and a structured spatial layout system is constructed. Specifically, the process of airport density division based on KNN algorithm is as follows: Data preparation: collect the geographic coordinates (longitude , latitude ) information of each airport, and construct an airport position data set , where is the total number of airports.

[0135] Density calculation: the density of each airport is calculated by KNN algorithm. A suitable value (k=6 in this scheme) is selected, the average distance of the airport and other nearest neighbor airports is calculated, and the formula is as follows: ; Where represents the set of nearest neighbor airports of airport . The smaller the average distance , the more densely the airports around the airport are distributed, that is, the higher the density; otherwise, the lower the density.

[0136] Density classification: the density of the airport is classified according to the calculated average distance . A suitable distance threshold , is set to divide the airport into a high-density region, a low-density region, and a backup relationship is constructed. ​Get all The average.

[0137] The process of sub-region partitioning based on graph connected vectors is described in detail below: 1. Construct an airport graph model: Abstract all airports as nodes V in a graph G = (V, E), where each node... Each airport corresponds to a specific airport. If the distance between two airports is within a set backup relationship threshold (within 10km in high-density areas or within 15km in low-density areas), then an undirected edge is added between the corresponding nodes to form an edge set E, thereby constructing an airport association network.

[0138] 2. Calculate the graph connectivity vector: For each node in graph G Calculate its graph connected vectors The formula is as follows: ; in, This is the damping coefficient (value 0.8), used to balance the influence gained from adjacent nodes and the impact of random jumps; Represents a node The number of neighboring nodes; The total number of nodes; Let be the number of iterations, when The iteration stops when the value converges, and the final graph connectivity vector of each node is obtained.

[0139] 3. Sub-region partitioning: Based on the calculated graph connectivity vectors... K-Means clustering is used to group airports with similar graph connectivity vectors into the same sub-region. Airports in the same sub-region serve as backup airports for each other.

[0140] 4. Redundant Airport Deployment: Low-density areas have few airports, and flight scheduling disruptions may affect mission execution. Therefore, each sub-area in the low-density area will have one additional charging airport. This airport will not house drones, will not perform inspection tasks, and will only be responsible for charging in emergency situations.

[0141] When a drone is performing an inspection task and receives a new emergency task, it determines whether the target of the emergency task is currently being inspected. If so, the task will not be issued again. Otherwise, search the adjacent airport table of the airport, issue the task to the airport's backup airport, and other airports calculate whether the remaining flight time of the aircraft meets the time required for the task. Select the airport closest to the inspection target from the airports with the required remaining flight time and issue the task to it.

[0142] When the UAV encounters a power limit abnormal situation and the inspection task is not completed, if the current power of the UAV cannot return to the original airport, the remaining task time is calculated, a task is issued to the standby airport of the airport, other airports calculate whether the remaining flight time of the aircraft meets the required time of flying to the task point + the remaining task time + the return time to the original airport, from the airports whose remaining flight time meets the requirements, the airport closest to the interruption position is selected, and the task is continued at the airport. The UAV without power charges at the airport.

[0143] When the UAV encounters a power limit abnormal situation and the inspection task is completed, in a set high-density area, it is judged whether the current power of the UAV can return to the original airport. If yes, the original airport is returned. Otherwise, a task is issued to the standby airport of the airport, other airports calculate whether the remaining flight time of the aircraft meets the required time of flying to the original airport, from the airports whose remaining flight time meets the requirements, the airport closest to the current UAV is selected and the UAV flies to the airport for charging, and the UAV of the airport closest to the current UAV flies to the original airport.

[0144] When the UAV encounters a power limit abnormal situation and the inspection task is completed, in a set high-density area, it is judged whether the current power of the UAV can return to the original airport. If yes, the original airport is returned. Otherwise, it is judged whether the current power of the UAV can fly to a charging airport. If yes, the charging airport is flown to for charging, and after the UAV is fully charged, the original airport is returned. Otherwise, a task is issued to the standby airport of the airport, other airports calculate whether the remaining flight time of the aircraft meets the required time of flying to the original airport, from the airports whose remaining flight time meets the requirements, the airport A closest to the current UAV is selected, the current UAV flies to the airport A for charging, and the UAV of the airport A flies to the original airport.

[0145] When the UAV encounters a power limit abnormal situation and the inspection task is not completed, it is judged whether the current power of the UAV can return to the original airport. If yes, the original airport is returned, the task interruption point is saved, and the remaining task time is calculated. A task is issued to the standby airport of the airport, other airports calculate whether the remaining flight time of the aircraft meets the required time of flying to the task point + the remaining task time + the return time to the respective airport, and from the airports meeting the requirements, the airport closest to the interruption point is selected to continue the task.

[0146] In other embodiments, a multi-agent reinforcement learning (MARL) algorithm is used to train the path conflict avoidance decision of multiple UAVs in the inspection process, so that the UAV cluster can autonomously learn and form a conflict avoidance strategy in a dynamic environment. In this way, the safety and smoothness of multi-machine cooperative operation can be improved.

[0147] In this embodiment, a QMIX algorithm of centralized training and decentralized execution is adopted. Each UAV maintains a local Q network , output the value of each action in the current state. The local Q value is aggregated into the global Q value by the mixing network , coordinate multi-agent strategy. Train the network by minimizing the loss function: ; where, represents the state, represents the action, is the reward function, is the network parameter, value 0.9; state, action at the next training step in the network training process, is the target network parameter.

[0148] The detailed design is as follows: (1) State including self-state, neighbor-state: Self-state: three-dimensional coordinates , task state (take-off / landing / in-flight / task), remaining power .

[0149] Neighbor state: relative coordinates and velocity vector of all neighboring aircraft within 50 meters , where is the neighbor index.

[0150] (2) Action including 7 discrete actions, each action corresponding to 1 meter displacement: Horizontal avoidance: left yaw, right yaw, i.e. translate 1 meter in the horizontal direction.

[0151] Vertical avoidance: climb, descend, i.e. translate 1 meter in the vertical direction.

[0152] Speed adjustment: acceleration, deceleration (adjust cruise speed, equivalent displacement 1 meter per action).

[0153] Maintain the current state: maintain the original heading and speed.

[0154] (3) Reward including collision penalty, distance gain, task delay penalty, power saving reward; Collision penalty: if the distance to neighboring aircraft or obstacles is meters, reward ; where, is the safety distance.

[0155] Distance gain: if the distance is increased by avoidance, reward where , is a constant coefficient. represents the distance to the previous and adjacent machine or obstacle.

[0156] Task delay penalty: task delay caused by avoidance seconds, reward wherein , is a constant coefficient.

[0157] Power saving reward: reward wherein encourages low-energy flight; , are the initial power and the current power, respectively.

[0158] The present application realizes efficient management of large-scale unmanned aerial vehicle clusters, rapid response to abnormal situations and continuous execution of tasks by fusing spatial region division algorithms, innovative scheduling rules and intelligent decision-making training technologies, while improving the autonomous processing capability of path conflicts during multi-unmanned aerial vehicle cooperative operation, ensuring the stability and safety of the inspection task under complex working conditions. Embodiment two As shown in Figure 6 , the embodiment provides a power unmanned aerial vehicle multi-modal full-autonomous inspection system, which specifically comprises the following modules: An inspection task issuing module 601 is configured to utilize multi-agent cooperative processing of power inspection task instructions within a regional grid range and issue the instructions to unmanned aerial vehicles of corresponding airports; A modal inspection data sensing module 602 is configured to control the unmanned aerial vehicles to fly and autonomously sense the dynamic environment of the inspection area, autonomously query the target of interest, guide the unmanned aerial vehicles to autonomously inspect the unknown environment, and sense multi-modal inspection data of the target of interest; A multi-modal inspection data fusion module 603 is configured to cross-modally calibrate the multi-modal inspection data of the target of interest, extract the spatio-temporal joint features of each modality after calibration, combine the modality weights matched with the dynamic environment, and then obtain the fusion features by weighting; A health condition evaluation module 604 is configured to evaluate the health condition of the target of interest based on the fusion features by using a multi-modal large model; An active prevention decision module 605 is configured to construct a correlation matrix based on the correlation of each modality, update a preset knowledge base after secondary correction of the correlation matrix, and obtain an active prevention decision of the target of interest driven by multi-modal fusion and correlation degree in combination with the health condition evaluation result of the target of interest.

[0159] It should be noted that the modules in the embodiment of the present application have the same implementation process as the steps in the above-mentioned embodiment one, and will not be repeated here.

[0160] Embodiment three As Figure 7 shown, the embodiment provides a power unmanned aerial vehicle multi-modal full autonomous inspection system, comprising: A data sensing unit 701 is carried on the unmanned aerial vehicle, and is used for autonomously sensing multi-modal inspection data of a dynamic environment and an interesting target. A data processing unit 702 comprises a memory, a processor, and a program stored in the memory and executable on the processor, and the processor implements the steps in the above power unmanned aerial vehicle multi-modal full autonomous inspection method when executing the program.

[0161] In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing Figure 1 the method shown. In such embodiments, the computer program can be downloaded and installed from a network by a communication part, and / or installed from a detachable medium. When the computer program is executed by a central processing unit, various functions defined in the device of the present application are executed.

[0162] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0163] The above is only the preferred embodiment of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A power unmanned aerial vehicle multi-modal full autonomous inspection method, characterized in that, The method comprises the following steps: using multi-agent to process power inspection task instructions in the regional grid range and issue the instructions to the corresponding unmanned aerial vehicle (UAV); controlling the UAV to fly and autonomously perceive the dynamic environment of the inspection area, to autonomously query the target of interest, guide the UAV to autonomously inspect the unknown environment, and perceive the multi-modal inspection data of the target of interest; cross-modal calibration is performed on the multi-modal inspection data of the target of interest, spatio-temporal joint features of each modality after calibration are extracted, and then the fusion features are obtained by weighting in combination with the modality weights matched with the dynamic environment; based on the fusion features, a multi-modal large model is used to evaluate the health status of the target of interest; according to the correlation of each modality, a correlation matrix is constructed, and after secondary correction, the preset knowledge base is updated, and then in combination with the evaluation result of the health status of the target of interest, an active prevention decision of the target of interest driven by multi-modal fusion and correlation is obtained.

2. The power drone multi-modal fully autonomous inspection method of claim 1, wherein, the correlation between modalities is calculated based on the spatio-temporal joint features, and the modality weights matched with the dynamic environment are calculated in combination with the noise level of the dynamic environment and the data acquisition sensor of each modality, and the expression is: ； wherein, denotes the temperature coefficient; the overall score ; the noise level of the sensor , is the noise standard deviation, is the sensitivity coefficient; the inter-modal correlation score , m denotes the total number of modalities, is the Pearson correlation coefficient of modalities i and j; the scene prior factor ; is the indicator function, is the scene gain coefficient. 3.The power unmanned aerial vehicle multi-modal full-autonomous inspection method of claim 1, wherein, the multi-modal inspection data of the target of interest is calibrated by a cross-modal calibration model; the training of the cross-modal calibration model adopts a weighted sum of local feature alignment loss and global context matching loss, and the cross-modal calibration model learns model parameters and adjustment factors for weighting through meta-learning. 4.The power unmanned aerial vehicle multi-modal full-autonomous inspection method of claim 3, wherein, The weighted sum is: ; wherein, is an adjustment factor; local feature alignment loss , represents a set of local features extracted by modality A, represents a set of local features extracted by modality B, N represents a local feature dimension extracted by modality A, and M represents a local feature dimension extracted by modality B, is a cosine similarity, is a constant coefficient; global context matching loss , is a global feature of modality A, is a global feature of modality B; represents a global feature of modality B of a negative sample, and K represents a number of negative samples.

5. The power drone multi-modal fully autonomous inspection method of claim 1, wherein, the elements in the correlation matrix are: ； wherein, denotes the spatio-temporal joint feature of the modality and the attention weight between the spatio-temporal joint feature of the modality S denotes the total number of modalities, d is the feature dimension; W Q and W K is the projection matrix.​​​ 6. The electric power unmanned aerial vehicle multi-modal fully autonomous inspection method of claim 1, wherein, the process of updating the preset knowledge base is: a causal graph is generated by using a causal discovery algorithm in combination with the original correlation matrix and the historical failure data of the target of interest; the original correlation matrix is filtered to obtain a first modified correlation matrix; the consistency between modalities is enhanced by contrast learning based on the first modified correlation matrix, and the preset low-confidence correlation degrees in the contrast learning are removed by using the causal graph to obtain a second modified correlation matrix; the preset knowledge base is updated by using the second modified correlation matrix.

7. The electric power unmanned aerial vehicle multi-modal fully autonomous inspection method of claim 6, wherein, A contrastive loss function employed in a contrastive learning process is: ； wherein, is the cosine similarity, is a constant coefficient; is the number of modalities mapped to the shared space i and j their features; denotes an element in the once revised affinity matrix; H is the number of modalities once revised.

8. The power drone multi-modal fully autonomous inspection method of claim 1, wherein, the process of guiding the UAV to autonomously inspect the unknown environment is: obtaining UAV multi-source perception data of the unknown environment and extracting obstacle information therefrom to plan a preliminary global path from the current point to the target point; dynamically adjusting the weight of the UAV multi-source perception data according to the current environmental information of the unknown environment and in combination with a preset fuzzy rule base, and performing weighted fusion on the UAV multi-source perception data, and then combining the current IMU data to jointly optimize and calculate the current pose of the UAV; based on the current pose of the UAV calculated by optimization, when flying along the preliminary global path, whether there is a new obstacle is determined according to the real-time UAV multi-source perception data, and the preliminary global path is optimized to realize the autonomous inspection of the UAV in the unknown environment. 9.The power unmanned aerial vehicle multi-modal full-autonomous inspection method of claim 8, wherein, The D* Lite algorithm is used to plan a preliminary global path from the current point to the target point. The D* Lite determines the priority of node update through a key value The expression is: ； wherein, is the current estimated cost from node s to the goal; is a more forward looking cost estimate based on successor nodes of node s ; is a heuristic function from the current location of the drone to node s, ensuring that the search prioritizes updating nodes that are most important to the current location of the drone; is an accumulator that records the sum of the heuristic cost added by all start moves since the algorithm was started.

10. The electric power unmanned aerial vehicle multi-modal fully autonomous inspection method of claim 1, wherein, when the UAV encounters an abnormal power limit situation, the inspection task is not completed, and the current power of the UAV cannot return to the original airport, the nearest airport to the position where the inspection task is interrupted is searched, the battery is replaced, and then the UAV continues to perform the inspection task.

11. The power drone multi-modal fully autonomous inspection method of claim 10, wherein, During the battery swapping operation, the robot searches for nearby grid positions from a set starting position, selecting the grid with the smallest estimated cost function value as the next starting point. This selected grid is called the current node. The process is repeated, searching for the next grid with the smallest estimated cost function value as the new current node, and so on, until the current node reaches the set ending position, thus planning the obstacle avoidance path for the battery swapping robot arm. The estimated cost function... for: ； ； n is the current node, g(n) is the actual cost function from the start point to the current node, and h(n) is the heuristic function from the current node to the end node; is the corner contact penalty function; k is the penalty intensity coefficient, is the steepness factor, is the minimum safety distance, is the safety distance threshold.

12. The power drone multi-modal fully autonomous inspection method of claim 10, wherein, in the process of searching for the nearest airport to the position where the inspection task is interrupted and performing the battery replacement, the battery is also managed and dispatched, and the process is as follows: Acquire the real-time temperature of the battery cabin of the battery swap machine nest, the state of the batteries in the cabin, and the working mode of the battery swap machine nest; According to the state of the batteries in the cabin and the real-time temperature, the overall temperature in the cabin and the temperature of the abnormal batteries are regulated; According to different working modes, the charging management of the batteries in the battery swap machine nest is carried out, and the state of the batteries in the cabin is updated in real time; Using the battery recycling strategy, the unmanned aerial vehicle with the battery to be replaced is operated according to the updated state of the batteries in the cabin, wherein the battery recycling strategy is that when the difference between the recycling times of the batteries is within the recycling threshold range, the battery closest to the gripper and meeting the power requirement is selected for replacement, and when the difference between the recycling times of the batteries exceeds the recycling threshold range, the batteries meeting the power requirement are evenly scheduled according to the recycling times.

13. The power drone multi-modal fully autonomous inspection method of claim 1, wherein, The power unmanned aerial vehicle multi-modal full-autonomous inspection method further comprises: abstracting all airports in a set region as nodes of a graph, constructing an airport correlation relationship network according to a standby relationship threshold corresponding to airport density, and calculating a graph connectivity vector according to the airport correlation relationship network, and then dividing the set region into sub-regions; wherein the airports in the same sub-region are mutual standby airports; scheduling the unmanned aerial vehicles according to the principles of task priority and task connection execution, and when the unmanned aerial vehicles encounter power limitation abnormal conditions, searching for the airport closest to the position where the inspection task is interrupted based on the divided sub-regions, and continuing to execute the inspection task by the unmanned aerial vehicle in the airport.

14. A power unmanned aerial vehicle multi-modal fully autonomous inspection system, characterized in that, It comprises: An inspection task issuing module for processing power inspection task instructions in a regional grid range by using multi-agent collaboration and issuing the instructions to the unmanned aerial vehicles of the corresponding airports; A modal inspection data sensing module for controlling the unmanned aerial vehicles to fly and autonomously sense the dynamic environment of the inspection region, autonomously query the interest target, guide the unmanned aerial vehicles to autonomously inspect the unknown environment, and sense the multi-modal inspection data of the interest target; A multi-modal inspection data fusion module for cross-modal calibration of the multi-modal inspection data of the interest target, extraction of the spatio-temporal joint features of each modality after calibration, combination of the modality weights matching the dynamic environment, and then weighted fusion of the fusion features; A health condition evaluation module for evaluating the health condition of the interest target based on the fusion features by using a multi-modal large model; An active prevention decision module for constructing a correlation matrix according to the correlation of each modality, performing secondary correction on the correlation matrix, updating a preset knowledge base, combining the health condition evaluation result of the interest target, and obtaining the active prevention decision of the interest target driven by multi-modal fusion and correlation degree.

15. An electric power unmanned aerial vehicle multi-modal fully autonomous inspection system, characterized in that, It comprises: A data sensing unit carried on the unmanned aerial vehicle, for autonomously sensing the multi-modal inspection data of the dynamic environment and the interest target; A data processing unit comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps of the power unmanned aerial vehicle multi-modal full-autonomous inspection method according to any one of claims 1-13 when executing the program.

Images