Multi-modal full autonomous inspection method and system for power unmanned aerial vehicle

Abstract

The present application belongs to the field of electric power unmanned aerial vehicle inspection, and provides an electric power unmanned aerial vehicle multi-modal full autonomous inspection method and system to solve the problem that traditional electric power unmanned aerial vehicle inspection is not applicable to unknown environments. The electric power unmanned aerial vehicle multi-modal full autonomous inspection method comprises: using multi-agent to cooperatively process electric power inspection task instructions within a regional grid range; guiding the unmanned aerial vehicle to conduct autonomous inspection of unknown environments; cross-modal calibration of multi-modal inspection data of an interest target; after calibration, the spatio-temporal joint features of each modality are combined with each modality weight matched with a dynamic environment to obtain a fusion feature; using a multi-modal large model to evaluate the health condition of the interest target; updating a preset knowledge base, combining the health condition evaluation result of the interest target to obtain an interest target active prevention decision jointly driven by multi-modal fusion and correlation degree. It can enable the unmanned aerial vehicle to conduct autonomous inspection in unknown environments and actively provide corresponding decisions for the stable operation of the inspection target.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of electric power unmanned aerial vehicle inspection, and particularly relates to an electric power unmanned aerial vehicle multi-modal full autonomous inspection method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the expansion of the scale of power grid equipment year by year and the increasing complexity of the operating environment, huge economic losses are caused by power outage accidents every year, and it is urgent to accurately perceive the operating state of power grid equipment and actively provide risk warning. Unmanned aerial vehicle inspection is an indispensable technical means for fine operation and maintenance of power grid equipment. In the traditional unmanned aerial vehicle inspection mode, the unmanned aerial vehicle is carried to the scene by manual operation for inspection, and the data is collected, which is labor-intensive and the operation quality is affected by the experience of personnel, and is uneven. The contradiction between fine inspection of power grid equipment and the traditional unmanned aerial vehicle inspection mode is becoming more and more prominent.

[0004] The traditional unmanned aerial vehicle inspection mode needs to manually re-map and plan the route every time the environment changes. In the inspection process, the static obstacles are generally stopped, and the processing capacity for dynamic obstacles encountered in the flight process is insufficient. Over-reliance on a single main sensor such as vision or laser radar can only inspect specific areas or specific targets, and is easily invalid in vision degradation or laser radar degradation scenarios. The flight process relies on the route and signal, and when the positioning signal in the flight process is poor, it will affect the flight safety, cannot fly in an environment without prior information or can rely on external positioning signals, and is not suitable for power inspection in unknown environments.

[0005] In addition, the traditional unmanned aerial vehicle inspection mode has poor data intelligent recognition effect and needs a large amount of manual checking. There is a lack of grid-based multi-specialty collaborative scheduling means, and the resource utilization rate is low. The detection scheme based on visible light, infrared and other multi-modal data fusion in the prior art usually uses multi-modal fusion methods such as simple splicing and fixed weight summation, and combines target detection technology to identify power target faults. Due to the influence of various factors on the reliability of multi-modal data, the fixed weight fusion method will introduce invalid information to limit the feature quality. Moreover, only image and text are fused, and device operating parameters and environmental data are not integrated, resulting in one-sided information and insufficient use of multi-modal data, which ultimately affects the accuracy of the inspection recognition result and cannot actively provide corresponding decisions for the stable operation of the inspection target. SUMMARY

[0006] In order to solve the above technical problems, the present application provides an electric power unmanned aerial vehicle multi-modal full autonomous inspection method and system, which can enable the unmanned aerial vehicle to perform autonomous inspection in an unknown environment and actively provide corresponding decisions for the stable operation of the inspection target.

[0007] To achieve the above object, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a power unmanned aerial vehicle multi-modal full autonomous inspection method.

[0009] In one or more embodiments, a power unmanned aerial vehicle multi-modal full autonomous inspection method comprises:

[0010] A multi-agent is used to cooperatively process power inspection task instructions within a regional grid range and issue them to unmanned aerial vehicles at corresponding airports.

[0011] The unmanned aerial vehicles are controlled to fly and autonomously perceive the dynamic environment of the inspection area to autonomously query the target of interest, guide the unmanned aerial vehicles to autonomously inspect the unknown environment, and perceive multi-modal inspection data of the target of interest.

[0012] The multi-modal inspection data of the target of interest is cross-modally calibrated, the spatio-temporal joint features of each modality after calibration are extracted, and then the fusion features are obtained by weighting in combination with the modality weights matched with the dynamic environment.

[0013] Based on the fusion features, a multi-modal large model is used to evaluate the health status of the target of interest.

[0014] According to the relevance of each modality, a correlation matrix is constructed and then updated after being twice modified, and the preset knowledge base is updated, and then the active prevention decision of the target of interest driven by multi-modal fusion and correlation is obtained in combination with the evaluation result of the health status of the target of interest.

[0015] The second aspect of the present application provides a power unmanned aerial vehicle multi-modal full autonomous inspection system.

[0016] In one or more embodiments, a power unmanned aerial vehicle multi-modal full autonomous inspection system comprises:

[0017] An inspection task issuing module is used to cooperatively process power inspection task instructions within a regional grid range by a multi-agent and issue them to unmanned aerial vehicles at corresponding airports.

[0018] A modality inspection data perception module is used to control the unmanned aerial vehicles to fly and autonomously perceive the dynamic environment of the inspection area to autonomously query the target of interest, guide the unmanned aerial vehicles to autonomously inspect the unknown environment, and perceive multi-modal inspection data of the target of interest.

[0019] A multi-modal inspection data fusion module is used to cross-modally calibrate the multi-modal inspection data of the target of interest, extract the spatio-temporal joint features of each modality after calibration, and then obtain the fusion features by weighting in combination with the modality weights matched with the dynamic environment.

[0020] a health condition evaluation module configured to evaluate the health condition of the target object by using a multi-modal large model based on the fused features;

[0021] an active prevention decision module configured to construct a correlation matrix according to the correlation of each modality, update a preset knowledge base after secondary correction of the correlation matrix, and obtain an active prevention decision of the target object driven by multi-modal fusion and correlation in combination with the health condition evaluation result of the target object.

[0022] In one or more embodiments, a power unmanned aerial vehicle (UAV) multi-modal full autonomous inspection system comprises:

[0023] a data sensing unit mounted on the UAV and configured to autonomously sense multi-modal inspection data of a dynamic environment and a target object;

[0024] a data processing unit comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps of the power UAV multi-modal full autonomous inspection method as described above when executing the program.

[0025] A third aspect of the present application provides a computer-readable storage medium.

[0026] A computer-readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the steps of the power UAV multi-modal full autonomous inspection method as described above.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] (1) The present application innovatively proposes a power UAV multi-modal full autonomous inspection technology, designs a real-time sensing and path planning method for UAV flight operation environment in unknown environment, develops a grid-based power swap airport, constructs a multi-target multi-scale multi-modal fusion feature model of power equipment, proposes a causal reasoning hidden danger decision analysis and early warning method, designs a multi-agent collaboration and dynamic resource scheduling model, realizes grid-based collaborative operation regulation of multi-nest multi-machine multi-task in a large area, and the UAV adaptively carries out the inspection operation, intelligently completes the image defect diagnosis and decision analysis after the inspection, constructs a power UAV "autonomous inspection-intelligent analysis-collaborative regulation" full autonomous operation system, solves the contradiction between personnel shortage and rapid increase of inspection demand, and improves the intelligent and unmanned inspection quality and efficiency.

[0029] (2) The application innovatively proposes a power equipment multi-modal data dynamic weighted fusion and correlation degree combined driving preventive decision method, constructs a multi-source data dynamic calibration deep fusion and equipment preventive decision generation model, realizes multi-target multi-scale defect accurate diagnosis and early warning decision analysis through a meta-learning framework and a spatio-temporal joint feature model, reduces the workload of manual review of post-inspection data analysis, and improves the abnormal defect detection accuracy and the reliability of equipment preventive decision.

[0030] (3) The application innovatively proposes a large-range multi-nest multi-machine multi-task grid regulation technology, constructs an airport-equipment relationship graph model, takes single-flight operation inspection benefit maximization as the goal, realizes collaborative regulation of inspection tasks and operation unmanned aerial vehicles, and improves the unmanned aerial vehicle inspection operation efficiency and safety.

[0031] (4) The application innovatively proposes an unmanned aerial vehicle intelligent inspection technology in unknown environment, realizes autonomous exploration and safe flight of the unmanned aerial vehicle in the unknown environment through multi-modal data fuzzy matching and trajectory prediction technology, self-adaptive dynamic adjustment of multi-source sensing data fusion weight, generation of global inspection path and dynamic prediction of inspection trajectory.

[0032] (5) The application innovatively proposes an unmanned aerial vehicle airport battery replacement method and system, designs an unmanned aerial vehicle battery replacement control model based on a grid model, realizes rapid energy supply of the unmanned aerial vehicle after inspection and uninterrupted unmanned inspection. BRIEF DESCRIPTION OF DRAWINGS

[0033] The drawings accompanying the specification of the application form a part of the application and serve to further understand the application. The schematic embodiments of the application and the description thereof are used to explain the application, and do not constitute an improper limitation on the application.

[0034] Figure 1 is a flowchart of the power unmanned aerial vehicle multi-modal full-autonomous inspection method in the embodiment one of the application;

[0035] Figure 2 is a process of guiding the unmanned aerial vehicle to perform autonomous inspection in unknown environment in the embodiment one of the application;

[0036] Figure 3 is a process of updating the preset knowledge base in the embodiment one of the application;

[0037] Figure 4 is a process of finding the airport closest to the position of interrupting the inspection task to perform the battery replacement operation in the embodiment one of the application;

[0038] Figure 5 is a process of managing the battery in the embodiment one of the application;

[0039] Figure 6is a structural schematic diagram of the power unmanned aerial vehicle multi-modal full autonomous inspection system in the second embodiment of the present application.

[0040] Figure 7 is a structural schematic diagram of the power unmanned aerial vehicle multi-modal full autonomous inspection system in the third embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as would be commonly understood by one of ordinary skill in the art to which the present application belongs.

[0043] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the present specification, they indicate the presence of a feature, step, operation, device, component and / or combinations thereof.

[0044] Embodiment One

[0045] Figure 1 is a flowchart of a power unmanned aerial vehicle multi-modal full autonomous inspection method according to an embodiment of the present application, as Figure 1 The power unmanned aerial vehicle multi-modal full autonomous inspection method of the present embodiment specifically includes the following steps:

[0046] Step S101: Utilize multi-agent to cooperatively process power inspection task instructions within the regional grid range and issue to the corresponding unmanned aerial vehicle of the airport.

[0047] In the specific implementation process of step S101, the process of utilizing multi-agent to cooperatively process power inspection task instructions is as follows:

[0048] The user's inspection task instruction is parsed into an inspection area determination subtask, an airport and unmanned aerial vehicle query subtask, and an inspection task list generation subtask;

[0049] The inspection resource management agent is called from the agent library to execute the inspection area determination subtask, the inspection area table is queried, and the inspection area is obtained;

[0050] The device management agent is called from the agent library to execute the airport and unmanned aerial vehicle query subtask, the airport table and the unmanned aerial vehicle table are queried, and the airport and the unmanned aerial vehicle therein to be executed for the inspection task are obtained;

[0051] The visual agent is called from the agent library to execute the inspection task order to generate a subtask, and the inspection area, the airport to be executed, and the unmanned aerial vehicle in the airport are displayed in the form of a task order to form an inspection task order to receive a confirmation instruction for the task order.

[0052] It should be noted that the inspection resource management agent, the device management agent, and the visual agent can be implemented using an existing large model.

[0053] The embodiment utilizes multi-agent division of labor and cooperation to decompose complex inspection tasks into specialized subtasks, and each agent focuses on its own field of expertise, thereby improving the overall response speed and processing accuracy.

[0054] Step S102: Control the unmanned aerial vehicle to fly and autonomously perceive the dynamic environment of the inspection area to autonomously query the target of interest, guide the unmanned aerial vehicle to perform autonomous inspection in the unknown environment, and perceive the multi-modal inspection data of the target of interest.

[0055] Specifically, the unmanned aerial vehicle of the corresponding airport is controlled to fly to a set position in the inspection area according to the inspection task order, and the dynamic environment of the position is autonomously perceived and the target of interest is autonomously queried.

[0056] In the specific implementation process, as shown in Figure 2 The process of guiding the unmanned aerial vehicle to perform autonomous inspection in the unknown environment is as follows:

[0057] Step S201: Obtain unmanned aerial vehicle multi-source perception data of an unknown environment and extract obstacle information therefrom to plan a preliminary global path from a current point to a target point.

[0058] In the unknown environment without prior environment information, the unmanned aerial vehicle multi-source perception data can be obtained by using a heterogeneous sensor array (solid-state laser radar, thermal imaging camera, depth camera, millimeter wave radar, ultrasonic sensor). The sensor array is shown in Table 1:

[0059] Table 1 Sensor array

[0060]

[0061] The D*Lite algorithm is used to plan a preliminary global path from the current point to the target point.

[0062] The D*Lite algorithm is a backward search algorithm that searches from the target point to the current position of the unmanned aerial vehicle. The core of the algorithm is to assume that the unknown area is all free space, and based on this, incrementally implement path planning to find the shortest distance from the target point to each node by minimizing the rhs value.

[0063] D*Lite uses a key-value pair to determine the priority of node updates. It consists of two parts: and composition:

[0064] ;

[0065] in, The current estimated cost from node s to the target point; For the successor node of node s A more forward-looking cost estimate; To start from the current location of the drone The heuristic function for node s ensures that the search prioritizes updating the nodes that are most important to the current drone position; It is an accumulator that records the sum of the heuristic costs increased by all starting point moves since the algorithm began running; initially... Whenever starting from the previous point Moving to a new starting point At that time, calculate the heuristic cost of this step. and added to superior, .

[0066] Among them, based on the successor node of node s The more forward-looking cost estimate is expressed as: Where Succ(s) is the set of successor nodes of node s. From s to The cost, if If the node s is consistent, it means that the value of g(s) is up-to-date and reflects the optimal cost to reach the goal. This means finding a new path from s to the target point with a lower cost than the previous g(s), setting... To update; if This means that the path cost previously used to compute rhs(s) has increased, and the cost of s needs to be re-evaluated.

[0067] Step S202: Based on the current environmental information of the unknown environment and combined with the preset fuzzy rule base, dynamically adjust the weight of the UAV multi-source perception data, perform weighted fusion of the UAV multi-source perception data, and then combine it with the current IMU data to jointly optimize and calculate the current pose of the UAV.

[0068] In the specific implementation process, the weights of the UAV's multi-source perception data are dynamically adjusted based on a preset fuzzy rule base; the input of the fuzzy rule base is the fuzzy value of the current environmental information, and the output is the direction of adjustment of the weights of the UAV's multi-source perception data.

[0069] The calculation of the fusion data is based on a preset fuzzy rule base which defines the mapping relationship of the reliability of each sensor under different environmental conditions.

[0070] For example, the input variable set , wherein : environmental conditions, such as illumination conditions, weather conditions, space conditions, etc. measured by sensors. The output variable set , wherein : solid-state laser radar weight, : thermal imaging camera weight, : depth camera weight, : millimeter wave radar weight, : ultrasonic sensor weight; for each input variable , define its fuzzy set , : the kth fuzzy set of the ith input variable; for each output variable , define its fuzzy set , : the jth fuzzy set of the ith output variable.

[0071] Input variable fuzzification: illumination intensity membership function (trapezoidal function):

[0072] ;

[0073] wherein x: illumination intensity, a, b, c, d: parameters of the trapezoidal function, : the membership degree of the illumination intensity in the fuzzy set.

[0074] Rain and snow weather condition membership function (linear approximation-trapezoidal function):

[0075] ;

[0076] wherein y: precipitation, i: lower limit of precipitation, j: upper limit of precipitation.

[0077] Fog condition membership function (Z function):

[0078] ;

[0079] wherein z: visibility, m: upper limit of visibility, p: lower limit of visibility, n: function midpoint: .

[0080] Specifically, the process of dynamically adjusting the weight of the multi-source perception data of the unmanned aerial vehicle based on the preset fuzzy rule base is as follows:

[0081] ​For the current input, calculate its matching degree to each fuzzy set in the antecedent of each rule,

[0082] Using Takagi-Sugeno type fuzzy model, for each rule, its total activation strength is the aggregation of all the matching degrees of the antecedent conditions, using the product operator to calculate the activation rule strength,

[0083] Using the ratio of the product of the aggregation of all the matching degrees of the consequent and the activation rule strength and the accumulation of all the activation rule strengths, the weight of the UAV multi-source perception data is calculated.

[0084] For example, the fuzzy rule base: Each rule is in the form of:

[0085] if is and is and...then is , is ,...;

[0086] For the current input , calculate its matching degree to each fuzzy set in the antecedent of each rule, where : the matching degree of the input value to the fuzzy set , : the matching degree of the input to the th antecedent condition in the rule .

[0087] Using Takagi-Sugeno type (TS type) fuzzy model, for each rule , its total activation strength is the aggregation of all the matching degrees of the antecedent conditions, using the product operator to calculate the activation rule strength, .

[0088] The consequent of the TS type rule is a function, the rule form: if is and...and is then , using the first-order TS model: where : the constant term of the tth rule, : the weight of the input variable in the tth rule, : The aggregation of the matching degrees of all conditions in the consequent.

[0089] Weight of UAV multi-source perception data: ;

[0090] According to the environmental conditions, the sensor weight is dynamically adjusted, the multi-source perception data of the UAV is fused according to the adjusted UAV multi-source perception data, and the fused multi-source data and the IMU data are jointly optimized to calculate a most reliable UAV pose.

[0091] Step S203: Based on the current pose of the UAV optimized by calculation, when flying along the preliminary global path, whether there is a new obstacle is judged according to real-time UAV multi-source perception data, and the preliminary global path is optimized to realize the autonomous inspection of the UAV in the unknown environment.

[0092] During the flight of the UAV according to the global planning path, the motion state of each dynamic object is estimated and predicted through real-time sensor data. The state vector of the moving object usually includes position and velocity: , wherein is a three-dimensional position, is a three-dimensional velocity.

[0093] For each tracked dynamic obstacle, an interacting multiple model filter (IMM) is established;

[0094] The filtering results of all interacting multiple model filters are fused to obtain the overall optimal estimation and uncertainty measure of the obstacle state at the current time, and based on the optimal estimation and model probability at the current time, the future motion trajectory of the dynamic obstacle is predicted.

[0095] The IMM includes a uniform speed model, a uniform acceleration model, and a coordinated turning model. The core of the IMM is to calculate the posterior probability of each model , which is used as the weight.

[0096] First, for the current model m, the mixing probability of the model n at the previous time is calculated:

[0097] ;

[0098] , wherein represents the contribution weight of the estimation of the model n at the previous time to the model m at the current time, k: current time, Q: total number of models; : probability of the model n at the previous time; : probability of the model m at the current time; is a normalization constant representing the predicted prior probability of the model m.

[0099] The mixed initial state and covariance are calculated for each model m, and the initial state: , Covariance: , is the estimation uncertainty of model n at the last time step, representing the internal error of each model. is the outer product of the deviation of the estimated value of model n and the mixed mean , representing the divergence between different models. Each model m starts with its mixed initial condition, independently performs the standard Kalman filtering process, and obtains the posterior probability at the current time step: , m and n are model indices, is the likelihood of model m, is the predicted mixed probability of model m. The filtering results of all models are fused to obtain the overall optimal estimation and uncertainty measure of the obstacle state at the current time step, and based on the optimal estimation and model probability at the current time step, the future trajectory of the obstacle is predicted.

[0100] The predicted dynamic obstacle trajectory is fed back to the D* Lite algorithm as temporary obstacle information. After receiving the information, D* Lite triggers the replanning process to generate a new path that can avoid both static obstacles and predicted dynamic obstacles.

[0101] The embodiment proposes a UAV inspection method in an unknown environment, which uses a multi-source perception data fusion scheme of the UAV, uses dynamic weight distribution, and adjusts the weight factor in real time according to environmental information. The most reliable UAV pose is calculated by optimizing the fused multi-source data and IMU data together, solving the problem that the UAV cannot fly in an environment without prior information or external positioning signals, and enabling preliminary detection of an unknown environment, thereby improving the environmental adaptability of UAV inspection.

[0102] Step S103: Cross-modal calibration is performed on the multi-modal inspection data of the target of interest, spatio-temporal joint features of each modality after calibration are extracted, and then each modality weight matched with the dynamic environment is combined to obtain the fused features through weighting.

[0103] In step S103, the multi-modal inspection data of the target of interest is calibrated by a cross-modal calibration model, and then time sequence feature extraction and spatio-temporal joint feature splicing operations are sequentially performed to obtain the spatio-temporal joint features of each modality.

[0104] The training of the cross-modal calibration model uses a weighted sum of local feature alignment loss and global context matching loss, and the cross-modal calibration model learns model parameters and adjustment factors for weighting through meta-learning training; the weighted sum is: ; wherein, is an adjustment factor; the local feature alignment loss , denotes a set of local features extracted from modality A, represents the local feature set proposed by modality B, N represents the dimension of the local feature extracted by modality A, M is the dimension of the local feature extracted by modality B, is a cosine similarity, is a constant coefficient; global context matching loss , is the global feature of modality A, is the global feature of modality B; represents the global feature of modality B of the negative sample, and K is the number of negative samples.

[0105] After the cross-modal calibration model adopts different branch encoders to extract the local features and global features of different modal data, the similarity matrix of the local features between different modal data is calculated to obtain a local similarity matrix, the similarity matrix of the global features between different modal data is calculated to obtain a global similarity matrix, based on the local similarity matrix and the global similarity matrix, the matching probability between different modal data is output through a classifier, and based on the matching probability, each modal data after cross-modal calibration is obtained through an attention-based decoder.

[0106] It should be noted here that the embodiment utilizes the spatio-temporal graph neural network model ST-GNN to extract time sequence features and spatial topological features, and those skilled in the art can also use other existing networks to extract time sequence features and spatial topological features.

[0107] Specifically, the inter-modality correlation is calculated based on the spatio-temporal joint features, and the inter-modality correlation is calculated based on the dynamic environment and the noise level of the data acquisition sensor of each modality.

[0108] wherein the inter-modality correlation of the dynamic environment is calculated as: ; represents a temperature coefficient; comprehensive score ; noise level of the sensor , is the noise standard deviation, is a sensitivity coefficient; inter-modality correlation score , m represents the total number of modalities, is the Pearson correlation coefficient of modality i and j; scene prior factor , is an indicator function, is a scene gain coefficient.

[0109] This embodiment proposes a meta-learning-driven cross-modal feature calibration, spatiotemporal joint feature extraction, and dynamic weighted fusion method. It improves cross-modal consistency through dual loss constraints, and on this basis, it automatically filters noise, weakens interference, and focuses on reliable modalities by dynamically adjusting weights and fusing features. This avoids invalid information from interfering with subsequent multimodal large models, improves feature quality, and thus improves the anomaly detection accuracy of the target of interest.

[0110] Step S104: Based on the fusion features, use a multimodal large model to assess the health status of the target of interest.

[0111] The multimodal large model here can be implemented using a multi-task prediction head designed for classification and regression tasks to obtain health type and health score.

[0112] (1) Global average pooling followed by Softmax is used to predict fault types:

[0113] ;

[0114] Features for the fusion of multimodal data; For average pooling layers; The weight matrix for the classification task; This is the bias term for the classification task.

[0115] (2) The fully connected layer outputs a scalar to predict the health score s:

[0116] ;

[0117] in, This represents the weight matrix for the regression task; This represents the bias term of the regression task; s∈(0, 1). When the output health score is >0.8, the device is in a healthy state; when the health score is in [0.4, 0.8), the device needs attention; when the health score is below 0.4, the device has a high risk of failure.

[0118] Step S105: Based on the correlation of each modality, construct the correlation degree matrix and make secondary corrections to it before updating the preset knowledge base. Then, combine the health status assessment results of the interest target to obtain the proactive prevention decision of the interest target driven by multimodal fusion and correlation degree.

[0119] Specifically, attention weights between spatiotemporal joint features of each modality are calculated based on a cross-modal attention mechanism, and used as the correlation degree to construct an initial correlation degree matrix. The initial correlation degree matrix is ​​then corrected a second time, and the preset knowledge base is updated.

[0120] In the embodiment, the attention weight between the spatio-temporal joint features of each modality is calculated based on a cross-modal attention mechanism, as a correlation degree to construct an initial correlation degree matrix, and an element in the matrix is: ; wherein, denotes the attention weight between the spatio-temporal joint feature of modality and the spatio-temporal joint feature of modality ; S denotes the total number of modalities, d is the feature dimension; W Q and W K is a projection matrix.

[0121] In the embodiment, as shown in Figure 3 , the process of updating the preset knowledge base is:

[0122] Step S301: using a causal discovery algorithm, combining the correlation degree matrix and the historical failure data of the interest target, to generate a causal graph.

[0123] The historical failure data of the interest target here includes but is not limited to visible light images, infrared images, vibration data, three-dimensional point cloud data, flight path files, weather data, defect data, defect elimination records, expert experience, etc.

[0124] Step S302: pseudo-correlation filtering is performed on the initial correlation degree matrix to obtain a first modified correlation degree matrix.

[0125] According to the rules of physical completeness, statistical consistency, knowledge base coverage, etc., pseudo-correlation filtering is performed, specifically considering dimensional consistency test, time shift independence test, confidence interval test and failure mode constraint, to obtain the purified correlation relationship:

[0126] (a) dimensional consistency test;

[0127] If the dimensions of modalities i and j and cannot be associated by a physical equation, they are forced to be zero:

[0128] ;

[0129] denotes the correlation weight of modality data i and modality data j.

[0130] (b) time shift independence test;

[0131] True correlation should satisfy the uniqueness of time shift mutual information peak:

[0132] ;​​

[0133] Typical threshold . denotes the modal i data at time t; denotes the modal j data at time t; denotes the modal j data at time t; I denotes the mutual information between and

[0134] (c) Bootstrap confidence interval;

[0135] 95% confidence interval of is calculated by resampling , if it contains zero, filter:

[0136] ;

[0137] denotes the lower limit of the 95% confidence interval of calculated by resampling denotes the upper limit of the 95% confidence interval of calculated by resampling

[0138] (d) Fault mode constraint;

[0139] Construct a whitelist matrix that allows association :

[0140] ;

[0141] Based on the purified association, estimate the causal utility, estimate the causal effect strength through the SEM structural equation model, and finally output the causal diagram and the corrected association matrix R* (i.e. the first corrected association).

[0142] Step S303: Strengthen the consistency between the modes by contrast learning on the first corrected association matrix, and remove the preset low-confidence association in contrast learning using the causal diagram to obtain the second corrected association matrix; wherein the contrast loss function used in the contrast learning process is:

[0143] ;

[0144] wherein, is the cosine similarity, is a constant coefficient; is the feature of the modal i and j mapped to the shared space; denotes an element in the once revised correlation matrix; H is the number of modes after the first revision.

[0145] The input is a pair of multi-modal positive samples (Fi, Fj+) and negative samples (Fi, Fj-). The positive sample pair is multi-modal data at the same time on the same device (such as image-vibration pairs), and the negative sample pair is data of different devices or different states. First, map the multi-modal features to a shared space: ;

[0146] wherein, denotes the feature of the n-th modal data; denotes the multi-modal data mapped to the shared space; denotes a d-dimensional real number space.

[0147] Then, through the contrast loss function, the positive sample distance is minimized, and the negative sample distance is maximized.

[0148] In this embodiment, the second revision adopts the way of contrast learning. The correlation relationship after the first revision participates in the generation of the similarity matrix through the contrast loss function. The main participation way is to embed the correlation relationship after the first revision as a weighted controller into the contrast loss function to directly act on the loss calculation, realizing two-level control.

[0149] Step S304: updating the pre-knowledge base by using the correlation matrix after the second revision.

[0150] For example, based on the data of power equipment design drawings, nameplate parameters, material properties, etc., a device ontology library is constructed; based on the power industry standards (DL / T 664, IEC 60567, etc.) and manufacturer technical manuals, a defect rule library is constructed, and new defect modes will automatically trigger rule generation to realize dynamic updating of the rule library; knowledge extraction is performed from historical maintenance reports to construct an expert experience library; historical case data is structured to construct a historical case library; finally, the above-mentioned device ontology library, defect rule library, expert experience library and historical case library are combined to construct a structured knowledge base.

[0151] The secondary revised correlation degree matrix and the causal diagram are input into the knowledge base, the equipment parameter is fed back according to the modal correlation, and the equipment ontology library is supplemented; if a new correlation mode is found in the secondary revised correlation degree matrix, a new rule is automatically generated and added to the defect rule library, and the confidence is marked, so that the rule library is dynamically updated; the secondary revised correlation degree matrix is compared with the existing expert experience, if the data-driven rule conflicts with the expert experience, manual review is triggered, the rule is determined to be consistent, the weight of the rule is increased, and the expert experience library is enhanced; the high-confidence correlation in the secondary revised correlation degree matrix generates a new case template, and the historical case library is expanded. The knowledge base provides a physically interpretable constraint for the secondary revised correlation degree matrix, and the secondary revised correlation degree matrix provides data-driven update materials for the knowledge base.

[0152] When the health of the equipment is predicted to be low, a similar historical case is retrieved using the Faiss vector database, the root cause of affecting the health of the equipment is analyzed, a warning is issued, and a maintenance decision suggestion is given.

[0153] Every week, the missed cases are analyzed, new defect modes are extracted, the knowledge base is dynamically updated, and continuous learning is formed to form a closed loop.

[0154] The embodiment uses a cross-modal attention mechanism to calculate the attention weight between multi-modalities to construct an initial correlation degree matrix, the initial correlation degree matrix is revised to update a preset knowledge base, the health status of the interest target is combined, and a preventive decision is actively provided for the interest target, thereby ensuring stable operation of the inspection.

[0155] In some optional embodiments, when the unmanned aerial vehicle encounters an abnormal power limit situation, the inspection task is not completed, and the current power of the unmanned aerial vehicle cannot return to the original airport, the nearest airport to the position where the inspection task is interrupted is searched, and the unmanned aerial vehicle is controlled to continue the inspection task after the power exchange operation is performed. As shown in Figure 4 The process of searching for the nearest airport to the position where the inspection task is interrupted and performing the power exchange operation is as follows:

[0156] Step S401: A grid model is established according to the actual environment inside the power exchange airport; wherein the unmanned aerial vehicle position and the battery compartment position inside the power exchange airport are arranged on the same plane, so that the movement form of the power exchange mechanical arm is decomposed into two-dimensional plane movement;

[0157] Step S402: Based on the grid model, the position of the unmanned aerial vehicle position, the position of the empty battery compartment position, and the position of the battery compartment position of the replaceable battery, an obstacle avoidance path of the power exchange mechanical arm is planned; the obstacle avoidance path of the power exchange mechanical arm includes an obstacle avoidance path from the starting unmanned aerial vehicle position to the empty battery compartment position, an obstacle avoidance path from the battery compartment position to the battery compartment position of the replaceable battery, and an obstacle avoidance path from the battery compartment position to the starting unmanned aerial vehicle position.

[0158] In the embodiment, the A* algorithm is used for path planning. Specifically, a nearby grid position of a set start position is searched, a nearby grid with the minimum estimated cost function value is selected as the next start position, the selected grid is referred to as a current node, then a nearby grid with the minimum estimated cost function value is searched again as a new current node, and the calculation is repeated in this way until the current node reaches a set end position, so as to plan an obstacle avoidance path of the battery replacement robot arm.

[0159] Since the A* algorithm only considers the node information of the obstacle when planning the path, the planned route intersects with the corners of the obstacle, which may cause friction and collision between the robot arm and the obstacle of the fuselage or the obstacle of the unmanned aerial vehicle landing platform, and needs to be avoided. Here, the traditional A* algorithm is improved, and a corner contact penalty function is added to the estimated cost formula .

[0160] In the embodiment, the estimated cost function is the sum of an actual cost function from the start point to the current node, a heuristic function from the current point to the end point, and a corner contact penalty function, that is,

[0161] ;

[0162] n is the current node, g(n) is the actual cost function from the start point to the current node, and h(n) is the heuristic function from the current point to the end point.

[0163] The corner contact penalty function is obtained by multiplying a penalty intensity coefficient and the exponential power of e; the power exponent of e is obtained by multiplying the difference between the minimum safety distance and the safety distance threshold by the multiple of the steepness factor.

[0164] The specific expression of the corner contact penalty function is: ;

[0165] wherein k is the penalty intensity coefficient (the constant value is 100), is the steepness factor (the constant value is 10), is the minimum safety distance, wherein is the set of all obstacle corner points, because each obstacle grid in the grid environment is a square with a side length of d, the four corners of the obstacle grid are: ;

[0166] Therefore , wherein is the set of obstacle grids.

[0167] The actual cost function is: ,

[0168] wherein Here, the grid is set as a square with side length d.

[0169] P K-1 is the start point of the motion step, P K is the end point of the motion step.

[0170] The heuristic function uses the Euclidean distance: ,

[0171] where is the current node coordinate, is the end point coordinate.

[0172] Minimum safety distance monitoring: ,

[0173] is the path node position vector; is the obstacle corner position vector.

[0174] is the safety distance threshold: , is the safety margin.

[0175] Then ,

[0176] The path segment needs to meet the path segment safety constraint, that is,

[0177]

[0178] In the established 5-row 13-column grid map, taking the left lower corner of the unmanned aerial vehicle at the coordinate point (3, 0) as the starting position and the battery compartment at the coordinate point (11, 4) as the end point position, the path planning of the battery carrying mechanical arm is simulated. According to the above A* algorithm, the planning and calculation process is as follows:

[0179] Define the node data structure, including position, value, value, value, value and parent node; initialize the open list and add the start point to the open list; loop until the end point is found or the open list is empty: take the node with the smallest value as the current node; if the current node is the end point, backtrack the path; otherwise, mark it as closed; generate 8-direction neighbor nodes.

[0180] For each direction:

[0181] Calculate the neighbor node coordinates;

[0182] Check if the coordinate is within [0, 4] x [0, 12] and is a free grid (not an obstacle);

[0183] If it is a straight direction (four directions: up, down, left, and right), i.e. , directly enter the next step;

[0184] If it is a diagonal direction, check if the two adjacent grids of the current node (the neighbor grids in the two straight directions corresponding to the diagonal moving direction) are both obstacles: if not, enter the next step; if so, compare and , if , enter the next step, if , skip this adjacent grid;

[0185] Calculate the value, value (Euclidean distance) and value of the neighbor node, and obtain the value;

[0186] If the neighbor node is in the open list and the new value is smaller, update; otherwise, add the neighbor to the open list.

[0187] The above algorithm takes four drones as an example. If the battery swap airport needs to adapt to a larger number of drones, more than four drones can be placed in the above layout in a set direction (such as upwards or to the right) in turn, and the travel distance of the corresponding module can be lengthened according to the actual layout scheme, and then a grid model is re-established based on the layout environment. The robot obstacle avoidance path planning scheme based on the A* algorithm is also applicable to the layout environment after increasing the number of drones.

[0188] It should be noted here that in other embodiments, in addition to the A* algorithm (grid search), the APF algorithm (artificial potential field), the PRM algorithm (probabilistic roadmap), the RRT (rapidly-exploring random tree), etc. can also be used for robot path planning.

[0189] In one or more embodiments, when an obstacle appears in the obstacle avoidance running path of the battery swap robot arm, the battery swap robot arm is controlled to stop running. In this way, mechanical damage to the battery swap robot arm can be prevented, and running safety can be ensured. In this way, the process of the battery being pulled out of the drone can be avoided, and the risk of the battery falling without support can be avoided.

[0190] The process of replacing the battery of the UAV is as follows: after one UAV lands, the battery replacement mechanical arm moves to the UAV position, the battery replacement mechanical arm turns off the UAV, the battery clamping jaw pulls out the battery in the UAV, the system detects the position of the existing empty battery compartment, and an obstacle avoidance path from the current UAV position to the empty battery compartment position is planned based on the A* algorithm, the battery replacement mechanical arm moves to the empty battery compartment position according to the planned path, the battery clamping jaw inserts the UAV battery into the battery compartment for charging, and after confirming that the inserted battery has started charging, the battery compartment position of the replaceable battery (generally, the battery charging capacity in the battery compartment reaches more than 95%) is detected;

[0191] An obstacle avoidance path from the current battery compartment position to the battery compartment position of the replaceable battery is planned based on the A* algorithm, the battery replacement mechanical arm moves to the battery compartment position of the replaceable battery according to the planned path, and the battery clamping jaw pulls out the battery in the battery compartment.

[0192] An obstacle avoidance path from the current battery compartment position to the starting UAV position is planned based on the A* algorithm, the battery replacement mechanical arm moves to the starting UAV position according to the planned path, and the battery clamping jaw inserts the UAV battery into the UAV.

[0193] If the UAV has no task to continue to perform, the UAV battery replacement process ends; if the UAV has a flight task to continue to perform, the battery clamping jaw turns on the UAV, the UAV flies out to continue to perform the task, and the UAV battery replacement process ends.

[0194] The above-mentioned scheme solves the problems of obstacle avoidance path planning of the battery replacement mechanical arm, safety hazards of collision between the battery replacement mechanical arm and the parked UAV and the airport structure when the battery replacement airport replaces the batteries of multiple UAVs, plans the obstacle avoidance path of the battery replacement mechanical arm based on the grid model, the position of the UAV position, the position of the empty battery compartment position and the position of the battery compartment position of the replaceable battery, improves the adaptability of the battery replacement mechanical arm to the complex battery replacement environment, and realizes the safety control of the battery replacement process of the battery replacement airport with multiple UAVs.

[0195] In one or more embodiments, as shown in Figure 5 The process of finding the airport closest to the patrol task interruption position and performing the battery replacement operation further includes managing and scheduling the batteries, and the process is as follows:

[0196] Step S501: Obtain the real-time temperature of the battery compartment of the battery replacement machine nest, the state of the batteries in the compartment and the working mode of the battery replacement machine nest; wherein the working mode includes a storage mode and a standby mode, in the storage mode, the unique standby battery is kept above the set storage threshold, and in the standby mode, the battery capacity of the other batteries is kept within the set standby threshold.

[0197] Step S502: Temperature control of the overall temperature in the compartment and the temperature abnormal batteries according to the state of the batteries in the compartment and the real-time temperature.

[0198] Specifically, the specific steps of temperature regulation of the overall temperature in the cabin and the temperature of the battery with temperature anomaly according to the battery status in the cabin and the real-time temperature are as follows:

[0199] Step S5021: adjusting the real-time temperature in the cabin according to the overall temperature threshold range;

[0200] In the management of the battery in the battery cabin, the temperature information in the nest cabin is acquired in real time, and the air conditioner in the cabin is controlled in real time according to the temperature data to adjust the temperature, so that the temperature in the nest cabin is kept at a temperature suitable for battery storage for a long time. The system dynamically controls the temperature according to the real-time temperature in the cabin and the temperature of each battery, so that the temperature in the cabin is kept as much as possible in the overall temperature threshold range of 15-35℃. When the temperature in the cabin is greater than 35℃, the cold power is controlled and adjusted, when the temperature drops to 25℃, it is kept, and when the temperature is lower than 15℃, the hot power is controlled and adjusted, and when the temperature rises to 25℃, it is kept.

[0201] Step S5022: detecting the battery with temperature anomaly, starting the single battery temperature control channel, and using the adaptive PID control algorithm to adjust the temperature of the battery with temperature anomaly.

[0202] When the temperature in the cabin is between 15-35℃, some batteries are put into the battery replacement nest after just completing a task, and the battery temperature is too high. Therefore, when a battery with a temperature that is too high is detected, the single battery temperature control channel is started, and the temperature control power is adjusted according to the temperature in the cabin, and a large amount of cold air is preferentially sent to the overheated battery. The system uses an adaptive PID control algorithm to adjust the temperature of the battery with temperature anomaly:

[0203] .

[0204] wherein, represents the temperature deviation of the battery with temperature anomaly, integrates on [0, t], is the temperature deviation, , is the set temperature is the actual temperature; is the output control amount (temperature control power); is the proportional term, which provides linear amplification according to the current error, and if there is still a large deviation between the system and the set value, the proportional term can quickly make an adjustment, ; is the integral term, which accumulates historical errors to eliminate static errors, so that the system can accurately reach the set value when stable, ; is the differential term, which predicts the error trend and has a certain anti-disturbance ability, and can suppress system overshoot and oscillation, .

[0205] Step S503: According to different working modes, the battery in the battery nest is charged and managed, and the battery state in the cabin is updated in real time.

[0206] In this embodiment, the working modes of the battery changing machine nest include storage mode and standby mode. In the storage mode, only one battery needs to ensure the power to execute the task, which can protect the service life of the battery without affecting the execution of the task. In the storage mode, the power of the only standby battery is kept above the set storage threshold power. In the standby mode, the power of the other batteries is kept within the set standby threshold.

[0207] The charging management steps of the battery in the storage mode are as follows: setting the storage threshold, when there is no battery in the battery cabin that meets the storage threshold power, finding the battery with the minimum cycle number and setting it as the only standby battery, and charging the only standby battery to the storage threshold power or above, when the only standby battery is used, finding the battery with the minimum cycle number in the battery cabin again as the only standby battery and charging it to the threshold power, and using the battery with the minimum cycle number in turn, which is beneficial to the balanced management of the battery pack.

[0208] In this embodiment, the storage threshold SOC is set to 80%. In the storage mode, the power of the other batteries except the only standby battery is kept at the appropriate storage power (40%-60%) for a long time. When the power of the battery is lower than 40%, the battery is charged, and the charging is stopped after the power is charged to 60%, so that the battery is kept at the appropriate storage power, which is beneficial to prolong the service life of the battery.

[0209] The charging management steps of the battery in the standby mode are as follows: setting the standby threshold, charging all batteries to the standby threshold power or above, first traversing all batteries to find all batteries below the standby threshold, traversing the batteries below the standby threshold to find the battery with the highest power, suspending the charging of other batteries first, and charging the battery with the highest power to the standby threshold power or above; such a cycle is repeated until all batteries are charged to the standby threshold power or above, and all batteries that meet the standby threshold power are selected and used during the task.

[0210] Step S504: Using the battery cycle use strategy to perform battery changing operation on the unmanned aerial vehicle with the to-be-changed battery according to the updated battery state in the cabin, wherein the battery cycle use strategy is to select the battery closest to the gripper and meeting the power requirement for replacement when the cycle number difference between the batteries is within the cycle threshold range, and to use the battery meeting the power requirement according to the cycle number for balanced scheduling when the cycle number difference between the batteries exceeds the cycle threshold range.

[0211] When the battery is replaced, the comprehensive calculation is performed according to the battery cycle number and the distance of each battery from the claw, and each battery is balanced and scheduled for use. According to the balanced scene of the battery cycle number, the minimum cycle number priority algorithm is used: . wherein, represents the selected battery to be replaced, i is the battery index, i = 1, 2, 3,..., n; represents the cycle number of the battery with index i, and n is the total number of batteries.

[0212] At the same time, the distance of the battery from the claw is comprehensively considered: ; wherein, represents the distance of the battery with index i from the claw.

[0213] In this embodiment, the influence of the battery cycle number is considered. The fewer the cycle numbers, the higher the score. When selecting a battery, the battery with fewer cycle numbers is preferentially selected. Therefore, the comprehensive battery cycle number use optimization term is designed: ; wherein, is the minimum cycle number of the battery in the cabin.

[0214] In addition, the influence between the claw distance is considered. The closer the battery is to the claw, the higher the score. When selecting a battery, the battery close to the claw is preferentially selected. Therefore, the comprehensive battery distance from the claw optimization term is designed: ; wherein, is the farthest distance of the battery in the cabin from the claw; is the closest distance of the battery in the cabin from the claw.

[0215] In order to balance the priority relationship between the battery cycle number and the claw distance, the comprehensive battery cycle number use optimization term and the comprehensive battery distance from the claw optimization term are combined to obtain the comprehensive calculation formula:

[0216] ;

[0217] wherein, is a weight coefficient, and the battery with the maximum value is selected as the battery to be replaced this time.

[0218] The battery cycle use strategy is used to perform battery replacement operation on the unmanned aerial vehicle to be replaced according to the updated battery state in the cabin.

[0219] Wherein, the battery cycle use strategy is that when the cycle number difference between the batteries is within the cycle threshold range, the battery closest to the claw and meeting the power requirement is selected for replacement, and when the cycle number difference between the batteries exceeds the cycle threshold range, the battery meeting the power requirement is balanced and scheduled for use according to the cycle number.

[0220] Estimate the drone's battery level to determine if a battery replacement is necessary. Use a self-discharge compensation equation to quantitatively describe the battery's capacity decay when the drone is idle, and determine whether the drone needs a battery replacement based on this decay pattern.

[0221] In this embodiment, the voltage and capacity of the drone lithium battery exhibit non-linear decay over time during static storage. The decay rate is initially rapid (high state of charge (SOC)) and then gradually slows down. This phenomenon is related to electrolyte decomposition, electrode side reactions, and changes in ion migration rates. Self-discharge is affected by many factors, including state of charge, temperature, humidity, and open-circuit storage time. This embodiment uses a self-discharge compensation equation to quantitatively describe the static capacity decay law of the lithium battery and manages the battery based on this decay law.

[0222] ;

[0223] in, Indicates the remaining battery power. Indicates the initial state of charge; denoted by temperature-dependent self-discharge coefficient; p characterizes the nonlinearity of self-discharge rate over time (typical value 0.5 for lithium iron phosphate, 0.55 for ternary lithium); t represents the cumulative time after the battery leaves the charge / discharge state.

[0224] Introducing a temperature correction term : ;

[0225] in, This represents the baseline value, which is the inherent decay rate when left to stand at 25°C (0.0003 for ternary lithium batteries). This corresponds to approximately 2.2% capacity loss per month, while lithium iron phosphate has a loss of 0.0001%. This corresponds to a capacity loss of approximately 0.7% per month; k is the material sensitivity coefficient (0.08 / ℃ for ternary lithium and 0.06 / ℃ for lithium iron phosphate).

[0226] Introducing aging correction factors:

[0227] ;

[0228] Where N represents the number of iterations; Indicates the cumulative storage time (in years).

[0229] By calculating and estimating the remaining battery power, the system comprehensively assesses whether the battery power is greater than the threshold set in the background to meet the task execution requirements. If the threshold is met, the battery will not be replaced and the system will directly enter the corresponding task process, which helps to shorten task preparation time and improve inspection efficiency.

[0230] Self-discharge compensation equation: .

[0231] According to the power attenuation law, during the monitoring process of the battery, the battery with the power lower than the threshold is replaced when performing the task. When there is no task for a long time, the battery replacement is automatically performed when the battery power is too low. In the embodiment, the cycle number of the battery in all battery compartments is monitored, and when the cycle number of the battery exceeds a warning threshold (such as 200 times), an alarm is reported to prompt the system to maintain, check or replace the battery.

[0232] The embodiment can adaptively adjust the temperature in the nest compartment, so that the battery is at a suitable storage temperature. The battery pack can also be managed according to the battery mode, so that the battery is at a suitable storage power, which is beneficial to prolong the service life of the battery. At the same time, the battery is balanced and used according to the cycle number of the battery during the task, which is beneficial to reduce the operation and maintenance cost.

[0233] In the process of power unmanned aerial vehicle inspection, the current multi-unmanned aerial vehicle inspection scheduling scheme only focuses on task allocation and path planning under normal circumstances. Once an exception occurs, the task is often interrupted or cannot be completed on time due to the lack of effective task re-allocation and path re-planning mechanism, which seriously affects the inspection efficiency and quality. In order to solve this problem, in one or more embodiments, all airports in a set region are abstracted as nodes of a graph, an airport correlation network is constructed according to a standby relationship threshold corresponding to an airport density, and a graph connectivity vector is calculated according to the airport correlation network, and then the set region is divided into sub-regions; wherein the airports in the same sub-region are mutual standby airports.

[0234] In the embodiment of the application, the density of large-scale airports is calculated by KNN algorithm, and hierarchical processing is implemented according to the density difference. For example, for a high-density region, a range is delimited with 10km as a limit, so that the airports within 10km apart form a standby relationship network; for a low-density region, the range is expanded to 15km to ensure that the airports within 15km are mutual standby. At the same time, the graph connectivity vector method is used to divide the airports into sub-regions, and a structured spatial layout system is constructed.

[0235] Specifically, the process of airport density division based on KNN algorithm is as follows:

[0236] Data preparation: collect the geographic coordinates (longitude , latitude ) information of each airport, and construct an airport position data set , wherein is the total number of airports.

[0237] Density calculation: the density of each airport is calculated by KNN algorithm. A suitable Value (k = 6 in this scheme), calculate the field The average distance to other nearest neighbor airports The formula is as follows:

[0238] ;

[0239] Where represents the set of nearest neighbor airports of airport The smaller the average distance , the more densely distributed the airports around the airport, i.e. the higher the density; otherwise, the lower the density.

[0240] Density classification: classify the airports according to the calculated average distance . Set appropriate distance thresholds , to divide the airports into high-density areas, low-density areas, and construct backup relationships. Take the average of all .

[0241] The process of sub-region division based on graph connectivity vectors is given in detail as follows:

[0242] 1. Construct the airport graph model: abstract all airports as nodes V in the graph G = (V, E), and each node corresponds to an airport. If the distance between two airports is within the set backup relationship threshold (within 10 km in high-density areas or within 15 km in low-density areas), add an undirected edge between the corresponding nodes to form the edge set E, and construct the airport correlation relationship network.

[0243] 2. Calculate the graph connectivity vector: for each node in the graph G, calculate its graph connectivity vector . The formula is as follows: ;

[0244] Where, is the damping coefficient (value 0.8) to balance the influence from adjacent nodes and random jumps; represents the number of neighbor nodes of node ; is the total number of nodes; is the number of iterations, and when the value of converges, stop iteration to get the final graph connectivity vector of each node.

[0245] 3. Sub-region division: according to the calculated graph connectivity vector , the K-Means clustering is used to divide the airports with similar graph connectivity vectors into the same sub-region. The airports in the same sub-region are mutual backup airports.

[0246] 4. Redundant airport deployment: the number of airports in the low-density area is small, and the flight caused by scheduling may affect the task execution. Therefore, one charging airport is added to each sub-region in the low-density area, which does not contain a UAV and does not perform a patrol task, but is responsible for charging in an emergency.

[0247] When the UAV is performing a patrol task and receives a new emergency task, it is determined whether the patrol target in the emergency task has an ongoing patrol task. If yes, the task is not issued again.

[0248] Otherwise, the adjacent airport table of the airport is searched, the task is issued to the standby airport of the airport, and the remaining flight time of the other airports is calculated to determine whether it meets the required time of the task. The airport closest to the patrol target is selected from the airport with the remaining flight time meeting the requirement and the task is issued to the airport.

[0249] When the UAV encounters an abnormal power limitation situation and the patrol task is not completed, if the current power of the UAV cannot return to the original airport, the remaining time of the task + the time of returning to the original airport is calculated, the task is issued to the standby airport of the airport, and the remaining flight time of the other airports is calculated to determine whether it meets the required time of the task + the remaining time of the task + the time of returning to the original airport. The airport closest to the interruption position is selected from the airport with the remaining flight time meeting the requirement, the airport continues to perform the task, and the UAV without power charges at the airport.

[0250] When the UAV encounters an abnormal power limitation situation and the patrol task is completed, in the set high-density area, it is determined whether the current power of the UAV can return to the original airport. If yes, the UAV returns to the original airport.

[0251] Otherwise, the task is issued to the standby airport of the airport, and the remaining flight time of the other airports is calculated to determine whether it meets the time of flying to the original airport. The airport closest to the current UAV is selected from the airport with the remaining flight time meeting the requirement, and the UAV flies to the airport for charging. The UAV of the airport closest to the current UAV flies to the original airport.

[0252] When the UAV encounters an abnormal power limitation situation and the patrol task is completed, in the set high-density area, it is determined whether the current power of the UAV can return to the original airport. If yes, the UAV returns to the original airport.

[0253] Otherwise, it is judged whether the current power of the UAV can fly to the charging airport. If yes, the UAV flies to the charging airport for charging. After the UAV is fully charged, it returns to the original airport. Otherwise, a task is issued to the standby airport of the airport, and it is calculated whether the remaining flight time of the aircraft meets the time of flying to the original airport. From the airport meeting the requirement, the airport A closest to the current UAV is selected. The current UAV flies to the airport A for charging, and the UAV of the airport A flies to the original airport.

[0254] When the UAV encounters an abnormal power situation and the inspection task is not completed, it is judged whether the current power of the UAV can return to the original airport. If yes, the UAV returns to the original airport, saves the task breakpoint, and calculates the remaining time of the task. A task is issued to the standby airport of the airport, and it is calculated whether the remaining flight time of the aircraft meets the time required for flying to the task point + the remaining time of the task + the time of returning to the respective airport. From the airport meeting the requirement, the airport closest to the breakpoint is selected to continue the task.

[0255] In other embodiments, a multi-agent reinforcement learning (MARL) algorithm is used to train the path conflict avoidance decision of the multi-UAV in the inspection process, so that the UAV cluster can autonomously learn and form a conflict avoidance strategy in a dynamic environment. In this way, the safety and fluency of multi-machine cooperative operation can be improved.

[0256] In the present embodiment, a centralized training-decentralized execution QMIX algorithm is adopted. Each UAV maintains a local Q network , which outputs the value of each action in the current state. The local Q values are aggregated into global Q values by a mixing network to coordinate the strategies of multiple machines. The network is trained by minimizing the loss function:

[0257] ;

[0258] wherein, represents the state, represents the action, is the reward function, is the network parameter, takes the value 0.9; are the state and action at the next training step in the network training process, respectively; is the target network parameter.

[0259] The detailed design is as follows:

[0260] (1) State includes the self-state and the neighbor-state:

[0261] Self-state: three-dimensional coordinates , task state (take-off / landing / flying / inspection), and remaining power .

[0262] Neighbor state: relative coordinates and velocity vectors of all neighbors within 50 meters . where is the neighbor index.

[0263] (2) Actions include 7 discrete actions, each corresponding to a 1-meter displacement:

[0264] Horizontal evasion: left yaw, right yaw, i.e., translate 1 meter horizontally.

[0265] Vertical evasion: climb, descend, i.e., translate 1 meter vertically.

[0266] Speed adjustment: accelerate, decelerate (adjust cruise speed, equivalent displacement 1 meter per action).

[0267] Maintain current state: maintain original heading and speed.

[0268] (3) Rewards include collision penalty, distance gain, task delay penalty, power saving reward;

[0269] Collision penalty: if the distance to a neighbor or obstacle is less than meters, reward ; where is the safety distance.

[0270] Distance gain: if the distance is increased by evasion, reward where , is a constant factor; denotes the previous distance to a neighbor or obstacle.

[0271] Task delay penalty: if evasion causes a task delay of seconds, reward , where , is a constant factor.

[0272] Power saving reward: reward , where encourages low-energy flight; , are the initial and current power levels, respectively.

[0273] The present application realizes efficient management of large-scale unmanned aerial vehicle clusters, rapid response to abnormal situations, and continuous execution of tasks by integrating spatial region division algorithms, innovative scheduling rules, and intelligent decision-making training techniques. At the same time, it enhances the autonomous processing capability of path conflicts during multi-unmanned aerial vehicle cooperative operation, ensuring the stability and safety of the inspection task under complex working conditions.

[0274] Example 2

[0275] like Figure 6 As shown, this embodiment provides a multimodal fully autonomous inspection system for power unmanned aerial vehicles (UAVs), which specifically includes the following modules:

[0276] The inspection task issuing module 601 is used to process power inspection task instructions within the regional grid area using multi-agent collaboration and issue them to the corresponding UAVs at the airport.

[0277] The modal inspection data perception module 602 is used to control the UAV to autonomously perceive the dynamic environment of the inspection area, so as to autonomously query the target of interest, guide the UAV to conduct autonomous inspection of the unknown environment, and perceive the multimodal inspection data of the target of interest.

[0278] The multimodal inspection data fusion module 603 is used to perform cross-modal calibration on the multimodal inspection data of the target of interest, extract the spatiotemporal joint features of each mode after calibration, and then combine them with the modal weights that match the dynamic environment to obtain the fusion features.

[0279] Health status assessment module 604 is used to assess the health status of the target of interest based on fusion features and using a multimodal large model;

[0280] The proactive prevention decision module 605 is used to construct a correlation matrix based on the correlation of each modality, make secondary corrections to it, update the preset knowledge base, and then combine it with the health status assessment results of the interest target to obtain the proactive prevention decision of the interest target driven by multimodal fusion and correlation.

[0281] It should be noted that the implementation process of each module in this embodiment is the same as that of each step in the above embodiment one, and will not be repeated here.

[0282] Example 3

[0283] like Figure 7 As shown, this embodiment provides a multimodal fully autonomous inspection system for power unmanned aerial vehicles (UAVs), including:

[0284] The data sensing unit 701, which is mounted on a drone, is used to autonomously sense multimodal inspection data of dynamic environment and targets of interest.

[0285] The data processing unit 702 includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the above-described multimodal fully autonomous inspection method for electric unmanned aerial vehicles.

[0286] In particular, according to an embodiment of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing Figure 1 the processes of the methods described above with reference to the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network by a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit, various functions defined in the apparatus of the present application are executed.

[0287] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (system) and computer program products of embodiments of the present application. It is understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 The flowcharts and / or block diagrams can also be implemented by or in connection with one or more systems containing one or more processors and memory. Figure 1 The flowcharts and / or block diagrams can also be implemented by or in connection with one or more systems containing one or more processors and memory.

[0288] The above merely provides preferred embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.

Claims

1. A multimodal fully autonomous inspection method for power unmanned aerial vehicles (UAVs), characterized in that, include: The system utilizes multi-agent collaborative processing to handle power inspection task instructions within a regional grid area and distributes them to the corresponding drones at the airport. Control the drone to autonomously perceive the dynamic environment of the inspection area, autonomously query targets of interest, guide the drone to conduct autonomous inspection of unknown environments, and perceive multimodal inspection data of targets of interest. Cross-modal calibration is performed on the multimodal inspection data of the target of interest. The spatiotemporal joint features of each mode after calibration are extracted and then combined with the modal weights that match the dynamic environment to obtain the fusion features. Based on fusion features, a multimodal large model is used to assess the health status of interest targets; Based on the correlation of each modality, a correlation degree matrix is ​​constructed and pseudo-correlation is filtered to obtain a first-corrected correlation degree matrix. The first-corrected correlation degree matrix is ​​then enhanced with contrastive learning to improve the consistency between modalities. Causal graphs are used to remove the pre-set low-confidence correlation degrees in the contrastive learning to obtain a second-corrected correlation degree matrix. The pre-knowledge base is updated accordingly. Finally, combined with the health status assessment results of interest targets, a proactive prevention decision for interest targets driven by multimodal fusion and correlation is obtained.

2. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, Based on the spatiotemporal joint features, the intermodal correlation is calculated, and combined with the dynamic environment and the noise levels of the sensors acquiring data for each modality, the modal weights matching the dynamic environment are calculated. The expression is as follows: ； in, Indicates temperature coefficient; composite score Sensor noise level , It is the noise standard deviation. It is the sensitivity coefficient; intermodal correlation score. m represents the total number of modes. The Pearson correlation coefficients of modes i and j; scene prior factors. ; It is an indicator function. It is the scene gain coefficient.

3. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, Cross-modal calibration is performed on multimodal inspection data of targets of interest using a cross-modal calibration model; The training of the cross-modal calibration model employs a weighted sum of local feature alignment loss and global context matching loss, and learns the model parameters and the adjustment factors used for weighting through meta-learning training.

4. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 3, characterized in that, The weighted sum is: ;in, The adjustment factor; local feature alignment loss , This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, Constant coefficients; global context matching loss , For the global features of mode A, This represents the global features of mode B; Let K represent the global feature of mode B of the negative samples, where K is the number of negative samples.

5. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, The elements in the correlation matrix are: ； in, Representing modes spatiotemporal joint characteristics and modality spatiotemporal joint characteristics Attention weights between them; S Indicates the total number of modes. d For feature dimensions; W Q and W K This is the projection matrix.

6. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, Using a causal discovery algorithm, a causal graph is generated by combining the original correlation matrix and historical fault data of the target of interest.

7. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 6, characterized in that, The contrastive loss function used in the contrastive learning process for: ； in, It is cosine similarity. It is a constant coefficient; For mapping to the shared space of modalities i and j Features; represents the elements in the correlation matrix after one correction; H is the number of modes after one correction.

8. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, The process of guiding a drone to conduct autonomous inspections in unknown environments is as follows: Acquire multi-source perception data of the drone in an unknown environment and extract obstacle information from it to plan a preliminary global path from the current point to the target point; The weights of the UAV's multi-source perception data are dynamically adjusted based on the current environmental information of the unknown environment and a preset fuzzy rule base. The multi-source perception data of the UAV is then weighted and fused, and combined with the current IMU data to optimize the calculation of the UAV's current pose. Based on the current pose of the UAV calculated by optimization, while flying along the preliminary global path, the UAV determines whether there are new obstacles based on real-time multi-source perception data, and optimizes the preliminary global path to achieve autonomous inspection of the UAV in unknown environments.

9. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 8, characterized in that, The D*Lite algorithm is used to plan a preliminary global path from the current point to the target point. D*Lite uses a key value to determine the priority of node updates. The expression is: ； in, The current estimated cost from node s to the target point; For the successor node based on node s A more forward-looking cost estimate; To start from the current location of the drone The heuristic function for node s ensures that the search prioritizes updating the nodes that are most important to the current drone position; It is an accumulator that records the sum of the heuristic costs added by all starting point moves since the algorithm began running.

10. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, If a drone encounters an abnormal power limit and the inspection mission is not yet completed and the drone's current power level prevents it from returning to the original airport, the drone can be controlled to continue the inspection mission after the nearest airport to the location where the inspection mission was interrupted is located and a battery swap operation is performed.

11. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 10, characterized in that, During the battery swapping operation, the robot searches for nearby grid positions from a set starting position, selecting the grid with the smallest estimated cost function value as the next starting point. This selected grid is called the current node. The process is repeated, searching for the next grid with the smallest estimated cost function value as the new current node, and so on, until the current node reaches the set ending position, thus planning the obstacle avoidance path for the battery swapping robot arm. The estimated cost function... for: ； ； n is the current node, g(n) is the actual cost function from the starting point to the current node, and h(n) is the heuristic function from the current point to the ending point. Here, k is the penalty function for corner contact; k is the penalty intensity coefficient. Steepness factor For minimum safe distance, This is the safe distance threshold.

12. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 10, characterized in that, The process of locating the nearest airport to the point where the inspection mission was interrupted and performing a battery swap also includes battery management and scheduling, the process of which is as follows: Get the real-time temperature of the battery compartment of the motor swapping nest, the status of the batteries inside the compartment, and the working mode of the motor swapping nest; Temperature regulation is performed on the overall temperature inside the cabin and on batteries with abnormal temperatures based on the status of the batteries inside the cabin and the real-time temperature. The battery charging management in the motor swapping compartment is carried out according to different working modes, and the battery status in the compartment is updated in real time. The battery recycling strategy is used to perform battery replacement operations on the drone whose battery needs to be replaced based on the updated battery status in the cabin. The battery recycling strategy is as follows: when the difference in the number of cycles between batteries is within the cycle threshold range, the battery closest to the gripper and meeting the power requirements is selected for replacement; when the difference in the number of cycles between batteries exceeds the cycle threshold range, the batteries that meet the power requirements are balanced and used according to the number of cycles.

13. The multimodal fully autonomous inspection method for power unmanned aerial vehicles as described in claim 1, characterized in that, The multimodal fully autonomous inspection method for power drones also includes: abstracting all airports within a designated area as nodes in a graph; constructing an airport association network based on the backup relationship threshold corresponding to airport density; calculating the graph connectivity vector based on the airport association network; and then dividing the designated area into sub-regions; wherein airports in the same sub-region serve as backup airports for each other; scheduling drones according to the principles of task priority and task continuation; when a drone encounters an abnormal situation of power limitation, based on the divided sub-regions, finding the airport closest to the interruption point of the inspection task and having the drone in that airport continue to perform the inspection task.

14. A multimodal fully autonomous inspection system for power unmanned aerial vehicles (UAVs), characterized in that, include: The inspection task distribution module is used to process power inspection task instructions within the regional grid area using multi-agent collaboration and distribute them to the corresponding drones at the airport. The modal inspection data perception module is used to control the UAV to autonomously perceive the dynamic environment of the inspection area, so as to autonomously query targets of interest, guide the UAV to conduct autonomous inspection of unknown environment, and perceive multimodal inspection data of targets of interest. The multimodal inspection data fusion module is used to perform cross-modal calibration on the multimodal inspection data of the target of interest, extract the spatiotemporal joint features of each mode after calibration, and then combine them with the modal weights that match the dynamic environment to obtain the fused features. The health status assessment module is used to assess the health status of targets of interest based on fusion features and utilizing a multimodal large model. The proactive prevention decision-making module is used to construct a correlation matrix based on the correlation of each modality and filter out pseudo-correlation to obtain a first-corrected correlation matrix. The first-corrected correlation matrix is ​​then enhanced with contrastive learning to improve the consistency between modalities. A causal graph is used to remove the pre-set low-confidence correlations in the contrastive learning to obtain a second-corrected correlation matrix. The pre-knowledge base is updated accordingly. Finally, combined with the health status assessment results of the interest target, a proactive prevention decision for the interest target driven by multimodal fusion and correlation is obtained.

15. A multimodal fully autonomous inspection system for power unmanned aerial vehicles (UAVs), characterized in that, include: The data sensing unit, mounted on the drone, is used to autonomously sense multimodal inspection data of the dynamic environment and targets of interest. A data processing unit includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multimodal fully autonomous inspection method for electric unmanned aerial vehicles as described in any one of claims 1-13.

Images