Unmanned aerial vehicle inspection cooperative control method, device and equipment and storage medium

Abstract

The invention provides an unmanned aerial vehicle inspection cooperative control method, device and equipment and a storage medium, and the method comprises the steps: splitting an inspection task into subtasks, employing a matching algorithm to distribute the subtasks to an unmanned aerial vehicle, and obtaining a task distribution result; based on a task allocation result, solving an initial inspection path of each unmanned aerial vehicle by adopting a path planning algorithm under a reinforcement learning architecture, and setting a flight constraint condition in a path planning solving process; executing a task based on the initial inspection path, obtaining fusion data influenced by the outside in real time, generating a physical force model for automatic obstacle avoidance according to the fusion data, solving the physical force model by adopting a gradient descent method, and adjusting a flight strategy from the current position of the initial inspection path to a next task point in real time; predicting the remaining power of the unmanned aerial vehicle through a power loss model, and repatriating the unmanned aerial vehicle according to the remaining power. According to the invention, a data closed loop of path initial planning, safety control adjustment and emergency return flight is formed on the whole.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle cooperative control, and in particular to an unmanned aerial vehicle inspection cooperative control method, device, equipment and storage medium. BACKGROUND

[0002] With the continuous expansion of key infrastructure such as power, transportation, and energy, traditional manual inspection methods have been difficult to meet the needs of large-scale, high-frequency, and high-efficiency inspection.

[0003] In related technologies, with the development of unmanned aerial vehicle technology, multi-vehicle cooperative inspection is increasingly widely used in the fields of power, oil and gas, transportation, etc. Currently, there are problems such as unreasonable task allocation, poor path planning adaptability, and delayed safety control response in multi-vehicle cooperation. The specific defects are described as follows: Task allocation is mostly dependent on static rules and does not fully combine historical execution efficiency and task difficulty to allocate, resulting in resource waste or task overload; Path planning is often determined before the unmanned aerial vehicle performs inspection, and it is difficult to respond to complex geographical environments and dynamic obstacles during the inspection process, lacking an adaptive adjustment mechanism based on real-time data; In addition, there is also a lack of multi-sensor data fusion accuracy, and the power loss prediction does not consider multi-dimensional environmental factors, further restricting the safety and efficiency of cooperative operation.

[0004] Based on the above analysis of the development status of the technical field, the existing technology lacks a scheme for dynamically matching and allocating based on historical data and task difficulty, and real-time acquisition of fusion data affected by the outside world to adjust the path during the inspection process. SUMMARY

[0005] The purpose of the present application is to provide an unmanned aerial vehicle inspection cooperative control method, device, equipment and storage medium, which aims to solve the above-mentioned problems in the prior art, and the core technical solution includes: decomposing the inspection task into subtasks, using a matching algorithm to allocate the subtasks to the unmanned aerial vehicles; based on the allocation result, using a reinforcement learning framework to construct an initial inspection path; in the inspection process, collecting flight data and environmental data through multi-sensor fusion technology, using a synthetic physical potential field model to automatically trigger safety strategies such as speed adjustment, obstacle avoidance, and adjustment; combining multi-dimensional factors to construct a power loss model as the basic rule for the return of the unmanned aerial vehicle.

[0006] According to a first aspect of an embodiment of the present application, an unmanned aerial vehicle inspection cooperative control method is provided, comprising: decomposing the inspection task into subtasks, using a matching algorithm to allocate the subtasks to the unmanned aerial vehicles, and obtaining a task allocation result, specifically including: obtaining historical execution records of the unmanned aerial vehicles, and based on the historical execution records, calculating the historical task completion efficiency of each unmanned aerial vehicle through a quantitative formula; An environment complexity level and an environment climate of a work area where the subtask is located are acquired, and index data contained in the environment complexity level and the environment climate are weighted and summed to obtain an estimated difficulty coefficient of each subtask; The historical task completion efficiency, the estimated difficulty coefficient, and the UAV state information are input into a dynamic matching algorithm, a matching matrix is constructed through the dynamic matching algorithm, the matching matrix is filled by using multi-objective optimization, and a task allocation result is obtained.

[0007] Based on the task allocation result, an initial inspection path of each UAV is solved by using a path planning algorithm under a reinforcement learning architecture, and flight constraint conditions are set in the path planning solving process, specifically including: A Markov decision model including a state space, an action space, and a reward function is constructed as the reinforcement learning architecture, wherein the state space includes a real-time position of the UAV, the action space includes an adjustment range of a flight direction, a speed, and a height, and the reward function includes a path length, an obstacle avoidance success rate, and a task completion progress; The maximum flight radius, the minimum turning radius, the no-fly area, and the flight conflict between UAVs of the UAV are set as the flight constraint conditions; An output and evaluation dual network model is used as the path planning algorithm to solve the initial inspection path of each UAV, wherein the initial inspection path is a frog-leaping inspection path connecting the sequence of each subtask; in the solving process, an experience replay mechanism and a target network separation mechanism are used.

[0008] The output and evaluation dual network model used as the path planning algorithm specifically includes: An Actor-Critic network is used as the dual network model, the Actor network outputs a continuous control flight action space according to the current state space, the Critic network evaluates the reward function according to the output of the Actor network, and the reward function result is fed back to the Actor network to optimize the output of the action space.

[0009] Based on the initial inspection path, the task is executed, the fusion data affected by the outside world is acquired in real time, the physical potential model for automatic obstacle avoidance is generated according to the fusion data, the gradient descent method is used to solve the physical potential model, and the flight strategy of the initial inspection path from the current position to the next task point is adjusted in real time, specifically including: A state space model of a laser radar, a visual SLAM, and a millimeter wave radar under a federated Kalman architecture, wherein the laser radar is used to acquire three-dimensional point cloud information of obstacles, the visual SLAM is used to estimate the UAV pose, and the millimeter wave radar is used to measure the real-time distance from the UAV to the obstacles; Timestamps of data collected by laser radar, visual SLAM and millimeter wave radar are aligned, and the data is calibrated in a spatial coordinate system with the UAV pose as a directional reference; a nearest neighbor association algorithm is used to realize spatial registration of the laser radar and the millimeter wave radar, and three-dimensional positions of obstacles and obstacle velocity vectors are obtained; Obtain environmental information through a weather type sensor; After chi-square test anomaly detection and elimination, output fusion data including three-dimensional positions of obstacles, obstacle velocity vectors and environmental information.

[0010] A physical force model is constructed, which includes an obstacle repulsive force field, an adverse environment repulsive force field, a next sub-task target point attractive force field and a path constraint field; Based on the fusion data, the distribution of obstacles and adverse environments is fitted through a Gaussian mixture model as an avoidance area, and a repulsive force vector of the avoidance area to the UAV is simulated, wherein the repulsive force vector increases as the distance to the UAV decreases; A gravitational force vector is simulated between the current position of the UAV and the next sub-task target point, and a distance attenuation factor is set during the simulation of the gravitational force vector, wherein the gravitational force vector increases as the distance to the UAV decreases; The turning angle and acceleration of the UAV are limited by setting a path constraint field; An LM nonlinear algorithm is used as a gradient descent method to solve the physical force model, automatically generating an optimized path to avoid obstacles and adverse environments, and adjusting the flight strategy according to the optimized path.

[0011] The remaining power of the UAV is predicted through a power consumption model, and the UAV is repatriated according to the remaining power, specifically including: A multi-dimensional time series of power consumption is constructed; Wavelet decomposition and reconstruction are performed on the multi-dimensional time series to extract features at different time scales to form short-term fluctuation features and long-term trend features, and a power consumption model composed of an ARIMA model and an LSTM model is obtained, wherein the order of the ARIMA model is determined by ADF test, and the parameters are trained by maximum likelihood estimation method; The long-term trend features are processed by the ARIMA model to output a first prediction result; The short-term fluctuation features are processed by the LSTM model to output a second prediction result; The first prediction result and the second prediction result are weighted and fused to obtain the predicted remaining power, and the corresponding UAV is repatriated when the remaining power is lower than the power threshold.

[0012] According to a second aspect of the embodiment of the present application, a UAV inspection cooperative control device is provided, comprising: The task splitting module is configured to split the inspection task into subtasks, assign the subtasks to the UAVs using a matching algorithm, and obtain a task assignment result, and is specifically configured to: obtain historical execution records of the UAVs, calculate historical task completion efficiencies of the UAVs based on the historical execution records using a quantitative formula; obtain an environmental complexity level and an environmental climate of a work area in which the subtasks are located, and obtain an estimated difficulty coefficient of each subtask by weighted summation of index data contained in the environmental complexity level and the environmental climate; input the historical task completion efficiencies, the estimated difficulty coefficients, and UAV state information into a dynamic matching algorithm, construct a matching matrix using the dynamic matching algorithm, fill the matching matrix using multi-objective optimization, and obtain the task assignment result.

[0013] The initial solution module is configured to obtain an initial inspection path of each UAV based on the task assignment result using a path planning algorithm under a reinforcement learning framework, and set flight constraints during path planning, and is specifically configured to: construct a Markov decision model including a state space, an action space, and a reward function as the reinforcement learning framework, wherein the state space includes real-time positions of the UAVs, the action space includes adjustment ranges of flight directions, speeds, and heights, and the reward function includes path lengths, obstacle avoidance success rates, and task completion progress; set a maximum flight radius, a minimum turning radius, a no-fly area, and avoidance of flight conflicts between UAVs as the flight constraints; use a dual-network model of output and evaluation as the path planning algorithm to obtain the initial inspection path of each UAV, wherein the initial inspection path is a frog-leap inspection path connecting the order of the subtasks; and use an experience replay mechanism and a target network separation mechanism during the solving process.

[0014] The dual-network model of output and evaluation as the path planning algorithm specifically includes: use an Actor-Critic network as the dual-network model, output a continuous control flight action space from the Actor network according to a current state space, evaluate a reward function from the output of the Actor network using the Critic network, and feed the reward function result back to the Actor network to optimize the output of the action space.

[0015] The real-time adjustment module is configured to execute the task based on the initial inspection path, obtain fusion data affected by the outside world in real time, generate a physical potential model for automatic obstacle avoidance based on the fusion data, solve the physical potential model using a gradient descent method, and adjust a flight strategy of the initial inspection path from a current position to a next task point in real time, and is specifically configured to: The state space model of the laser radar, the visual SLAM and the millimeter wave radar under the federal Kalman framework, wherein the laser radar is used to acquire three-dimensional point cloud information of the obstacle, the visual SLAM is used to estimate the pose of the unmanned aerial vehicle, and the millimeter wave radar is used to measure the real-time distance from the obstacle; The data collected by the laser radar, the visual SLAM and the millimeter wave radar are time-stamped and aligned, the data is calibrated in the spatial coordinate system with the pose of the unmanned aerial vehicle as the direction reference, the nearest neighbor association algorithm is used to realize the spatial registration of the laser radar and the millimeter wave radar, and the three-dimensional position of the obstacle and the velocity vector of the obstacle are obtained; The environmental information is acquired through the weather type sensor; After the chi-square test anomaly detection and elimination, the fusion data including the three-dimensional position of the obstacle, the velocity vector of the obstacle and the environmental information are output.

[0016] A physical force model including an obstacle repulsive field, an adverse environment repulsive field, a next sub-task target point attractive field and a path constraint field is constructed; The distribution of the obstacle and the adverse environment is fitted through the Gaussian mixture model based on the fusion data as an avoidance area, and a repulsive vector of the avoidance area to the unmanned aerial vehicle is simulated, wherein the repulsive vector increases with the decrease of the distance from the unmanned aerial vehicle; A gravitational vector is simulated between the current position of the unmanned aerial vehicle and the next sub-task target point, and a distance attenuation factor is set in the simulation process of the gravitational vector, wherein the gravitational vector increases with the decrease of the distance from the unmanned aerial vehicle; The turning angle and the acceleration of the unmanned aerial vehicle are limited by setting the path constraint field; The LM nonlinear algorithm is used as the gradient descent method to solve the physical force model, an optimized path avoiding the obstacle and the adverse environment is automatically generated, and the flight strategy is adjusted according to the optimized path.

[0017] A repatriation module is configured to predict the remaining power of the unmanned aerial vehicle through a power consumption model, and repatriate the unmanned aerial vehicle according to the remaining power, and specifically configured to: A multi-dimensional time series of power consumption is constructed; The multi-dimensional time series is decomposed and reconstructed by wavelet, features at different time scales are extracted to form short-term fluctuation features and long-term trend features, and a power consumption model composed of an ARIMA model and an LSTM model is obtained, wherein the order of the ARIMA model is determined by ADF test, and the parameters are trained by maximum likelihood estimation method; The long-term trend features are processed by the ARIMA model, and a first prediction result is output; The short-term fluctuation features are processed by the LSTM model, and a second prediction result is output; The first prediction result and the second prediction result are weighted and fused to obtain the predicted remaining power, and the corresponding unmanned aerial vehicle is repatriated when the remaining power is lower than the power threshold.

[0018] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable in the processor, which, when executed by the processor, implements the steps of the UAV inspection cooperative control method provided by the first aspect of the present application.

[0019] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores an implementation program of information transmission, which, when executed by a processor, implements the steps of the UAV inspection cooperative control method provided by the first aspect of the present application.

[0020] The technical scheme provided by the embodiments of the present application has the following beneficial effects: the inspection task is divided into subtasks, the historical task completion efficiency of each UAV is calculated accurately, and the difficulty coefficient is estimated, the matching algorithm is used to distribute the subtasks to the UAVs, which helps to preferentially distribute high-difficulty tasks to high-efficiency and reliable UAVs, so that the swarm can complete the inspection in a shorter time and with a higher success rate as a whole; based on the distribution result, an initial inspection path is constructed using a reinforcement learning architecture, adaptive adjustment of the frog jump type path is realized, the Actor-Critic network avoids the traditional policy gradient to optimize the action every round, and the advantages of policy guidance and value evaluation are combined to realize a continuous learning mechanism with stable efficiency; in the inspection process, flight data and environmental data are collected through multi-sensor fusion technology, a synthetic physical potential field model is used to intelligently avoid areas with dynamic obstacles or bad environments, and safety strategies such as speed adjustment, obstacle avoidance, and adjustment are automatically triggered, which can ensure smooth and coherent adjustment process; a power consumption model is constructed in combination with multi-dimensional factors as the basic rule for UAV return, which ensures the safety of power management.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical scheme in the one or more embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0023] Figure 1 is a flowchart of the UAV inspection cooperative control method of the embodiments of the present application; Figure 2 is a schematic diagram of the inspection architecture of the embodiments of the present application; Figure 3is a schematic diagram of a UAV inspection cooperative control device according to an embodiment of the present application; Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable personnel in the art to better understand the technical solutions in one or more embodiments of the present specification, the technical solutions in one or more embodiments of the present specification will be described clearly and completely below in conjunction with the drawings in one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on one or more embodiments of the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the protection scope of the present document.

[0025] Method embodiments According to an embodiment of the present application, a UAV inspection cooperative control method is provided, Figure 1 is a flowchart of a UAV inspection cooperative control method according to an embodiment of the present application, as Figure 1 shown, the UAV inspection cooperative control method according to an embodiment of the present application specifically includes: In step S110, the inspection task is split into subtasks, a matching algorithm is used to assign the subtasks to the UAVs, and a task assignment result is obtained, specifically including: The inspection task is disassembled into subtasks according to geographical areas, object types, and specific inspection items, and a subtask library including geographical coordinates and operation requirements is generated; The historical execution records of the UAVs are obtained, and detailed logs of all past tasks are comprehensively extracted. The subtask types are various and cover specific operation categories such as power tower inspection, oil and gas pipeline monitoring, and traffic facility aerial photography. Based on the historical execution records, the historical task completion efficiency of each UAV is calculated through a quantitative formula; In the embodiment of the present application, the ratio of the task completion amount to the total operation time, or the ratio of the flight distance to the unit energy consumption, is calculated as the historical task completion efficiency, which reflects the operation ability of the UAV in different task scenarios. The flight distance is calculated through satellite positioning trajectory data, and the operation time is accurate to the full process time from takeoff to completion of data back transmission.

[0026] The environmental complexity level and climate of the work area where the subtask is located are obtained. The environmental complexity is divided into three levels: simple, medium and complex based on parameters such as terrain slope, vegetation coverage and building density of the work area, and is assigned a corresponding score. The climate is obtained by combining real-time data such as wind force level, precipitation probability and visibility range. The data types are weighted according to data precision and data dimension. The precision is similar to centimeter or meter level, and the data dimension includes two-dimensional impact or three-dimensional point cloud. The estimated difficulty coefficient of each sub-task is obtained by weighting and summing the data of environmental complexity level and environmental climate indicators.

[0027] The historical task completion efficiency, estimated difficulty coefficient, and UAV status information are input into the dynamic matching algorithm. The matching matrix is ​​constructed through the dynamic matching algorithm, and the matching matrix is ​​filled by multi-objective optimization to obtain the task allocation result. Preferably, in addition to UAV status information, the capacity and charging status of UAV airports can be combined as logistics support points as part of the multi-objective optimization conditions. The capacity avoids airport saturation, and the charging status prioritizes airports with high charging efficiency as logistics support points. Historical task completion efficiency reflects the drone's operational capabilities, the estimated difficulty coefficient reflects the difficulty of sub-task execution, and drone status information includes the remaining battery power to support the flight time, the straight-line distance between the current location and the sub-task area, the type and weight of the payload equipment, etc. In other words, in this embodiment of the invention, the columns of the matching matrix are the numbers of each UAV and the numbers of each subtask. The initial matching matrix is ​​an empty matrix, or it can be a random matching situation constructed based on historical execution. After solving, matching is achieved by checking boxes in the empty matrix or modifying existing relationships. A UAV can execute a series of subtasks. Multi-objective optimization is the core of dynamic matching algorithms. UAV state information serves as a constraint, and the goal is to match UAVs with higher operational capabilities with more difficult sub-tasks within the allowable range, while minimizing the overall task completion time and resource consumption. This matching can be done automatically through computer programs.

[0028] In step S120, based on the task allocation results, a path planning algorithm is used under the reinforcement learning architecture to solve the initial inspection path for each UAV, and flight constraints are set during the path planning process, specifically including: A Markov decision model containing a state space, action space, and reward function is constructed as a reinforcement learning architecture. The state space includes the real-time position of the UAV. Preferably, the state space may also include information such as remaining battery power, distribution of surrounding obstacles, and no-fly zone boundaries. However, at this stage, such information refers to fixed external influences and does not conflict with subsequent online adjustments. The action space includes the adjustment range of flight direction, speed, and altitude. The reward function includes path length, obstacle avoidance success rate, and task completion progress. The maximum flight radius, minimum turning radius, no-fly zone and flight collision avoidance of the unmanned aerial vehicle are set as flight constraints to ensure the efficiency and safety of the multi-collaborative inspection path. The output and evaluation dual network model is used as the path planning algorithm to solve the initial inspection path of each unmanned aerial vehicle, wherein the initial inspection path is a frog-leaping inspection path connecting the sequence of each sub-task, that is, the frog-leaping point is regarded as the execution point of the sub-task, and the path between two sub-tasks is approximately a straight line except for the start and end smoothing and path designation caused by the flight constraints. During the solving process, the experience replay mechanism and the target network separation mechanism are used. The experience replay mechanism refers to storing state transition samples for reference by the path planning solution strategy. The target network technology separates the copy of the dual network model. Online decision-making real-time adjustment of network parameters is easily affected by external environment, while the copy adjustment time step is longer. Therefore, when using the dual network model, the output result is as close as possible to the output result of the copy.

[0029] Preferably, after completing a predetermined number of inspection tasks, such as 10 times, the network parameters are iteratively updated based on newly collected environmental data such as changes in obstacle position, updates to weather conditions, etc., so that the path planning strategy can adapt to different working environments and task requirements.

[0030] The dual network model of output and evaluation is used as the path planning algorithm, which specifically includes: The Actor-Critic network is used as the dual network model. The Actor network outputs the action space for continuous flight control according to the current state space, the Critic network evaluates the reward function according to the output of the Actor network, and the reward function result is fed back to the Actor network to optimize the output of the action space.

[0031] In step S130, the task is executed based on the initial inspection path, the fusion data affected by the outside world is obtained in real time, the physical potential model for automatic obstacle avoidance is generated according to the fusion data, the gradient descent method is used to solve the physical potential model, and the flight strategy of the initial inspection path from the current position to the next task point is adjusted in real time. Specifically includes: The state space model of the laser radar, visual SLAM and millimeter wave radar under the federated Kalman architecture, wherein the laser radar is used to obtain three-dimensional point cloud information of obstacles, the visual SLAM is used to estimate the pose of the unmanned aerial vehicle, and the millimeter wave radar is used to measure the real-time distance from the obstacle; The data collected by the laser radar, visual SLAM and millimeter wave radar are time-stamped to ensure the consistency of different sensor data in the time dimension; The data is calibrated in a unified spatial coordinate system by taking the pose of the unmanned aerial vehicle as a direction reference, and a nearest neighbor association algorithm is used to realize spatial registration of the laser radar and the millimeter wave radar, that is, the laser radar and the millimeter wave radar are arranged at different positions of the unmanned aerial vehicle, it is needed to determine whether the data measured by the two belongs to the same obstacle, to calculate the distance from the laser radar to the unmanned aerial vehicle in the unified coordinate, and to compare the distance with the data measured by the millimeter wave radar, if the distance difference is less than a distance threshold, it is determined that they belong to the same obstacle, that is, the spatial registration is realized, and the three-dimensional position of the obstacle and the velocity vector of the obstacle are obtained; Preferably, an adaptive noise covariance matrix is designed, and the fusion weight is dynamically adjusted according to the real-time measurement residual of each sensor, for example, when the measurement error of a certain sensor increases, the weight proportion of the sensor in data fusion is reduced, that is, if the error of the millimeter wave radar is large, the corresponding confidence is reduced, and the measurement result is expanded to a larger range interval; The environment information is obtained through a weather type sensor, for example, a wind speed sensor, a humidity sensor and the like; After the chi-square test anomaly detection and elimination, the fusion data including the three-dimensional position of the obstacle, the velocity vector of the obstacle and the environment information are output, the three-dimensional position integrates the size and distance of the obstacle, and the velocity vector can be calculated according to the continuous frames of data, the fusion data provides input for the subsequent flight safety evaluation model, and supports the triggering of safety strategies such as adjusting speed, changing flight path to avoid obstacles, adjusting flight height and the like.

[0032] During the flight inspection process, the above-mentioned external influences are easy to be affected, therefore, the path can be adjusted in real time during the flight, in the embodiment of the present application, the execution order of the subtask target points is not changed, but the flight route is adjusted under the condition that the next subtask target point is determined; A physical force model composed of an obstacle repulsive force field, a bad environment repulsive force field, a next subtask target point attractive force field and a path constraint field is constructed, the physical force model is a virtual force field, the unmanned aerial vehicle automatically avoids the area with large repulsive force and moves forward to the target with large attractive force; The distribution of the obstacle and the bad environment is fitted through a Gaussian mixture model based on the fusion data as an avoidance area, and a repulsive force vector of the avoidance area to the unmanned aerial vehicle is simulated, wherein the repulsive force vector increases with the decrease of the distance from the unmanned aerial vehicle; An attractive force vector between the current position of the unmanned aerial vehicle and the next subtask target point is simulated, and a distance attenuation factor is set in the simulation process of the attractive force vector, the attenuation factor avoids falling into a local minimum value, wherein the attractive force vector increases with the decrease of the distance from the unmanned aerial vehicle; In the embodiment of the present application, the attractive force value is the original attractive force divided by the attenuation factor; The turning angle and the acceleration of the unmanned aerial vehicle are limited by setting a path constraint field; The LM nonlinear algorithm, i.e., the Levenberg-Marquardt algorithm, is used as a gradient descent method to solve the physical force model, and the LM nonlinear algorithm can automatically generate an optimized path avoiding obstacles and adverse environments, and the flight strategy is adjusted according to the optimized path, and the flight measurement is adjusted, including adjusting the flight height, changing the flight path direction, etc., so that the path is fitted to the physical performance of the unmanned aerial vehicle.

[0033] According to the demand, the online update can be enabled in real time according to the fusion data affected by the outside world. Preferably, in order to further improve the effect of online optimization, the current flight state can be judged in advance by an evaluation model, and when the flight state does not meet the safety threshold, the adjustment is triggered immediately, otherwise, according to the demand, the result of the initial inspection path can also be directly executed, in order to improve the rapid judgment efficiency of the evaluation model, a lightweight rule-based matching model is used in the embodiment of the present application.

[0034] In step S140, the remaining power of the unmanned aerial vehicle is predicted by the power loss model, and the unmanned aerial vehicle is returned according to the remaining power, specifically including: A power loss multi-dimensional time series including power, flight speed, height change rate, load weight and environmental wind force is constructed; The multi-dimensional time series is decomposed and reconstructed by wavelet, and the features at different time scales are extracted to form short-term fluctuation features and long-term trend features, and the power loss model composed of an ARIMA model and an LSTM model is obtained, wherein the order of the ARIMA model is determined by ADF test, and the parameters are trained by maximum likelihood estimation method; The long-term trend feature is processed by the ARIMA model, and the first prediction result is output, which is used to fit the linear change rule of power loss; The short-term fluctuation feature is processed by the LSTM model, and the second prediction result is output, which is used to fit the nonlinear change rule of power loss, the LSTM sets a three-layer network structure, and uses the Adam optimizer and the mean square error loss function to train the model to adapt to the influence of complex environmental factors; The first prediction result and the second prediction result are weighted and fused, the weight coefficient is dynamically updated by Kalman filtering, the predicted remaining power is obtained, and when the remaining power is lower than the power threshold, the corresponding unmanned aerial vehicle is returned, and is returned to the airport closest in straight line distance and having a charging space. It should be noted that the fitting of this part of the remaining power is for future period prediction, which is different from the current remaining power.

[0035] Preferably, the embodiment of the present application further includes a task reassignment mechanism. When a certain unmanned plane fails, the geographical environment complexity of its unfinished sub-tasks, the data type to be collected and other parameters are extracted, the real-time power of the remaining unmanned planes, the positions, the remaining flight time and the charging status of the nearest airport are combined, a matching algorithm is used to re-allocate the sub-tasks to other unmanned planes, since the flight tasks of the unmanned planes have been executed to half at this time, therefore, the embodiment of the application sets a more convenient and fast way to obtain the historical task completion efficiency of the remaining unmanned planes and the estimated difficulty coefficient of all unfinished sub-tasks, that is, to obtain all the unfinished tasks and the available unmanned planes, re-call the matching algorithm for task allocation, quickly generate an optimal allocation scheme, synchronously update the task queue of each unmanned plane, set the maximum flight radius, the minimum turning radius, the no-fly area and the flight conflict avoidance between unmanned planes as the flight constraint conditions in the subsequent path planning of the unmanned planes, and ensure the efficiency and safety of the multi-collaborative inspection path.

[0036] The above technical solutions of the embodiment of the application are exemplarily illustrated in combination with the following drawings.

[0037] Figure 2 is a schematic diagram of the inspection architecture of the embodiment of the application, as shown in Figure 2 , a complete process of unmanned plane inspection and collaborative control is shown, a mode of pre-judgment of the current flight state by an evaluation model to determine whether to start online adjustment safely is adopted, the collaborative control can be divided into a task initialization layer, an allocation and execution layer, an online safety control layer and an emergency response layer, the task initialization layer is the starting point of the process, including two core modules of task multi-level decomposition and sub-task allocation, the allocation and execution layer includes an initial frog jump type inspection path planning, the online safety control layer includes real-time data collection and safety strategy adjustment parts, and the emergency response layer includes monitoring power consumption and unmanned plane return parts.

[0038] To sum up, in view of the existing problems, the unmanned aerial vehicle inspection cooperative control method of the present application divides the inspection task into subtasks, calculates the historical task completion efficiency of each unmanned aerial vehicle and estimates the difficulty coefficient, and uses a matching algorithm to assign the subtasks to the unmanned aerial vehicles, which helps to assign high-difficulty tasks to high-efficiency and reliable unmanned aerial vehicles first, so that the cluster can complete the inspection in a shorter time and with a higher success rate as a whole; based on the assignment result, an initial inspection path is constructed using a reinforcement learning framework to realize adaptive adjustment of the frog-leaping path, and the Actor-Critic network avoids the optimization of actions in each round of the traditional policy gradient, combines the advantages of policy guidance and value evaluation, and realizes a continuous learning mechanism with stable efficiency; an experience replay mechanism is used to store state transition samples, and a target network technology is used to improve the training stability; in the inspection process, flight data and environmental data are collected through multi-sensor fusion technology, a synthetic physical potential field model is used to intelligently avoid areas with dynamic obstacles or adverse environments, and safety strategies such as speed adjustment, obstacle avoidance and adjustment are automatically triggered to ensure smooth and coherent adjustment process; and the multi-sensor fusion is realized through a federal framework, which can accurately distinguish, associate and continuously track multiple static and dynamic obstacles; a power loss model is constructed based on multiple factors to serve as the basic rule for the return of the unmanned aerial vehicle, ensuring the safety of power management.

[0039] Device embodiment According to the embodiment of the present application, an unmanned aerial vehicle inspection cooperative control device is provided, Figure 3 is a schematic diagram of the unmanned aerial vehicle inspection cooperative control device of the embodiment of the present application, as Figure 3 shown, the unmanned aerial vehicle inspection cooperative control device according to the embodiment of the present application specifically comprises: The task splitting module 30 is used to split the inspection task into subtasks, assign the subtasks to the unmanned aerial vehicles using a matching algorithm, and obtain the task assignment result, and is specifically used for: The inspection task is split into subtasks according to geographical areas, object types and specific inspection items, and a subtask library including geographical coordinates and operation requirements is generated; The historical execution records of the unmanned aerial vehicles are obtained, and detailed logs of all past tasks are comprehensively extracted, the subtask types are various and cover specific operation categories such as power tower inspection, oil and gas pipeline monitoring and traffic facility aerial photography, and the historical task completion efficiency of each unmanned aerial vehicle is calculated based on the historical execution records through a quantitative formula; In the embodiment of the present application, the ratio of the task completion amount to the total operation time, or the ratio of the flight distance to the unit energy consumption, is calculated as the historical task completion efficiency, which reflects the operation ability of the unmanned aerial vehicle in different task scenarios, the flight distance is calculated through satellite positioning trajectory data, and the operation time is accurately calculated as the whole process time from take-off to completion of data return.

[0040] Obtain the environment complexity level and the environment climate of the work area where the subtask is located, the environment complexity is divided into three levels of simple, medium and complex according to the parameters such as the terrain slope, vegetation coverage, building density of the work area, and corresponding scores are given; the environment climate is obtained in combination with real-time data such as wind level, precipitation probability, and visibility range, the data types are set with weights according to data accuracy and data dimension, the accuracy is similar to centimeter level or meter level, and the data dimension includes two-dimensional influence or three-dimensional point cloud; The index data contained in the environment complexity level and the environment climate are weighted and summed to obtain the estimated difficulty coefficient of each subtask.

[0041] The historical task completion efficiency, the estimated difficulty coefficient and the unmanned aerial vehicle state information are input into the dynamic matching algorithm, a matching matrix is constructed through the dynamic matching algorithm, the matching matrix is filled by using multi-objective optimization, and a task allocation result is obtained; The historical task completion efficiency reflects the work capacity of the unmanned aerial vehicle, the estimated difficulty coefficient reflects the execution difficulty of the subtask, and the unmanned aerial vehicle state information includes the endurance time supported by the remaining power, the straight-line distance between the current position and the subtask area, the load device type and weight, etc. That is, in the embodiment of the application, the behavior of the matching matrix is that the unmanned aerial vehicle number column is the subtask number, the initial matching matrix is an empty matrix, or a random matching condition constructed according to historical execution conditions, and the matching is realized by checking or modifying the existing relationship in the empty matrix after solving, and one unmanned aerial vehicle can execute a series of subtasks. Multi-objective optimization is the core of the dynamic matching algorithm, the unmanned aerial vehicle state information is used as a constraint, and the goal of solving is to make the unmanned aerial vehicle with higher work capacity match and execute the subtask with greater difficulty in the allowed range, and the overall task completion time is the shortest and the resource consumption is the lowest, which can be automatically matched through a computer program.

[0042] An initial solving module 32 is configured to solve an initial inspection path of each unmanned aerial vehicle based on the task allocation result and using a path planning algorithm under a reinforcement learning architecture, and set flight constraint conditions in the path planning solving process, and specifically configured to: A Markov decision model including a state space, an action space and a reward function is constructed as the reinforcement learning architecture, wherein the state space includes the real-time position of the unmanned aerial vehicle, and preferably, the state space further includes information such as the remaining power, the distribution of surrounding obstacles and the boundary of the no-fly area, but such information does not conflict with subsequent online adjustment at this stage, the action space includes the adjustment range of the flight direction, speed and height, and the reward function includes the path length, the obstacle avoidance success rate and the task completion progress; The maximum flight radius, minimum turning radius, no-fly area and flight collision avoidance of the unmanned aerial vehicle are set as flight constraints to ensure the efficiency and safety of the multi-collaborative inspection path. An output and evaluation dual network model is used as a path planning algorithm to solve the initial inspection path of each unmanned aerial vehicle, wherein the initial inspection path is a frog jump inspection path connecting the sequence of each subtask, that is, the frog jump point is regarded as the execution point of the subtask, and the path between two subtasks is approximately a straight line except for the start and end point smoothing and path designation caused by the flight constraints. During the solving process, an experience replay mechanism and a target network separation mechanism are used. The experience replay mechanism refers to storing state transition samples for path planning strategy reference. The target network technology separates the copy of the dual network model. Online decision real-time adjustment of network parameters is easily affected by external environment, while the copy adjustment time step is longer. Therefore, when using the dual network model, the output result is as close as possible to the output result of the copy.

[0043] Preferably, after completing a predetermined number of inspection tasks, such as 10 times, the network parameters are iteratively updated based on newly collected environmental data such as obstacle position changes, weather condition updates, etc., so that the path planning strategy can adapt to different working environments and task requirements.

[0044] The dual network model of output and evaluation is used as a path planning algorithm, which specifically includes: The Actor-Critic network is used as the dual network model. The Actor network outputs the action space of continuous control flight according to the current state space, the Critic network evaluates the reward function according to the output of the Actor network, and the reward function result is fed back to the Actor network to optimize the output of the action space.

[0045] The real-time adjustment module 34 is used to execute tasks based on the initial inspection path, to obtain fusion data affected by the outside world in real time, to generate an automatic obstacle avoidance physical force model according to the fusion data, to solve the physical force model using the gradient descent method, and to adjust the flight strategy of the initial inspection path from the current position to the next task point in real time. Specifically used for: Through the state space model of the laser radar, visual SLAM and millimeter wave radar under the federal Kalman architecture, wherein the laser radar is used to obtain three-dimensional point cloud information of obstacles, the visual SLAM is used to estimate the pose of the unmanned aerial vehicle, and the millimeter wave radar is used to measure the real-time distance from the obstacle; The data collected by the laser radar, visual SLAM and millimeter wave radar are time-stamped to ensure the consistency of different sensor data in the time dimension; The data is calibrated in a unified spatial coordinate system by taking the pose of the unmanned aerial vehicle as a direction reference, and a nearest neighbor association algorithm is used to realize spatial registration of the laser radar and the millimeter wave radar, that is, the laser radar and the millimeter wave radar are arranged at different positions of the unmanned aerial vehicle, it is needed to determine whether the data measured by the two belongs to the same obstacle, to calculate the distance from the laser radar to the unmanned aerial vehicle in the unified coordinate, and to compare the distance with the data measured by the millimeter wave radar, if the distance difference is less than a distance threshold, it is determined that they belong to the same obstacle, that is, the spatial registration is realized, and the three-dimensional position of the obstacle and the velocity vector of the obstacle are obtained; Preferably, an adaptive noise covariance matrix is designed, and the fusion weight is dynamically adjusted according to the real-time measurement residual of each sensor, for example, when the measurement error of a certain sensor increases, the weight proportion of the sensor in data fusion is reduced, that is, if the error of the millimeter wave radar is large, the corresponding confidence is reduced, and the measurement result is expanded to a larger range interval; The environment information is obtained through a weather type sensor, for example, a wind speed sensor, a humidity sensor and the like; After the chi-square test anomaly detection and elimination, the fusion data including the three-dimensional position of the obstacle, the velocity vector of the obstacle and the environment information are output, the three-dimensional position integrates the size and distance of the obstacle, and the velocity vector can be calculated according to the continuous frames of data, the fusion data provides input for the subsequent flight safety evaluation model, and supports the triggering of safety strategies such as adjusting speed, changing flight path to avoid obstacles, adjusting flight height and the like.

[0046] During the flight inspection process, the above-mentioned external influences are easy to be affected, therefore, the path can be adjusted in real time during the flight, in the embodiment of the present application, the execution order of the subtask target points is not changed, but the flight route is adjusted under the condition that the next subtask target point is determined; A physical force model composed of an obstacle repulsive force field, an adverse environment repulsive force field, a next subtask target point attractive force field and a path constraint field is constructed, the physical force model is a virtual force field, the unmanned aerial vehicle automatically avoids the area with large repulsive force and moves forward to the target with large attractive force; The distribution of the obstacle and the adverse environment is fitted through a Gaussian mixture model based on the fusion data as an avoidance area, and a repulsive force vector of the avoidance area to the unmanned aerial vehicle is simulated, wherein the repulsive force vector increases with the decrease of the distance from the unmanned aerial vehicle; A gravitational force vector is simulated between the current position of the unmanned aerial vehicle and the next subtask target point, and a distance attenuation factor is set in the simulation process of the gravitational force vector, the attenuation factor avoids falling into a local minimum value, wherein the gravitational force vector increases with the decrease of the distance from the unmanned aerial vehicle; In the embodiment of the present application, the gravitational force value is the original gravitational force divided by the attenuation factor; The turning angle and the acceleration of the unmanned aerial vehicle are limited by setting a path constraint field; The LM nonlinear algorithm, i.e., the Levenberg-Marquardt algorithm, is used as a gradient descent method to solve the physical force model, the LM nonlinear algorithm can automatically generate an optimized path avoiding obstacles and adverse environments, and the flight strategy is adjusted according to the optimized path, and the flight measurement is adjusted, including adjusting the flight height, changing the flight path direction, etc., so that the path is fitted to the physical performance of the unmanned aerial vehicle.

[0047] According to the demand, the online update can be enabled in real time according to the fusion data affected by the outside world, preferably, in order to further improve the effect of online optimization, the current flight state can be judged in advance by an evaluation model, when the flight state does not meet the safety threshold, the adjustment is triggered immediately, otherwise, according to the demand, the result of the initial inspection path can also be directly executed, in order to improve the rapid judgment efficiency of the evaluation model, a lightweight rule-based matching model is used in the embodiment of the present application.

[0048] The prediction return module 36 is used for predicting the remaining power of the unmanned aerial vehicle through the power loss model, and returning the unmanned aerial vehicle according to the remaining power, and is specifically used for: A power loss multi-dimensional time series including power, flight speed, height change rate, load weight and environmental wind force is constructed; The multi-dimensional time series is decomposed and reconstructed by wavelet, features at different time scales are extracted to form short-term fluctuation features and long-term trend features, and a power loss model composed of an ARIMA model and an LSTM model is obtained, wherein the order of the ARIMA model is determined by ADF test, and the parameters are trained by maximum likelihood estimation method; The long-term trend features are processed by the ARIMA model to output a first prediction result, which is used to fit the linear variation law of power loss; The short-term fluctuation features are processed by the LSTM model to output a second prediction result, which is used to fit the nonlinear variation law of power loss, the LSTM sets a three-layer network structure, and uses an Adam optimizer and a mean square error loss function to train the model to adapt to the influence of complex environmental factors; The first prediction result and the second prediction result are weighted and fused, the weight coefficient is dynamically updated by Kalman filtering, the predicted remaining power is obtained, and when the remaining power is lower than the power threshold, the corresponding unmanned aerial vehicle is returned, and is returned to the airport closest in straight line distance and having a charging space, it should be noted that the fitting of this part of the remaining power is for future period prediction, which is different from the current remaining power.

[0049] Preferably, the embodiment of the present application further includes a task reassignment mechanism. The redistribution module 38 is used for extracting the geographical environment complexity, the data type to be collected and other parameters of the unfinished sub-tasks of the unmanned aerial vehicle when the unmanned aerial vehicle fails, combining the real-time power, position, remaining flight time and the charging state of the nearest airport of the remaining unmanned aerial vehicles, and using a matching algorithm to redistribute the sub-tasks to other unmanned aerial vehicles. Since the flight tasks of the unmanned aerial vehicles have been performed to half at this time, the embodiment of the application sets a more convenient and fast way to obtain the historical task completion efficiency of the remaining unmanned aerial vehicles and the estimated difficulty coefficient of all unfinished sub-tasks, that is, to obtain all the unfinished tasks and the available unmanned aerial vehicles at present, and to call the matching algorithm again for task distribution, to quickly generate an optimal distribution scheme, to update the task queue of each unmanned aerial vehicle synchronously, and to set the maximum flight radius, the minimum turning radius, the no-fly area and the flight conflict avoidance between unmanned aerial vehicles as the flight constraint conditions in the subsequent path planning, so as to ensure the efficiency and safety of the multi-collaborative inspection path.

[0050] In summary, in view of the existing problems, the unmanned aerial vehicle inspection collaborative control device of the application divides the inspection task into sub-tasks, calculates the historical task completion efficiency and the estimated difficulty coefficient of each unmanned aerial vehicle, and uses a matching algorithm to distribute the sub-tasks to the unmanned aerial vehicles, which helps to preferentially distribute high-difficulty tasks to high-efficiency and reliable unmanned aerial vehicles, so that the swarm can complete the inspection in a shorter time and with a higher success rate as a whole. Based on the distribution result, an initial inspection path is constructed using a reinforcement learning architecture, adaptive adjustment of the frog jump path is realized, the Actor-Critic network avoids optimizing actions in each round of the traditional policy gradient, combines the respective advantages of policy guidance and value evaluation, and realizes a continuous learning mechanism with stable efficiency. The experience replay mechanism is used to store state transition samples, and the target network technology is used to improve the training stability. In the inspection process, flight data and environmental data are collected through multi-sensor fusion technology, a synthetic physical potential field model is used to intelligently avoid areas with dynamic obstacles or bad environments, and safety strategies such as speed adjustment, obstacle avoidance and adjustment are automatically triggered, which can ensure smooth and coherent adjustment process. The multi-sensor fusion is realized through a federal architecture, which can accurately distinguish, associate and continuously track multiple static and dynamic obstacles. A power loss model is constructed based on multiple factors, which is used as a basic rule for the return of the unmanned aerial vehicle, and ensures the safety of the operation.

[0051] Electronic device embodiment Figure 4is a schematic diagram of an electronic device according to an embodiment of the present application. The electronic device 400 can include at least one processor 410 and a memory 420. The processor 410 can execute instructions stored in the memory 420. The processor 410 is communicatively connected to the memory 420 through a data bus. In addition to the memory 420, the processor 410 can be communicatively connected to an input device 430, an output device 440, and a communication device 450 through the data bus.

[0052] The processor 410 can be any conventional processor, such as commercially available CPUs. The processor can also include a graphics processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0053] The memory 420 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0054] In the embodiments of the present disclosure, the memory 420 stores executable instructions. The processor 410 can read the executable instructions from the memory 420 and execute the instructions to implement all or part of the steps of the UAV inspection cooperative control method of any of the above example embodiments.

[0055] Computer-readable storage medium embodiments In addition to the above method and device, the example embodiments of the present disclosure can also be a computer program product or a computer readable storage medium storing the computer program product, the computer program product including computer program instructions executable by a processor to implement all or part of the steps described in the UAV inspection cooperative control method of any of the above example embodiments.

[0056] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages, and scripting languages (e.g., Python). The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server.

[0057] The computer readable storage medium can be a combination of one or more types of computer readable storage media. The computer readable storage medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium include static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, or any suitable combination of the above.

[0058] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for cooperative control of unmanned aerial vehicle (UAV) inspection, characterized in that, The application relates to a method for unmanned aerial vehicle (UAV) task allocation and path planning. The method comprises the following steps: splitting a patrol task into subtasks, assigning the subtasks to UAVs by using a matching algorithm to obtain a task allocation result; based on the task allocation result, solving an initial patrol path of each UAV by using a path planning algorithm under a reinforcement learning architecture, and setting flight constraint conditions in the path planning solving process; based on the initial patrol path, performing a task, obtaining fusion data affected by the outside world in real time, generating a physical potential model for automatic obstacle avoidance according to the fusion data, solving the physical potential model by using a gradient descent method, and adjusting a flight strategy of the initial patrol path from a current position to a next task point in real time; 2. The method of claim 1, wherein, predicting the remaining power of the UAV by using a power loss model, and returning the UAV according to the remaining power. The task allocation result obtained by assigning the subtasks to the UAVs by using the matching algorithm comprises the following steps: obtaining historical execution records of the UAVs, calculating the historical task completion efficiency of each UAV by using a quantitative formula based on the historical execution records; obtaining the environmental complexity level and the environmental climate of a work area where the subtasks are located, and obtaining an estimated difficulty coefficient of each subtask by weighted summing index data contained in the environmental complexity level and the environmental climate; 3. The method of claim 1, wherein, inputting the historical task completion efficiency, the estimated difficulty coefficient and UAV state information into a dynamic matching algorithm, constructing a matching matrix by using the dynamic matching algorithm, filling the matching matrix by using multi-objective optimization, and obtaining the task allocation result. The initial patrol path of each UAV is solved by using the path planning algorithm under the reinforcement learning architecture based on the task allocation result, and the flight constraint conditions are set in the path planning solving process, which comprises the following steps: constructing a Markov decision model containing a state space, an action space and a reward function as the reinforcement learning architecture, wherein the state space comprises a real-time position of the UAV, the action space comprises an adjustment range of a flight direction, a speed and a height, and the reward function comprises a path length, an obstacle avoidance success rate and a task completion progress; setting the maximum flight radius, the minimum turning radius, the no-fly area and the flight conflict between UAVs of the UAV as the flight constraint conditions; 4. The method of claim 3, wherein, using a double-network model of output and evaluation as the path planning algorithm to solve the initial patrol path of each UAV, wherein the initial patrol path is a frog-leap type patrol path connecting the sequence of the subtasks; and using an experience replay mechanism and a target network separation mechanism in the solving process. The double-network model of output and evaluation used as the path planning algorithm comprises the following steps:

5. The method of claim 1, wherein, using an Actor-Critic network as the double-network model, outputting a continuous control flight action space from the Actor network according to a current state space, evaluating a reward function from the output of the Actor network by using the Critic network, and feeding back the reward function result to the Actor network to optimize the output of the action space. The fusion data affected by the outside world is obtained in real time, which comprises the following steps: The state space model of the laser radar, the visual SLAM and the millimeter wave radar under the federal Kalman framework, wherein the laser radar is used to acquire three-dimensional point cloud information of obstacles, the visual SLAM is used to estimate the pose of the unmanned aerial vehicle, and the millimeter wave radar is used to measure the real-time distance from the unmanned aerial vehicle to the obstacles; The data collected by the laser radar, the visual SLAM and the millimeter wave radar are time-stamped and aligned, the data is calibrated in a spatial coordinate system with the pose of the unmanned aerial vehicle as a directional reference, a nearest neighbor association algorithm is used to realize spatial registration of the laser radar and the millimeter wave radar, and three-dimensional positions of the obstacles and velocity vectors of the obstacles are obtained; The environmental information is acquired through a weather type sensor; After chi-square test anomaly detection and elimination, the fusion data including the three-dimensional positions of the obstacles, the velocity vectors of the obstacles and the environmental information are output.

6. The method of claim 1, wherein, The physical potential model for automatic obstacle avoidance is generated according to the fusion data, the gradient descent method is used to solve the physical potential model, and the flight strategy from the current position of the initial inspection path to the next task point is adjusted in real time, and the flight strategy from the current position of the initial inspection path to the next task point specifically includes: A physical potential model composed of an obstacle repulsive field, an adverse environment repulsive field, a next sub-task target point attractive field and a path constraint field is constructed; The distribution of the obstacles and the adverse environment is fitted through a Gaussian mixture model based on the fusion data as an avoidance area, and a repulsive vector of the avoidance area to the unmanned aerial vehicle is simulated, wherein the repulsive vector increases as the distance to the unmanned aerial vehicle decreases; A gravitational vector between the current position of the unmanned aerial vehicle and the next sub-task target point is simulated, and a distance attenuation factor is set in the simulation process of the gravitational vector, wherein the gravitational vector increases as the distance to the unmanned aerial vehicle decreases; A path constraint field is set to limit the turning angle and acceleration of the unmanned aerial vehicle; The LM nonlinear algorithm is used as the gradient descent method to solve the physical potential model, an optimized path for avoiding obstacles and adverse environments is automatically generated, and the flight strategy is adjusted according to the optimized path.

7. The method of claim 1, wherein, The remaining power of the unmanned aerial vehicle is predicted through a power consumption model, and the unmanned aerial vehicle is returned according to the remaining power, and the method specifically includes: A multi-dimensional time series of power consumption is constructed; Wavelet decomposition and reconstruction are performed on the multi-dimensional time series to extract features at different time scales to form short-term fluctuation features and long-term trend features, and a power consumption model composed of an ARIMA model and an LSTM model is obtained, wherein the order of the ARIMA model is determined through ADF test, and the parameters are trained by maximum likelihood estimation method; The long-term trend features are processed by the ARIMA model to output a first prediction result; The short-term fluctuation features are processed by the LSTM model to output a second prediction result; The first prediction result and the second prediction result are weighted and fused to obtain a predicted remaining power, and the corresponding unmanned aerial vehicle is returned when the remaining power is lower than a power threshold.

8. An unmanned aerial vehicle inspection cooperative control device, characterized in that, The method includes: A task splitting module is configured to split an inspection task into sub-tasks, and a matching algorithm is used to assign the sub-tasks to the unmanned aerial vehicles to obtain a task assignment result. An initial solution module is configured to solve an initial inspection path of each unmanned aerial vehicle based on the task allocation result by using a path planning algorithm under a reinforcement learning architecture, and set a flight constraint condition during path planning solution; A real-time adjustment module is configured to execute a task based on the initial inspection path, acquire fusion data affected by an external environment in real time, generate a physical potential model for automatic obstacle avoidance according to the fusion data, solve the physical potential model by using a gradient descent method, and adjust a flight strategy from a current position to a next task point of the initial inspection path in real time. A prediction return module is configured to predict a remaining power of the unmanned aerial vehicle by using a power consumption model, and return the unmanned aerial vehicle according to the remaining power.

9. An electronic device, comprising: The unmanned aerial vehicle inspection cooperative control method comprises the following steps: A computer readable storage medium stores an implementation program of information transmission, and the program is executed by a processor to implement the steps of the unmanned aerial vehicle inspection cooperative control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an implementation program of information transmission, and the program is executed by a processor to implement the steps of the unmanned aerial vehicle inspection cooperative control method according to any one of claims 1 to 7.

Images