Unmanned aerial vehicle inspection path planning method based on artificial intelligence, medium and server

Abstract

The invention provides an unmanned aerial vehicle routing inspection path planning method based on artificial intelligence, a medium and a server, and the method comprises the steps: obtaining an initial routing inspection path in a power routing inspection region, the initial routing inspection path being a double-side routing inspection path, and a power transmission cable at each side corresponding to a single-side routing inspection path, each single-side inspection path comprises a plurality of I-class inspection path segments for performing power inspection on the power transmission towers and II-class inspection path segments for performing power inspection along power transmission cables between two adjacent power transmission towers, and the I-class inspection path segments and the II-class inspection path segments are connected in sequence; and for a current inspection path segment where the unmanned aerial vehicle is located, obtaining electric power inspection data of the unmanned aerial vehicle in a current time period, and adjusting the current inspection path segment based on the electric power inspection data in the current time period, so that the unmanned aerial vehicle performs electric power inspection based on the adjusted inspection path segment, the power inspection data comprises position data, attitude data and state data of the unmanned aerial vehicle and image data shot by the unmanned aerial vehicle.

Description

Technical Field

[0001] The present application relates to the field of electric power inspection technology, and more specifically, to an artificial intelligence-based drone inspection path planning method, medium, and server. Background Art

[0002] During the UAV power inspection process, the UAV power inspection route planning is a very important link. The transmission line is mainly composed of two parts: the transmission tower and the transmission cable. The transmission cable is connected between two transmission towers and usually has a certain sag. The farther the distance between the transmission towers, the greater the line height difference caused by the sag. For the power inspection of the transmission line, it is necessary to maintain a safe distance during the UAV inspection, and the clarity of the image taken during the UAV inspection must also be considered.

[0003] However, existing drone power inspections usually require staff to plan inspection routes. If refined inspections are to be achieved, more accurate inspection routes must be designed manually, which is a heavy burden for staff. Therefore, how to achieve intelligent inspection route planning is a technical problem that needs to be solved in this field.

[0004] To achieve intelligent inspection route planning, there are currently two directions that can be considered for development. One is to perform real-time data processing and route adjustment, which relies on data processing capabilities and communication capabilities. The other is to perform precise position measurement and spatial estimation, which relies on high-precision positioning and spatial estimation (for example, accurately estimating the height distribution characteristics of transmission cables caused by the sag between any two adjacent transmission towers), but it cannot cope with real-time conditions and has poor flexibility. This patent adopts the first idea to solve the problem and realize intelligent drone inspection route planning. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an artificial intelligence-based drone inspection path planning method, medium and server, which uses artificial intelligence to quickly process various data collected by the drone in real time during the inspection process based on the initial path, so as to adjust the drone's power inspection path in real time.

[0006] In order to achieve the above purpose, the embodiments of the present application are implemented in the following manner:

[0007] In a first aspect, an embodiment of the present application provides an artificial intelligence-based UAV inspection path planning method, comprising: obtaining an initial inspection path within a power inspection area, wherein the initial inspection path is a bilateral inspection path, each side of the transmission cable corresponds to a unilateral inspection path, each unilateral inspection path contains a number of Class I inspection path segments for power inspection of transmission towers and Class II inspection path segments for power inspection along the transmission cables between two adjacent transmission towers, and the Class I inspection path segments and the Class II inspection path segments are connected in sequence; for the current inspection path segment where the UAV is located: obtaining the power inspection data of the UAV in the current time period, and based on the power inspection data of the current time period, adjusting the current inspection path segment, so that the UAV performs power inspection based on the adjusted inspection path segment, wherein the power inspection data includes the UAV's position data, attitude data, status data and image data taken by the UAV.

[0008] In combination with the first aspect, in a first possible implementation method of the first aspect, when the current inspection path segment where the drone is located is a Class II inspection path segment, the current inspection path segment is adjusted based on the power inspection data of the current time period, including: for the current time period T: detecting the image data in the power inspection data, determining the position and direction of the transmission cable on this side in the image data, and obtaining the cable characteristics corresponding to the transmission cable on this side; based on the cable characteristics corresponding to the transmission cable on this side, combined with the position data, attitude data, and status data of the drone, determining the inspection path adjustment amount corresponding to the current time period T; based on the inspection path adjustment amount corresponding to the current time period T, adjusting the segment path of the next time period T+1 in the current inspection path segment.

[0009] In combination with the first possible implementation method of the first aspect, in a second possible implementation method of the first aspect, the image data includes n high-definition images, the number of transmission cables on each side is 6, the image data in the power inspection data is detected, the position and direction of the transmission cables on this side in the image data are determined, and the cable features corresponding to the transmission cables on this side are obtained, including: preprocessing and edge detection of each high-definition image in the image data of the power inspection data, and determining the transmission cables on this side in each high-definition image; for each high-definition image: numbering each transmission cable on this side in the high-definition image, and identifying the position and direction of the numbered transmission cables on this side to form a cable component corresponding to each transmission cable on this side; integrating the cable components corresponding to each transmission cable on this side in all high-definition images to obtain the cable features corresponding to the transmission cables on this side in the image data.

[0010] In combination with the second possible implementation method of the first aspect, in a third possible implementation method of the first aspect, the position and direction of the numbered local transmission cables are identified to form a cable component corresponding to each local transmission cable, including: for each numbered local transmission cable: determining the pixel coordinate group of the local transmission cable with this number in the current high-definition image, and further determining the cable starting coordinates, cable starting width, cable ending coordinates and cable ending width; based on the pixel coordinate group, cable starting coordinates, cable starting width, cable ending coordinates and cable ending width of the local transmission cable with this number, generating the cable component corresponding to the local transmission cable with this number.

[0011] In combination with the second possible implementation method of the first aspect, in a fourth possible implementation method of the first aspect, based on the cable characteristics corresponding to the transmission cable on this side and combined with the position data, attitude data, and status data of the drone, the inspection path adjustment amount corresponding to the current time period T is determined, including: based on each data collection time node within the current time period T, the cable characteristics corresponding to the transmission cable on this side, the position data, attitude data, and status data of the drone are processed to obtain input data containing 6 groups of input vectors, wherein each number corresponds to a group of input vectors; the input data is input into the staged path planning model to obtain the inspection path adjustment amount corresponding to the current time period T.

[0012] In combination with the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the staged path planning model includes an input layer, an LSTM module, a fully connected layer and an output layer, wherein the input layer is used to receive input data; the LSTM module includes 6 parallel LSTM layers, each LSTM layer is used to receive a set of input vectors input by the input layer, and perform feature extraction on the input vector to obtain a corresponding feature vector; the fully connected layer includes 3 fully connected units, each fully connected unit is used to receive the feature vectors extracted by the 6 LSTM layers, and predict the 6 feature vectors to predict the corresponding inspection path adjustment components, wherein the 3 fully connected units are respectively used to predict the x-axis adjustment component, the y-axis adjustment component and the z-axis adjustment component in the inspection path adjustment amount; the output layer is used to output the inspection path adjustment amount corresponding to the current time period T.

[0013] In combination with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, an attention module is inserted between the LSTM module and the fully connected layer, and the attention module includes 3 attention units, each of which is connected to a fully connected unit, for assigning a weight to each feature vector to the fully connected unit when predicting the corresponding adjustment component.

[0014] In combination with the first possible implementation method of the first aspect, in the seventh possible implementation method of the first aspect, based on the inspection path adjustment amount corresponding to the current time period T, the segmented path of the next time period T+1 in the current inspection path segment is adjusted, including: obtaining the end point coordinates of the segmented path of time period T and the end point coordinates of the segmented path of time period T+1 in the current inspection path segment; updating the end point coordinates of the segmented path of time period T+1 based on the inspection path adjustment amount to obtain the updated coordinates of the segmented path of time period T+1; generating a smooth segmented path based on the end point coordinates of the segmented path of time period T and the updated coordinates of the segmented path of time period T+1 to obtain the adjusted segmented path of the next time period T+1.

[0015] In a second aspect, an embodiment of the present application provides a storage medium, which is arranged in a drone or a server and includes a stored program, wherein when the program is running, the drone or the server where the storage medium is located is controlled to execute the artificial intelligence-based drone inspection path planning method described in the first aspect or any one of the possible implementation methods of the first aspect.

[0016] In a third aspect, an embodiment of the present application provides a server comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the steps of the artificial intelligence-based drone inspection path planning method described in the first aspect or any one of the possible implementation methods of the first aspect are implemented.

[0017] Beneficial effects:

[0018] 1. This scheme obtains the initial inspection path in the power inspection area, obtains the power inspection data of the drone in the current period during the inspection process (including the location data, attitude data, status data and image data taken by the drone), and adjusts the current inspection path segment based on the power inspection data of the current period, so that the drone can perform power inspection based on the adjusted inspection path segment. This scheme can use the initial inspection path (if the section is the first inspection, the inspection path of the transmission tower and the inspection path of the transmission cable between the transmission towers can be set by locating the transmission tower; if it is not the first inspection, the historical inspection path can be used as the initial inspection path), and then use the real-time data collected during the drone inspection process to adjust the inspection path in real time, reducing the burden on the staff, and the automated path planning can quickly respond to environmental changes, such as the sag change of the transmission cable, the influence of wind, etc., so that the drone can inspect along the optimal path, improving the inspection efficiency.

[0019] 2. This solution adopts the method of real-time data processing and path adjustment, and dynamically adjusts the subsequent inspection path according to the inspection data of the current period of the drone (including position, posture, state and image, etc.) (for example, adjusting the segmented path of the next period T+1 according to the data of the current period T), which can improve the adaptability of the drone to complex inspection environments. In addition, by adjusting the inspection path in real time, it can ensure that the drone always maintains a safe distance from the transmission cable, optimize the shooting distance, and improve the clarity of the captured image, thereby improving the inspection quality. In the feature extraction process, the high-definition image is used for preprocessing and edge detection to extract the relevant features (such as features reflecting the position and direction) of each transmission cable on this side (and number each transmission cable on this side), integrate the cable components corresponding to each transmission cable on this side in all high-definition images, and obtain the cable features corresponding to the transmission cable on this side in the image data (such as the pixel coordinate group, cable start coordinates, cable start width, cable end coordinates and cable end width of each numbered transmission cable on this side in the current high-definition image). Combined with the drone's position data (such as the drone's real-time position coordinates), attitude data (such as the drone's yaw angle, pitch angle, roll angle, etc.) and state data (such as flight speed, acceleration, wind force and direction, etc.), the processing is performed. The final processing is input data containing 6 sets of input vectors, which are input into the staged path planning model to predict the inspection path adjustment amount corresponding to the current period T. This feature extraction scheme can take into account a variety of factors and can comprehensively reflect the various elements of drone inspections along the transmission cables. Each data collection node in the current period is recorded and reflected, and the accuracy of the data can be considered, which is conducive to improving the prediction accuracy of the inspection path adjustment amount corresponding to the current period.

[0020] 3. The independently developed phased path planning model uses the input layer to receive input data; the LSTM module contains 6 parallel LSTM layers, each LSTM layer is used to receive a set of input vectors input by the input layer, and extract features from the input vectors to obtain corresponding feature vectors; the fully connected layer contains 3 fully connected units, each fully connected unit is used to receive the feature vectors extracted by the 6 LSTM layers, and predict the 6 feature vectors to predict the corresponding inspection path adjustment components. The 3 fully connected units are used to predict the x-axis adjustment component, y-axis adjustment component and z-axis adjustment component in the inspection path adjustment amount respectively; the output layer outputs the inspection path adjustment amount corresponding to the current time period T. In addition, considering the positional relationship of different transmission cables under the perspective of drone shooting and the key points that can be reflected (for example, the components of the middle cable on the x and y axis coordinates have better reflection ability, because the cables in the middle tend to be a straight line forward, while the transmission cables on both sides have better reflection ability on the z axis coordinates, because the transmission cables on both sides can reflect the sag of the transmission cables to a certain extent under the perspective of drone shooting), an attention module is inserted between the LSTM module and the fully connected layer. The attention module contains 3 attention units, each of which is connected to a fully connected unit, which is used to assign the weight of each feature vector to the fully connected unit when predicting the corresponding adjustment component. In this way, the trained staged path planning model can more effectively take into account the distance in the height direction to predict a more accurate and appropriate inspection path adjustment amount, so that the inspection path of the drone can be intelligently planned, ensuring that the drone adapts to the sag of the transmission cables in various complex environments and always maintains a safe and high-quality inspection distance with the transmission cables.

[0021] 4. When updating the inspection path of the next stage, obtain the end point coordinates of the segmented path of time period T and the end point coordinates of the segmented path of time period T+1 in the current inspection path segment; update the end point coordinates of the segmented path of time period T+1 based on the inspection path adjustment amount, and obtain the updated coordinates of the segmented path of time period T+1; generate a smooth segmented path based on the end point coordinates of the segmented path of time period T and the updated coordinates of the segmented path of time period T+1, and obtain the adjusted segmented path of the next time period T+1. This method can realize smooth inspection path planning, ensure the stability and safety of the drone during flight, and reduce the flight risk caused by sudden path changes. In addition, this solution can give priority to the safety of the drone when the communication is poor. For example, if the inspection path adjustment amount after real-time processing cannot be received in the current time period, the drone will transition to the initial path (for example, gradually restore to the initial path within 3 time periods), avoiding the danger caused by the drone still flying at a low altitude in the arc rise section and entering the safe distance of the transmission cable.

[0022] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of an artificial intelligence-based drone inspection path planning method provided in an embodiment of the present application;

[0025] Figure 2 Image data captured by drones during single-side power inspections;

[0026] Figure 3 A schematic diagram for cable identification and numbering for high-definition images;

[0027] Figure 4 A schematic diagram of the width coordinates of a numbered transmission cable at a certain height coordinate in a high-definition image;

[0028] Figure 5 This is a schematic diagram of the structure of the staged path planning model. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0030] The artificial intelligence-based drone inspection path planning method provided in this embodiment uses the server as the execution subject, and the drone cooperates with the server to complete the artificial intelligence-based drone inspection path planning and guide the drone to perform power inspection. Of course, in the case of a higher configuration of the drone, this method can also be directly run by the drone to avoid problems caused by poor communication status.

[0031] See also Figure 1 , Figure 1 A flowchart of a method for planning a drone inspection path based on artificial intelligence provided in an embodiment of the present application. In this embodiment, the method for planning a drone inspection path based on artificial intelligence may include steps S10 and S20.

[0032] First, the server may execute step S10.

[0033] Step S10: Obtain an initial inspection path within the power inspection area, wherein the initial inspection path is a double-sided inspection path, each side of the transmission cable corresponds to a single-sided inspection path, and each single-sided inspection path contains a number of Class I inspection path segments for power inspection of transmission towers and Class II inspection path segments for power inspection along the transmission cables between two adjacent transmission towers, and the Class I inspection path segments and the Class II inspection path segments are connected in sequence.

[0034] In this embodiment, the server can first obtain the initial inspection path in the power inspection area. Since the transmission lines in the power inspection area are mainly the transmission towers and the transmission cables between the transmission towers (the transmission cables are distributed on both sides, and this embodiment takes 6 transmission cables distributed on each side as an example), the focus of the UAV power inspection is usually the transmission towers and transmission cables. Transmission towers usually have similar inspection trajectories, and the points that need to be inspected are usually relatively fixed. Therefore, the focus of this embodiment is on the real-time adjustment of the inspection path of the transmission cables. Then, the initial inspection path is a bilateral inspection path, and each side of the transmission cable corresponds to a unilateral inspection path. Each unilateral inspection path contains several Class I inspection path segments for power inspection of transmission towers and Class II inspection path segments for power inspection along the transmission cables between two adjacent transmission towers. The Class I inspection path segments and the Class II inspection path segments are connected in sequence.

[0035] Based on this, for the power inspection area of ​​the initial inspection, the specific location of the transmission tower (as well as the specific parameters such as the type and height of the transmission tower) can be located to design a Class I inspection path segment for power inspection of the transmission tower. Since the parameters of the transmission tower generally do not change, there is no need to consider the changing factors when designing the inspection path, so there is no need to adjust the Class I inspection path segment corresponding to the transmission tower. Class II inspection path segments are easily affected by factors such as wind and sag, and changes are more common, so Class II inspection path segments need to be adjusted in real time. Then, in the power inspection area of ​​the initial inspection, for any Class II inspection path segment, the end point of the Class I inspection path segment corresponding to the previous transmission tower can be the starting point of this Class II inspection path segment (this embodiment takes the coordinate position of the upper central set height of the Class II inspection path segment as an example), and the starting point of the Class I inspection path segment corresponding to the transmission cable on this side and the next transmission tower is the end point of the Class II inspection path segment (also taking the coordinate position of the upper central set height of the Class II inspection path segment as an example). At this time, the starting point to the end point of the Class II inspection path segment are at the same height, and subsequent adjustments are implemented through data collected in real time during the drone inspection.

[0036] For power inspection areas that are not inspected for the first time, the historical inspection path of the drone power inspection can be retrieved as the initial inspection path for this power inspection.

[0037] After obtaining the initial inspection path, the server may execute step S20.

[0038] Step S20: For the current inspection path segment where the UAV is located: obtain the UAV's power inspection data in the current time period, and adjust the current inspection path segment based on the power inspection data in the current time period, so that the UAV performs power inspection based on the adjusted inspection path segment, wherein the power inspection data includes the UAV's position data, attitude data, status data and image data taken by the UAV.

[0039] In this embodiment, take the current inspection path segment where the drone is located as an example: the server can obtain the power inspection data of the drone in the current time period (for example, one second as a time period, of course, 3 seconds as a time period, or 100 milliseconds as a time period, etc.), and the power inspection data includes the drone's position data (three-dimensional coordinates, collected n times per second), attitude data (such as yaw angle, pitch angle, roll angle, etc., also collected n times per second), state data (such as flight speed, acceleration, collected n times per second, and there may also be data such as wind force and direction, but wind force and direction data are generally recorded separately) and image data taken by the drone (n frames of high-definition images are taken per second). Of course, the remaining power is an important data of the drone, which will be obtained and monitored at all times to ensure that the drone has enough power to return.

[0040] The power inspection data collected by the drone in the current period T (including n time nodes) is shown in Table 1 below:

[0041] Table 1. Power inspection data for the current period T

[0042]

[0043] The high-definition images taken by drones are Figure 2 shown.

[0044] After obtaining the power inspection data of the current period, the server can adjust the current inspection path segment based on the power inspection data of the current period, so that the drone performs power inspection based on the adjusted inspection path segment.

[0045] In this embodiment, only the Class II inspection path segments need to be adjusted in real time. Therefore, for the current time period T: the server can detect the image data in the power inspection data, determine the position and direction of the transmission cable on this side in the image data, and obtain the cable characteristics corresponding to the transmission cable on this side.

[0046] Since the image data contains n high-definition images, and the number of power transmission cables on each side is 6, the server can pre-process and edge detect each high-definition image in the image data of the power inspection data to determine the power transmission cables on this side in each high-definition image. The pre-processing here can include denoising, grayscale (binarization can also be used), etc., and edge detection can be implemented using an edge detection algorithm (such as the Sobel operator) to detect the power transmission cables on this side in the high-definition image (in the high-definition image, the size, direction, area, etc. of the power transmission cables on this side are obviously different from the power transmission cables on the opposite side, and are easy to identify).

[0047] After identifying the local transmission cables in each high-definition image, the server can number each local transmission cable in the high-definition image (such as Figure 3 As shown, the cables are numbered in a clockwise direction, and the numbers in the previous frame are compared to achieve target tracking and avoid wrong numbering), and the positions and directions of the numbered transmission cables on this side are identified to form the cable components corresponding to each transmission cable on this side.

[0048] For example, for each numbered local power transmission cable, the server can determine the pixel coordinate group of the local power transmission cable with this number in the current high-definition image. For example, the height direction in the image can be divided into multiple coordinates, such as 64 height coordinates (or 12), and when these height coordinates are determined, the width coordinates of the local power transmission cable with this number at these height coordinates are determined. Figure 4 As shown, at the same height coordinate, the transmission cable on this side with this number has multiple pixels at this height coordinate, and the average width coordinate of the transmission cable on this side with this number when it is at the same height is calculated as the width coordinate corresponding to the height coordinate, thereby obtaining the coordinate of the transmission cable on this side with this number at the height, thereby obtaining 64 pixel coordinates (or 12) as the pixel coordinate group of the transmission cable on this side with this number in the current high-definition image.

[0049] Then, the server can further determine the starting coordinates of the cable (for example, the pixel coordinates at a height of 0), the starting width of the cable (when the height is zero, the width coordinate difference of the power transmission cable on this side with this number), the ending coordinates of the cable (the pixel coordinates at the maximum height), and the ending width of the cable (when the height is the maximum, the width coordinate difference of the power transmission cable on this side with this number). It should be noted that this embodiment does not consider the situation where the power transmission cable enters the high-definition image from the side, because the shooting angle of the drone is limited, and this situation will not exist. The power transmission cable that enters the high-definition image from the side is generally the power transmission cable on the opposite side, which does not need to be identified and processed by this solution.

[0050] According to the pixel coordinate group, cable start coordinate, cable start width, cable end coordinate and cable end width of the transmission cable on this side with this number, the cable component corresponding to the transmission cable on this side with this number is generated, which is recorded as l. For example, in the first frame of the high-definition image in the current period, the transmission cable on this side with the number 1 generates the cable component corresponding to the transmission cable on this side with the number 1 through the pixel coordinate group (taking 64 pixel coordinates as an example), the cable start coordinate, the cable start width, the cable end coordinate and the cable end width, which is recorded as:

[0051]

[0052] in, is the cable component corresponding to the transmission cable numbered 1 in the first frame of the HD image in the current period, u1,v1 is the pixel coordinate value when the height is 0, that is, the starting coordinate of the cable, u 64 ,v 64 is the pixel coordinate value when the height is maximum, that is, the end coordinate of the cable, w1 is the starting width of the cable, and w 64 The end width of the cable.

[0053] Based on this, the cable component corresponding to each numbered transmission cable on this side in each high-definition image of the current period can be obtained: By integrating the cable components corresponding to each transmission cable on this side in all these high-definition images, the cable features corresponding to the transmission cable on this side in the image data can be obtained. For example, the integration is:

[0054] L={L 1 ,L 2 ,…,L i ,…,L 6}, (2)

[0055]

[0056] L is the cable feature corresponding to the transmission cable on this side in the image data of the current period, L i is the characteristic of the transmission cable on this side numbered i, is the j-th cable component of the transmission cable on this side numbered i, i∈[1,6], j∈[1,n], i, j, n are all positive integers.

[0057] Afterwards, the server can determine the inspection path adjustment amount corresponding to the current time period T based on the cable characteristics corresponding to the transmission cable on this side and the location data, attitude data, and status data of the drone.

[0058] In this embodiment, the server can process the cable characteristics corresponding to the transmission cable on this side, the position data, attitude data and status data of the drone based on each data collection time node in the current period T, and obtain input data including 6 groups of input vectors, where each number corresponds to a group of input vectors. Specifically, the input data including 6 groups of input vectors is as follows:

[0059] S={S 1 ,S 2 ,…,S i ,…,S 6}, (4)

[0060]

[0061] Among them, S is the input data, S i is the i-th input vector in the input data, i∈[1,6], The input vector S i The jth feature in is the jth cable component of the transmission cable numbered i on this side, d j =(x j ,y j ,z j ) is the jth time point T in the current period j The collected drone location data, A j =(Yaw j ,Pitch j ,Roll j ) is the jth time point T in the current period j The collected drone attitude data, v j =(vx j ,vy j ) is the jth time point T in the current period j The speed data collected by the drone, a j =(ax j ,ay j ) is the jth time point T in the current period j Acceleration data collected from the drone.

[0062] After determining the input data, the server can input the input data into the staged path planning model to obtain the inspection path adjustment amount corresponding to the current time period T.

[0063] See also Figure 5 , Figure 5 Schematic diagram of the structure of the staged path planning model. In this embodiment, the staged path planning model may include an input layer, an LSTM module, a fully connected layer and an output layer.

[0064] The input layer is used to receive input data and input the six input vectors of the input data into the corresponding LSTM layers in the LSTM module respectively.

[0065] The LSTM module contains 6 parallel LSTM layers. Each LSTM layer is used to receive a set of input vectors from the input layer and extract features from the input vectors to obtain the corresponding feature vectors, which are recorded as h. i , the eigenvector h i The input vector S is the i-th LSTM layer i Feature extraction is performed.

[0066] The fully connected layer includes three fully connected units. Each fully connected unit is used to receive the feature vectors extracted by the six LSTM layers, and predict the six feature vectors to predict the corresponding inspection path adjustment components. Among them, the three fully connected units are respectively used to predict the x-axis adjustment component, y-axis adjustment component and z-axis adjustment component in the inspection path adjustment amount.

[0067] The output layer is used to output the inspection path adjustment amount corresponding to the current time period T.

[0068] The transmission cables in the middle (such as No. 1 and No. 4) have better reflective ability in the x- and y-axis coordinates, because the transmission cables in the middle tend to be in a straight line forward (such as Figure 3 As shown), the transmission cables on both sides (such as numbers 2, 3, 5, and 6) have a better ability to reflect the components on the z-axis coordinate, because the transmission cables on both sides can reflect the sag of the transmission cables to a certain extent under the perspective of the drone. Then, in order to improve the accuracy of the prediction, this embodiment inserts an attention module between the LSTM module and the fully connected layer. The attention module also includes 3 attention units, each of which is connected to a fully connected unit, which is used to assign a weight to each feature vector when the fully connected unit predicts the corresponding adjustment component.

[0069] Specifically, for the attention unit connected to the fully connected unit that predicts the component on the x-axis coordinate, the parameters of the attention weight are designed as:

[0070]

[0071]

[0072] in, represents the i-th eigenvector h i The attention weight of the component prediction on the x-axis coordinate (corresponding to the i-th numbered transmission cable), i∈[1,6], β x is the basic weight vector for component prediction on the x-axis coordinate, γx is the learning weight vector for component prediction on the x-axis coordinate, is the basic weight vector β x The i-th basic weight factor in, To learn the weight vector γ k The i-th learning weight factor in, It is calculated and The product of x γ x is to calculate β x With γ x The dot product of .

[0073] Similarly, for the attention unit connected to the fully connected unit that predicts the component on the y-axis coordinate, the parameters of the attention weight are designed as:

[0074]

[0075] in, represents the i-th eigenvector h i The attention weight of the component prediction on the y-axis coordinate (corresponding to the i-th numbered transmission cable), i∈[1,6], β y is the basic weight vector for component prediction on the y-axis coordinate, γ y is the learning weight vector for component prediction on the y-axis coordinate, is the basic weight vector β y The i-th basic weight factor in, To learn the weight vector γ y The i-th learning weight factor in, It is calculated and The product of y γ y is to calculate β y With γ y The dot product of .

[0076] For the attention unit connected to the fully connected unit that predicts the component on the z-axis coordinate, the parameters of the attention weight are designed as:

[0077]

[0078]

[0079] in, represents the i-th eigenvector h i The attention weight of the component prediction on the z-axis coordinate (corresponding to the i-th numbered transmission cable), i∈[1,6], β zis the basic weight vector for component prediction on the z-axis coordinate, γ z is the learning weight vector for component prediction on the z-axis coordinate, is the basic weight vector β z The i-th basic weight factor in, To learn the weight vector γ z The i-th learning weight factor in, It is calculated and The product of z γ z is to calculate β z With γ z The dot product of .

[0080] For the training of the staged path planning model, the training data can collect inspection data of multiple inspection areas (this embodiment also performs initial path planning on some sections of the inspected area, and collects data at various flight altitudes between the flight altitude in the initial path and the flight altitude in the optimal path, so as to improve the richness of the training data and enhance the adaptability of the model), perform feature processing in the above manner, process it into training data, and use the corresponding optimal path as the response data in the model, which is processed into a verification result (similar to a label), so that in the training stage of the staged path planning model, the mean square error is used to calculate the loss, and the back propagation algorithm is used to update the model parameters (including the parameter update of the learning weight vector of each attention unit in the attention module) to complete the training of the staged path planning model. The specific training process will not be repeated here.

[0081] After obtaining the inspection path adjustment amount corresponding to the current time period T (for example, expressed as Δd=(Δx, Δy, Δz)), the server can adjust the segmented path of the next time period T+1 in the current inspection path segment based on the inspection path adjustment amount corresponding to the current time period T.

[0082] In this embodiment, the server can obtain the end point coordinates of the segmented path of time period T and the end point coordinates of the segmented path of time period T+1 in the current patrol path segment, and then update the end point coordinates of the segmented path of time period T+1 based on the patrol path adjustment amount (add Δx, Δy, Δz to the end point coordinates of the segmented path of time period T+1, and the patrol path adjustment amount accumulated in the previous time period in the current patrol path segment, but in the case of poor communication status, if there is a patrol path adjustment amount missing in a certain time period, then the previously accumulated patrol path adjustment amount is cleared and re-accumulated from that time period onwards to avoid safety hazards), and obtain the updated coordinates of the segmented path of time period T+1.

[0083] In addition, when communication is poor, priority is given to ensuring the safety of the drone. For example, in the current period, the drone cannot receive the inspection path adjustment after real-time processing. In this case, the drone will transition to the initial path (for example, gradually return to the initial path within three periods) to avoid the danger of the drone maintaining low-altitude flight during the sag rising section and entering the safe distance of the transmission cable.

[0084] Afterwards, the server may generate a smooth segmented path based on the end point coordinates of the segmented path of time period T and the updated coordinates of the segmented path of time period T+1, and obtain an adjusted segmented path of the next time period T+1.

[0085] In summary, the embodiment of the present application provides a method, medium and server for planning the inspection path of a drone based on artificial intelligence, by obtaining the initial inspection path in the power inspection area, obtaining the power inspection data of the drone in the current period during the inspection process (including the position data, attitude data, status data and image data taken by the drone), and adjusting the current inspection path segment based on the power inspection data of the current period, so that the drone performs power inspection based on the adjusted inspection path segment. This scheme can use the initial inspection path (if the section is the first inspection, the inspection path of the transmission tower and the inspection path of the transmission cable between the transmission towers can be set by locating the transmission tower; if it is not the first inspection, the historical inspection path can be used as the initial inspection path), and then the data collected in real time during the drone inspection process can be used to adjust the inspection path in real time, which reduces the burden on the staff, and the automated path planning can quickly respond to environmental changes, such as the sag change of the transmission cable, the influence of wind, etc., so that the drone can inspect along the optimal path, improving the inspection efficiency.

[0086] This solution adopts the method of real-time data processing and path adjustment, and dynamically adjusts the subsequent inspection path according to the inspection data of the current period of the drone (including position, posture, state and image, etc.) (for example, adjusting the segmented path of the next period T+1 according to the data of the current period T), which can improve the adaptability of the drone to complex inspection environments. In addition, by adjusting the inspection path in real time, it can ensure that the drone always maintains a safe distance from the transmission cable, optimize the shooting distance, and improve the clarity of the captured image, thereby improving the inspection quality. In the feature extraction process, the high-definition image is used for preprocessing and edge detection to extract the relevant features (such as features reflecting the position and direction) of each transmission cable on this side (and number each transmission cable on this side), integrate the cable components corresponding to each transmission cable on this side in all high-definition images, and obtain the cable features corresponding to the transmission cable on this side in the image data (such as the pixel coordinate group, cable start coordinates, cable start width, cable end coordinates and cable end width of each numbered transmission cable on this side in the current high-definition image). Combined with the drone's position data (such as the drone's real-time position coordinates), attitude data (such as the drone's yaw angle, pitch angle, roll angle, etc.) and state data (such as flight speed, acceleration, wind force and direction, etc.), the processing is performed. The final processing is input data containing 6 sets of input vectors, which are input into the staged path planning model to predict the inspection path adjustment amount corresponding to the current period T. This feature extraction scheme can take into account a variety of factors and can comprehensively reflect the various elements of drone inspections along the transmission cables. Each data collection node in the current period is recorded and reflected, and the accuracy of the data can be considered, which is conducive to improving the prediction accuracy of the inspection path adjustment amount corresponding to the current period.

[0087] The independently developed phased path planning model uses the input layer to receive input data; the LSTM module contains 6 parallel LSTM layers, each of which is used to receive a set of input vectors from the input layer, and extract features from the input vectors to obtain corresponding feature vectors; the fully connected layer contains 3 fully connected units, each of which is used to receive the feature vectors extracted by the 6 LSTM layers, and predict the 6 feature vectors to predict the corresponding inspection path adjustment components. The 3 fully connected units are used to predict the z-axis adjustment component, y-axis adjustment component and z-axis adjustment component in the inspection path adjustment amount, respectively; the output layer outputs the inspection path adjustment amount corresponding to the current time period T. In addition, considering the positional relationship of different transmission cables under the perspective of drone shooting and the key points that can be reflected (for example, the components of the middle cable on the x and y axis coordinates have better reflection ability, because the transmission cable located in the middle tends to be a straight line forward, while the transmission cables located on both sides have better reflection ability for the components on the z axis coordinate, because the transmission cables located on both sides can reflect the sag of the transmission cables to a certain extent under the perspective of drone shooting), an attention module is inserted between the LSTM module and the fully connected layer. The attention module contains 3 attention units, each of which is connected to a fully connected unit, which is used to assign the weight of each feature vector to the fully connected unit when predicting the corresponding adjustment component. In this way, the trained staged path planning model can more effectively consider the distance in the height direction to predict a more accurate and appropriate inspection path adjustment amount, so that the inspection path of the drone can be intelligently planned, ensuring that the drone adapts to the sag of the transmission cables in various complex environments and always maintains a safe and high-quality inspection distance with the transmission cables.

[0088] When updating the inspection path of the next stage, obtain the end point coordinates of the segmented path of time period T and the end point coordinates of the segmented path of time period T+1 in the current inspection path segment; update the end point coordinates of the segmented path of time period T+1 based on the inspection path adjustment amount, and obtain the updated coordinates of the segmented path of time period T+1; generate a smooth segmented path based on the end point coordinates of the segmented path of time period T and the updated coordinates of the segmented path of time period T+1, and obtain the adjusted segmented path of the next time period T+1. This method can realize smooth inspection path planning, ensure the stability and safety of the drone during flight, and reduce the flight risk caused by sudden path changes. In addition, this solution can give priority to the safety of the drone when the communication is poor. For example, if the inspection path adjustment amount after real-time processing cannot be received in the current time period, the drone will transition to the initial path (for example, gradually restore to the initial path within 3 time periods), avoiding the danger caused by the drone still flying at a low altitude in the arc rise section and entering the safe distance of the transmission cable.

[0089] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for unmanned aerial vehicle inspection path planning based on artificial intelligence, characterized in that: include: Obtain an initial inspection path within the power inspection area, wherein the initial inspection path is a double-sided inspection path, each side of the transmission cable corresponds to a single-sided inspection path, each single-sided inspection path includes a number of Class I inspection path segments for power inspection of transmission towers and Class II inspection path segments for power inspection along the transmission cables between two adjacent transmission towers, and the Class I inspection path segments and the Class II inspection path segments are connected in sequence; For the current inspection path segment where the UAV is located: obtain the UAV's power inspection data in the current time period, and adjust the current inspection path segment based on the power inspection data in the current time period, so that the UAV performs power inspection based on the adjusted inspection path segment, wherein the power inspection data includes the UAV's position data, attitude data, status data and image data taken by the UAV.

2. The artificial intelligence-based UAV inspection path planning method according to claim 1 is characterized in that: When the current inspection path segment where the drone is located is a Class II inspection path segment, the current inspection path segment is adjusted based on the power inspection data of the current period, including: For the current period T: Detect the image data in the power inspection data, determine the position and direction of the transmission cable on this side in the image data, and obtain the cable features corresponding to the transmission cable on this side; Based on the cable characteristics corresponding to the power transmission cable on this side, combined with the position data, attitude data, and status data of the UAV, the inspection path adjustment amount corresponding to the current period T is determined; Based on the inspection path adjustment amount corresponding to the current time period T, the segmented path of the next time period T+1 in the current inspection path segment is adjusted.

3. The artificial intelligence-based UAV inspection path planning method according to claim 2 is characterized in that: The image data contains n high-definition images. The number of transmission cables on each side is 6. The image data in the power inspection data is detected to determine the position and direction of the transmission cables on this side in the image data, and the cable features corresponding to the transmission cables on this side are obtained, including: Preprocess and detect edges of each high-definition image in the image data of the power inspection data, and determine the power transmission cables on this side in each high-definition image; For each high-definition image: number each local power transmission cable in the high-definition image, and identify the position and direction of the numbered local power transmission cable to form a cable component corresponding to each local power transmission cable; The cable components corresponding to each transmission cable on this side in all high-definition images are integrated to obtain the cable features corresponding to the transmission cables on this side in the image data.

4. The method for planning a drone inspection path based on artificial intelligence according to claim 3 is characterized in that: The position and direction of the numbered local transmission cables are identified to form the cable components corresponding to each local transmission cable, including: For each numbered transmission cable on this side: Determine the pixel coordinate group of the transmission cable on this side with this number in the current high-definition image, and further determine the cable start coordinates, cable start width, cable end coordinates and cable end width; The cable component corresponding to the power transmission cable on this side with this number is generated according to the pixel coordinate group, cable start coordinate, cable start width, cable end coordinate and cable end width of the power transmission cable on this side with this number.

5. The artificial intelligence-based UAV inspection path planning method according to claim 3 is characterized in that: Based on the cable characteristics corresponding to the transmission cable on this side, combined with the position data, attitude data, and status data of the drone, the inspection path adjustment amount corresponding to the current period T is determined, including: Based on each data collection time node in the current period T, the cable features corresponding to the transmission cable on this side, the position data, attitude data and status data of the UAV are processed to obtain input data containing 6 groups of input vectors, where each number corresponds to a group of input vectors; The input data is input into the staged path planning model to obtain the inspection path adjustment amount corresponding to the current time period T.

6. The artificial intelligence-based UAV inspection path planning method according to claim 5 is characterized in that: The staged path planning model includes an input layer, an LSTM module, a fully connected layer, and an output layer. The input layer is used to receive input data; The LSTM module includes 6 parallel LSTM layers, each LSTM layer is used to receive a set of input vectors input by the input layer, and perform feature extraction on the input vectors to obtain corresponding feature vectors; The fully connected layer includes three fully connected units, each of which is used to receive the feature vectors extracted by the six LSTM layers, and predict the six feature vectors to predict the corresponding inspection path adjustment components, wherein the three fully connected units are respectively used to predict the x-axis adjustment component, the y-axis adjustment component and the z-axis adjustment component in the inspection path adjustment amount; The output layer is used to output the inspection path adjustment amount corresponding to the current time period T.

7. The method for unmanned aerial vehicle inspection path planning based on artificial intelligence according to claim 6 is characterized in that: An attention module is inserted between the LSTM module and the fully connected layer. The attention module contains three attention units. Each attention unit is connected to a fully connected unit to assign a weight to each feature vector when the fully connected unit predicts the corresponding adjustment component.

8. The artificial intelligence-based UAV inspection path planning method according to claim 2 is characterized in that: Based on the inspection path adjustment amount corresponding to the current time period T, the segment path of the next time period T+1 in the current inspection path segment is adjusted, including: Obtain the end point coordinates of the segment path of time period T and the end point coordinates of the segment path of time period T+1 in the current inspection path segment; Based on the inspection path adjustment amount, the end point coordinates of the segmented path in time period T+1 are updated to obtain the updated coordinates of the segmented path in time period T+1; Based on the end point coordinates of the segmented path of time period T and the updated coordinates of the segmented path of time period T+1, a smooth segmented path is generated to obtain the adjusted segmented path of the next time period T+1.

9. A storage medium, characterized in that: The storage medium is set in a drone or a server, and includes a stored program, wherein when the program is running, the drone or the server where the storage medium is located is controlled to execute the artificial intelligence-based drone inspection path planning method described in any one of claims 1 to 8.

10. A server comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, characterized in that: When the program instructions are loaded and executed by the processor, the steps of the artificial intelligence-based drone inspection path planning method described in any one of claims 1 to 8 are implemented.

Images