An operation monitoring method based on drone inspection

Abstract

The present application belongs to the field of drone patrol application technology, and in particular, relates to an operation monitoring method based on drone patrol. It includes the following steps: marking the object to be measured by a marker, and monitoring the marker that changes with the movement of the target by drone patrol; grayscale conversion processing of the image collected by the drone patrol; constructing a grayscale overlay information map by grayscale compression method; determining the spatial matrix; determining the required feature points; generating the angle feature description of the aforementioned feature points based on the accelerated robust feature algorithm; determining the scale parameters, displacement or movement trajectory of the marked object. The operation monitoring method based on drone patrol of the present application provides a low-cost, convenient and fast technical solution for monitoring and positioning of actual operation objects on the basis of drone patrol technology. The solution is highly flexible and the monitoring method is simple and efficient.

Description

Technical Field

[0001] The present application belongs to the technical field of drone patrol applications, and in particular, relates to an operation monitoring method based on drone patrol. Background Art

[0002] In the process of temporary and emergency power distribution operations for specific projects and tasks, in order to increase the speed of operation and shorten the operation cycle, it is often necessary to carry out multi-point different operations in the extension area of ​​the distribution network, so as to establish and improve the complete power distribution infrastructure in a shorter time. In this process, in order to ensure the integrity of the operation projects in different areas, it is necessary to coordinate and unify the specific targets distributed in different operation areas to ensure the correct coordination between the transmission lines or structures after the operation technology, and the scientific and stable line structure. This requires continuous monitoring and control of factors such as the target spatial position and displacement information. Due to the large operation range, the long spatial distance between different operation areas, and the different altitudes and environmental characteristics, it is often difficult to implement effectively, resulting in large errors in the monitoring and control data finally collected. Summary of the invention

[0003] The purpose of this application is to provide an operation monitoring method that can be used to continuously monitor specific target spaces and position information of multiple operation projects based on drone inspection operations.

[0004] To achieve the above objectives, this application adopts the following technical solutions.

[0005] A method for monitoring operations based on drone inspection, comprising the following steps:

[0006] a1. Mark the object to be tested with a marker, and monitor the marker as it moves with the target through drone inspection;

[0007] a2. Perform grayscale conversion on the images collected by the drone inspection to reduce the impact of exposure and weather changes on image feature information;

[0008] a3. The grayscale information at different coordinates of the image is rectangularly compressed by the grayscale compression method, and the grayscale superposition information map I is constructed from the extracted grayscale features. i (x,y);

[0009]

[0010] I(x,y) refers to the grayscale value of the point with pixel coordinates (x,y) before image compression; I i (x, y) is the pixel coordinate of the compressed image (x i ,y i ) The origin of pixel coordinates is located at the corner point of the image;

[0011] a4. Get grayscale superimposed information map I i (x, y), in order to facilitate the determination of the feature point location of the image in the scale space r, the spatial matrix R(x, y, r) is determined by superimposing the information map with grayscale;

[0012]

[0013] Where R xx It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution of R yy It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution of R xy It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution; where G(r) is the convolution function;

[0014] a5. After determining the spatial matrix R(x, y, r) at different spatial scales r, find the feature point judgment function The extreme point I r,j (x,y), and the extreme point I r,j The pixel value of (x, y) and the grayscale superposition information of adjacent pixels I i (x, y) The pixels of the corresponding points are compared. If the extreme point is the extreme point I r,j The pixel value of (x, y) is the largest, which is determined to be the required feature point;

[0015] a6. Generate the angle feature description of the aforementioned feature points based on the accelerated robust feature algorithm and achieve matching and positioning of the marker features based on the aforementioned feature points and angle feature descriptions; the directional feature vector of each feature point needs to be determined during the calculation of the angle description feature;

[0016] a7. Acquire a monitoring image, locate the marker in the image based on the above steps, extract the area and coordinates of the marker, and determine the scale parameter, displacement or movement trajectory of the marked object based on the area and coordinates of the marker.

[0017] A further improvement or preferred implementation scheme of the above-mentioned operation monitoring method based on drone inspection, the step a6 specifically refers to:

[0018] a61. Based on the feature points determined in the preceding step a4, a monitoring area with a radius of 6S and a feature point as the center is constructed based on an accelerated robust feature algorithm;

[0019] a62. Calculate the Haar response values ​​of all feature points in the monitoring area in the coordinate direction, where S is the scale factor;

[0020] a63. Establish a monitoring area divided into 6 equally divided sectors with the feature point as the center and 6S as the radius, determine the Haar response of the feature point inside each sector to the accumulated amount, and take the direction with the largest amount of class addition as the feature point direction;

[0021] a64. Based on the above steps, the main direction of each feature point is obtained. For each feature point, the image space coordinates are converted to the main direction (one of the axis directions is adjusted to the main direction);

[0022] a65. Establish the scale (20S) based on the transformed image 2 The rectangular feature matching area is divided into 4 2 Feature extraction partitioning;

[0023] a66. Calculate the Haar response value d of each area at 2S scale 2s,x ,d 2s,y , and accumulate to get the main direction eigenvector u=(∑d 2s,x ,∑d 2s,y ,|∑d 2s,x |,|∑d 2s,y |); perform normalization to obtain the directional feature vector of each feature point; a56, based on the feature points and directional feature vectors, the points in the image that meet the characteristics of the marker can be finally determined based on the nearest neighbor search, thereby locating the marker.

[0024] A further improvement or preferred implementation scheme of the above-mentioned operation monitoring method based on drone inspection is that the convolution function G(r) adopts a box filter or a Gaussian filter

[0025] A further improvement or preferred implementation scheme of the aforementioned operation monitoring method based on drone inspection utilizes a deep learning network to realize automatic recognition and processing of large amounts of data, and monitors the displacement of the object under test or the dimensional change in a specific direction based on the displacement information of the marker.

[0026] A further improvement or preferred implementation scheme of the above-mentioned operation monitoring method based on drone inspection specifically refers to replacing step a6 with the following step a8, wherein step a8 specifically includes:

[0027] Based on the feature points extracted in the previous steps, a feature map is formed, and the upsampling layer parameters of the deep learning network are initialized according to the feature map to carry out training iterations;

[0028] The downsampling layer based on the deep learning network reduces the training process and the degree of model overfitting;

[0029] The fully connected layer based on the deep learning network maps the extracted features to the label space, calculates the dot product based on the input vector and weight, and passes it to the activation function;

[0030] The candidate regions are generated based on the feature map based on the regional recommendation network RPN. In the process of generating the candidate regions, several anchor boxes that are fully connected to the spatial window of the input convolution feature map are used to slide on the feature map output by the final convolution layer, and the data of the sliding window is mapped to the low-dimensional vector of the intermediate layer. A regression layer consisting of corner point coordinates that are 4 times the number of anchor boxes and a classification layer consisting of attribution probabilities that are 2 times the number of anchor boxes are obtained. The attribution probability includes two parameters: the probability that the candidate region belongs to or does not belong to the monitoring target; the category of each candidate region is calculated based on the fully connected layer and the attribution probability, and the coordinates of the candidate region are calculated based on the corner point coordinates to correct the monitoring area.

[0031] A further improvement or preferred implementation scheme of the above-mentioned operation monitoring method based on drone inspection is to use a loss function to calculate the error between the predicted value and the sample value during the training process of the regional recommendation network RPN. In this embodiment, the loss function is defined as:

[0032]

[0033] Where P i It refers to the predicted probability that the element in the i-th anchor box belongs to the target, P i * is the true label. When the element in the anchor box belongs to a positive sample, P i * = 1, when the element in the anchor box belongs to a negative sample, P i * =0;L c is the loss value of the classification layer, L r is the loss value of the regression layer, t i is a vector used to represent the coordinates of the corner points of the prediction box, is a vector used to represent the coordinates of the corner points of the real box; the specific calculation method is as follows:

[0034] t i =[t x ,t y ,t w ,t h ]=[(xx i ) / w i ,(xx i ) / h i ,log(w / w i ),log(h / h i )]

[0035]

[0036] Where x, y refers to the center coordinates of the prediction box, w, h refers to the width and height of the prediction box respectively; i ,y i refers to the center point coordinates of the anchor box, w i ,h i are the width and height of the anchor box respectively; refers to the coordinates of the center point of the real frame, They refer to the width and height of the real box respectively.

[0037] Its beneficial effects are:

[0038] The operation monitoring method based on drone inspection in the present application provides a low-cost, convenient and fast technical solution for monitoring and positioning of actual operation objects on the basis of drone inspection technology. The solution is highly flexible and the monitoring method is simple and efficient. Through efficient processing and analysis of multi-point data, it is convenient to realize the rapid collection and processing of various parameter structure data at the operation site. It has good versatility and low equipment requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the sector monitoring area division method. DETAILED DESCRIPTION

[0040] The present application is described in detail below in conjunction with specific embodiments.

[0041] Taking into account the various types of equipment, facilities or structures that need to be monitored and controlled at the actual work site, in order to facilitate the unified management and analysis of actual targets, combined with the widespread application of current drone inspection monitoring and management equipment, the present invention eliminates the operation monitoring method based on drone inspection that can provide mobile terminal intelligent collection, combined with target markers and marker displacement extraction.

[0042] a1. In the specific implementation, it is first necessary to mark the object to be measured with a specific marker, and extract and express the displacement information of the marker in the series of monitoring images that changes with the movement of the target through the drone monitoring equipment. The marker is a known object that can be used to express and define the content or progress of the operation project. It is best to be a specific building structure or specific equipment that needs to be used stably during the operation, or other types of objects that can be easily identified and extracted in the monitoring image. For example, for the distribution network line laying project, it can be a crossarm with a series of dimensional standards that are directly related to the extension direction and height distribution of the line, or an insulator on the crossarm, etc. In actual implementation, it is necessary to select the object based on the difficulty of identifying the object and the degree of correlation with the data to be measured. Priority is given to objects with lower identification difficulty and higher correlation with the object to be measured. Lower identification difficulty means more obvious identifiable elements;

[0043] After the markers are determined and selected, it is necessary to establish a recognition framework for the corresponding markers based on the identifiable elements of the markers. The establishment of the recognition framework can be achieved based on machine recognition technology.

[0044] Although the markers can be quickly located from various monitoring images through manual identification, in view of the massive monitoring data and machine recognition technology, it is necessary to use methods such as feature matching recognition to extract various features of the markers so that the computer can effectively identify the corresponding markers;

[0045] a2. In order to improve recognition efficiency, the collected images need to be grayscale converted to reduce the impact of exposure and weather changes on image feature information;

[0046] a3. The grayscale information at different coordinates of the image is rectangularly compressed by grayscale compression method, and grayscale features are extracted from (training images containing markers) to construct grayscale superimposed information map I i (x,y);

[0047]

[0048] I(x,y) refers to the grayscale value of the point with pixel coordinates (x,y) before image compression; I i (x, y) is the pixel coordinate of the compressed image (x i ,y i ) The origin of pixel coordinates is located at the corner point of the image;

[0049] a4. Get grayscale superimposed information map I i (x, y), in order to facilitate the determination of the feature point location of the image in the scale space r, the spatial matrix R(x, y, r) is determined by superimposing the information map with grayscale;

[0050]

[0051] Where R xx It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution of R yy It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution of R xy It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution; wherein G(r) is the convolution function; in this embodiment, the convolution function G(r) adopts a box filter or a Gaussian filter

[0052] a5. After determining the spatial matrix R(x, y, r) at different spatial scales r, find the feature point judgment function The extreme point I r,j (x,y), and the extreme point I r,j The pixel value of (x, y) and the grayscale superposition information of adjacent pixels I i (x, y) The pixels of the corresponding points are compared. If the extreme point is the extreme point I r,j The pixel value of (x, y) is the largest, which is determined to be the required feature point;

[0053] a6. In the actual marker extraction process, due to the operation or image acquisition, the angle of the marker in the original image is not consistent. In order to accurately locate the marker in all images, the present application generates the angle feature description of the aforementioned feature points based on the accelerated robust feature algorithm and realizes the matching and positioning of the marker features based on the aforementioned feature points and angle feature description. In the calculation process of the angle description feature, it is necessary to determine the direction feature vector of each feature point. The specific steps are:

[0054] a67. Based on the feature points determined in the preceding step a4, a monitoring area with a radius of 6S and a feature point as the center is constructed based on an accelerated robust feature algorithm;

[0055] a68. Calculate the Haar response values ​​of all feature points in the monitoring area in the coordinate direction, where S is the scale factor;

[0056] a69. Establish a monitoring area with the feature point as the center and 6S as the radius to form 6 equally divided sectors, such as Figure 1 As shown, the Haar response of the feature point inside each sector area is determined to the cumulative amount, and the direction with the largest cumulative amount is taken as the feature point direction;

[0057] a610. Based on the above steps, the main direction of each feature point is obtained. For each feature point, the image space coordinate is converted to the main direction (one of the axis directions is adjusted to the main direction);

[0058] a611. Establish the scale (20S) based on the transformed image 2 The rectangular feature matching area is divided into 4 2 Feature extraction partitioning;

[0059] a612. Calculate the Haar response value d of each area at 2S scale 2s,x ,d 2s,y , and accumulate to get the main direction eigenvector u=(∑d 2s,x ,∑d 2s,y ,|∑d 2s,x |,|∑d 2s,y |); normalize the directional feature vector of each feature point; a66 can finally determine the points in the image that meet the characteristics of the marker based on the feature points and directional feature vectors, and then locate the marker;

[0060] a7. Acquire a monitoring image, locate the marker in the image based on the above steps, extract the area and coordinates of the marker, and determine the scale parameter, displacement or movement trajectory of the marked object based on the area and coordinates of the marker.

[0061] In particular, in the actual operation monitoring process based on drone inspection, it often involves a large amount of content that only continuously monitors the displacement parameters of the object being measured in real time. In this case, it involves a large amount of continuous monitoring of the displacement of the marker in the image, and generally does not need to consider the feature direction problem involved in step a6. For this situation, the following step a8 can be used to replace the original step a6, and automatic recognition and processing of a large amount of data can be realized based on the deep learning network, and the displacement of the object being measured or the size change in a specific direction can be monitored according to the displacement information of the marker. Step a8 specifically includes:

[0062] Based on the feature points extracted in the previous steps, a feature map is formed, and the upsampling layer parameters of the deep learning network are initialized according to the feature map to carry out training iterations;

[0063] The downsampling layer based on the deep learning network reduces the training process and the degree of model overfitting;

[0064] The fully connected layer based on the deep learning network maps the extracted features to the label space, calculates the dot product based on the input vector and weight, and passes it to the activation function;

[0065] Based on the feature map, the candidate region is generated based on the regional recommendation network RPN. In the process of generating the candidate region, several anchor point boxes that are completely connected to the spatial window of the input convolution feature map are used to slide on the feature map output by the final convolution layer, and the data of the sliding window is mapped to the low-dimensional vector of the intermediate layer. The regression layer composed of the corner point coordinates of 4 times the number of anchor point boxes and the classification layer composed of the attribution probability of 2 times the number of anchor point boxes are obtained. The attribution probability includes two parameters: the probability that the candidate region belongs to or does not belong to the monitoring target; the category of each candidate region is calculated based on the fully connected layer and the attribution probability, and the coordinates of the candidate region are calculated based on the corner point coordinates to correct the monitoring area;

[0066] In particular, during the training process of the region recommendation network RPN, the error between the predicted value and the sample value is calculated using the loss function. In this embodiment, the loss function is defined as:

[0067]

[0068] Where P i It refers to the predicted probability that the element in the i-th anchor box belongs to the target, P i * is the true label. When the element in the anchor box belongs to a positive sample, P i * = 1, when the element in the anchor box belongs to a negative sample, P i * =0;L c is the loss value of the classification layer, L r is the loss value of the regression layer, t i is a vector used to represent the coordinates of the corner points of the prediction box, is a vector used to represent the coordinates of the corner points of the real box; the specific calculation method is as follows:

[0069] t i =[t x ,t y ,t w ,t h ]=[(xx i ) / w i ,(xx i ) / h i ,log(w / w i ),log(h / h i )]

[0070]

[0071] Where x, y refers to the center coordinates of the prediction box, w, h refers to the width and height of the prediction box respectively; i ,y irefers to the center point coordinates of the anchor box, w i ,h i are the width and height of the anchor box respectively; refers to the coordinates of the center point of the real frame, They refer to the width and height of the real box respectively;

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, rather than to limit the scope of protection of the present application. Although the present application has been described in detail with reference to the preferred embodiments, ordinary technicians in this field should understand that the technical solution of the present application can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present application.

Claims

1. A method for monitoring operations based on drone inspection, characterized in that: The following steps are included a1. Mark the object to be tested with a marker, and monitor the marker as it moves with the target through drone inspection; a2. Perform grayscale conversion on the images collected by the drone inspection to reduce the impact of exposure and weather changes on image feature information; a3. The grayscale information at different coordinates of the image is rectangularly compressed by the grayscale compression method, and the grayscale superposition information map I is constructed from the extracted grayscale features. i (x,y); I(x,y) refers to the grayscale value of the point with pixel coordinates (x,y) before image compression; I i (x, y) is the pixel coordinate of the compressed image (x i ,y i ) The origin of pixel coordinates is located at the corner point of the image; a4. Get grayscale superimposed information map I i (x, y), in order to facilitate the determination of the feature point location of the image in the scale space r, the spatial matrix R(x, y, r) is determined by superimposing the information map with grayscale; Where R xx It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution of R yy It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution of R xy It refers to the second-order partial derivative of the point with pixel coordinates (x, y) in the grayscale superposition information map Convolution; where G(r) is the convolution function; a5. After determining the spatial matrix R(x, y, r) at different spatial scales r, find the feature point judgment function The extreme point I r,j (x,y), and the extreme point I r,j The pixel value of (x, y) and the grayscale superposition information of adjacent pixels I i (x, y) to compare the pixels of the corresponding points. If the extreme point I r,j The pixel value of (x, y) is the largest, which is determined to be the required feature point; a6. Generate the angle feature description of the aforementioned feature points based on the accelerated robust feature algorithm and achieve matching and positioning of the marker features based on the aforementioned feature points and angle feature descriptions; the directional feature vector of each feature point needs to be determined during the calculation of the angle description feature; The step a6 specifically refers to: a61. Based on the feature points determined in the preceding step a4, a detection zone with a radius of 6S and a feature point as the center is constructed based on an accelerated robust feature algorithm; a62. Calculate the Haar response values ​​of all feature points in the detection area in the coordinate direction, where S is the scale factor; a63. Establish a detection area with a feature point as the center and a radius of 6S to form 6 equally divided sectors, determine the Haar response of the feature point inside each sector to the accumulated amount, and take the direction with the largest amount of added as the feature point direction; a64. Based on the above steps, the main direction of each feature point is obtained, and for each feature point, the image space coordinates are converted to the main direction; a65. Establish the scale (20S) based on the transformed image 2 The rectangular feature matching area is divided into 4 2 Feature extraction partitioning; a66. Calculate the Haar response value d of each area at 2S scale 2s,x ,d 2s,y , and accumulate to get the main direction eigenvector u=(∑d 2s,x ,∑d 2s,y ,|∑d 2s,x |,|∑d 2s,y |); perform normalization to obtain the directional feature vector of each feature point; a56, based on the feature points and directional feature vectors, the points in the image that meet the characteristics of the marker can be finally determined based on the nearest neighbor search, thereby locating the marker; a7. Acquire a monitoring image, locate the marker in the image based on the above steps, extract the area and coordinates of the marker, and determine the scale parameter, displacement or movement trajectory of the marked object based on the area and coordinates of the marker.

2. The operation monitoring method based on drone inspection according to claim 1 is characterized in that: The convolution function G(r) uses a box filter or a Gaussian filter 3. The operation monitoring method based on drone inspection according to claim 1 is characterized in that: The deep learning network is used to realize automatic recognition and processing of large amounts of data, and the displacement of the object being measured or the dimensional change in a specific direction is monitored based on the displacement information of the marker.

4. The operation monitoring method based on drone inspection according to claim 3 is characterized in that: Specifically, step a6 is replaced by the following step a8, wherein step a8 specifically includes: Based on the feature points extracted in the previous steps, a feature map is formed, and the upsampling layer parameters of the deep learning network are initialized according to the feature map to carry out training iterations; The downsampling layer based on the deep learning network reduces the training process and the degree of model overfitting; The fully connected layer based on the deep learning network maps the extracted features to the label space, calculates the dot product based on the input vector and weights, and passes it to the activation function; The region proposal network (RPN) generates candidate regions based on feature maps. In the process of generating candidate regions, several anchor boxes that are fully connected to the spatial window of the input convolution feature map are used to slide on the feature map output by the final convolution layer, and the data of the sliding window is mapped to the low-dimensional vector of the intermediate layer. A regression layer consisting of corner point coordinates that are 4 times the number of anchor boxes and a classification layer consisting of attribution probabilities that are 2 times the number of anchor boxes are obtained. The attribution probability includes two parameters: the probability that the candidate region belongs to or does not belong to the detection target; the category of each candidate region is calculated based on the fully connected layer and the attribution probability, and the coordinates of the candidate region are calculated based on the corner point coordinates to correct the detection area.

5. The operation monitoring method based on drone inspection according to claim 4 is characterized in that: During the training process of the regional recommendation network RPN, the error between the predicted value and the sample value is calculated using the loss function, which is defined as: Where P i It refers to the predicted probability that the element in the i-th anchor box belongs to the target, P i * is the true label. When the element in the anchor box belongs to a positive sample, P i * = 1, when the element in the anchor box belongs to a negative sample, P i * =0;L c is the loss value of the classification layer, L r is the loss value of the regression layer, t i is a vector used to represent the coordinates of the corner points of the prediction box, is a vector used to represent the coordinates of the corner points of the real box; the specific calculation method is as follows: t i =[t x ,t y ,t w ,t h ]=[(x-x i ) / w i ,(x-x i ) / h i ,log(w / w i ),log(h / h i )] Where x, y refers to the center coordinates of the prediction box, w, h refers to the width and height of the prediction box respectively; i ,y i refers to the center point coordinates of the anchor box, w i ,h i are the width and height of the anchor box respectively; refers to the coordinates of the center point of the real frame, They refer to the width and height of the real box respectively.

Images