Unmanned aerial vehicle automatic classification method and system based on power transmission line, and storage medium

Abstract

The invention discloses an unmanned aerial vehicle automatic classification method and system based on a power transmission line and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: S1, obtaining an image information set of a scanning region and a corresponding point cloud information set; s2, establishing a neural network model, outputting a semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector, and labeling the label information of the sub-regions; s3, sequentially integrating the point cloud data in the point cloud information set based on a time sequence to generate extended point cloud data of the scanning area; and S4, dividing the point data in the sub-regions in the extended point cloud data into one class based on the scanning angle of the scanning region, and dividing the extended point cloud data into a plurality of point cloud classes based on the number of the sub-regions in the two-dimensional classification region. According to the invention, the point cloud data of the extended scanning area can be integrated and then automatically classified, so that unstable point cloud data is avoided, and the error rate of a point cloud classification result is relatively high.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an automatic classification method, system, and storage medium for unmanned aerial vehicles (UAVs) based on power transmission lines. Background Technology

[0002] With the rapid development of drone technology, drones are being used more and more widely in many fields, such as military, security, surveying and mapping, and agriculture. In the process of drone application, effective classification and management of drone point cloud data is of great significance for ensuring the safety and application of drones.

[0003] For example, Chinese patent application "CN113989571A" discloses a method, apparatus, electronic device, and storage medium for classifying point cloud data. The method includes: acquiring point cloud data from a lidar system; removing useless data from the point cloud data to obtain target point cloud data; registering the target point cloud data according to coordinate systems of different angles to obtain registered point cloud data; classifying the registered target point cloud data according to a support vector machine method and feature information of the point cloud data to obtain classified point cloud data; constructing a 3D model of the classified point cloud data; and displaying the 3D model. This invention improves the classification accuracy of point cloud data through a point cloud data classification method. For example, Chinese patent application CN114119625A discloses the segmentation and classification of point cloud data. A system may include a computer, which includes a processor and a memory. The memory stores instructions executable by the processor to receive point cloud data. The instructions also include instructions for generating multiple feature maps based on the point cloud data. Each feature map in the multiple feature maps corresponds to a parameter of the point cloud data. The instructions also include instructions for aggregating the multiple feature maps into an aggregated feature map. The instructions also include instructions for generating at least one of a segmentation output or a classification output based on the aggregated feature map via a feedforward neural network.

[0004] However, the existing technologies described above classify the feature information and feature maps generated from point cloud data. In practice, the usability and randomness of point cloud data can affect the accuracy of point cloud data classification. Therefore, it is necessary to classify point cloud data in conjunction with the corresponding image information set to improve the accuracy of automatic classification of point cloud data. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides an automatic classification method, system, and storage medium for unmanned aerial vehicles (UAVs) based on power transmission lines, thereby resolving the issues in the prior art.

[0006] To achieve the aforementioned objectives, this invention proposes an automatic classification method for unmanned aerial vehicles (UAVs) based on power transmission lines, comprising: The image information set and the corresponding point cloud information set of the scanned area are acquired based on a preset time interval. The image information set and the point cloud information set respectively contain multiple time series image data and point cloud data. A neural network model is established, the image information set is input into the neural network model, a synthetic feature vector is set, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled. The point cloud data in the point cloud information set are sequentially integrated based on the time series to generate extended point cloud data of the scanned area; Based on the label information, the sub-regions of all the image data are summarized to generate a two-dimensional classification region of the scanning area. Based on the scanning angle of the scanning area, the point data located in the sub-regions of the extended point cloud data are classified into one category. Based on the number of the sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories.

[0007] Further generation of the extended point cloud data includes the following steps: The point cloud data includes coordinate information of point data. The point cloud data is sorted sequentially based on the time series. The point cloud data of the first time series is set as the baseline point cloud data, and the point cloud data of the next time series is set as the comparison point cloud data. Based on the coordinate information, the distance between the point data in the baseline point cloud data and the point data in the comparison point cloud data are obtained respectively. The magnitude of the distance between the point data is compared, and a first threshold and a second threshold are set respectively. If the minimum distance between the point data is greater than or equal to zero and less than the first threshold, then the point data in the baseline point cloud data is... The data is set as a fixed point cloud. If the minimum distance between the point data is greater than or equal to the second threshold, the point data of the reference point cloud data and the comparison point cloud data are set as new point clouds respectively. The fixed point cloud and the new point cloud are combined into new reference point cloud data. The new reference point cloud data is set as the reference point cloud data and integrated with the point data of the next comparison point cloud data. This step is repeated based on the number of time series to integrate the point data of the point cloud data in the point cloud information set. The generated new reference point cloud data is set as the extended point cloud data.

[0008] Furthermore, the first threshold and the second threshold are set respectively based on the following steps: The first threshold E is calculated based on the first formula, which is: Where t1 is the baseline point cloud data and t2 is the comparison point cloud data. The mean planar accuracy of the reference point cloud data. The mean planar accuracy of the compared point cloud data. The planar alignment error between the reference point cloud data and the comparison point cloud data. These are the weighting coefficients. The first deviation value of the reference point cloud data. This is the second deviation value of the compared point cloud data; The subplane containing the most point data in the reference point cloud data and the comparison point cloud data is set as an approximate plane. The point distance between the point data of the approximate plane in the reference point cloud data and the point data of the approximate plane in the comparison point cloud data is obtained. The average value of the minimum point distance between the point data in the two sets of approximate planes is set as the second threshold.

[0009] Furthermore, the mean plane accuracy, the plane alignment error, the first deviation value, and the second deviation value are obtained based on the following steps: Based on the coordinate information, two adjacent points in the point cloud data are connected to generate the curvature value of the point data. A curvature threshold is set, and the points with curvature values ​​less than or equal to the curvature threshold are connected sequentially to generate the subplane. If the curvature value is greater than the curvature threshold, the point data is used as the starting point to reconnect with other points so that the curvature values ​​of two adjacent points are less than or equal to the curvature threshold. This step is repeated to divide all the points in the point cloud data into multiple subplanes. In the point cloud data, the first distance between the point data and the subplane is obtained, and the subplane with the smallest first distance is set as the nearest subplane of the point data. The average of the first distances from all the point data to the nearest subplane in the same nearest subplane is obtained and set as the subplane error of the nearest subplane. Based on the number of subplanes in the point cloud data, the average of all the subplane errors is obtained and set as the average plane accuracy of the point cloud data. The distance between each subplane in the reference point cloud data and each subplane in the comparison point cloud data is sequentially obtained and defined as the second distance. The minimum second distance is obtained based on the magnitude of the second distance. The mean of all the minimum second distances is calculated based on the number of subplanes in the reference point cloud data. The mean of the minimum second distance is set as the plane alignment error between the reference point cloud data and the comparison point cloud data. The measurement error r of the nearest sub-plane in the point cloud data is calculated based on the second formula, which is: ,in, Here, N is the measurement error parameter of the scanning instrument, and N is the number of point data points in the nearest subplane. Let D be the distance from the i-th point data in the nearest subplane to the scanning device, and let D be the distance from the nearest subplane to the scanning device. Based on the number of nearest subplanes contained in the point cloud data and the measurement error, the average standard deviation is obtained. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

[0010] Furthermore, generating the point cloud category includes the following steps: Based on the same scanning angle, the extended point cloud data is sequentially superimposed onto the two-dimensional classification region of the scanning area, and the label information of the two-dimensional classification region is marked on the point data contained in the extended point cloud data. Based on the same label information, the extended point cloud data is classified to generate the point cloud category.

[0011] This invention also provides an automatic classification system based on UAV point cloud data. This system is used to implement the aforementioned automatic UAV classification method based on power transmission lines. The system mainly includes: The data collection module acquires image information sets and corresponding point cloud information sets of the scanned area based on a preset time interval. The image information sets and the point cloud information sets respectively contain multiple time-series image data and point cloud data. The image processing module establishes a neural network model, inputs the image information set into the neural network model, sets a synthetic feature vector, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled. The point cloud integration module integrates the point cloud data in the point cloud information set sequentially based on the time series to generate extended point cloud data of the scanned area. The point cloud classification module summarizes the sub-regions of all the image data based on the label information to generate a two-dimensional classification region of the scanned area. Based on the scanning angle of the scanned area, the point data located in the sub-regions of the extended point cloud data are classified into one category. Based on the number of the sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories.

[0012] The present invention also provides a computer storage medium storing program instructions, wherein the program instructions, when executed, control the device where the computer storage medium is located to execute the above-described automatic classification method for unmanned aerial vehicles based on power transmission lines.

[0013] Compared with the prior art, the beneficial effects of the present invention are at least as follows: This invention first acquires image information sets and corresponding point cloud information sets of the scanned area simultaneously at preset time intervals, ensuring overlapping areas. This effectively improves the usability of point cloud data and facilitates registration and integration. Then, by sequentially comparing two adjacent sets of point cloud data in a time series, different values ​​for the first and second thresholds are set. Based on the minimum distance between points, the point data in the two sets are divided into fixed point clouds and newly added point clouds. Further, all point cloud information sets are integrated to generate extended point cloud data, eliminating point cloud data generated by moving objects. Finally, a neural network model performs image recognition and semantic segmentation on the image information sets to generate multiple sub-regions and corresponding label information. Based on the scanning angle, point data from the extended point cloud data can be superimposed onto the sub-regions, and point data with the same label information can be set as point cloud categories, further classifying the extended point cloud data.

[0014] This invention also improves the accuracy of semantic segmentation of neural network models by setting synthetic feature vectors, which can effectively improve the classification accuracy of point cloud data located at the edge of objects. This invention can automatically classify point cloud data scanned multiple times at different times within an unrestricted spatial range with high accuracy. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of the automatic classification method for unmanned aerial vehicles based on power transmission lines according to the present invention. Figure 2 This is a schematic diagram of the UAV scanning process in this invention; Figure 3 This is a schematic diagram showing the connection between point data in two sets of point cloud data in this invention; Figure 4 This is a schematic diagram illustrating the calculation of the average planar accuracy of point cloud data in this invention; Figure 5 This is a schematic diagram illustrating the calculation of the average alignment error between two sets of point cloud data in this invention. Figure 6 This is a structural diagram of the automatic classification system based on UAV point cloud data of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this application, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script.

[0018] like Figure 1 As shown, the automatic classification method for UAVs based on power transmission lines includes: Step S1: Based on a preset time interval, acquire the image information set and the corresponding point cloud information set of the scanned area. The image information set and the point cloud information set each contain multiple time series image data and point cloud data.

[0019] Specifically, in this embodiment, the drone can simultaneously collect image data by installing devices such as LiDAR and cameras, and generate point cloud data corresponding to the image data. The scanning area refers to the target area acquired by the drone through scanning, including but not limited to the substation area. By setting a time interval, the drone can continuously and intermittently acquire image data and corresponding point cloud data of the scanning area, combining them to generate an image information set and a corresponding point cloud information set. The time interval is set based on the drone's flight speed and flight orientation. The generated image information set has a time series. To improve the classification accuracy of the point cloud data, it is necessary for two consecutive adjacent image data to have overlapping areas, such as... Figure 2 As shown, the UAV scans the scanning area Q from scanning angles E1 and E2, respectively. The dashed line represents the scanning range, and the shaded area Q1 is the overlapping area scanned by the UAV. Point cloud data refers to a set of vectors in a three-dimensional coordinate system, recorded in the form of point data. Each point data packet contains three-dimensional coordinates and can carry other information about the point's attributes, such as color, reflectivity, and intensity. Each point contains three-dimensional coordinates and other features. Point cloud data can be used to represent objects, paths, scenes, etc. in space. Point cloud data plays an important role in fields such as 3D modeling. Point cloud data can usually be refined and classified to improve its usability. In this embodiment, the number of image information sets is not limited, that is, the range of the scanning area is not limited, allowing the scanning area to expand at set time intervals. For example, the larger the overlapping area of ​​the UAV's scanning area, the more complete the generated image information set, and the higher the usability of the corresponding point cloud data.

[0020] Step S2: Establish a neural network model. Input the image information set into the neural network model, set the synthetic feature vector, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled.

[0021] Specifically, in this embodiment, the neural network model refers to a convolutional neural network used for image recognition and semantic segmentation. The synthesized feature vector refers to the feature vector generated by setting the feature extraction method in the neural network model in this invention. The neural network model obtains image information by synthesizing feature vectors to achieve semantic segmentation of image data, which can effectively improve the accuracy of semantic segmentation. The label information refers to the class information of the image recognition after the neural network model trains on the image information set, such as building a, building b, road 1, power transmission line and ground, etc. The sub-region refers to each image region generated after the image data is semantically segmented. Among them, the class information is the feature information generated by the neural network model after performing image recognition on the image information set, including the category, texture and edge information of each pixel in the image. Therefore, the label information is used to label each sub-region.

[0022] Step S3: Integrate the point cloud information and point cloud data sequentially based on time series to generate extended point cloud data of the scanned area.

[0023] Specifically, in this embodiment, extended point cloud data refers to generating more usable point cloud data by dividing and integrating all point cloud data contained in the scanned area. That is, extended point cloud data includes all point cloud data to be classified, and point cloud data generated by object movement is removed, which facilitates the effective classification of point cloud data in the future. Since point cloud information is concentrated in the same overlapping area, the process of dividing and integrating all point cloud data in this embodiment also belongs to the point cloud registration process.

[0024] Step S4: Summarize the sub-regions of all image data based on the label information to generate a two-dimensional classification region of the scanned area. Based on the scanning angle of the scanned area, classify the point data located in the sub-regions of the extended point cloud data into one category. Based on the number of sub-regions in the two-dimensional classification region, divide the extended point cloud data into multiple point cloud categories.

[0025] Specifically, in this embodiment, the image information set and the point cloud information set are data acquired from the same time series and the same scanning angle. Therefore, each image data in the image information set corresponds to the point cloud data in the point cloud information set. Summarizing the sub-regions in all image data is to integrate the object regions generated from different scanning angles. Since the same object may be scanned multiple times, it is necessary to summarize the sub-regions generated by the same object in the image data. During the summarization process, the resolution of different image data is set so that the pixel size of each image data is consistent, thereby summarizing and generating a two-dimensional classification region of the scanning area. Based on the label information of each sub-region in the scanning area, the point data at the corresponding position in the extended point cloud data can be labeled with label information. The point cloud data with the same label information are classified, so that all the point data in the extended point cloud data generates multiple point cloud categories, completing the automatic classification process of the point cloud data.

[0026] Generating extended point cloud data includes the following steps: Point cloud data includes the coordinate information of point data. The point cloud data is sorted sequentially based on a time series. The point cloud data of the first time series is set as the baseline point cloud data, and the point cloud data of the next time series is set as the comparison point cloud data. Based on the coordinate information, the distance between the point data in the baseline point cloud data and the point data in the comparison point cloud data are obtained respectively. The magnitude of the distance between the point data is compared, and a first threshold and a second threshold are set respectively. If the minimum distance between the point data is greater than or equal to zero and less than the first threshold, the point data in the baseline point cloud data is set as the fixed point cloud. If the minimum distance between the point data is greater than or equal to the second threshold, the point data in the baseline point cloud data and the comparison point cloud data are set as the new point cloud. The fixed point cloud and the new point cloud are combined into a new baseline point cloud data. The new baseline point cloud data is set as the baseline point cloud data and integrated with the point data of the next comparison point cloud data. This step is repeated based on the number of time series to integrate the point cloud data in the point cloud information set. The generated new baseline point cloud data is set as the extended point cloud data.

[0027] Specifically, in this embodiment, the point cloud information set is multiple point cloud data generated by composing a time series from scanning times. Each time series corresponds to a set of point cloud data. To facilitate the use of the continuity of the point cloud data, the point cloud data of the first time series is set as the reference point cloud data, and the point cloud data of the next adjacent time series is set as the comparison point cloud data. The inter-point distance refers to the distance between two point data calculated based on coordinate information. The inter-point distance can be calculated using the Euclidean distance formula. The formula for calculating the inter-point distance l is: The minimum inter-point distance refers to the minimum distance between points. By obtaining the minimum inter-point distance between each point in the baseline point cloud data and the comparison point cloud data, it is easier to distinguish the positional relationship of each point in the two time series. There may be overlapping point cloud data, so it is necessary to set a first threshold and a second threshold to judge the minimum inter-point distance and determine the positional relationship between the point data in the baseline point cloud data and the comparison point cloud data. The first threshold and the second threshold are set according to the point cloud data of different time series, that is, the first threshold and the second threshold are not fixed values; minimum inter-point distance If the minimum distance between points is greater than 0 and less than the first threshold, it indicates that the point data in the two sets of point cloud data may belong to the same object. Therefore, this point data in the reference point cloud data is set as a fixed point cloud. If the minimum distance between points is greater than or equal to the second threshold, it indicates that the point data in the two sets of point cloud data belongs to newly generated point data in non-overlapping areas. Therefore, the corresponding point data in the two sets of point cloud data are set as newly added point clouds. The first threshold is less than the second threshold. If the minimum distance between points is greater than the first threshold and less than the second threshold, it indicates that there may be duplicate connections between the point data in the two sets of point cloud data. Figure 3 As shown, the same point data h1 in the baseline point cloud data Q2 is connected by multiple point data in the comparison point cloud data Q3. Therefore, point data with the minimum distance between points between the first threshold and the second threshold are removed. This step is repeated until the point cloud data in all point cloud information sets are registered and combined in sequence, and all fixed point clouds and newly added point clouds are integrated to generate extended point cloud data.

[0028] The first threshold and the second threshold are set based on the following steps: The first threshold E is calculated based on the first formula, which is: Where t1 is the baseline point cloud data and t2 is the comparison point cloud data, The mean planar accuracy of the reference point cloud data. To compare the average planar accuracy of point cloud data, The planar alignment error between the reference point cloud data and the comparison point cloud data. These are the weighting coefficients. The first deviation value of the reference point cloud data. This is the second deviation value for comparing point cloud data; The subplane containing the most point data in the reference point cloud data and the comparison point cloud data is set as the approximate plane. The point distance between the point data of the approximate plane in the reference point cloud data and the point data of the approximate plane in the comparison point cloud data is obtained. The average value of the minimum point distance between the point data in the two sets of approximate planes is set as the second threshold.

[0029] Specifically, in this embodiment, due to changes in the time series, the positions of each subplane in the reference point cloud data and the subplane in the comparison point cloud data change. If only a fixed first threshold and second threshold are set to determine the distance between point data, there will be a large error when integrating the point data in the reference point cloud data and the comparison point cloud data, which is not conducive to the subsequent classification of point cloud data. Therefore, in the first formula, various values ​​of the reference point cloud data and the comparison point cloud data are statistically analyzed, and the first threshold is calculated by the distance between the subplanes and the average alignment error of the two sets of point cloud data. This facilitates the integration of each point data in the two sets of point cloud data. Among them, the weighting coefficient... Based on the measurement error parameters of the scanning machine If the measurement error parameter is large, the weighting coefficient is small. The mean plane accuracy refers to the average accuracy of all sub-planes in the point cloud data. The plane alignment error refers to the distance deviation between point data in the overlapping area of ​​two sets of point cloud data. The first deviation value and the second deviation data are the average standard deviation of the measurement error of each nearest sub-plane calculated after calibrating the scanning instrument, which can describe the degree of dispersion of the measurement error of each sub-plane.

[0030] An approximate plane is a subplane containing the most points in point cloud data. By comparing the approximate planes in two sets of point cloud data, the displacement expansion range of the two sets of point cloud data can be compared. By calculating the average of the minimum distances between the point data in the two approximate planes, the displacement expansion range of the two sets of point cloud data can be obtained. Therefore, the average of the minimum inter-point distances between the point data in the two sets of approximate planes is set as the second threshold.

[0031] The mean plane accuracy, plane alignment error, first deviation value, and second deviation value are obtained based on the following steps: Based on coordinate information, connect two adjacent points in the point cloud data to generate the curvature value of the point data. Set a curvature threshold, and connect the point data with curvature values ​​less than or equal to the curvature threshold in sequence to generate a subplane. If the curvature value is greater than the curvature threshold, start from the point data and reconnect with other point data so that the curvature values ​​of two adjacent point data are less than or equal to the curvature threshold. Repeat this step to divide all the point data of the point cloud data into multiple subplanes. In the point cloud data, obtain the first distance between the point data and the subplane, set the subplane with the smallest first distance as the nearest subplane of the point data, obtain the average of the first distances from all point data to the nearest subplane in the same nearest subplane, and set it as the subplane error of the nearest subplane, and obtain the average of all subplane errors based on the number of subplanes in the point cloud data, and set it as the average plane accuracy of the point cloud data. The distance between each subplane in the reference point cloud data and each subplane in the comparison point cloud data is sequentially obtained and defined as the second distance. The minimum second distance is obtained based on the magnitude of the second distance. The mean of all minimum second distances is calculated based on the number of subplanes in the reference point cloud data. The mean of the minimum second distances is set as the plane alignment error between the reference point cloud data and the comparison point cloud data. The measurement error r of the nearest subplane in the point cloud data is calculated based on the second formula, which is: ,in, Here, N represents the measurement error parameter of the scanning instrument, and N is the number of nearest subplane midpoint data points. Let be the distance from the i-th point data in the nearest subplane to the scanning device, and D be the distance from the nearest subplane to the scanning device. The average standard deviation is obtained based on the number of nearest subplanes contained in the point cloud data and the measurement error. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

[0032] Specifically, in this embodiment, the point data in the point cloud data has normals. A normal is a line segment bound to the point data and used to represent the surface direction between points. Each point data in the point cloud data corresponds to a three-dimensional coordinate and a normal. The distance and angle between points are obtained by calculating the distance between points and the angle between the normals of these two points. Then, the final curvature value is obtained through mathematical transformation. All adjacent point data on the point cloud surface determine the shape of the local subplane. For example, let the unit normal vector of point data h3 be H3, and let point data h4 be the nearest neighbor of point data h3. The normal vector is H4 The curvature value k3 of point data h3 is calculated using the osculating circle of point data h3. The calculation formula is as follows: ,in, Normal vector H4 The curvature is calculated by tracing a line segment from the origin of the point data. Similarly, the curvature values ​​of all point data can be calculated. By setting a curvature threshold, multiple point data can be divided into the same plane. The distance between each point data and the plane is within a set range, and the plane is set as a subplane. If the curvature value is greater than the preset curvature threshold, the point data reselects adjacent point data to connect in order to calculate the corresponding curvature value. This step is repeated to divide each point data in the point cloud into multiple subplanes, so that each point data is bound.

[0033] Average error is a quantifiable measure of observation accuracy. In this invention, average error is used to measure the measurement accuracy of all sub-planes generated from point cloud data. The smaller the average error, the higher the measurement accuracy, the more data points closest to that sub-plane, and the more suitable the sub-plane is. Therefore, the average error is set as the mean of plane accuracy. The formula for calculating the average error P is: ,in, The number of point data in the point cloud data. Let be the distance from the j-th point data to the nearest subplane. This method is used to calculate the average planar accuracy of the reference point cloud data and the comparison point cloud data, for example, as... Figure 4 As shown, point cloud data Q5 contains two subplanes q51 and q52, where the nearest subplane to point data h is subplane q52, and the distance is indicated by the dashed line. Let h be the distance from the point data to the nearest subplane q52. Based on the above formula, the average distance between all point data in the point cloud data Q5 and the nearest subplane can be calculated, and this value is set as the average planar accuracy of the point cloud data.

[0034] The average distance refers to the average distance between each subplane in the reference point cloud data and the nearest subplane in the comparison point cloud data. If two subplanes are not parallel, the distance is calculated by taking the cross product of their normal vectors and dividing by the magnitude of the normal vector. The calculation formula is: ,in For subplane normal vector and subplane normal vector cross product between Normal vector The model, Normal vector The average distance can be used to illustrate the alignment error between the corresponding sub-planes. Therefore, the average distance is set as the average alignment error between the reference point cloud data and the comparison point cloud data, for example, as... Figure 5 As shown, the nearest connection distances of each subplane in the reference point cloud data Q2 and the comparison point cloud data Q3 are L1, L2 and L3, respectively. The average alignment error between the reference point cloud data Q2 and the comparison point cloud data Q3 is the result of (L1+L2+L3) / 3.

[0035] Because scanning instruments have a factory-set error coefficient when generating point cloud data, in order to improve the accuracy of point cloud data, the measurement error of a sub-plane is calculated by summing the average measurement error of all point data in the sub-plane using a second formula. In the second formula, the measurement error parameter of the scanning instrument refers to the factory-set error coefficient of the device. By setting the scanning instrument as the shooting coordinate point, the distance D from the shooting coordinate point to the nearest sub-plane can be calculated. The measurement error can indicate the measurement accuracy of the sub-plane in the point cloud data. Therefore, the average standard deviation of the measurement error of all sub-planes in the point cloud data can indicate the degree of dispersion between measurement errors. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

[0036] Setting the synthetic feature vector includes the following steps: The image information set contains multiple time-series image data. Any two adjacent image data sets have overlapping scan areas. A neural network model learns and trains on the image information set to extract feature quantities and corresponding feature contribution rates, and obtains class information from the image information set. The feature quantities are multiplied by the feature contribution rates to generate advanced feature quantities. Based on these advanced feature quantities, the neural network model extracts the feature map corresponding to the highest channel and generates a synthetic feature vector C based on the third formula: Where h is the height of the feature map and w is the width of the feature map. Let be the feature vector of the advanced feature quantity at position j. It is a parametric exponential function, specifically a function , Let be the feature map at position q.

[0037] Specifically, in this embodiment, the feature quantities in the neural network model need to be set to improve the accuracy of semantic segmentation during image recognition, and further improve the accuracy of point cloud data classification. In the image information set, any two adjacent image data sets have overlapping regions, which is to ensure that the corresponding point cloud data have connected regions, effectively dividing and integrating the corresponding point cloud data. The neural network model trains the image information set through convolution. During image recognition, it obtains the class information of the image information set. Class information refers to the category labels contained in the image information set. In semantic segmentation, the feature contribution rate refers to the degree of contribution of each feature vector to the segmentation result. The feature contribution rate is output by the neural network model. For each feature vector, the model calculates its contribution to the segmentation result. The score in the image segmentation result is divided by the probability of the feature vector appearing in the image data to calculate the contribution rate of each feature to the segmentation result. By calculating the contribution rate of the feature, the degree of contribution of each feature vector to the image segmentation result can be evaluated more accurately. The feature is multiplied by the corresponding feature contribution rate to generate advanced feature, and the weight of each feature in the semantic segmentation process can be calculated. The feature map refers to the image generated on the image data by each advanced feature in the neural network model. Through the third formula, all feature maps can be aggregated to generate an aggregate map, which is a graph representing the relationship between different pixels in the input image data. The synthetic feature vector is calculated by using the advanced feature and the aggregate map.

[0038] The neural network model performs semantic segmentation on the image information set using a set of synthetic feature vectors to generate multiple sub-regions. The number of sub-regions corresponds to the number of class information, so the class information is set as the label information of the sub-region.

[0039] Generating point cloud categories involves the following steps: Based on the same scanning angle, the extended point cloud data is sequentially superimposed onto the two-dimensional classification area of ​​the scanning area, and the label information of the two-dimensional classification area is marked in the point data contained in the extended point cloud data. Based on the same label information, the extended point cloud data is classified to generate point cloud categories.

[0040] Specifically, in this embodiment, the extended point cloud data and each sub-region have the same scanning angle. Therefore, the point data in the extended point cloud data are superimposed onto the sub-region based on the scanning angle, so that the point data in the same sub-region are labeled with the corresponding label information. Furthermore, the extended point cloud data is classified into point cloud categories based on the same label information.

[0041] This invention first acquires image information sets and corresponding point cloud information sets of the scanned area simultaneously at preset time intervals, ensuring overlapping areas. This effectively improves the usability of point cloud data and facilitates registration and integration. Then, by sequentially comparing two adjacent sets of point cloud data in a time series, different values ​​for the first and second thresholds are set. Based on the minimum distance between points, the point data in the two sets are divided into fixed point clouds and newly added point clouds. Further, all point cloud information sets are integrated to generate extended point cloud data, eliminating point cloud data generated by moving objects. Finally, a neural network model performs image recognition and semantic segmentation on the image information sets to generate multiple sub-regions and corresponding label information. Based on the scanning angle, point data from the extended point cloud data can be superimposed onto the sub-regions, and point data with the same label information can be set as point cloud categories, further classifying the extended point cloud data.

[0042] This invention also improves the accuracy of semantic segmentation of neural network models by setting synthetic feature vectors, which can effectively improve the classification accuracy of point cloud data located at the edge of objects. This invention can automatically classify point cloud data scanned multiple times at different times within an unrestricted spatial range with high accuracy.

[0043] Of particular note is that this invention can integrate point cloud data from an extended scanning area to achieve automatic classification, thereby avoiding unstable point cloud data that would result in a high error rate in the classification results.

[0044] like Figure 6 As shown, the present invention also provides an automatic classification system based on UAV point cloud data. This system is used to implement the above-mentioned automatic UAV classification method based on power transmission lines. The system mainly includes: The data collection module acquires image information sets and corresponding point cloud information sets of the scanned area based on preset time intervals. The image information sets and point cloud information sets each contain multiple time-series image data and point cloud data. The image processing module establishes a neural network model, inputs the image information set into the neural network model, sets the synthetic feature vector, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions and the label information of the sub-regions is labeled. The point cloud integration module integrates point cloud information and point cloud data sequentially based on time series to generate extended point cloud data of the scanned area. The point cloud classification module summarizes the sub-regions of all image data based on label information to generate a two-dimensional classification region of the scanned area. Based on the scanning angle of the scanned area, the point data located in the sub-regions in the extended point cloud data are classified into one category. Based on the number of sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories.

[0045] The present invention also provides a computer storage medium storing program instructions, wherein, when the program instructions are executed, the device where the computer storage medium is located executes the above-described automatic classification method for unmanned aerial vehicles based on power transmission lines.

[0046] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0047] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

[0050] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automatic classification method for unmanned aerial vehicles (UAVs) based on power transmission lines, characterized in that, The method includes the following steps: The image information set and the corresponding point cloud information set of the scanned area are acquired based on a preset time interval. The image information set and the point cloud information set respectively contain multiple time series image data and point cloud data. A neural network model is established, the image information set is input into the neural network model, a synthetic feature vector is set, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled. The point cloud data in the point cloud information set are sequentially integrated based on the time series to generate extended point cloud data of the scanned area; Based on the label information, the sub-regions of all the image data are summarized to generate a two-dimensional classification region of the scanning area. Based on the scanning angle of the scanning area, the point data located in the sub-regions of the extended point cloud data are classified into one category. Based on the number of the sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories.

2. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 1, characterized in that, Generating the extended point cloud data includes the following steps: The point cloud data includes coordinate information of point data. The point cloud data is sorted sequentially based on the time series. The point cloud data of the first time series is set as the baseline point cloud data, and the point cloud data of the next time series is set as the comparison point cloud data. Based on the coordinate information, the distance between the point data in the baseline point cloud data and the point data in the comparison point cloud data are obtained respectively. The magnitude of the distance between the point data is compared, and a first threshold and a second threshold are set respectively. If the minimum distance between the point data is greater than or equal to zero and less than the first threshold, then the point data in the baseline point cloud data is... The data is set as a fixed point cloud. If the minimum distance between the point data is greater than or equal to the second threshold, the point data of the reference point cloud data and the comparison point cloud data are set as new point clouds respectively. The fixed point cloud and the new point cloud are combined into new reference point cloud data. The new reference point cloud data is set as the reference point cloud data and integrated with the point data of the next comparison point cloud data. This step is repeated based on the number of time series to integrate the point data of the point cloud data in the point cloud information set. The generated new reference point cloud data is set as the extended point cloud data.

3. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 2, characterized in that, The first threshold and the second threshold are set based on the following steps: The first threshold E is calculated based on the first formula, which is: Where t1 is the baseline point cloud data and t2 is the comparison point cloud data. The mean planar accuracy of the reference point cloud data. The mean planar accuracy of the compared point cloud data. The planar alignment error between the reference point cloud data and the comparison point cloud data. These are the weighting coefficients. The first deviation value of the reference point cloud data. This is the second deviation value of the compared point cloud data; The subplane containing the most point data in the reference point cloud data and the comparison point cloud data is set as an approximate plane. The point distance between the point data of the approximate plane in the reference point cloud data and the point data of the approximate plane in the comparison point cloud data is obtained. The average value of the minimum point distance between the point data in the two sets of approximate planes is set as the second threshold.

4. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 3, characterized in that, The mean plane accuracy, the plane alignment error, the first deviation value, and the second deviation value are obtained based on the following steps: Based on the coordinate information, two adjacent points in the point cloud data are connected to generate the curvature value of the point data. A curvature threshold is set, and the points with curvature values ​​less than or equal to the curvature threshold are connected sequentially to generate the subplane. If the curvature value is greater than the curvature threshold, the point data is used as the starting point to reconnect with other points so that the curvature values ​​of two adjacent points are less than or equal to the curvature threshold. This step is repeated to divide all the points in the point cloud data into multiple subplanes. In the point cloud data, the first distance between the point data and the subplane is obtained, and the subplane with the smallest first distance is set as the nearest subplane of the point data. The average of the first distances from all the point data to the nearest subplane in the same nearest subplane is obtained and set as the subplane error of the nearest subplane. Based on the number of subplanes in the point cloud data, the average of all the subplane errors is obtained and set as the average plane accuracy of the point cloud data. The distance between each subplane in the reference point cloud data and each subplane in the comparison point cloud data is sequentially obtained and defined as the second distance. The minimum second distance is obtained based on the magnitude of the second distance. The mean of all the minimum second distances is calculated based on the number of subplanes in the reference point cloud data. The mean of the minimum second distance is set as the plane alignment error between the reference point cloud data and the comparison point cloud data. The measurement error r of the nearest sub-plane in the point cloud data is calculated based on the second formula, which is: ,in, Here, N is the measurement error parameter of the scanning instrument, and N is the number of point data points in the nearest subplane. Let D be the distance from the i-th point data in the nearest subplane to the scanning device, and let D be the distance from the nearest subplane to the scanning device. Based on the number of nearest subplanes contained in the point cloud data and the measurement error, the average standard deviation is obtained. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

5. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 1, characterized in that, Generating the point cloud category includes the following steps: Based on the same scanning angle, the extended point cloud data is sequentially superimposed onto the two-dimensional classification region of the scanning area, and the label information of the two-dimensional classification region is marked on the point data contained in the extended point cloud data. Based on the same label information, the extended point cloud data is classified to generate the point cloud category.

6. An automatic classification system based on UAV point cloud data, used to implement the automatic UAV classification method based on power transmission lines as described in any one of claims 1-5, characterized in that, The system includes the following modules: The data collection module acquires image information sets and corresponding point cloud information sets of the scanned area based on a preset time interval. The image information sets and the point cloud information sets respectively contain multiple time-series image data and point cloud data. The image processing module establishes a neural network model, inputs the image information set into the neural network model, sets a synthetic feature vector, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled. The point cloud integration module integrates the point cloud data in the point cloud information set sequentially based on the time series to generate extended point cloud data of the scanned area. The point cloud classification module summarizes the sub-regions of all the image data based on the label information to generate a two-dimensional classification region of the scanned area. Based on the scanning angle of the scanned area, the point data located in the sub-regions of the extended point cloud data are classified into one category. Based on the number of the sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories.

7. A computer storage medium, characterized in that, The computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the automatic classification method for unmanned aerial vehicles based on power transmission lines as described in any one of claims 1-5.

Images