Unmanned aerial vehicle automatic classification method and system based on power transmission line, and storage medium

Abstract

The application discloses a kind of unmanned vehicle automatic classification method, system and storage medium based on transmission line, belong to image processing technical field, include: step S1: the image information set of scanning area is obtained and corresponding point cloud information set;Step S2: neural network model is established, based on the semantic segmentation result of image data and corresponding label information of synthesis feature vector output, and the label information of sub-region is labeled;Step S3: point cloud data in point cloud information set is sequentially integrated based on time sequence, and the extended point cloud data of scanning area is generated;Step S4: the point data in extended point cloud data located in sub-region is classified into a class based on the scanning angle of scanning area, and the extended point cloud data is divided into multiple point cloud categories based on the number of sub-region in two-dimensional classification area.By the application, the point cloud data of extended scanning area can be integrated to realize automatic classification, so as to avoid the occurrence of unstable point cloud data and make the error rate of point cloud classification result higher.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an unmanned aerial vehicle automatic classification method and system based on a power transmission line and a storage medium. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, unmanned aerial vehicles are increasingly widely used in many fields, such as the military, security, surveying and mapping, agriculture, and the like. In the process of application of unmanned aerial vehicles, effective classification and management of unmanned aerial vehicle point cloud data are of great significance to ensuring the safety and application of unmanned aerial vehicles.

[0003] For example, Chinese patent application "CN113989571A" discloses a point cloud data classification method, device, electronic equipment and storage medium, which comprises the following steps: obtaining point cloud data of a laser radar system, eliminating useless data in the point cloud data to obtain target point cloud data, registering the target point cloud data according to different angle coordinate systems to obtain registered point cloud data, classifying the registered target point cloud data according to a support vector machine method and feature information of the point cloud data to obtain classified point cloud data, and constructing a 3D model of the classified point cloud data and displaying the 3D model. The application improves the classification accuracy of point cloud data through a point cloud data classification method. Another example is Chinese patent application "CN114119625A", which discloses segmentation and classification of point cloud data. A system can include a computer, the computer including a processor and a memory, the memory storing instructions executable by the processor to receive point cloud data, the instructions further including instructions to generate a plurality of feature maps based on the point cloud data, each feature map of the plurality of feature maps corresponding to a parameter of the point cloud data, the instructions further including instructions to aggregate the plurality of feature maps into an aggregated feature map, and the instructions further including instructions to generate at least one of a segmentation output or a classification output via a feedforward neural network based on the aggregated feature map.

[0004] However, the above-mentioned prior art classifies the feature information and feature maps generated from point cloud data. In actual situations, the usability and randomness of point cloud data can affect the classification accuracy of point cloud data. Therefore, it is necessary to classify point cloud data in combination with corresponding image information sets to improve the automatic classification accuracy of point cloud data. SUMMARY

[0005] To solve the above-mentioned problems, the application provides an unmanned aerial vehicle automatic classification method and system based on a power transmission line and a storage medium to solve the problems in the prior art.

[0006] To achieve the above-mentioned application purposes, the application provides an unmanned aerial vehicle automatic classification method based on a power transmission line, which comprises the following steps:

[0007] acquire image information set and corresponding point cloud information set of a scanning area based on a preset time interval, the image information set and the point cloud information set respectively contain multiple time series of image data and point cloud data;

[0008] establish a neural network model, input the image information set into the neural network model, set a synthetic feature vector, the neural network model outputs the semantic segmentation result of the image data and the corresponding label information based on the synthetic feature vector, divides the image data into multiple sub-regions based on the semantic segmentation result, and labels the label information of the sub-regions;

[0009] integrate the point cloud data in the point cloud information set based on the time sequence in turn to generate extended point cloud data of the scanning area;

[0010] based on the label information, all the sub-regions of the image data are summarized to generate a two-dimensional classification region of the scanning area, the point data in the sub-regions of the extended point cloud data are divided into a class based on the scanning angle of the scanning area, and the extended point cloud data is divided into multiple point cloud categories based on the number of sub-regions in the two-dimensional classification region.

[0011] Further, the generation of the extended point cloud data includes the following steps:

[0012] The point cloud data includes coordinate information of point data, the point cloud data is sequentially sorted based on the time sequence, the point cloud data of the first time sequence is set as the reference point cloud data, the point cloud data of the next time sequence is set as the comparison point cloud data, the inter-point distance between the point data in the reference point cloud data and the point data in the comparison point cloud data is obtained based on the coordinate information, the sizes of the inter-point distances are compared, the first threshold and the second threshold are set respectively, if the minimum inter-point distance between the point data is greater than or equal to zero and less than the first threshold, the point data in the reference point cloud data is set as fixed point cloud, if the minimum inter-point distance between the point data is greater than or equal to the second threshold, the point data of the reference point cloud data and the comparison point cloud data is set as new point cloud respectively, the fixed point cloud and the new point cloud are combined as new reference point cloud data, the new reference point cloud data is set as the reference point cloud data, and the point data of the next comparison point cloud data is integrated, the step is repeated based on the number of time sequences to integrate the point data in the point cloud data in the point cloud information set, and the generated new reference point cloud data is set as the extended point cloud data.

[0013] Further, the first threshold and the second threshold are set based on the following steps respectively:

[0014] The first threshold value E is calculated based on a first formula, and the first formula is: wherein t1 is the reference point cloud data, t2 is the comparison point cloud data, is a plane precision mean value of the reference point cloud data, is the plane precision mean value of the comparison point cloud data, is a plane alignment error between the reference point cloud data and the comparison point cloud data, is a weight coefficient, is a first deviation value of the reference point cloud data, is a second deviation value of the comparison point cloud data;

[0015] The subplane containing the most point data in the reference point cloud data and the comparison point cloud data is set as an approximate plane, the point-to-point distance between the point data of the approximate plane in the reference point cloud data and the point data of the approximate plane in the comparison point cloud data is obtained, and the average value of the minimum point-to-point distance between the point data in the two sets of approximate planes is set as the second threshold value.

[0016] Further, the plane precision mean value, the plane alignment error, the first deviation value and the second deviation value are obtained based on the following steps:

[0017] Based on the coordinate information, two adjacent point data in the point cloud data are connected to generate a curvature value of the point data, a curvature threshold value is set, the point data with a curvature value less than or equal to the curvature threshold value are sequentially connected to generate the subplane, if the curvature value is greater than the curvature threshold value, the point data is taken as a starting point and is reconnected with other point data, so that the curvature values of two adjacent point data are less than or equal to the curvature threshold value, and the step is repeated, and all point data in the point cloud data are divided into a plurality of subplanes;

[0018] In the point cloud data, a first distance between the point data and the subplane is obtained, the subplane with the minimum first distance is set as the nearest subplane of the point data, a mean value of the first distances from all point data in the same nearest subplane to the nearest subplane is obtained and is set as a subplane error of the nearest subplane, a mean value of all subplane errors based on the number of subplanes in the point cloud data is obtained and is set as a plane precision mean value of the point cloud data;

[0019] The distances between the sub-planes in the reference point cloud data and each of the sub-planes in the comparison point cloud data are sequentially obtained, and the distances are defined as second distances; the minimum second distances are obtained based on the sizes of the second distances; the mean of all the minimum second distances is calculated based on the number of the sub-planes in the reference point cloud data; and the mean of the minimum second distances is set as the plane alignment error between the reference point cloud data and the comparison point cloud data.

[0020] The measurement error r of the nearest sub-plane in the point cloud data is calculated based on a second formula, and the second formula is: wherein, is a measurement error parameter of a scanning instrument, N is the number of the point data in the nearest sub-plane, is the distance from the i-th point data in the nearest sub-plane to the scanning instrument, D is the distance from the nearest sub-plane to the scanning instrument, the average standard deviation is obtained based on the number of the nearest sub-planes contained in the point cloud data and the measurement error, the average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

[0021] Further, the generation of the point cloud category includes the following steps:

[0022] The extended point cloud data is sequentially superimposed into the two-dimensional classification region of the scanning region based on the same scanning angle, and the label information of the two-dimensional classification region is marked in the point data contained in the extended point cloud data; and the extended point cloud data is classified based on the same label information to generate the point cloud category.

[0023] The application also provides an automatic classification system based on unmanned aerial vehicle point cloud data, which is used to realize the unmanned aerial vehicle automatic classification method based on power transmission lines described above, and mainly includes:

[0024] A data collection module is configured to obtain an image information set and a corresponding point cloud information set of a scanning region based on a preset time interval, and the image information set and the point cloud information set respectively contain multiple time series of image data and point cloud data;

[0025] An image processing module is configured to establish a neural network model, input the image information set into the neural network model, set a synthetic feature vector, and output a semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector, divide the image data into multiple sub-regions based on the semantic segmentation result, and mark the label information of the sub-regions.

[0026] The point cloud integration module integrates the point cloud data of the point cloud information set based on the time sequence in sequence to generate extended point cloud data of the scanning area.

[0027] The point cloud classification module aggregates the sub-regions of all the image data based on the label information to generate a two-dimensional classification region of the scanning area, classifies the point data in the extended point cloud data located in the sub-regions into a category based on the scanning angle of the scanning area, and divides the extended point cloud data into multiple point cloud categories based on the number of the sub-regions in the two-dimensional classification region.

[0028] The application also provides a computer storage medium storing program instructions, wherein the program instructions control a device where the computer storage medium is located to execute the unmanned aerial vehicle automatic classification method based on a power transmission line.

[0029] Compared with the prior art, the application has at least the following advantages:

[0030] The application first acquires an image information set and a corresponding point cloud information set of a scanning area based on a preset time interval, so that the two sets contain overlapping regions, which can effectively improve the usability of the point cloud data and facilitate the registration and integration of the point cloud data, then compares two groups of point cloud data adjacent in a time sequence in sequence, sets first and second thresholds with different values, divides the point data in the two groups of point cloud data into fixed point clouds and new point clouds according to the size of the minimum point distance, further integrates all the point cloud information sets to generate extended point cloud data, which can eliminate the point cloud data generated by moving objects, finally performs image recognition and semantic segmentation on the image information set through a neural network model to generate multiple sub-regions and corresponding label information, which can be based on a scanning angle to superimpose each point data in the extended point cloud data into a sub-region, set the point data with the same label information as a point cloud category, and further classify the extended point cloud data.

[0031] The application also sets a synthetic feature vector to improve the accuracy of semantic segmentation by the neural network model, which can effectively improve the classification accuracy of point cloud data located at the edges of objects, and the application can automatically classify point cloud data scanned multiple times at different times in an unlimited spatial range with high accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A step flowchart of the unmanned aerial vehicle automatic classification method based on a power transmission line of the application;

[0033] Figure 2 A schematic diagram of the unmanned aerial vehicle scanning process in the application;

[0034] Figure 3A connection diagram of point data in two groups of point cloud data in the application;

[0035] Figure 4 A diagram for calculating the mean of the plane precision of point cloud data in the application;

[0036] Figure 5 A diagram for calculating the average alignment error of two groups of point cloud data in the application;

[0037] Figure 6 A structure diagram of an automatic classification system based on point cloud data of a UAV in the application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the application clearer and more understandable, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0039] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various elements, but unless specifically stated, these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first xx script can be referred to as the second xx script, and similarly, the second xx script can be referred to as the first xx script.

[0040] As shown in Figure 1 The automatic classification method of the UAV based on the power transmission line comprises:

[0041] Step S1: obtaining an image information set and a corresponding point cloud information set of a scanning area based on a preset time interval, the image information set and the point cloud information set comprising a plurality of time series of image data and point cloud data respectively.

[0042] Specifically, in the present embodiment, the UAV can simultaneously collect image data by installing devices such as laser radars and cameras, and generate point cloud data corresponding to the image data. The scanning area refers to a target area obtained by scanning and photographing by the UAV, including but not limited to a substation area. The time interval is set to enable the UAV to continuously and intermittently obtain image data and corresponding point cloud data of the scanning area, and to combine to generate an image information set and a corresponding point cloud information set. The time interval is set based on the flight speed and flight direction of the UAV. The generated image information set has a time sequence. In order to improve the classification accuracy of the point cloud data, it is necessary to make the two adjacent image data have an overlapping area. Figure 2As shown, the unmanned aerial vehicle scans the scanning area Q from the scanning angle E1 and the scanning angle E2 respectively, wherein the dashed line is the scanning range, and the shaded area Q1 is the overlapping area scanned by the unmanned aerial vehicle; the point cloud data refers to a set of vectors in a three-dimensional coordinate system, recorded in the form of point data, each point data contains three-dimensional coordinates, and can carry other information about the attributes of the point, such as color, reflectivity, intensity, etc., wherein each point contains three-dimensional coordinates and other features. Point cloud data can be used to represent objects, paths, scenes, etc. in space. In the field of three-dimensional modeling, point cloud data plays an important role, and point cloud data can usually be classified to improve the usability of point cloud data. In the embodiment, the number of image information sets is not limited, that is, the range of the scanning area is not limited, so that the scanning area can be expanded at a set time interval. For example, the larger the overlapping area of the scanning area scanned by the unmanned aerial vehicle, the more complete the generated image information set, and the higher the usability of the corresponding point cloud data.

[0043] Step S2: Establishing a neural network model, inputting the image information set into the neural network model, setting a synthetic feature vector, and outputting the semantic segmentation result and the corresponding label information of the image data by the neural network model based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into a plurality of sub-regions, and the label information of the sub-regions is labeled.

[0044] Specifically, in the embodiment, the neural network model refers to a convolutional neural network used for image recognition and semantic segmentation, the synthetic feature vector refers to a feature vector generated by setting the feature extraction method in the neural network model in the application, the neural network model obtains image information through the synthetic feature vector, realizes semantic segmentation of image data, and can effectively improve the accuracy of semantic segmentation. Label information refers to class information recognized by the neural network model after training the image information set, for example, building a, building b, road 1, power transmission line and ground, etc. The sub-region refers to each image region generated after the image data is subjected to semantic segmentation. The class information is the feature information generated by the neural network model after image recognition of the image information set, including the class, texture and edge information of each pixel in the image, therefore, the label information is used to label each sub-region.

[0045] Step S3: Integrating the point cloud data in the point cloud information set based on the time sequence in sequence to generate the extended point cloud data of the scanning area.

[0046] Specifically, in the embodiment, the extended point cloud data refers to the point cloud data that is generated after the division and integration of all point cloud data contained in the scanning area, and the usability of the point cloud data is high. That is, the extended point cloud data contains all point cloud data to be classified, and the point cloud data generated by object movement is removed, so as to facilitate subsequent effective classification of the point cloud data. Since the point cloud data in the same overlapping area is concentrated in the point cloud information set, the division and integration process of all point cloud data in the embodiment also belongs to the point cloud registration process.

[0047] Step S4: aggregating all sub-regions of the image data based on the label information to generate a two-dimensional classification region of the scanning area, classifying the point data in the extended point cloud data located in the sub-region based on the scanning angle of the scanning area, and dividing the extended point cloud data into multiple point cloud categories based on the number of sub-regions in the two-dimensional classification region.

[0048] Specifically, in the embodiment, the image information set and the point cloud information set are data obtained at the same time sequence and the same scanning angle. Therefore, each image data in the image information set corresponds to the point cloud data in the point cloud information set. The sub-regions in all image data are aggregated to integrate the object regions generated at different scanning angles. Since the same object may be scanned multiple times, the sub-regions generated by the same object in the image data need to be aggregated. The resolutions of different image data are set in the aggregation process to make the pixel sizes of each image data consistent, so as to aggregate and generate a two-dimensional classification region of the scanning area. Based on the label information of each sub-region in the scanning area, the point data at the corresponding position in the extended point cloud data can be labeled with the label information. The point cloud data with the same label information is classified, so that all point data in the extended point cloud data generates multiple point cloud categories, and the automatic classification process of the point cloud data is completed.

[0049] Generating the extended point cloud data includes the following steps:

[0050] The point cloud data includes coordinate information of point data, the point cloud data is sequentially sorted based on time sequences, point cloud data of a first time sequence is set as reference point cloud data, and point cloud data of a next time sequence is set as comparison point cloud data. The inter-point distance between point data in the reference point cloud data and point data in the comparison point cloud data is obtained based on the coordinate information. The inter-point distance is compared in size. A first threshold value and a second threshold value are set respectively. If the minimum inter-point distance between the point data is greater than or equal to zero and less than the first threshold value, the point data in the reference point cloud data is set as fixed point cloud. If the minimum inter-point distance between the point data is greater than or equal to the second threshold value, the point data in the reference point cloud data and the comparison point cloud data is set as new point cloud respectively. The fixed point cloud and the new point cloud are combined as new reference point cloud data. The new reference point cloud data is set as the reference point cloud data and is integrated with point data of a next comparison point cloud data. The step is repeated based on the number of time sequences to integrate point data in the point cloud information set. The generated new reference point cloud data is set as extended point cloud data.

[0051] Specifically, in the embodiment, the point cloud information set is a plurality of point cloud data generated by scanning time to form time sequences. Each time sequence corresponds to a group of point cloud data. In order to facilitate the continuity of the point cloud data, the point cloud data of the first time sequence is set as reference point cloud data, and the point cloud data of the adjacent next time sequence is set as comparison point cloud data. The inter-point distance refers to the distance between two point data calculated according to the coordinate information. The Euclidean distance formula can be used to calculate the inter-point distance. The calculation formula of the inter-point distance l is: The minimum inter-point distance refers to the minimum value of the inter-point distance. The minimum inter-point distance between each point data in the reference point cloud data and the comparison point cloud data is obtained to facilitate the position relationship of each point data in the two time sequences. There may be point cloud data in the overlap. Therefore, the first threshold value and the second threshold value are set to judge the minimum inter-point distance to determine the position relationship between the point data in the reference point cloud data and the point data in the comparison point cloud data. The first threshold value and the second threshold value are set according to the point cloud data of different time sequences, that is, the first threshold value and the second threshold value do not belong to fixed values. If the minimum inter-point distance is greater than 0 and less than the first threshold value, it means that the point data in the two groups of point cloud data may belong to the point cloud data of the same object. Therefore, the point data in the reference point cloud data is set as fixed point cloud. If the minimum inter-point distance is greater than or equal to the second threshold value, it means that the point data in the two groups of point cloud data belongs to newly generated point data in the non-overlapping area. Therefore, the corresponding point data in the two groups of point cloud data is set as new point cloud respectively. The first threshold value is less than the second threshold value. If the minimum inter-point distance is greater than the first threshold value and less than the second threshold value, it means that there may be repeated connection between the point data in the two groups of point cloud data, such as Figure 3As shown, the same point data h1 in the reference point cloud data Q2 is connected with multiple point data in the comparison point cloud data Q3, therefore, the point data with the minimum point distance between the first threshold value and the second threshold value is removed; this step is repeated until the point cloud data in all point cloud information sets are sequentially registered and combined, and all fixed point clouds and new point clouds are integrated to generate extended point cloud data.

[0052] The first threshold value and the second threshold value are respectively set based on the following steps:

[0053] The first threshold value E is calculated based on the first formula, and the first formula is: wherein t1 is the reference point cloud data, t2 is the comparison point cloud data, is the average plane precision of the reference point cloud data, is the average plane precision of the comparison point cloud data, is the plane alignment error between the reference point cloud data and the comparison point cloud data, is the weight coefficient, is the first deviation value of the reference point cloud data, is the second deviation value of the comparison point cloud data;

[0054] The subplane with the most point data in the reference point cloud data and the comparison point cloud data is set as the approximate plane, the point distance between the point data of the approximate plane in the reference point cloud data and the point data of the approximate plane in the comparison point cloud data is obtained, and the average value of the minimum point distance between the point data in the two sets of approximate planes is set as the second threshold value.

[0055] Specifically, in this embodiment, due to the time sequence change, each subplane in the reference point cloud data and the subplane of the comparison point cloud data have position changes, if only the fixed first threshold value and the second threshold value are set to judge the point distance between the point data, there will be a large error when integrating the point data in the reference point cloud data and the comparison point cloud data, which is not conducive to the subsequent classification of the point cloud data, therefore, in the first formula, the various values of the reference point cloud data and the comparison point cloud data are counted, the first threshold value is calculated through the distance between the subplanes and the average alignment error of the two sets of point cloud data, and the point data in the two sets of point cloud data is integrated, wherein the weight coefficient According to the measurement error parameters of the scanning instrument It is set that if the measurement error parameter is large, the weight coefficient is small, the average plane precision refers to the average precision of all subplanes in the point cloud data, the plane alignment error refers to the distance deviation between the point data in the overlapping area of the two sets of point cloud data, and the first deviation value and the second deviation data are the average standard deviation of the measurement error of each nearest subplane calculated after correcting the scanning instrument, which can describe the dispersion degree of the measurement error of each subplane.

[0056] The approximate plane refers to a subplane containing the most points in the point cloud data. By comparing the approximate planes in two sets of point cloud data, the displacement expansion range of the two sets of point cloud data can be compared. The displacement expansion range of the two sets of point cloud data can be obtained by calculating the average of the minimum distances of the point data in the two approximate planes. Therefore, the average of the minimum point-to-point distances between the point data in the two approximate planes is set as the second threshold value.

[0057] The plane precision average, the plane alignment error, the first deviation value and the second deviation value are obtained based on the following steps:

[0058] Based on the coordinate information, the curvature values of the point data are connected between the adjacent two point data in the point cloud data. The curvature threshold value is set. The point data with a curvature value less than or equal to the curvature threshold value are sequentially connected to generate a subplane. If the curvature value is greater than the curvature threshold value, the point data is taken as a starting point to be reconnected with other point data, so that the curvature values of the adjacent two point data are less than or equal to the curvature threshold value. This step is repeated to divide all the point data in the point cloud data into multiple subplanes.

[0059] The first distance of the point data to the subplane is obtained in the point cloud data. The subplane with the minimum first distance is set as the nearest subplane of the point data. The average of the first distances of all the point data in the same nearest subplane to the nearest subplane is obtained and set as the subplane error of the nearest subplane. The average of all the subplane errors is obtained based on the number of subplanes in the point cloud data, and set as the plane precision average of the point cloud data.

[0060] The distances between the subplanes in the reference point cloud data and each subplane in the comparison point cloud data are sequentially obtained and defined as the second distance. The minimum second distance is obtained based on the size of the second distance. The average of all the minimum second distances is calculated based on the number of subplanes in the reference point cloud data, and the average of the minimum second distance is set as the plane alignment error between the reference point cloud data and the comparison point cloud data.

[0061] The measurement error r of the nearest subplane in the point cloud data is calculated based on the second formula. The second formula is: wherein, is the measurement error parameter of the scanning instrument, N is the number of point data in the nearest subplane, is the distance from the i-th point data in the nearest subplane to the scanning instrument, D is the distance from the nearest subplane to the scanning instrument. The average standard deviation is obtained based on the number of nearest subplanes contained in the point cloud data and the measurement error. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

[0062] Specifically, in the embodiment, the point data in the point cloud data has a normal, the normal is a line segment bound with the point data, used to represent the surface direction between points, each point data in the point cloud data corresponds to a three-dimensional coordinate information and a normal, the distance between points and the angle between the normals of the two points are obtained by calculating, and then the final curvature value is obtained through mathematical transformation, all adjacent point data on the point cloud surface determines the shape of the local subplane, for example, assuming that the unit normal vector of point data h3 is H3, and point data h4 is the nearest neighbor point of point data h3, the normal vector of point data h4 is H4, the curvature value k3 of point data h3 is calculated through the osculating circle of point data h3, and the calculation formula is: The normal vector H4 is calculated by the line segment from the scanning origin along the point data, and the curvature value of all point data can be calculated by the same method; by setting the curvature threshold, a plurality of point data can be divided into the same plane, wherein the distance of each point data from the plane is within the set range, the plane is set as a subplane, if the curvature value is greater than the preset curvature threshold, the point data is reselected to connect the adjacent point data to calculate the corresponding curvature value, and the step is repeated to divide each point data in the point cloud data into a plurality of subplanes, so that each point data is bound.

[0063] The average error refers to a measurement index for quantifying the observation accuracy, and in the application, the average error can be used to measure the measurement accuracy of all subplanes generated in the point cloud data, the smaller the average error, the higher the measurement accuracy, the more the point data closest to the subplane, and the more suitable the subplane generated, therefore the average error is set as the plane accuracy mean, and the calculation formula of the average error P is: wherein, is the number of point data in the point cloud data, Figure 4 is the distance from the jth point data to the nearest subplane, and the plane accuracy mean of the reference point cloud data and the comparison point cloud data is calculated by the method, for example, as shown in the point cloud data Q5 includes two subplanes q51 and q52, wherein the nearest subplane of point data h is the subplane q52, and the dashed line distance

[0064] ​​​​​The average distance refers to the average distance between each subplane in the reference point cloud data and the nearest subplane in the comparison point cloud data. If two subplanes are not parallel, the distance is calculated by taking the cross product of their normal vectors and dividing by the magnitude of the normal vector. The calculation formula is: ,in For subplane normal vector and subplane normal vector cross product between Normal vector The model, Normal vector The average distance can be used to illustrate the alignment error between the corresponding sub-planes. Therefore, the average distance is set as the average alignment error between the reference point cloud data and the comparison point cloud data, for example, as... Figure 5 As shown, the nearest connection distances of each subplane in the reference point cloud data Q2 and the comparison point cloud data Q3 are L1, L2 and L3, respectively. The average alignment error between the reference point cloud data Q2 and the comparison point cloud data Q3 is the result of (L1+L2+L3) / 3.

[0065] Because scanning instruments have a factory-set error coefficient when generating point cloud data, in order to improve the accuracy of point cloud data, the measurement error of a sub-plane is calculated by summing the average measurement error of all point data in the sub-plane using a second formula. In the second formula, the measurement error parameter of the scanning instrument refers to the factory-set error coefficient of the device. By setting the scanning instrument as the shooting coordinate point, the distance D from the shooting coordinate point to the nearest sub-plane can be calculated. The measurement error can indicate the measurement accuracy of the sub-plane in the point cloud data. Therefore, the average standard deviation of the measurement error of all sub-planes in the point cloud data can indicate the degree of dispersion between measurement errors. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

[0066] Setting the synthetic feature vector includes the following steps:

[0067] The image information set contains multiple time-series image data. Any two adjacent image data sets have overlapping scan areas. A neural network model learns and trains on the image information set to extract feature quantities and corresponding feature contribution rates, and obtains class information from the image information set. The feature quantities are multiplied by the feature contribution rates to generate advanced feature quantities. Based on these advanced feature quantities, the neural network model extracts the feature map corresponding to the highest channel and generates a synthetic feature vector C based on the third formula: Where h is the height of the feature map and w is the width of the feature map. is an eigenvector of the advanced feature quantity at position j, is a parameter exponential function, specifically a function , is a feature map at position q.

[0068] Specifically, in this embodiment, the feature quantity in the neural network model needs to be set to improve the accuracy of semantic segmentation in the image recognition process, which can further improve the accuracy of point cloud data classification. In the image information set, any two adjacent image data have overlapping regions, which is to connect the corresponding point cloud data and effectively divide and integrate the corresponding point cloud data. The neural network model trains the image information set through convolution, and obtains the class information of the image information set in the image recognition process. The class information refers to the category label contained in the image information set. In semantic segmentation, the feature contribution rate refers to the contribution degree of each eigenvector to the segmentation result. The feature contribution rate is output by the neural network model. For each eigenvector, the model calculates its score in the image segmentation result, and then divides the score by the probability of the occurrence of the eigenvector in the image data to calculate the contribution rate of each feature quantity to the segmentation result. By calculating the contribution rate of the feature quantity, the contribution degree of each eigenvector to the image segmentation result can be more accurately evaluated. The feature quantity is multiplied by the corresponding feature contribution rate to generate an advanced feature quantity, and the weight of each feature quantity in the semantic segmentation process can be calculated. The feature map refers to the image generated by each advanced feature quantity on the image data. Through the third formula, all feature maps can be aggregated to generate an aggregated map, which represents the relationship between different pixels in the input image data. The composite eigenvector is calculated by the advanced feature quantity and the aggregated map.

[0069] The neural network model performs semantic segmentation on the image information set based on the set composite eigenvector to generate a plurality of sub-regions. The number of sub-regions corresponds to the number of class information, so the class information is set as the label information of the sub-region.

[0070] The generation of the point cloud category includes the following steps:

[0071] The extended point cloud data is sequentially superimposed into the two-dimensional classification region of the scanning region based on the same scanning angle, and the label information of the two-dimensional classification region is labeled in the point data contained in the extended point cloud data. The extended point cloud data is classified based on the same label information to generate a point cloud category.

[0072] Specifically, in this embodiment, the extended point cloud data has the same scanning angle as each sub-region, so each point data in the extended point cloud data is superimposed into the sub-region based on the scanning angle, so that the point data of the same sub-region is labeled as the corresponding label information, and the extended point cloud data is further classified into a point cloud category based on the same label information.

[0073] The present application is first based on a preset time interval to simultaneously obtain an image information set and a corresponding point cloud information set of a scanning region, so that it contains an overlapping region, which can effectively improve the usability of point cloud data, facilitate registration and integration of point cloud data, and then by sequentially comparing two groups of point cloud data adjacent in time sequence, setting different values of a first threshold and a second threshold, and dividing the point data in the two groups of point cloud data into fixed point clouds and new point clouds according to the size of the minimum point distance, further integrating all point cloud information sets to generate extended point cloud data, which can eliminate point cloud data generated by moving objects, finally, through a neural network model, the image information set is subjected to image recognition and semantic segmentation to generate a plurality of sub-regions and corresponding label information, and based on the scanning angle, each point data in the extended point cloud data is superimposed into a sub-region, the point data with the same label information is set as a point cloud category, and the extended point cloud data is further classified.

[0074] The present application further sets a synthetic feature vector to improve the accuracy of semantic segmentation by the neural network model, which can effectively improve the classification accuracy of point cloud data located at the edge of an object, and the present application can automatically classify point cloud data scanned multiple times at different times in an unrestricted spatial range with high accuracy.

[0075] It is particularly noted that through the present application, the point cloud data of an extended scanning region can be integrated and then automatically classified, thereby avoiding the occurrence of unstable point cloud data with a high error rate of classification results.

[0076] As shown in Figure 6 The present application further provides an automatic classification system based on unmanned aerial vehicle point cloud data, which is used to implement the above-mentioned unmanned aerial vehicle automatic classification method based on a power transmission line, and mainly comprises:

[0077] A data collection module obtains an image information set and a corresponding point cloud information set of a scanning region based on a preset time interval, and the image information set and the point cloud information set respectively contain a plurality of time sequence image data and point cloud data;

[0078] An image processing module establishes a neural network model, inputs the image information set into the neural network model, sets a synthetic feature vector, and the neural network model outputs a semantic segmentation result of the image data and corresponding label information based on the synthetic feature vector, divides the image data into a plurality of sub-regions based on the semantic segmentation result, and labels the label information of the sub-regions;

[0079] A point cloud integration module sequentially integrates the point cloud data in the point cloud information set based on the time sequence to generate extended point cloud data of the scanning region;

[0080] The point cloud classification module aggregates all sub-regions of the image data based on the label information to generate a two-dimensional classification region of the scanning region, classifies the point data in the extended point cloud data located in the sub-regions into a category based on the scanning angle of the scanning region, and divides the extended point cloud data into multiple point cloud categories based on the number of the sub-regions in the two-dimensional classification region.

[0081] The application further provides a computer storage medium, which stores program instructions, and when the program instructions are executed, controls a device where the computer storage medium is located to perform the unmanned aerial vehicle automatic classification method based on a power transmission line.

[0082] It should be understood that, although each step in the flowchart of each embodiment of the application is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in each embodiment can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0083] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The above-mentioned program can be stored in a non-volatile computer readable storage medium, and when the program is executed, it can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered within the scope of the present disclosure.

[0085] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as limiting the patent scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

[0086] The above is only a preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An automatic classification method for unmanned aerial vehicles (UAVs) based on power transmission lines, characterized in that, The method includes the following steps: The image information set and the corresponding point cloud information set of the scanned area are acquired based on a preset time interval. The image information set and the point cloud information set respectively contain multiple time series image data and point cloud data. A neural network model is established, the image information set is input into the neural network model, a synthetic feature vector is set, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled. The point cloud data in the point cloud information set are sequentially integrated based on the time series to generate extended point cloud data of the scanned area; Based on the label information, the sub-regions of all the image data are summarized to generate a two-dimensional classification region of the scanning region. Based on the scanning angle of the scanning region, the point data located in the sub-regions of the extended point cloud data are classified into one category. Based on the number of the sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories. Generating the extended point cloud data includes the following steps: The point cloud data includes coordinate information of point data. The point cloud data is sorted sequentially based on the time series. The point cloud data of the first time series is set as the baseline point cloud data, and the point cloud data of the next time series is set as the comparison point cloud data. Based on the coordinate information, the distance between the point data in the baseline point cloud data and the point data in the comparison point cloud data are obtained respectively. The magnitude of the distance between the point data is compared, and a first threshold and a second threshold are set respectively. If the minimum distance between the point data is greater than or equal to zero and less than the first threshold, then the point data in the baseline point cloud data is... The data is set as a fixed point cloud. If the minimum distance between the point data is greater than or equal to the second threshold, the point data of the reference point cloud data and the comparison point cloud data are set as new point clouds respectively. The fixed point cloud and the new point cloud are combined into new reference point cloud data. The new reference point cloud data is set as the reference point cloud data and integrated with the point data of the next comparison point cloud data. This step is repeated based on the number of time series to integrate the point data of the point cloud data in the point cloud information set. The generated new reference point cloud data is set as the extended point cloud data.

2. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 1, characterized in that, The first threshold and the second threshold are set based on the following steps: The first threshold E is calculated based on the first formula, which is: Where t1 is the baseline point cloud data and t2 is the comparison point cloud data. The mean planar accuracy of the reference point cloud data. The mean planar accuracy of the compared point cloud data. The planar alignment error between the reference point cloud data and the comparison point cloud data. These are the weighting coefficients. The first deviation value of the reference point cloud data. This is the second deviation value of the compared point cloud data; The subplane containing the most point data in the reference point cloud data and the comparison point cloud data is set as an approximate plane. The point distance between the point data of the approximate plane in the reference point cloud data and the point data of the approximate plane in the comparison point cloud data is obtained. The average value of the minimum point distance between the point data in the two sets of approximate planes is set as the second threshold.

3. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 2, characterized in that, The mean plane accuracy, the plane alignment error, the first deviation value, and the second deviation value are obtained based on the following steps: Based on the coordinate information, two adjacent points in the point cloud data are connected to generate the curvature value of the point data. A curvature threshold is set, and the points with curvature values ​​less than or equal to the curvature threshold are connected sequentially to generate the subplane. If the curvature value is greater than the curvature threshold, the point data is used as the starting point to reconnect with other points so that the curvature values ​​of two adjacent points are less than or equal to the curvature threshold. This step is repeated to divide all the points in the point cloud data into multiple subplanes. In the point cloud data, the first distance between the point data and the subplane is obtained, and the subplane with the smallest first distance is set as the nearest subplane of the point data. The average of the first distances from all the point data to the nearest subplane in the same nearest subplane is obtained and set as the subplane error of the nearest subplane. Based on the number of subplanes in the point cloud data, the average of all the subplane errors is obtained and set as the average plane accuracy of the point cloud data. The distance between each subplane in the reference point cloud data and each subplane in the comparison point cloud data is sequentially obtained and defined as the second distance. The minimum second distance is obtained based on the magnitude of the second distance. The mean of all the minimum second distances is calculated based on the number of subplanes in the reference point cloud data. The mean of the minimum second distance is set as the plane alignment error between the reference point cloud data and the comparison point cloud data. The measurement error r of the nearest sub-plane in the point cloud data is calculated based on the second formula, which is: ,in, Here, N is the measurement error parameter of the scanning instrument, and N is the number of point data points in the nearest subplane. Let D be the distance from the i-th point data in the nearest subplane to the scanning device, and let D be the distance from the nearest subplane to the scanning device. Based on the number of nearest subplanes contained in the point cloud data and the measurement error, the average standard deviation is obtained. The average standard deviation of the reference point cloud data is set as the first deviation value, and the average standard deviation of the comparison point cloud data is set as the second deviation value.

4. The automatic classification method for unmanned aerial vehicles based on power transmission lines according to claim 1, characterized in that, Generating the point cloud category includes the following steps: Based on the same scanning angle, the extended point cloud data is sequentially superimposed onto the two-dimensional classification region of the scanning area, and the label information of the two-dimensional classification region is marked on the point data contained in the extended point cloud data. Based on the same label information, the extended point cloud data is classified to generate the point cloud category.

5. An automatic classification system based on UAV point cloud data, used to implement the automatic UAV classification method based on power transmission lines as described in any one of claims 1-4, characterized in that, The system includes the following modules: The data collection module acquires image information sets and corresponding point cloud information sets of the scanned area based on a preset time interval. The image information sets and the point cloud information sets respectively contain multiple time-series image data and point cloud data. The image processing module establishes a neural network model, inputs the image information set into the neural network model, sets a synthetic feature vector, and the neural network model outputs the semantic segmentation result and corresponding label information of the image data based on the synthetic feature vector. Based on the semantic segmentation result, the image data is divided into multiple sub-regions, and the label information of the sub-regions is labeled. The point cloud integration module integrates the point cloud data in the point cloud information set sequentially based on the time series to generate extended point cloud data of the scanned area. The point cloud classification module summarizes the sub-regions of all the image data based on the label information to generate a two-dimensional classification region of the scanned area. Based on the scanning angle of the scanned area, the point data located in the sub-regions of the extended point cloud data are classified into one category. Based on the number of the sub-regions in the two-dimensional classification region, the extended point cloud data is divided into multiple point cloud categories.

6. A computer storage medium, characterized in that, The computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the automatic classification method for unmanned aerial vehicles based on power transmission lines as described in any one of claims 1-4.

Images