Automatic labeling system and method for UAV aerial survey data based on artificial intelligence

Abstract

The present invention relates to the field of aerial survey annotation technology, and in particular to an automatic annotation system and method for drone aerial survey data based on artificial intelligence, comprising: recording shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey; extracting i areas in the aerial survey image that cannot be automatically annotated and marking them as unrecognized areas; inputting the shooting environment data and shooting posture data corresponding to the areas that cannot be automatically annotated into a trained first machine learning model, and outputting the corresponding number of standard pixel blocks; dividing the i unrecognized areas in turn according to the number of standard pixel blocks to obtain sub-areas corresponding to each unrecognized area; inputting the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image into a trained second machine learning model in turn, and obtaining the annotation result corresponding to each sub-area; and effectively reducing the manual annotation intensity of the unrecognized areas.

Description

Technical Field

[0001] The present invention relates to the field of aerial survey annotation technology, and in particular to an automatic annotation system and method for unmanned aerial survey data based on artificial intelligence. Background Art

[0002] With the rapid development of drone technology, drone aerial surveys have been widely used in the fields of geographic information, urban planning, agricultural monitoring, etc. However, a large amount of aerial survey data needs to be accurately and efficiently labeled to extract valuable information. In the existing technology, although artificial intelligence is used to automatically label drone aerial survey data, there are still unidentified areas. These unidentified areas usually need to be reported directly and manually identified and labeled by staff, resulting in the workload and intensity of manual labeling being still high and not effectively reduced. Summary of the invention

[0003] In order to solve the above problems, the present invention provides an automatic annotation system and method for UAV aerial survey data based on artificial intelligence.

[0004] The present invention adopts the following technical scheme, a method for automatic labeling of drone aerial survey data based on artificial intelligence, including:

[0005] Record the shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey;

[0006] Extract i areas in the aerial survey image that have not been automatically annotated and mark them as unrecognized areas;

[0007] Input the shooting environment data and shooting posture data corresponding to the area that cannot be automatically labeled into the first machine learning model that has been trained, and output the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence;

[0008] Divide the i unrecognized regions in turn by the number of standard pixel blocks to obtain sub-regions corresponding to each unrecognized region;

[0009] The shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are input into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include the labeling name and labeling failure.

[0010] As a further description of the above technical solution: the method of dividing the sub-regions is: a square region is obtained with the number of standard pixel blocks as the length, and the unidentified region is divided into n sub-regions by the square region, where n is an integer greater than or equal to 1; a region that is not large enough to form a square region is also regarded as a sub-region.

[0011] As a further description of the above technical solution: the training method of the first machine learning model includes:

[0012] Pre-collect k groups of recognition unit training data, the recognition unit training data including recognition unit feature data and the number of standard pixel blocks corresponding to the recognition unit feature data, the recognition unit feature data including shooting environment data and shooting posture data, and convert the recognition unit training data into a first feature vector;

[0013] Each group of first feature vectors is used as the input of the first machine learning model. The first machine learning model uses the predicted standard pixel block number vector corresponding to each group of identification unit feature data vectors as the output, and the actual standard pixel block number vector corresponding to each group of identification unit feature data vectors as the prediction target; minimizing the sum of the first prediction errors of all identification unit feature data vectors is used as the training target; the first machine learning model is trained until the sum of the first prediction errors reaches convergence, and the training is stopped; the first machine learning model is a random forest model, a gradient boosting tree model or a recurrent neural network model.

[0014] As a further description of the above technical solution: the calculation formula of the first prediction error is Z K =(α K -μ K ) 2 , where Z K is the first prediction error, K is the group number of the identification unit feature data vector corresponding to the first feature vector, α K is the prediction standard pixel block quantity vector corresponding to the Kth group recognition unit feature data vector, μ K The actual standard pixel block quantity vector corresponding to the K-th group of identification unit feature data vectors.

[0015] As a further description of the above technical solution: the shooting environment data includes season, ambient brightness and weather, and the shooting attitude data includes the height above the ground, tilt angle and speed of the drone: the tilt angle includes pitch, yaw and roll.

[0016] As a further description of the above technical solution: the labeled names are highways, rivers, buildings, lakes, farmlands, airports, industrial areas and parks.

[0017] As a further description of the above technical solution: it also includes:

[0018] Add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image;

[0019] Input the secondary annotated images into the trained rationality recognition model and output the unreasonable annotation names.

[0020] As a further description of the above technical solution: the secondary labeled image includes a labeled name and an area corresponding to each labeled name.

[0021] As a further description of the above technical solution: the training method of the rationality recognition model includes:

[0022] Collect recognition data in advance, the recognition data includes secondary annotated pictures and unreasonable annotated names, convert the recognition data into a second feature vector, divide the second feature vector into a training set and a test set, use the secondary annotated picture vector in the second feature data as the input of the rationality recognition model, use the unreasonable annotated name vector in the training set as the output of the rationality recognition model, train the rationality recognition model to obtain an initial rationality recognition model, test the initial rationality recognition model with the test set, and output a rationality recognition model that meets the preset accuracy. The rationality recognition model is a convolutional neural network model.

[0023] The automatic labeling system for drone aerial survey data based on artificial intelligence is used to implement the automatic labeling method for drone aerial survey data based on artificial intelligence. The system includes:

[0024] The recording module is used to record the shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey;

[0025] The first processing module extracts i areas in the aerial survey image that have not been automatically marked and marks them as unrecognized areas;

[0026] The second processing module inputs the shooting environment data and shooting posture data corresponding to the area that cannot be automatically marked into the trained first machine learning model, and outputs the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence;

[0027] A third processing module is used to divide the i unrecognized areas in sequence according to the number of standard pixel blocks to obtain sub-areas corresponding to each unrecognized area;

[0028] A fourth processing module is used to input the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image into the trained second machine learning model in sequence, and obtain a labeling result corresponding to each sub-area, where the labeling result includes a labeling name and a labeling failure;

[0029] A fifth processing module is used to add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image;

[0030] The rationality processing module is used to input the secondary labeled images into the trained rationality recognition model and output the unreasonable labeling names.

[0031] Beneficial effects:

[0032] For areas that cannot be automatically identified, the shooting environment data and shooting posture data of the aerial survey image are obtained, and the appropriate number of standard pixel blocks is selected to improve the accuracy of subsequent annotation results; selecting the appropriate number of standard pixel blocks can effectively improve the recognition accuracy.

[0033] First, the unrecognized area is divided into square sub-regions according to the number of standard pixel blocks to obtain several sub-regions. This detailed division can pay more attention to details and improve the recognition ability of tiny objects and complex scenes. Each sub-region is processed using a standard recognition model. Since the sub-region is small, the model can focus more on the features within the region, reduce interference factors, and improve recognition accuracy.

[0034] Taking the shooting environment data, shooting posture data and several sub-areas corresponding to the aerial survey image as input data, the model can analyze the unidentified areas more comprehensively and accurately label the names. Through automated processing, it reduces the intensity of manual labeling of unidentified areas, and also improves the completeness and accuracy of the overall labeling of the aerial survey image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a module connection diagram of the automatic annotation system for drone aerial survey data based on artificial intelligence in Example 1 of the present invention;

[0036] Figure 2 This is a module connection diagram of the automatic annotation system for drone aerial survey data based on artificial intelligence in Example 2 of the present invention;

[0037] Figure 3 Schematic diagram of the process of the automatic labeling method of UAV aerial survey data based on artificial intelligence in Example 3 of the present invention. DETAILED DESCRIPTION

[0038] In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the present invention is further described below with reference to specific diagrams. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0039] See also Figure 1 An embodiment of the present invention provides an automatic annotation system for drone aerial survey data based on artificial intelligence, including: a recording module, a first processing module, a second processing module, a third processing module and a fourth processing module, and each module is connected by wire and / or wirelessly.

[0040] The recording module is used to record the shooting environment data and shooting attitude data corresponding to each aerial survey image during drone aerial survey; the shooting environment data includes season, ambient brightness and weather, and the shooting attitude data includes the drone's height from the ground, tilt angle and speed: the tilt angle includes pitch, yaw and roll.

[0041] The first processing module extracts i areas in the aerial survey image that cannot be automatically marked and marks them as unrecognized areas.

[0042] The second processing module inputs the shooting environment data and shooting posture data corresponding to the area that cannot be automatically marked into the trained first machine learning model, and outputs the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks adjacent to each other in sequence.

[0043] For areas that have not been automatically labeled, how to divide them into small areas for re-identification, that is, sub-areas below. The size of the sub-areas will directly affect the accuracy of the subsequent re-labeling results. The inventor uses season, ambient brightness, weather, altitude, tilt angle and speed of the drone as inputs to the recognition unit model for the following reasons:

[0044] Weather conditions such as sunny, cloudy, and haze will affect the clarity and contrast of the image. For example, a cloudy day will cause the image to be dark overall, while haze will cause the image to be blurred. When the image quality is low, it is divided into smaller areas to extract more details and reduce the negative impact of weather on the recognition results. The ambient brightness affects the shadows and brightness of the image. For example, strong sunlight will cast obvious shadows on the objects, while low light may make some objects difficult to identify. In poor lighting conditions, dividing into smaller areas can reduce the impact of shadows and lighting changes on recognition and improve recognition accuracy. Seasonal changes can cause changes in the appearance of objects, such as plant growth and withering, snow coverage, etc. For example, there is a big visual difference between farmland in summer and farmland covered with snow in winter. In the case of obvious seasonal changes, dividing into smaller areas can capture more details and help distinguish the characteristics of objects in different seasons.

[0045] The tilt angle of the drone will cause geometric distortion of the image, such as stretching, compression, rotation, etc. A larger tilt angle needs to be divided into smaller areas to reduce the impact of distortion on recognition. Using smaller areas can help better correct the image and maintain recognition accuracy. The speed of the drone will affect the clarity and overlap of the image. A faster speed will cause the image to be blurred, while a slower speed can obtain a clearer image. At a faster speed, dividing into smaller areas can reduce the impact of blur on recognition and ensure that the model can capture more details, and vice versa. The number of standard pixel blocks corresponding to dividing into smaller areas is smaller, and vice versa.

[0046] In summary, the shooting environment data and the shooting posture data are closely related to the number of selected unit pixel blocks.

[0047] The training method of the first machine learning model includes:

[0048] Collect k groups of recognition unit training data in advance, the recognition unit training data includes recognition unit feature data and the number of standard pixel blocks corresponding to the recognition unit feature data, the recognition unit feature data includes shooting environment data and shooting posture data, and convert the recognition unit training data into a first feature vector.

[0049] Each group of first feature vectors is used as the input of the first machine learning model. The first machine learning model uses the predicted standard pixel block quantity vector corresponding to each group of identification unit feature data vectors as the output, and uses the actual standard pixel block quantity vector corresponding to each group of identification unit feature data vectors as the prediction target; minimizing the sum of the first prediction errors of all identification unit feature data vectors is used as the training target; wherein the calculation formula of the first prediction error is Z K =(α K -μ K ) 2 , where Z K is the first prediction error, K is the group number of the identification unit feature data vector corresponding to the first feature vector, α K is the prediction standard pixel block quantity vector corresponding to the Kth group recognition unit feature data vector, μ K The actual standard pixel block quantity vector corresponding to the Kth group of identification unit feature data vectors; the first machine learning model is trained until the sum of the first prediction errors reaches convergence and the training is stopped; the first machine learning model is a random forest model, a gradient boosting tree model (such as XGBoost, LightGBM) or a recurrent neural network model.

[0050] The third processing module is used to divide the i unidentified areas in sequence according to the number of standard pixel blocks to obtain sub-areas corresponding to each unidentified area.

[0051] The method of dividing the sub-regions is: obtain a square region with the number of standard pixel blocks as the length, divide the unrecognized region with the square region to obtain n sub-regions, where n is an integer greater than or equal to 1; a region that is not enough to form a square region is also regarded as a sub-region.

[0052] The fourth processing module is used to input the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include labeling names and labeling failures; the sub-areas corresponding to the labeling failures are reported and manually reviewed and labeled; the labeling names include highways, rivers, buildings, lakes, farmlands, airports, industrial areas, parks, etc.

[0053] In this embodiment, for areas that cannot be automatically identified, the shooting environment data and shooting posture data corresponding to the aerial survey image are obtained to select an appropriate number of standard pixel blocks. The appropriate number of standard pixel blocks effectively improves the accuracy of subsequent re-annotation results; a square area is then obtained with the number of standard pixel blocks as the length, and the unidentified area is divided into n sub-areas by the square area. The identification of sub-areas in a small range can pay more attention to details and improve the recognition ability of tiny objects and complex scenes. During processing, the first machine learning model can focus more on the features in the area, reduce interference factors, and improve recognition accuracy; the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are then used as input data to obtain the labeling name of the unidentified area, thereby reducing the intensity of manual labeling of the unidentified area, and at the same time effectively improving the integrity and accuracy of the overall labeling of the aerial survey image.

[0054] Example 2

[0055] See also Figure 2 As shown, the automatic annotation system for drone aerial survey data based on artificial intelligence provided in this embodiment also includes:

[0056] The fifth processing module is used to add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image, which includes the annotation name and the region corresponding to each annotation name.

[0057] The rationality processing module is used to input the secondary labeled images into the trained rationality recognition model, output the unreasonable labeling names, report the unreasonable labeling names manually, check and re-label them, effectively improving the accuracy of automatic labeling and the efficiency of rationality screening.

[0058] For example, in a large farmland marking area, a high-rise building marking appears. There are obvious geographical and functional differences between farmland and high-rise buildings. It is impossible for a high-rise building to appear in the middle of farmland. The reason for the error may be that some structures in the farmland (such as irrigation facilities) are mistaken for buildings; in a highway marking area, an airport runway marking appears. The construction and use of highways and airport runways are completely different. It is impossible for an airport runway to be adjacent to a highway. The reason for the error is that the model confuses the characteristics of linear features, such as similar lengths, widths, and directions; in a park green space marking area, an industrial area marking appears. The interior of a park is usually green space, trails, and entertainment facilities. It is impossible for there to be large-scale industrial facilities. The reason for the error is that the model confuses some green areas of the park and the industrial area.

[0059] The training methods of the rationality recognition model include:

[0060] Collect recognition data in advance, the recognition data includes secondary annotated pictures and unreasonable annotated names, convert the recognition data into a second feature vector, divide the second feature vector into a training set and a test set, use the secondary annotated picture vector in the second feature data as the input of the rationality recognition model, use the unreasonable annotated name vector in the training set as the output of the rationality recognition model, train the rationality recognition model to obtain an initial rationality recognition model, test the initial rationality recognition model with the test set, and output a rationality recognition model that meets the preset accuracy. The rationality recognition model is a convolutional neural network model, such as VGG16, ResNet, etc.

[0061] Example 3

[0062] See also Figure 3 As shown, this embodiment provides an automatic labeling method for drone aerial survey data based on artificial intelligence, including:

[0063] Record the shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey;

[0064] Extract i areas in the aerial survey image that have not been automatically annotated and mark them as unrecognized areas;

[0065] Input the shooting environment data and shooting posture data corresponding to the area that cannot be automatically labeled into the first machine learning model that has been trained, and output the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence;

[0066] Divide the i unrecognized regions in turn by the number of standard pixel blocks to obtain sub-regions corresponding to each unrecognized region;

[0067] The shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are input into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include the labeling name and labeling failure.

[0068] The basic principles, main features and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and the description in the specification are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An automatic labeling method for drone aerial survey data based on artificial intelligence, characterized in that: include: Record the shooting environment data and shooting posture data corresponding to each aerial survey image during drone aerial survey; Extract i areas in the aerial survey image that have not been automatically annotated and mark them as unrecognized areas; Input the shooting environment data and shooting posture data corresponding to the area that cannot be automatically labeled into the first machine learning model that has been trained, and output the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence; Divide the i unrecognized regions in turn by the number of standard pixel blocks to obtain sub-regions corresponding to each unrecognized region; The method of dividing the sub-regions is as follows: a square region is obtained with the number of standard pixel blocks as the length, and the unrecognized region is divided into n sub-regions by the square region, where n is an integer greater than or equal to 1; a region that is not large enough to form a square region is also regarded as a sub-region; The shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image are input into the trained second machine learning model in sequence to obtain the labeling results corresponding to each sub-area, and the labeling results include the labeling name and labeling failure.

2. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 1 is characterized in that: The training method of the first machine learning model includes: Pre-collect k groups of recognition unit training data, the recognition unit training data including recognition unit feature data and the number of standard pixel blocks corresponding to the recognition unit feature data, the recognition unit feature data including shooting environment data and shooting posture data, and convert the recognition unit training data into a first feature vector; Each group of first feature vectors is used as the input of the first machine learning model. The first machine learning model uses the predicted standard pixel block number vector corresponding to each group of identification unit feature data vectors as the output, and the actual standard pixel block number vector corresponding to each group of identification unit feature data vectors as the prediction target; minimizing the sum of the first prediction errors of all identification unit feature data vectors is used as the training target; the first machine learning model is trained until the sum of the first prediction errors reaches convergence, and the training is stopped; the first machine learning model is a random forest model, a gradient boosting tree model or a recurrent neural network model.

3. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 2 is characterized in that: The calculation formula of the first prediction error is: ,in is the first prediction error, To identify the group number of the first eigenvector corresponding to the unit eigendata vector, For the The predicted standard pixel block quantity vector corresponding to the group identification unit feature data vector, For the The actual standard pixel block quantity vector corresponding to the group identification unit feature data vector.

4. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 1 is characterized in that: The shooting environment data includes season, environment brightness and weather, and the shooting attitude data includes the height from the ground, tilt angle and speed of the drone: the tilt angle includes pitch, yaw and roll.

5. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 1 is characterized in that: The labeled names include highways, rivers, buildings, lakes, farmlands, airports, industrial areas and parks.

6. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 1 is characterized in that: Also includes: Add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image; Input the secondary annotated images into the trained rationality recognition model and output the unreasonable annotation names.

7. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 6 is characterized in that: The secondary labeled image includes a labeled name and an area corresponding to each labeled name.

8. The method for automatic labeling of drone aerial survey data based on artificial intelligence according to claim 6 is characterized in that: The training method of the rationality recognition model includes: Collect recognition data in advance, the recognition data includes secondary annotated pictures and unreasonable annotated names, convert the recognition data into a second feature vector, divide the second feature vector into a training set and a test set, use the secondary annotated picture vector in the second feature data as the input of the rationality recognition model, use the unreasonable annotated name vector in the training set as the output of the rationality recognition model, train the rationality recognition model to obtain an initial rationality recognition model, test the initial rationality recognition model with the test set, and output a rationality recognition model that meets the preset accuracy. The rationality recognition model is a convolutional neural network model.

9. The automatic annotation system of UAV aerial survey data based on artificial intelligence is characterized by: The system for implementing the method for automatic labeling of drone aerial survey data based on artificial intelligence described in any one of items 1 to 8 comprises: The recording module is used to record the shooting environment data and shooting posture data corresponding to each aerial survey image during the UAV aerial survey; The first processing module extracts i areas in the aerial survey image that have not been automatically marked and marks them as unrecognized areas; The second processing module inputs the shooting environment data and shooting posture data corresponding to the area that cannot be automatically marked into the trained first machine learning model, and outputs the corresponding number of standard pixel blocks, where the pixel blocks are pixel blocks that are adjacent in sequence; The third processing module is used to divide the i unrecognized areas in sequence according to the number of standard pixel blocks to obtain sub-areas corresponding to each unrecognized area; The method of dividing the sub-regions is as follows: a square region is obtained with the number of standard pixel blocks as the length, and the unrecognized region is divided into n sub-regions by the square region, where n is an integer greater than or equal to 1; a region that is not large enough to form a square region is also regarded as a sub-region; A fourth processing module is used to input the shooting environment data, shooting posture data and n sub-areas corresponding to the aerial survey image into the trained second machine learning model in sequence, and obtain a labeling result corresponding to each sub-area, where the labeling result includes a labeling name and a labeling failure; A fifth processing module is used to add the annotation name to the corresponding sub-region position in the aerial survey image to form a secondary annotation image; The rationality processing module is used to input the secondary labeled images into the trained rationality recognition model and output the unreasonable labeling names.

Images