An image feature identification method using a UNET model

Abstract

The application provides an image ground object recognition method using a UNET model, a deep learning method for ground object recognition in a UNET model by taking multi-channel images containing elevation information and ground object labeled images as basic images, and the deep learning model is verified and calibrated by randomly dividing samples corresponding to the basic images, so that a more accurate multi-channel and ground object recognition model is obtained, the to-be-tested image is further input into the model, the prediction result is obtained, and the prediction result is further inlaid and corrected to obtain a result image containing multi-channel information and ground object recognition. Through the method, the automation degree of image processing is improved, the ground object in the image can be recognized by multiple channels, and the ground object is conveniently and better extracted and the holes after extraction are conveniently filled.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image feature recognition method, and particularly relates to an image feature recognition method using a UNET model. BACKGROUND

[0002] At present, the production of digital surface model (DSM) of visible light image by using oblique photogrammetry technology is relatively mature, but the oblique photogrammetry technology obtains a surface elevation containing feature height. In large-scale engineering design of square kilometers and above, the terrain contains many coverings such as vegetation, and if the contour line is directly generated by using the digital surface model (DSM), the abnormal elevation area will appear, which cannot reflect the real terrain condition. Therefore, in order to restore the ground elevation, it is usually necessary to filter out the feature height such as vegetation. In the vegetation filtering, the method of identifying first and then removing is usually used, and the main means at present is to remove by using artificial high points or directly single channel identification, which has low efficiency, poor identification effect, high misidentification probability and low automation degree, thereby affecting normal operation. SUMMARY

[0003] The present application provides an image feature recognition method using a UNET model, which uses multi-channel simultaneous recognition in the field of feature recognition extraction for the first time, and eliminates the misidentification of multi-channel recognition.

[0004] The technical means adopted by the present application is as follows:

[0005] An image feature recognition method using a UNET model, which obtains the recognition image of feature type and elevation of the image to be measured by the following steps:

[0006] Step 1: selecting an area, performing data processing on the photographed photos of the area to obtain a multi-channel image corresponding to the area, and performing feature labeling on the basis of the multi-channel image to obtain a classification label image corresponding to the area;

[0007] Step 2: performing cutting processing on the classification label image and the multi-channel image obtained in step 1 to obtain a sample data set, and randomly dividing the sample data set into a training data set for model training and a verification data set for model verification;

[0008] Step 3: inputting the training data set obtained in step 2 into the UNET model for multi-channel recognition training and feature classification recognition training to obtain an initial model;

[0009] Step 4: inputting the verification data set obtained in step 2 into the initial model generated in step 3 for verification training, and calculating the mean intersection over union (MIOU) of the predicted value and the true value of each channel and each feature class recognition, modifying the parameters of the initial model through the mean intersection over union (MIOU), and obtaining a final prediction model.

[0010] Step five: the to-be-tested image is subjected to the data processing of step one to obtain a corresponding to-be-tested multi-channel image; and the to-be-tested multi-channel image is subjected to the cutting processing of step two to obtain a to-be-tested data set;

[0011] Step six: the to-be-tested data set obtained in step five is input into the prediction model obtained in step four for prediction to obtain a prediction result data set containing feature classification and channel identification;

[0012] Step seven: the prediction result data set obtained in step six is subjected to image mosaic processing to obtain a result map containing multi-channel information and feature identification.

[0013] As a preferred, further comprising step eight: the result map obtained by the mosaic in step seven is subjected to interactive editing with the to-be-tested multi-channel image obtained in step five to obtain a corrected result map containing multi-channel information and feature identification.

[0014] As a preferred, the multi-channel image obtained in step one comprises at least RGB three channels and a slope channel.

[0015] As a preferred, the labeling processing in step one comprises adding a class name and a class label value field to each feature to be labeled after vectorization processing of the multi-channel image; and further rasterization processing is performed on the vectorized multi-channel image after the labeling processing to obtain a class label map corresponding to the multi-channel image.

[0016] As a preferred, the class name and the class label value field added when labeling the vectorized multi-channel image comprise: background 0, vegetation 1.

[0017] As a preferred, the cutting processing in step two comprises cutting the classification label map and the multi-channel image obtained in step one into sample maps of the same size, and further enhancing the sample maps obtained by cutting to obtain a sample data set.

[0018] As a preferred, in the identification training of step three using the UNET model, the backbone network structure selects a VGG16 model, and a Relu function is selected as an activation function, and the loss in the training is calculated by a soft function and cross entropy.

[0019] As a preferred, the mean intersection over union (MIOU) in step four is calculated by the following steps:

[0020] (1) calculate the confusion matrix;

[0021] (2) calculate the intersection over union (IOU_i) of each channel and feature class, and for each IOU_i, the formula is:

[0022] IOU_i = TP_i / (TP_i + FN_i + FP_i)

[0023] Wherein, TP_i is true value, FN_i is false negative value, FP_i is false positive value.

[0024] (3) Average IOU_i of each class to get mean intersection over union MIOU.

[0025] As preferred, in step seven, the mosaicked result map containing multi-channel information and ground object recognition is further raster-vector processed to convert into a vector file, and the vectorized result map is interactively edited and repaired.

[0026] As preferred, the data processing in step one is to use unmanned aerial vehicle data processing software to process the photos of the region into an orthophoto and a digital surface model corresponding to the region, and further perform slope calculation on the digital surface model to obtain a corresponding slope map, and integrate the orthophoto and the slope map into a multi-channel image including at least RGB channel and slope channel.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] The present method first introduces the UNET model in large-scale topographic map (1:500-1:2000) production, and improves it to construct a vegetation and other ground object fine classification algorithm suitable for unmanned aerial vehicle visible light image, which has more extension ability of ground object recognition; and the present method first introduces a correction method of multi-channel recognition in the field of ground object recognition, which can recognize multiple channels such as slope and classification simultaneously, and also reduces the misrecognition rate of multi-channel recognition, facilitates further editing processing after recognition, and the present method can be automatically completed through program, with less human intervention, high automation degree, and can greatly improve the work efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 It is a flowchart of ground elevation extraction method based on visible light image and deep learning algorithm.

[0030] Figure 2 It is a top view of a vegetation area in a certain survey area in an embodiment.

[0031] Figure 3 It is Figure 2 The corresponding contour map.

[0032] Figure 4 It is a corresponding survey area range and phase control point distribution map obtained by taking photos in an embodiment.

[0033] Figure 5 It is a Figure 4 Orthophoto DOM obtained by processing photos in the survey area.

[0034] Figure 6 For the orthographic image DOM format conversion and normalization processing after selecting a sample area dom13N. Figure 4 The digital surface model DSM obtained by processing the photos in the survey area.

[0035] Figure 7 For the orthographic image DOM format conversion and normalization processing after selecting a sample area dom13N.

[0036] Figure 8 For the orthographic image DOM format conversion and normalization processing after selecting a sample area dom13N. Figure 7 The sample area slope13N corresponding to the sample area position.

[0037] Figure 9 The vector image obtained after the multi-channel image annotation processing.

[0038] Figure 10 For Figure 9 The corresponding raster data image.

[0039] Figure 11 The sample data set obtained after cutting and enhancing the multi-channel image and the class label map.

[0040] Figure 12 The predicted data (sample) set obtained after cutting and enhancing the four-channel image to be predicted.

[0041] Figure 13 The prediction result vector map corresponding to the image to be predicted.

[0042] Figure 14 The DEM image containing holes obtained after mask processing according to the prediction result vector map.

[0043] Figure 15 The overlay map of the contour line and the contour line generated by the original digital surface model DSM corresponding to the DEM image obtained after the hole filling. DETAILED DESCRIPTION

[0044] Specifically, the following specific schemes are given in combination with the drawings in the specification. Figures 1-15

[0045] In combination with a certain gray field topographic map surveying project, the present method is described. The survey area of the project is about 12 square kilometers, and the mapping scale is 1:500. In the project, DJI Flyma D2000 unmanned aerial vehicle carrying SONY a6000 camera is used, and POS and IMU systems with RTK function are integrated. The SONY a6000 camera sensor size is 23.5X15.5mm, the effective pixels are 2400 million, and the focal length is 25mm.

[0046] In this project, the survey area is full of gullies,​Figure 2 Fig. 1 shows a top view of the vegetation area of the test area of this embodiment. The high and large trees scattered in the ditch are shown in the boxed area. If a digital surface model (DSM) is directly used to generate the contour lines as shown in Fig. 2, it can be seen that there are multiple elevation anomaly areas (black blocks), which cannot reflect the true terrain conditions, causing errors in the calculation of excavation and filling. Figure 3

[0047] Therefore, in view of the above problems, according to the actual operation steps, as shown in Fig. 3, the method proposes a specific embodiment: Figure 1

[0048] First, use a UAV carrying an optical camera to obtain a large number of centimeter-level spatial resolution photos in a selected area. The selected area is mainly a vegetation sparse area of several square kilometers. The interior and exterior orientation element information of the photos must be obtained synchronously, and the photo control points must be laid out to meet the specification requirements.

[0049] Specifically, the UAV flight time of this embodiment project is May 17, 2020, and the weather is sunny. The flight height is about 300 meters, the photo overlap degree in the flight direction is ≥70%, preferably 80%, and the photo overlap degree in the lateral direction is ≥60%, preferably 65%. A total of 3619 photos were taken in this project, with an average spatial resolution of 0.05m, a single photo range of about 280m*212m, and a total of 29 photo control points laid out in the test area, as shown in Fig. 4. Figure 4 Figure 4 Fig. 4(a) is a test area range and photo control point distribution diagram, Figure 4 Fig. 4(b) is a photo control point.

[0050] Second, use the UAV data processing software to input the photos, photo control points and interior orientation element information obtained in the first step into the UAV data processing software to obtain the orthophoto (DOM) and digital surface model (DSM) matched with the test area. The orthophoto (DOM) and digital surface model (DSM) obtained in this step are the basic images for subsequent processing. The UAV data processing software uses Context Capture software. As shown in Fig. 5 and Fig. 6, Figure 5 and Figure 6 respectively, are the orthophoto (DOM) and digital surface model (DSM) corresponding to the test area after processing using the software (tif format).

[0051] ​​​Thirdly, the digital surface model DSM obtained in the second step is converted into a corresponding slope map. Generally, the obtained DOM is an RGB three-channel 8-bit unsigned integer, and the digital surface model DSM is a 32-bit floating point. And because the DOM and the digital surface model DSM are both sensitive to vegetation, the vegetation in the DOM is represented by spectral and texture characteristics, and the digital surface model DSM is represented by height gradient mutation and texture characteristics. In this method, the slope of the digital surface model DSM is calculated pixel by pixel to obtain the slope map corresponding to the selected area. The slope calculation considers the 8-neighborhood of the center pixel, as shown in Table 1 below is the position diagram of the center pixel e and the surrounding 8 pixels:

[0052] a b c d e f g h i

[0053] Table 1

[0054] Suppose a, b, c, d, e, f, g, h, i are height values, x_cellsize and y_cellsize are the commonly used unit distances in the x direction and the y direction, respectively, in order to calculate the slope of the center pixel e, the change rate of the pixel e in the x direction (formula (1)) and the change rate of the pixel e in the y direction (formula (2)) are calculated respectively:

[0055] [dz / dx]=((c+2f+i)-(a+2d+g) / (8*x_cellsize (1)

[0056] [dz / dy]=((g+2h+i)-(a+2b+c)) / (8*y_cellsize) (2)

[0057] Further, the slope value at the pixel e is obtained (formula (3)):

[0058]

[0059] And by converting and normalizing the two kinds of data, a 4-channel floating point data image containing RGB channels and slope channels is obtained. This method is the first time to use both DOM and slope map data in the deep learning algorithm for vegetation classification, combined with class label map and other vegetation classification files, to extract vegetation classification deep learning. Of course, this method can also be applied to the classification of other ground objects or interference objects, and in this embodiment, the vegetation classification in mapping is mainly explained, and other channel images can also be introduced on this basis to integrate into multi-channel images for subsequent vegetation classification extraction.

[0060] The format conversion is to convert the orthophoto and the slope map into the same floating point data, and to integrate the RGB three channels of the orthophoto and the slope channel to obtain a 4-channel map; the normalization processing is to take a value between 0 and 1. Specifically, according to dsm.tif, a slope map slope.tif (value range 0.0000-90.0000) is generated, and the normalization processing is performed to obtain a single-channel floating point data slopeN.tif (value range 0.0000-1.0000). And the orthophoto DOM is an RGB three-channel 8-bit unsigned integer format (value range 0-255), and the normalization processing is performed on the channel data to obtain a 3-channel floating point data domN.tif (value range 0.0000-1.0000). From the output 3-channel floating point data domN.tif corresponding to the orthophoto DOM and the single-channel floating point data slopeN.tif corresponding to the slope map, a block of each is selected as a sample area, which is a sample area dom13N.tif as shown in Figure 7 and a sample area slope13N.tif as shown in Figure 8 Further, the data is spliced to obtain a 4-channel data sample.tif (value range 0.0000-1.0000) containing R, G, B and slope, and the projection and image row and column are kept unchanged. Figure 1

[0061] The fourth step is to label the vegetation and other ground objects in the four-channel image obtained in the third step (or using the orthophoto obtained in the second step). Specifically, the image is interpreted using a remote sensing data processing software, and the corresponding vegetation area is marked in detail using ArcGIS software, a vector polygon matching the image is generated, a class name and a class label value field (such as background 0, vegetation 1, and more classes can be expanded according to the ground object type) are added to the attribute value, and saved as a vector file (label.shp) as shown in Figure 9 Further, the vector file label.shp is converted into a raster file (label.tif) with the same size (resolution and row and column number are the same) and projection mode as the four-channel image using ArcGIS software, that is, a class label map. Figure 10

[0062] ​​Step 5, image cutting processing is performed on the class label image label.tif obtained in step 4 and the four-channel image sample.tif generated in step 3, that is, the classification label image obtained in step 4 and the multi-channel image obtained in step 3 are cut into sample images of the same size, and sample enhancement processing (such as rotation, blurring, adding noise, etc.) is performed at the same time. A feature of the enhancement processing is to enhance the display of the labeled part to be recognized, so as to better learn and recognize; the enhancement processing obtains a sample data set of four-channel images and corresponding class label images of the same size and satisfying 32 integer multiples, such as Figure 11 After the cutting and enhancement processing of the four-channel image and the class label image, a total of 6000 sample data sets are obtained, and the image size is 512*512 pixels.

[0063] Step 6, the sample data set generated in step 5 is randomly divided into two parts, one part is used for model training, that is, a training data set, and the other part is used for model verification, that is, a verification data set.

[0064] In this specific embodiment, the sample data set generated in step 5 is preferably divided into two parts according to a ratio of 9:1, the part with a ratio of 9 is used for model training, and the part with a ratio of 1 is used for model verification. The file name index of the training data set and the verification data set is saved to a file.

[0065] Step 7, multi-channel recognition and ground object classification recognition training is performed on the training data set in step 6 by using a UNET model. UNET can achieve high classification results with a small amount of training samples. It adjusts parameters according to task requirements, including the number of data channels, the number of classes, the number of iterations, the learning rate, etc. In this embodiment, the structure of the main network adopts a VGG16 model, uses a Relu function as an activation function, and uses a cross-entropy and a softmax function to calculate a loss. The training data set generated in step 6 and an initial model file (which can be used for transfer learning) are used as inputs to perform 400 rounds of training, and finally a UNET vegetation classification model capable of performing 4-channel data training and prediction of R, G, B, and slope is generated, and a series of initial model files (*.pth) are generated.

[0066] Step 8, the verification data set in step 6 and the initial model file generated in step 7 are used as inputs to perform verification training, and the mean intersection over union (MIOU) of the predicted value and the true value of each channel and each ground object class recognition is calculated. It is one of the reference factors for updating the model. The parameters of the initial model are modified through the mean intersection over union, preferably the calculated mean intersection over union is optimal, otherwise the model parameters are modified, the model is iterated and optimized, and the final prediction recognition model (M) is obtained by updating the model file (*.pth). The calculation steps of the mean intersection over union MIOU are as follows:

[0067] (1) Calculate the confusion matrix;

[0068] (2) Calculate the intersection over union (IOU_i) of each channel and feature class, the formula is:

[0069] IOU_i = TP_i / (TP_i + FN_i + FP_i)

[0070] Wherein:

[0071] TP_i is the true value, that is, the intersection of the true value and the predicted value;

[0072] FN_i is the false negative value, that is, the part of the true value minus the intersection of the true value and the predicted value;

[0073] FP_i is the false positive value, that is, the part of the predicted value minus the intersection of the true value and the predicted value;

[0074] (3) Average the IOU of each class to get the mean intersection over union (MIOU).

[0075] Step 9, perform the second and third steps on the image to be predicted to obtain a four-channel image to be predicted corresponding to the image to be predicted, which includes RGB three channels and a slope channel.

[0076] Step 10, then perform the fifth step of cutting and enhancing on the four-channel image to be predicted obtained in step 9 to obtain a data (sample) set to be predicted corresponding to the image to be predicted as shown in Figure 12 , each sample of which has the same size as the sample obtained after cutting and enhancing the four-channel image and the class label map in step 5. In addition, the predicted data file needs to retain the projection and geographic coordinate information of the original image to be measured to facilitate positioning and splicing. The naming rule of the data file is "filename_original total row number_original total column number_starting row number_starting column number.tif".

[0077] Step 11, send the predicted data set generated in step 10 and the recognition model file obtained in step 8 into the final prediction recognition model for prediction to obtain a vegetation prediction result data set.

[0078] Step 12, perform image mosaicking (i.e., splicing) on the prediction result obtained in step 11 to obtain a prediction result with the same size and projection as the original four-channel image to be predicted in step 9.

[0079] Step 13, perform raster-to-vector conversion on the prediction result obtained in step 12 to obtain a prediction result vector map corresponding to the image to be predicted, from which the boundary range of the circled vegetation can be obtained. Figure 13

[0080] ​The fourteenth step is to superimpose the prediction result vector file obtained in the thirteenth step on the four-channel image to be predicted of step 9, and to perform interactive editing, adding, deleting and modifying the missed and mistaken areas, so as to further improve the recognition rate of the vegetation, and to obtain a full-coverage ground object classification result image containing multi-channel information recognition and vegetation and other ground object classification recognition.

[0081] The fifteenth step is to take the classification result obtained in the fourteenth step as a mask, and to perform mask processing on the digital surface model (DSM) obtained by processing the image to be predicted in the unmanned aerial vehicle data processing software, so as to obtain a DEM image containing a plurality of holes (vegetation areas) as shown in Figure 14 .

[0082] The sixteenth step is to take the DEM image containing holes obtained in the fifteenth step as a basis, to extract the minimum rectangular boundary of each vegetation (or ground object) area in the DEM image, to perform appropriate outward expansion (2 pixels), and to record the top-left corner and bottom-right corner positions, so as to obtain a corresponding index file.

[0083] The seventeenth step is to sequentially fill the holes from left to right and from top to bottom according to the index file obtained in the sixteenth step. For each sub-area with a hole, a linear interpolation method is used to perform three-dimensional spatial interpolation. That is, the interpolation points are determined according to the existing rectangular boundary points determined in the sixteenth step by using the linear interpolation method, and further, a triangular basis is used to first find three points around the interpolation point to form a triangle, and the interpolation point is in the triangle, so as to construct a Delaunay triangular mesh surface that satisfies the Delaunay criterion through the determined interpolation points. A continuous and smooth surface, i.e., an elevation surface, is obtained by using this method. In the process of constructing the elevation surface, the algorithm retains the original pixel value while filling in the missing pixel value, that is, the pixel value of the interpolation point is determined according to the pixel value of the known point in the rectangular area by using the linear interpolation method, and the interpolation point is recursively assigned to the entire Delaunay triangular mesh surface according to the position and pixel value of the interpolation point, so as to obtain a smooth and continuous DEM data image of the area to be measured as shown in Figure 15 . In the figure, the discontinuous line is the contour line obtained by using the method, and the solid line is the contour line generated directly by using the digital surface model (DSM). The method first proposes a multi-hole DEM image filling algorithm, which fills the gap in this field.

[0084] The method is suitable for DEM extraction in sparse vegetation areas, and can be particularly used for generating large-scale topographic maps (1:500-1:2000) in a square kilometer area. By using the method, the vegetation can be filtered without changing the true ground elevation value, and the ground elevation of the corresponding area can be filled, which has a high automation level and can greatly improve the work efficiency of the industry personnel.

Claims

1.A method for identifying an image feature using a UNET model, the method comprising: inputting an image into the UNET model; and identifying the image feature based on a result of the inputting. The identification image of the ground object type and the elevation of the to-be-tested image is obtained through the following steps: Step one: selecting an area, performing data processing on the photographed photos of the area to obtain a multi-channel image corresponding to the area, and performing ground object labeling on the basis of the multi-channel image to obtain a classification label map corresponding to the area; Step two: performing cutting processing on the classification label map and the multi-channel image obtained in step one to obtain a sample data set, and randomly dividing the sample data set into a training data set for model training and a verification data set for model verification; Step three: inputting the training data set obtained in step two into a UNET model for multi-channel identification training and ground object classification identification training to obtain an initial model; Step four: inputting the verification data set obtained in step two into the initial model generated in step three for verification training, and calculating the mean intersection over union (MIOU) of the predicted value and the true value of each channel and each ground object class identification, modifying the parameters of the initial model through the MIOU to obtain a final prediction model; Step five: performing data processing on the to-be-tested image according to step one to obtain a corresponding to-be-tested multi-channel image; and performing cutting processing on the to-be-tested multi-channel image according to step two to obtain a to-be-tested data set; Step six: inputting the to-be-tested data set obtained in step five into the prediction model obtained in step four for prediction to obtain a prediction result data set containing ground object classification and channel identification; Step seven: performing image mosaic processing on the prediction result data set obtained in step six to obtain a result map containing multi-channel information and ground object identification; In step one, the labeling processing includes adding a class name and a class label value field to each ground object to be labeled after vectorization processing of the multi-channel image; and further rasterization processing of the vectorized multi-channel image to obtain a class label map corresponding to the multi-channel image; The class name and class label value field added when labeling the vectorized multi-channel image include: background 0, vegetation 1. 2.The image feature recognition method using the UNET model according to claim 1, wherein, Further, step eight is included: performing interactive editing on the result map obtained by mosaic in step seven and the to-be-tested multi-channel image obtained in step five to obtain a corrected result map containing multi-channel information and ground object identification. 3.The image feature recognition method using the UNET model according to claim 1, wherein, The multi-channel image obtained in step one includes at least RGB three channels and a slope channel. 4.The image feature recognition method using the UNET model according to claim 1, wherein, The cutting processing in step two includes cutting the classification label map and the multi-channel image obtained in step one into sample images of the same size, and further enhancing the sample images obtained by cutting to obtain a sample data set. 5.The image feature recognition method using the UNET model according to claim 1, wherein, In step three, the recognition training is performed using a UNET model, the VGG16 model is selected as the backbone network structure, the Relu function is selected as the activation function, and the loss in the training is calculated by the soft function and the cross entropy. 6.The image feature recognition method using the UNET model according to claim 1, wherein, In step four, the MIOU is calculated through the following steps: (1) calculating the confusion matrix; (2) calculating the intersection over union (IOU_i) of each channel and ground object class, and for each IOU_i, the formula is: IOU_i=TP_i / (TP_i+FN_i+FP_i) Wherein, TP_i is the true value, FN_i is the false negative value, FP_i is the false positive value; (3) Average IOU_i of each class to get the mean intersection over union MIOU. 7.The image feature recognition method using the UNET model according to claim 2, characterized in that, In step seven, the mosaicked result image containing multi-channel information and ground object recognition is further processed by raster-vector conversion to a vector file, and the vectorized result image is interactively edited and repaired. 8.The image feature recognition method using the UNET model according to claim 3, characterized in that, The data processing in step one is to process the photos of the region into an orthophoto and a digital surface model corresponding to the region by using a UAV data processing software, and further perform slope calculation on the digital surface model to obtain a corresponding slope map, and integrate the orthophoto and the slope map into a multi-channel image including at least an RGB channel and a slope channel.

Images