Image recognition method, illegal building detection method, terminal device and storage medium

Abstract

The application discloses an image recognition method, an illegal building detection method, a terminal device and a computer storage medium. The image recognition method comprises the following steps: acquiring a UAV image collected based on a target area; inputting the UAV image into a pre-trained instance segmentation network, wherein the instance segmentation network comprises a feature extraction model, a feature detection model and a mask generation model; extracting a feature image of the UAV image through the feature extraction model; generating a target building detection frame in the feature image through the feature detection model; generating a mask of the target building on the feature image according to the target building detection frame through the mask generation model; and acquiring building information in the UAV image based on the mask output by the instance segmentation network. The image recognition method disclosed by the application efficiently extracts a building mask in a UAV image by proposing an instance segmentation network for buildings.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image recognition method, a illegal building detection method, a terminal device and a computer storage medium. BACKGROUND

[0002] Illegal building supervision has always been one of the important work of city governance, the traditional way of artificial illegal building patrol time-consuming and laborious, for the supervision of illegal building is not timely and effective, at the same time, due to the limitation of the angle of view, can not find the illegal construction on the top of the building and the behavior of building, resulting in the need to invest a lot of time and energy to carry out the demolition and reconstruction work of illegal building. SUMMARY

[0003] The present application provides an image recognition method, a illegal building detection method, a terminal device and a computer storage medium.

[0004] One of the technical solutions adopted by the present application is to provide an image recognition method, which comprises:

[0005] acquiring a drone image collected based on a target area;

[0006] inputting the drone image into a pre-trained instance segmentation network, wherein the instance segmentation network comprises a feature extraction model, a feature detection model and a mask generation model;

[0007] extracting a feature image of the drone image through the feature extraction model;

[0008] generating a target building bounding box in the feature image through the feature detection model;

[0009] generating a mask of the target building on the feature image according to the target building bounding box through the mask generation model;

[0010] obtaining building information in the drone image based on the mask output by the instance segmentation network.

[0011] The mask generation model comprises a first multilayer perceptron, a second multilayer perceptron, a deconvolution layer and a convolution layer connected in sequence.

[0012] The mask generation model further comprises a maximization processing layer and an average processing layer, which are arranged side by side between the second multilayer perceptron and the deconvolution layer, and are used to remove the hole information of the target building bounding box in the process of generating the mask.

[0013] The loss function of the instance segmentation network comprises at least a mask loss, wherein the mask loss is calculated by using a sampling point loss function, and the sampling point loss function calculates the loss function of the instance segmentation network by using difference information between predicted mask output and actual mask output of a plurality of sampling points in the unmanned aerial vehicle image.

[0014] The image recognition method further comprises:

[0015] In the training process of the instance segmentation network, a training feature map of a training image is obtained;

[0016] Confidence of all pixel points in the training feature map is obtained;

[0017] A plurality of sampling points are selected in the training feature map according to the confidence of all pixel points;

[0018] The instance segmentation network is trained by using difference information between predicted mask output and actual mask output of the plurality of sampling points.

[0019] The plurality of sampling points are selected in the training feature map according to the confidence of all pixel points, comprising:

[0020] A first confidence that the all pixel points belong to a foreground category and a second confidence that the all pixel points belong to a background category are obtained;

[0021] An uncertainty of the pixel points is determined according to a sum of an absolute value of the first confidence and an absolute value of the second confidence;

[0022] The pixel points with an uncertainty greater than or equal to a preset threshold value are selected as the sampling points in the training feature map.

[0023] The image recognition method further comprises:

[0024] In the training process of the instance segmentation network, a training image and a building roof mask thereof are obtained;

[0025] A building mask of illegal construction is extracted from the training image or a database of illegal construction;

[0026] The building mask of illegal construction is attached to the building roof mask of the training image to form a new training image.

[0027] After the mask generation model generates the mask of the target building on the feature image according to the target building detection frame, the image recognition method further comprises:

[0028] A building roof mask output by the instance segmentation network is obtained;

[0029] generate a corresponding minimum bounding rectangle based on the building roof mask, and obtain a to-be-regularized region in the minimum bounding rectangle that does not belong to the building roof mask;

[0030] generate a maximum inscribed rectangle in the to-be-regularized region;

[0031] trim the minimum bounding rectangle according to the maximum inscribed rectangle to obtain a regularized building roof mask.

[0032] Another technical solution adopted by the present application is to provide a illegal building detection method, the illegal building detection method comprises:

[0033] obtain a UAV image collected based on a target region, and obtain a corresponding orthographic map slice of the UAV image;

[0034] obtain a building roof mask and an illegal building mask on the UAV image, wherein the mask obtaining manner is the same as the mask obtaining manner in the image recognition method described above;

[0035] obtain a reprojected image of the orthographic map slice reprojected to a coordinate system of the UAV image based on the building roof mask;

[0036] obtain a change detection region result of the target region based on image difference information of the reprojected image and the UAV image;

[0037] obtain illegal building detection information of the UAV image by combining the change detection region result and illegal building detection result of the illegal building mask.

[0038] Another technical solution adopted by the present application is to provide a terminal device, the terminal device comprises a memory and a processor coupled with the memory;

[0039] wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the image recognition method and / or the illegal building detection method described above.

[0040] Another technical solution adopted by the present application is to provide a computer storage medium, the computer storage medium is configured to store program data, and the program data is configured to implement the image recognition method and / or the illegal building detection method described above when executed by a computer.

[0041] The beneficial effects of the present application are: the terminal device acquires the unmanned aerial vehicle image based on the target area collection; the unmanned aerial vehicle image is input into the pre-trained instance segmentation network, wherein the instance segmentation network includes a feature extraction model, a feature detection model and a mask generation model; the feature image of the unmanned aerial vehicle image is extracted through the feature extraction model; the target building bounding box in the feature image is generated through the feature detection model; the mask of the target building on the feature image is generated according to the target building bounding box through the mask generation model; and the building information in the unmanned aerial vehicle image is acquired based on the mask output by the instance segmentation network. The image recognition method of the present application efficiently extracts the building mask in the unmanned aerial vehicle image by proposing an instance segmentation network for buildings. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a flowchart of the first embodiment of the image recognition method provided by the present application;

[0044] Figure 2 is a structural schematic diagram of an embodiment of the instance segmentation network provided by the present application;

[0045] Figure 3 is a flowchart of the second embodiment of the image recognition method provided by the present application;

[0046] Figure 4 is a flowchart of the third embodiment of the image recognition method provided by the present application;

[0047] Figure 5 is a schematic diagram of the simulation data generation process provided by the present application;

[0048] Figure 6 is a flowchart of the fourth embodiment of the image recognition method provided by the present application;

[0049] Figure 7 is a schematic diagram of the regularized building roof provided by the present application;

[0050] Figure 8 is a flowchart of an embodiment of the illegal building detection method provided by the present application;

[0051] Figure 9 is a schematic diagram of the total flow of the illegal building detection method provided by the present application;

[0052] Figure 10 is a structural schematic diagram of an embodiment of a terminal device provided by the present application.

[0053] Figure 11 is a structural schematic diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0055] Please refer to Figure 1 , Figure 1 is a flow schematic diagram of a first embodiment of an image recognition method provided by the present application.

[0056] The image recognition method of the present application is applied to an image recognition device. The image recognition device of the present application can be a server or a system cooperating with a local terminal. Correspondingly, each part of the image recognition device, such as each unit, sub-unit, module, and sub-module, can be arranged in the server or arranged in the server and the local terminal respectively.

[0057] Further, the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing a distributed server, or as a single software or software module, which is not limited here. In some possible implementation manners, the image recognition method of the embodiments of the present application can be implemented by a processor calling computer readable instructions stored in a memory.

[0058] Specifically, as shown in Figure 1 the image recognition method of the embodiments of the present application specifically includes the following steps:

[0059] Step S11: Obtain a UAV image collected based on a target area.

[0060] In the embodiments of the present application, the image recognition device reads historical UAV images or UAV video streams in the process of UAV inspection on a target area or a target region, and obtains a UAV image from the historical UAV images or the UAV video streams.

[0061] Step S12: inputting the UAV image into a pre-trained instance segmentation network, wherein the instance segmentation network comprises a feature extraction model, a feature detection model and a mask generation model.

[0062] In the embodiment of the present application, the image recognition device inputs the UAV image collected in step S11 into the pre-trained instance segmentation network, and the instance segmentation network identifies the building targets in the UAV image, such as building roofs and illegal buildings, and generates corresponding masks.

[0063] Among them, the present application puts forward a new type of instance segmentation network, please refer to Figure 2 , Figure 2 is the structural schematic diagram of an embodiment of the instance segmentation network provided by the present application.

[0064] As Figure 2 shown, the instance segmentation network of the present application specifically comprises a feature extraction model, a feature detection model and a mask generation model.

[0065] Among them, the feature extraction model uses swim-transform as the encoder to strengthen the network's ability to extract image features. As Figure 2 shown, swim-transform includes four pooling layers, and each pooling layer will reduce the resolution of the input feature map, similar to the CNN expanding the receptive field layer by layer. Through swim-transform, the feature image of the UAV image is reduced by one scale each time after passing through a pooling layer, and finally five feature images of different scales can be obtained.

[0066] The feature detection model uses the FCOS (Fully Convolutional One-Stage Object Detection) network to extract the detection frame of the target building, wherein the FCOS network is a one-stage network based on the idea of center-ness, which realizes target detection without anchor and proposal, and the recall rate is not inferior to the anchor-based detection algorithm.

[0067] The mask generation model employs a SAG-Mask network structure to predict a 28x28 mask, which is then scaled to the size of the detection bounding box. The decoder SAG-Mask part provided in this application uses an MLP instead; that is, the feature map of the last layer of the encoder's swim-transform is combined with the result of the FCOS network to extract an N*14*14 feature map, where N represents the number of predicted objects. The SAG-Mask network structure uses two MLPs (Multilayer Perceptrons), then uses max (maximization processing layer) and avg (average processing layer) to remove holes respectively, stacks them together, and finally uses deconvolution and a convolutional layer to output the final result.

[0068] The maximization processing layer and the averaging processing layer are arranged side by side between the second multilayer perceptron and the deconvolution layer. The maximization processing layer and the averaging processing layer are used to remove the hole information of the target building detection box during the mask generation process.

[0069] in, Figure 2 The loss function of the instance segmentation network shown consists of four parts: target classification loss, center location loss, regression loss, and mask loss. The mask loss is calculated using a sampling point loss function, which utilizes the difference between the predicted mask output and the actual mask output at several sampling points in the UAV image to calculate the mask loss of the instance segmentation network.

[0070] Specifically, for the process of the instance segmentation network selecting sampling points and calculating the loss function using the sampling points, please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the image recognition method provided in this application.

[0071] Specifically, such as Figure 3 As shown, the image recognition method of this application embodiment specifically includes the following steps:

[0072] Step S21: During the training process of the instance segmentation network, obtain the training feature map of the training image.

[0073] In the embodiments of this application, during the training process of the instance segmentation network, the image recognition device uses a feature extraction model to extract the 2*H*W training feature map of the training image.

[0074] Step S22: Obtain the confidence scores of all pixels in the training feature map.

[0075] In this embodiment, the image recognition device uses a feature detection model to predict a first confidence level for each pixel in the training feature map belonging to the foreground category and a second confidence level for belonging to the background category. The foreground includes illegal buildings and building rooftops.

[0076] Step S23: Selecting a plurality of sampling points in the training feature map according to the confidence of all pixel points.

[0077] In the embodiment of the present application, the image recognition device selects K sampling points according to the uncertainty of the current pixel point for each H*W training feature map, instead of the entire training feature map, to calculate the loss function.

[0078] The uncertainty of the pixel point can be determined by the confidence of the pixel point predicting different categories. Specifically, the image recognition device obtains the first confidence of all pixel points on the training feature map belonging to the foreground category, and the second confidence belonging to the background category, and then calculates the sum of the absolute values of the first confidence and the second confidence as the uncertainty of the pixel point. The specific formula is as follows:

[0079] uncertainty = -(torch.abs(gt_class_logits1))-(torch.abs(gt_class_logits2))

[0080] Where uncertainty is the uncertainty of the pixel point, gt_class_logits1 is the first confidence, gt_class_logits2 is the second confidence, and torch.abs() is the absolute value function.

[0081] The image recognition device selects the pixel point with an uncertainty higher than a certain threshold as the sampling point.

[0082] Step S24: Training the instance segmentation network using the difference information between the predicted mask output and the actual mask output of the plurality of sampling points.

[0083] In the embodiment of the present application, the image recognition device trains the instance segmentation network using the difference between the predicted mask output and the actual mask output of the sampling points selected in step S23, so that the predicted mask output of the instance segmentation network approximates the actual mask output as the training purpose.

[0084] In the image recognition method of the embodiment of the present application, the image recognition device uses the sampling point loss function, which can effectively improve the training speed of the network. In addition, when the image recognition device uses the simulated data set for training, there is a difference between the real standard data set, and the labeling accuracy will be much lower, so using the sampling point loss function can further reduce the influence of data set error.

[0085] The process of generating a simulated data set in the present application will be introduced below. For details, please refer to Figure 4 and Figure 5 , Figure 4is a flowchart of a third embodiment of the image recognition method provided by the present application, Figure 5 is a schematic diagram of the simulation data generation process provided by the present application.

[0086] The embodiment of the present application first trains the instance segmentation network using a small data set annotated with building roofs and illegal buildings in the roofs, and after training, predicts the building roofs and illegal buildings in the roofs from the unlabeled drone images by using the instance segmentation network. The illegal buildings are extracted from the images to form an illegal building data set.

[0087] Specifically, as Figure 4 shown, the image recognition method of the embodiment of the present application specifically includes the following steps:

[0088] Step S31: In the training process of the instance segmentation network, a training image and its building roof mask are obtained.

[0089] In the embodiment of the present application, the mask generation model of the instance segmentation network generates a building roof mask on the training image. As Figure 5 shown, the upper left corner is the original image of the training image, and the upper right corner is the building roof mask and illegal building mask of the training image.

[0090] Step S32: Extract the illegal building mask from the training image or the illegal building database.

[0091] In the embodiment of the present application, the image recognition device randomly extracts the illegal building mask from the training image or the illegal building database, as Figure 5 the illegal building mask extracted from the upper right corner of the training image in the lower left corner.

[0092] Step S33: Attach the illegal building mask to the building roof mask of the training image to form a new training image.

[0093] In the embodiment of the present application, the image recognition device attaches the illegal building mask to the building roof mask of the training image to form a new training image, as Figure 5 the generated simulation data in the lower right corner.

[0094] The attachment rules provided by the embodiment of the present application include but are not limited to:

[0095] 1. The illegal building mask must be entirely in the building roof mask;

[0096] 2. The illegal building mask must have one edge coinciding with the edge of the building roof mask;

[0097] 3. Cannot overlap with other illegal building masks.

[0098] Step S13: extracting a feature image of the UAV image through the feature extraction model.

[0099] Step S14: generating a target building detection frame in the feature image through the feature detection model.

[0100] Step S15: generating a mask of the target building on the feature image according to the target building detection frame through the mask generation model.

[0101] In the embodiments of the present application, the functions of the feature extraction model, the feature detection model and the mask generation model in the application are basically the same as those in the training process. For details, please refer to the introduction of the training process above. The mask generation model can generate a building roof mask and a illegal building mask.

[0102] Through the above process, the instance segmentation network outputs a building roof mask, but the building roof mask is generally irregular in shape and may contain a large number of unreasonable sharp corners and the like because it is the product of instance segmentation. However, the building roof is generally rectangular in shape, so the image recognition device of the present application can regularize the building roof by using a model-driven method. For details, please refer to Figure 6 , Figure 6 is a flowchart of the fourth embodiment of the image recognition method provided by the present application.

[0103] Specifically, as shown in Figure 6 , the image recognition method of the embodiments of the present application specifically includes the following steps:

[0104] Step S41: obtaining a building roof mask output by an instance segmentation network.

[0105] Step S42: generating a corresponding minimum bounding rectangle based on the building roof mask, and obtaining a regularization area in the minimum bounding rectangle that does not belong to the building roof mask.

[0106] In the embodiments of the present application, the image recognition device identifies the shape of the building roof mask, labels the area of the building roof mask in the minimum bounding rectangle, and marks the remaining area as the regularization area.

[0107] Step S43: generating a maximum inscribed rectangle in the regularization area.

[0108] In the embodiments of the present application, the image recognition device generates a maximum inscribed rectangle in each connected regularization area, and the number of maximum inscribed rectangles is consistent with the number of areas of connected regularization areas.

[0109] Step S44: cropping the minimum bounding rectangle according to the maximum inscribed matrix to obtain a regularized building roof mask.

[0110] In the embodiment of the present application, the image recognition device crops each belonging to the to-be-normalized region according to the maximum inscribed rectangle determined in step S43, and retains the region where the building roof mask is located, so as to realize the normalization of the building roof mask.

[0111] Further, the image recognition device can also continuously normalize the building roof mask in an iterative manner until the shape of the actual building roof is approached, that is, the normalized building roof as shown in the schematic diagram of the normalized building roof. Figure 7 The iteration number can be set in a preset number manner, or can be determined according to whether the area of the to-be-normalized region in each iteration exceeds a preset area threshold, wherein the preset area threshold can be determined according to the area of the building roof mask, that is, the preset area threshold is obtained by multiplying the area of the building roof mask by a proportionality coefficient.

[0112] Step S16: obtaining building information in the unmanned aerial vehicle image based on the mask output by the instance segmentation network.

[0113] In the embodiment of the present application, the image recognition device instance segments the building roof mask and the illegal building mask through steps S11 to S15, so as to obtain the building information in the unmanned aerial vehicle image, including the illegal building behavior position, the illegal building behavior time and the illegal building behavior type, etc.

[0114] In the embodiment of the present application, the image recognition device obtains the unmanned aerial vehicle image collected based on the target region; inputs the unmanned aerial vehicle image into a pre-trained instance segmentation network, wherein the instance segmentation network includes a feature extraction model, a feature detection model and a mask generation model; extracts a feature image of the unmanned aerial vehicle image through the feature extraction model; generates a target building detection frame in the feature image through the feature detection model; generates a mask of the target building on the feature image according to the target building detection frame through the mask generation model; and obtains building information in the unmanned aerial vehicle image based on the mask output by the instance segmentation network. The image recognition method of the present application efficiently extracts the building mask in the unmanned aerial vehicle image by proposing an instance segmentation network for buildings.

[0115] The present application efficiently extracts the building roof mask and the illegal building mask by proposing an instance segmentation network for building roofs and illegal buildings; automatically and efficiently simulates illegal buildings in the roof by proposing a data simulation method for illegal buildings, greatly improves the data volume, and reduces the complex task quantity caused by data labeling; finally, the building mask is normalized by using building normalization, so that the building mask is more beautiful in effect.

[0116] Furthermore, based on the aforementioned image recognition methods, this application also proposes a scheme for monitoring illegal construction based on UAV inspection images, comprehensively utilizing deep learning registration, instance segmentation, and historical image change analysis. This scheme can achieve automated extraction and analysis of illegal buildings within a specific area, effectively investigating illegal construction activities over a wide area in a comprehensive and low-cost manner, improving the detection rate and rapid response efficiency of illegal buildings, and enabling thorough investigation of existing illegal buildings, in-process detection of illegal construction activities, and pre-emptive prevention of additions.

[0117] Among them, deep learning instance segmentation is mainly used to detect existing illegal buildings, construction workers and related construction equipment; deep learning image registration is mainly used to register multiple phases of UAV imagery and maps; and deep learning change analysis is mainly used to detect change areas in multiple phases of UAV imagery and maps.

[0118] Please see Figure 8 and Figure 9 , Figure 8 This is a flowchart illustrating an embodiment of the illegal construction detection method provided in this application. Figure 9 This is a schematic diagram of the overall process of the illegal construction detection method provided in this application.

[0119] The present application proposes a deep learning-based method for detecting and monitoring illegal buildings using UAV imagery. For details of the technical process, please refer to [link to relevant documentation]. Figure 9 The overall process diagram is described above. The illegal construction detection method of this application is mainly divided into three modules, namely Module 1 (i.e.... Figure 2 Patent 2) Base map positioning and cropping transformation module, module two (i.e. Figure 2 Patent 3) Unmanned Aerial Vehicle Image / Video Stream Instance Segmentation Module and Module Three (i.e. Figure 2 Patent 1) Change detection and post-processing module. The following section combines... Figure 1 The flowchart of one embodiment of the illegal construction detection method describes the workflow of module one, module two and module three respectively.

[0120] The illegal construction detection method of this application is applied to an illegal construction detection device. This device can be a server or a system in which the server and the illegal construction detection device cooperate. Accordingly, all components of the illegal construction detection device, such as units, subunits, modules, and submodules, can be entirely housed in the server, or they can be separately housed in the server and the illegal construction detection device.

[0121] Further, the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules, for example, software or software modules used to provide a distributed server, or as a single software or software module, which is not limited here. In some possible implementation manners, the illegal building detection method of the embodiments of the application can be implemented by a processor calling computer readable instructions stored in a memory.

[0122] Specifically, as shown in Figure 8 The illegal building detection method of the embodiments of the application specifically includes the following steps:

[0123] Step S51: Obtain a UAV image collected based on a target area, and obtain an orthographic map slice corresponding to the UAV image.

[0124] In the embodiments of the application, the illegal building detection device cuts out an orthographic map slice corresponding to the range of the UAV image from the orthographic map according to GPS information in the exif (Exchangeable image file format) of the UAV image, such as the latitude and longitude of the target area.

[0125] In other embodiment modes, the illegal building detection device can expand a preset area based on the map area determined by the GPS information of the UAV image, so as to ensure that the orthographic map slice can provide all pixel information matched with the UAV image in the case of considering camera distortion and calculation error, etc., and to locate in the UAV image according to the GPS information and cut out the corresponding area of the base map according to the width and height of the UAV image, the cutting width and height being slightly larger than the image length and width, so as to make up for the positioning error of the UAV image and improve the success rate of subsequent registration of the orthographic map slice and the UAV image.

[0126] Specifically, the illegal building detection device obtains a first image size of the UAV image, determines a second image size of the orthographic map slice according to the first image size, wherein the second image size is slightly larger than the second image size. The illegal building detection device determines the positioning range of the orthographic map slice in the orthographic map according to the positioning information, and then cuts out the orthographic map slice from the orthographic map according to the second image size and the positioning range.

[0127] Step S52: Obtain a building roof mask and an illegal building mask on the UAV image.

[0128] In the embodiments of the application, the illegal building detection device obtains the building roof mask and the illegal building mask on the UAV image by the image recognition method in the above embodiments, and the specific process is not described here.

[0129] Step S53: Obtain a re-projection image of the ortho map slice re-projected to the coordinate system of the UAV image based on the building roof mask.

[0130] In the embodiments of the present application, as shown in Figure 9 The illegal building detection device calculates a first homography matrix between the ortho map slice and the UAV image according to the position of the building roof mask. Then, the illegal building detection device re-projects the ortho map slice onto the coordinate system of the UAV image according to the homography matrix to obtain a re-projection image.

[0131] The homography matrix is equivalent to the matrix used in perspective transformation. Perspective transformation describes the mapping relationship between two planes. The reason why it is called homography matrix is that the relationship between the two planes is determined, and only a unique matrix can be used to represent this transformation, so it is called homography matrix.

[0132] Step S54: Obtain a change detection area result of the target area based on the image difference information of the re-projection image and the UAV image.

[0133] Please continue to refer to Figure 9 In the embodiments of the present application, the illegal building detection device obtains the image or video key frame obtained by the UAV in the previous inspection and the ortho map slice after the homographic transformation as the input of the change detection module, and analyzes the changes in the corresponding area at two times.

[0134] In the embodiments of the present application, the illegal building detection device can obtain the image difference information between the re-projection image and the UAV image, such as pixel value difference, pixel value distribution and the like, so as to compare and obtain the change detection area result of the target area, that is, to analyze the change detection area of the target area.

[0135] Step S55: Obtain illegal building detection information of the UAV image by combining the change detection area result and the illegal building detection result of the illegal building mask.

[0136] In the embodiments of the present application, the illegal building detection device calculates the IoU (intersection over union) of the overlapping area by fusing the instance segmentation result of the illegal building, removes the area with large overlap rate, and obtains the final illegal building detection result to form the alarm information. The illegal building detection information includes the illegal building position, the illegal building time and the illegal building type, etc.

[0137] As shown in Figure 9As shown, on one hand, the illegal building detection device obtains the change detection result of the DOM slice and the UAV map through the twin change detection model, which is represented as a predicted change detection box. On the other hand, the illegal building detection device segments the roof and illegal building instance on the UAV image through the pre-trained target detection model to obtain the illegal building on the UAV image, which is represented as an illegal building mask. Further, the illegal building detection device fuses the predicted change detection box and the illegal building mask to post-process the change detection region result, thereby obtaining the final illegal building detection information.

[0138] Specifically, the illegal building detection device calculates the overlap rate of the predicted change detection box and the illegal building mask, removes the predicted change detection box or illegal building mask with an overlap rate greater than a preset threshold, and forms the illegal building detection information from the remaining predicted change detection box and illegal building mask.

[0139] The above embodiments are only one common case of the present application, and do not limit the technical scope of the present application. Therefore, any slight modification, equivalent change or modification of the above content according to the essence of the present application still belongs to the scope of the technical solution of the present application.

[0140] Please continue to see Figure 10 , Figure 10 FIG. 1 is a structural schematic diagram of an embodiment of a terminal device provided by the present application. The terminal device 600 of the embodiment of the present application includes a processor 61, a memory 62, an input / output device 63 and a bus 64.

[0141] The processor 61, the memory 62 and the input / output device 63 are respectively connected to the bus 64. The memory 62 stores program data, and the processor 61 is used to execute the program data to realize the image recognition method and / or the illegal building detection method described in the above embodiments.

[0142] In the embodiment of the present application, the processor 61 can also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 61 can be an integrated circuit chip with signal processing capability. The processor 61 can also be a general-purpose processor, a digital signal processor (DSP, Digital Signal Process), an application specific integrated circuit (ASIC, Application Specific Integrated Circuit), a field programmable gate array (FPGA, Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 61 can also be any conventional processor or the like.

[0143] The application also provides a computer storage medium, please continue to refer to Figure 11 , Figure 11 is a structural schematic diagram of an embodiment of the computer storage medium provided by the application. The computer storage medium 700 stores program data 71, which is used to implement the image recognition method and / or illegal building detection method of the above embodiment when executed by a processor.

[0144] When the embodiments of the application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in the various embodiments of the application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0145] The above is only an embodiment of the application, and does not limit the patent scope of the application. The equivalent structure or equivalent flow transformation made by the content of the specification and drawings of the application, or the direct or indirect application in other related technical fields, are also included in the patent protection scope of the application.

Claims

1. An image recognition method characterized by, The image recognition method comprises: acquiring a UAV image collected based on a target region; inputting the UAV image into a pre-trained instance segmentation network, wherein the instance segmentation network comprises a feature extraction model, a feature detection model, and a mask generation model; extracting a feature image of the UAV image by using the feature extraction model; generating a target building bounding box in the feature image by using the feature detection model; generating a mask of the target building on the feature image according to the target building bounding box by using the mask generation model; acquiring building information in the UAV image based on the mask output by the instance segmentation network; after the mask generation model generates the mask of the target building on the feature image according to the target building bounding box, the image recognition method further comprises: acquiring a building roof mask output by the instance segmentation network; generating a corresponding minimum bounding rectangle based on the building roof mask, and acquiring a to-be-regularized region in the minimum bounding rectangle that does not belong to the building roof mask; generating a maximum inscribed rectangle in the to-be-regularized region; cropping the minimum bounding rectangle according to the maximum inscribed rectangle to obtain a regularized building roof mask.

2. The image recognition method of claim 1, wherein the mask generation model comprises a first multilayer perceptron, a second multilayer perceptron, a deconvolution layer, and a convolution layer connected in sequence.

3. The image recognition method of claim 2, wherein the mask generation model further comprises a maximization processing layer and an average processing layer, the maximization processing layer and the average processing layer are arranged side by side between the second multilayer perceptron and the deconvolution layer, and the maximization processing layer and the average processing layer are used to remove hole information of the target building bounding box in the mask generation process.

4. The image recognition method of claim 1, wherein a loss function of the instance segmentation network at least comprises a mask loss, wherein the mask loss is calculated by using a sampling point loss function, and the sampling point loss function calculates the loss function of the instance segmentation network by using difference information between predicted mask output and actual mask output of a plurality of sampling points in the UAV image.

5. The image recognition method of claim 4, wherein the image recognition method further comprises: in the training process of the instance segmentation network, acquiring a training feature map of a training image; acquiring a confidence of all pixel points in the training feature map; selecting a plurality of sampling points in the training feature map according to the confidence of all pixel points; training the instance segmentation network by using difference information between predicted mask output and actual mask output of the plurality of sampling points.

6. The image recognition method of claim 5, wherein the selecting a plurality of sampling points in the training feature map according to the confidence of all pixel points comprises: acquiring a first confidence of all pixel points belonging to a foreground category and a second confidence of all pixel points belonging to a background category; ​ ​ ​ ​ ​ determine an uncertainty of the pixel point according to a sum of an absolute value of the first confidence and an absolute value of the second confidence; take, as the selected sampling points in the training feature map, the pixel points in the all pixel points whose uncertainty is greater than or equal to a preset threshold.

7. The image recognition method of claim 5, wherein the image recognition method further comprises: obtaining a training image and a building roof mask thereof in a training process of the instance segmentation network; extracting a illegal building mask from the training image or a illegal building database; attaching the illegal building mask to the building roof mask of the training image to form a new training image. The illegal building detection method comprises:

8. A method for detecting illegal construction, characterized by, obtaining a UAV image collected based on a target region, and obtaining an orthographic map slice corresponding to the UAV image; obtaining a building roof mask and a illegal building mask on the UAV image, wherein the mask obtaining manner is the mask obtaining manner in the image recognition method of any one of claims 1 to 7; obtaining a re-projection image of the orthographic map slice re-projected to a coordinate system of the UAV image based on the building roof mask; obtaining a change detection region result of the target region based on image difference information of the re-projection image and the UAV image; obtaining illegal building detection information of the UAV image in combination with the change detection region result and illegal building detection result of the illegal building mask. The terminal device comprises a memory and a processor coupled to the memory; 9. A terminal device, comprising: The memory is configured to store program data, and the processor is configured to execute the program data to implement the image recognition method of any one of claims 1 to 7 and / or the illegal building detection method of claim 8. The computer storage medium is configured to store program data, and the program data, when executed by a computer, is configured to implement the image recognition method of any one of claims 1 to 7 and / or the illegal building detection method of claim 8.

10. A computer storage medium, characterized in that, ​

Images