Factory target change detection method and device based on unmanned aerial vehicle image, and electronic equipment

Abstract

The invention relates to the technical field of intelligent image processing, and particularly provides a factory target change detection method and device based on an unmanned aerial vehicle image, and electronic equipment, and the method comprises the steps: obtaining two unmanned aerial vehicle images, shot by an unmanned aerial vehicle, of a factory in different periods; carrying out dodging processing on the two unmanned aerial vehicle images in different periods by using an improved Wallis dodging and color dodging algorithm; fusing the two unmanned aerial vehicle result images after dodging processing in different periods with the digital elevation model of the position of the factory in the corresponding period to obtain factory 2.5 D models in the two periods; calculating factory difference data according to the factory 2.5 D models of the two periods; and inputting the factory difference data into the trained factory target change detection model to obtain a factory target change result. According to the method, the problem of non-uniform illumination and contrast of the unmanned aerial vehicle image is solved, the quality of the unmanned aerial vehicle image is improved, the factory height information is detected by the factory target change detection model based on the 2.5 D model, and the factory target change detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent image processing technology, and in particular to a factory target change detection method, device and electronic equipment based on drone images. Background Art

[0002] With the development of drone technology, as well as the high resolution and portability of drone images, drone remote sensing technology has played an increasingly important role in various fields. Some factories have begun to use drone images for detection of illegal construction.

[0003] At present, most of the detection of illegal construction in factories based on drone images is still done by manually comparing drone images from different periods to find illegal construction, which has problems such as low efficiency and strong subjectivity. In recent years, a method of detecting illegal construction through image detection algorithms has emerged. Although this method improves the detection efficiency, the detection results are affected by the color and brightness differences of drone images, and the accuracy is difficult to guarantee. Color uniformity of drone images can eliminate the color and brightness differences of images to a certain extent, but there are many types of existing color uniformity algorithms with different effects. In addition, drone images can only detect illegal expansion and demolition on the horizontal surface of the factory, and it is difficult to detect changes in the height of the factory.

[0004] Therefore, how to design a color uniformity method suitable for UAV images to eliminate the color and brightness differences between UAV images to improve detection accuracy, and how to detect the height changes of factories to avoid missed detection, require further research. Summary of the invention

[0005] The present invention provides a factory target change detection method, device and electronic equipment based on drone images, which improves the traditional Wallis uniform light and color algorithm to solve the problems of uneven illumination and contrast of drone images, integrates elevation digital models with drone images, and realizes factory target change detection based on 2.5D models to improve detection accuracy.

[0006] In a first aspect, the present invention provides a method for detecting factory target changes based on drone images, comprising:

[0007] S1, obtain two drone images of the factory taken by drone at different times;

[0008] S2. Performing light balancing on two drone images taken at different times using an improved Wallis light balancing and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis light balancing and color balancing algorithm is used to perform overlapping segmentation of the two drone images taken at different times with a preset segmentation size to obtain all image blocks, and after performing light balancing on each image block, the preset segmentation size is changed until the light balancing is completed for a preset number of times;

[0009] S3, fusing the drone image results of two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods;

[0010] S4, calculating the factory difference data based on the factory 2.5D model of the two periods;

[0011] S5. Input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, heightened area and lowered area.

[0012] Optionally, the improved Wallis uniform light and color algorithm is used to process two drone images taken at different times to obtain two drone result images taken at different times, including:

[0013] S201, performing light index evaluation on a plurality of pre-collected drone images at different times to determine a template image with the highest quality score, and calculating the grayscale mean and standard deviation of the template image, wherein the light index evaluation is used to perform parameter index calculation on any drone image to obtain a quality score corresponding to the drone image, and the parameter index includes an image color parameter and an image brightness parameter;

[0014] S202, dividing two drone images taken at different times into blocks according to a preset segmentation size, setting overlapping areas between image blocks according to a preset ratio, and calculating the grayscale mean and standard deviation of each image block;

[0015] S203, using the Wallis light and color equalization algorithm to perform light equalization processing on each image block, as follows:

[0016]

[0017] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and mg is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0018] S204, stitching the image blocks after the homogenization processing, and smoothing the overlapping areas to obtain two drone result images of different periods;

[0019] S205, changing the preset segmentation size, and repeatedly executing steps S202-S204 to perform light homogenization on two drone result images of different periods obtained by the last light homogenization process, until the preset number of light homogenization processes is reached.

[0020] Furthermore, the parameter indicators include mean, standard deviation, average gradient, and information entropy.

[0021] Further, the preset number of homogenization processing times is 3 times, the preset segmentation size is 6×6 grids, the preset ratio is 1 / 3 to set the overlapping area, the preset segmentation size is changed, and steps S202-S204 are repeatedly performed to homogenize the two drone result images of different periods obtained by the previous homogenization processing until the preset number of homogenization processing times is reached, including:

[0022] Change the preset segmentation size from 6×6 grid to 12×12 grid, divide the drone image obtained by the first homogenization process into blocks, set the overlapping area between image blocks at a ratio of 1 / 3, calculate the grayscale mean and standard deviation of each image block, use the Wallis homogenization and color balancing algorithm to homogenize each image block, splice the homogenized image blocks, and smooth the overlapping area;

[0023] The preset segmentation size was changed from 12×12 grid to 36×36 grid, and the UAV image obtained by the second homogenization treatment was divided into blocks. The overlapping areas were set at a ratio of 1 / 3 between the image blocks, and the grayscale mean and standard deviation of each image block were calculated. The Wallis homogenization and color balancing algorithm was used to homogenize each image block, and the image blocks after homogenization were spliced, and the overlapping areas were smoothed.

[0024] Optionally, the two drone result images of different periods are fused with the digital elevation model of the factory location of the corresponding period to obtain the 2.5D model of the factory in the two periods, including:

[0025] S301, obtaining a digital elevation model of the factory location at a time period corresponding to two drone images at different times, and converting the coordinate systems of the two sets of digital elevation models at different times to the coordinate system of the drone images to obtain a converted digital elevation model;

[0026] S302, performing data sampling on the converted digital elevation model based on the position correspondence between the two drone images taken at different times and the corresponding converted digital elevation models, to obtain elevation values ​​corresponding to the two drone images taken at different times;

[0027] S303, superimposing the elevation values ​​corresponding to the two drone images taken at different periods onto the two drone result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

[0028] Optionally, the calculating of factory difference data according to the factory 2.5D model of the two periods includes:

[0029] S401, converting the 2.5D factory models of the two periods into raster data;

[0030] S402, superimposing and clipping the factory grid data of the two periods to obtain the valid factory grid data of the two periods;

[0031] S403, calculating the difference between the effective factory grid data of the next period and the effective factory grid data of the previous period to obtain factory difference data.

[0032] Optionally, the factory target change detection model is constructed based on a U-net network.

[0033] Optionally, before acquiring two drone images of the factory taken by the drone at different times, the method further includes:

[0034] Obtain multiple pre-collected drone images from different periods and the digital elevation model of the factory location at the corresponding period;

[0035] The coordinate system of the digital elevation models at different times is converted to the coordinate system of the drone image to obtain the converted digital elevation model;

[0036] Based on the position correspondence between the drone images of different periods and the corresponding converted digital elevation models, the converted digital elevation models are sampled to obtain the elevation values ​​corresponding to the drone images of different periods;

[0037] The elevation values ​​corresponding to the drone images of multiple different periods are superimposed on the drone images of the corresponding period to obtain the 2.5D model of the factory at multiple periods;

[0038] Calculate factory difference data based on factory 2.5D models of multiple periods, and classify and label the factory difference data to obtain a sample data set of factory target changes;

[0039] The factory target change detection model is trained using the factory target change sample data set to obtain the factory target change detection model.

[0040] In a second aspect, the present invention provides a factory target change detection device based on drone images, comprising:

[0041] An acquisition unit, used to acquire two drone images of the factory taken by the drone at different times;

[0042] A dodging processing unit is used to perform dodging processing on two drone images taken at different times using an improved Wallis dodging and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis dodging and color balancing algorithm is used to perform overlapping segmentation of the two drone images taken at different times with a preset segmentation size to obtain all image blocks, and after dodging processing is performed on each image block, the preset segmentation size is changed until dodging processing for a preset number of dodging processing times is completed;

[0043] The data fusion unit is used to fuse the drone image results of two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods;

[0044] A factory difference data calculation unit, used to calculate factory difference data based on a 2.5D factory model of two periods;

[0045] The change detection unit is used to input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, heightened area and lowered area.

[0046] Optionally, the light homogenization processing unit is specifically used to:

[0047] Performing a light index evaluation on multiple drone images collected in different periods in advance to determine a template image with the highest quality score, and calculating the grayscale mean and standard deviation of the template image, wherein the light index evaluation is used to calculate parameter indicators for any drone image to obtain the quality score corresponding to the drone image, and the parameter indicators include image color parameters and image brightness parameters;

[0048] The two drone images taken at different times are divided into blocks according to the preset segmentation size, the overlapping areas between the image blocks are set according to the preset ratio, and the grayscale mean and standard deviation of each image block are calculated;

[0049] The Wallis light and color uniformity algorithm is used to perform light uniformity processing on each image block, as follows:

[0050]

[0051] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0052] The image blocks after uniform light treatment are spliced ​​and the overlapping areas are smoothed to obtain two drone images taken at different times.

[0053] Change the preset segmentation size and repeat the previous two steps to perform homogenization on the two drone result images obtained from the last homogenization process at different times until the preset number of homogenization processes is reached.

[0054] Optionally, the data fusion unit is specifically used to:

[0055] Obtain the digital elevation model of the factory location at the time period corresponding to the two drone images at different time periods, and convert the coordinate systems of the two sets of digital elevation models at different time periods to the coordinate system of the drone images to obtain the converted digital elevation model;

[0056] Based on the position correspondence between the two UAV images taken at different times and the corresponding converted digital elevation models, the converted digital elevation model is sampled to obtain the elevation values ​​corresponding to the two UAV images taken at different times.

[0057] The elevation values ​​corresponding to the two UAV images taken at different periods are superimposed on the two UAV result images taken at different periods to obtain the 2.5D models of the factory in the two periods.

[0058] Optionally, the factory difference data calculation unit is specifically used to:

[0059] Convert the 2.5D factory models of the two periods into raster data;

[0060] The factory raster data of the two periods are superimposed and clipped to obtain the effective factory raster data of the two periods;

[0061] The difference between the effective factory grid data of the latter period and the effective factory grid data of the previous period is calculated to obtain the factory difference data.

[0062] Optionally, the device further comprises:

[0063] The training unit is used to classify and label the factory difference data to obtain a factory target change sample data set; the factory target change detection model is trained using the factory target change sample data set to obtain the factory target change detection model. In a third aspect, the present invention provides an electronic device, including: at least one processor and a memory;

[0064] The memory stores computer-executable instructions;

[0065] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the factory target change detection method described in any one of the first aspects.

[0066] The present invention provides a method, device and electronic equipment for detecting factory target changes based on drone images. Compared with the prior art, the scheme of the present invention improves the traditional Wallis homogenization and color balancing algorithm to perform homogenization processing on drone images, which not only solves the color and brightness difference problems between drone images, but also eliminates the brightness inconsistency problem between pixels inside drone images through homogenization processing after multiple overlapping blocks, thereby improving the quality of drone images, thereby improving the accuracy of factory target change detection, and fusing the elevation digital model with the drone images after homogenization processing to obtain a factory 2.5D model. Based on the factory 2.5D model, a factory target change detection model is trained to detect factory changes, which can not only detect illegal expansion and demolition of the factory horizontal surface, but also detect the height change of the factory, thereby avoiding missed detection of factory height changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0068] Figure 1 A schematic diagram of the scenario architecture provided by the present invention;

[0069] Figure 2A schematic diagram of a flow chart of a factory target change detection method based on drone images provided by the present invention;

[0070] Figure 3 This is a schematic diagram of the UAV image segmentation;

[0071] Figure 4 A schematic diagram of the structure of a factory target change detection device based on drone images provided by the present invention;

[0072] Figure 5 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Figure 1 A schematic diagram of the scene architecture provided by the present invention, such as Figure 1 As shown, a scenario architecture based on which the present disclosure is based may include a factory target change detection device 1 and a drone 2.

[0075] Among them, the factory target change detection device 1 can be hardware, software or a combination of software and hardware, which can be used to execute the factory target change detection method based on drone images described in the following embodiments.

[0076] When the plant target change detection device 1 is hardware, it can be an electronic device with a computing function. When the plant target change detection device 1 is software, it can be installed in an electronic device with a computing function. The electronic device includes but is not limited to a server, a notebook, a desktop computer, and the like.

[0077] The factory target change detection device 1 can be a server integrated on the drone 2 and provide factory target change detection services for the drone 2. At the same time, the factory target change detection device 1 can also use the shooting function of the drone 2 to obtain factory images taken by the drone 2.

[0078] Of course, in other usage scenarios, the factory target change detection device 1 can also be integrated into a server for processing drone images, such as an image processing server. At this time, the drone 2 can communicate and exchange data with the aforementioned factory target change detection device 1 through the network, and the drone 2 can transmit the drone images to be processed to the factory target change detection device 1, so that the factory target change detection device 1 processes the drone images in the manner shown below to obtain the factory target change detection results.

[0079] The following is a further description of the factory target change detection method, device and electronic equipment based on drone images provided by this application:

[0080] Figure 2 A schematic diagram of a process flow of factory target change detection based on drone images provided by an embodiment of the present disclosure. Figure 2 As shown, a method for detecting factory target changes based on drone images provided by an embodiment of the present disclosure includes:

[0081] S1, obtain two drone images of the factory taken by drone at different times;

[0082] In this embodiment, in order to detect whether there is any illegal construction in the factory before and after two periods, a drone equipped with a high-definition camera takes drone images of the factory from above the factory at two different periods.

[0083] S2, using the improved Wallis light and color uniformity algorithm to perform light uniformity processing on two drone images taken at different times, and obtain two drone result images taken at different times;

[0084] Since drone photography is affected by different shooting environments (such as solar altitude angle, clouds, etc.), terrain undulations (such as mountains, plains, etc.), camera characteristics (such as exposure, white balance, etc.), aerial photography scale and other factors, it is easy for the images to have uneven colors and inconsistent brightness between and within the images. If these differences are not processed, the images will have obvious color and brightness differences, which will bring corresponding difficulties to the subsequent image processing work. The uniform light and color algorithm between drone images is used to solve this problem. According to the processing ideas, it is mainly divided into three categories: uniform light and color algorithm based on histogram, uniform light and color algorithm based on statistical method and uniform light and color algorithm based on global. The traditional Wallis uniform light and color algorithm is a uniform light and color algorithm based on statistical method. The mean and variance of the template image are used as the standard. By transforming the grayscale value of the image, its mean and variance are approximate to the mean and variance of the template image, thereby realizing the image uniformity processing. However, the traditional Wallis uniformity algorithm is based on single pixel processing, so the brightness of different pixels will be inconsistent after uniformity, and a dividing line will appear. Therefore, based on the traditional Wallis uniformity algorithm, the drone image is segmented for multiple times with different preset segmentation sizes and the image block overlapping area is set. The preset segmentation size and the proportion of the overlapping area can be determined according to the size of the drone image, and then the uniformity processing is performed to solve the color and brightness difference problems between drone images and the brightness inconsistency problem between pixels inside drone images, thereby improving the overall quality of drone images.

[0085] Specifically, 100 pre-collected drone images from different periods were evaluated for light indexes to determine a template image with the highest quality score. Light index evaluation is a method of evaluating image quality by calculating parameter indicators that can reflect image color and brightness. Since the mean can reflect image brightness changes, the standard deviation can reflect the difference between image pixels, the average gradient can reflect the characteristics of tiny detail contrast and texture changes in the image, and also express the clarity of the image, and the information entropy can reflect the information richness of the image, the parameter indicators used are: mean, standard deviation, average gradient, and information entropy. The 100 drone images were scored using the evaluation indicators, and the drone image with the highest score was determined as the template image. The grayscale mean and standard deviation of the template image were calculated.

[0086] The two drone images from different periods are divided into 6×6 grids, and the overlapping areas between the image blocks are set at a ratio of 1 / 3. The overlapping modes include left-right overlap and top-bottom overlap, such as Figure 3As shown, the 6×6 red boundary image block and the 6×6 yellow boundary image block overlap left and right, and the 2×6 grid area where the two overlap represents the overlapping area. The 6×6 red boundary image block and the 6×6 blue boundary image block overlap up and down, and the 6×2 grid area where the two overlap represents the overlapping area. The grayscale mean and standard deviation of each image block are calculated;

[0087] The Wallis light and color uniformity algorithm is used to perform light uniformity processing on each image block, as follows:

[0088]

[0089] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0090] The image blocks after the homogenization processing are stitched together, and the overlapping areas are smoothed. The smoothing algorithm can be determined by comparing a variety of different smoothing algorithms through experiments to determine the optimal algorithm, and obtain two drone result images from different periods;

[0091] The drone image obtained by the first homogenization process is divided into blocks according to a 12×12 grid, and the overlapping areas are set at a ratio of 1 / 3 between the image blocks. The grayscale mean and standard deviation of each image block are calculated, and the Wallis homogenization and color homogenization algorithm is used to homogenize each image block. The image blocks after homogenization are spliced, and the overlapping areas are smoothed.

[0092] The drone image obtained by the second homogenization process is divided into 36×36 grids, and the overlapping areas are set at a ratio of 1 / 3 between the image blocks. The grayscale mean and standard deviation of each image block are calculated, and the Wallis homogenization and color homogenization algorithm is used to homogenize each image block. The image blocks after homogenization are spliced, and the overlapping areas are smoothed.

[0093] The original UAV images of the same period, the UAV images processed by the traditional Wallis algorithm, and the UAV images processed by the algorithm in this paper were respectively calculated to reflect the parameter indicators of image quality, and then the parameter indicators were quantitatively analyzed to determine the image quality. Since the mean can reflect the change of image brightness, the standard deviation can reflect the difference between image pixels, the average gradient can reflect the contrast of tiny details and texture change characteristics in the image, and also express the clarity of the image, the information entropy can reflect the information richness of the image, so the parameter indicators used are: mean, standard deviation, average gradient, and information entropy.

[0094] The results show that the mean, standard deviation, average gradient and information entropy of the UAV images obtained after being processed by the algorithm in this paper are ranked in the top two, indicating that the UAV images obtained after the third homogenization treatment have good color preservation, rich image information, good details, clear images, relatively concentrated image grayscale distribution, and small differences between pixels.

[0095] S3, fusing the drone image results of two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods;

[0096] Since drone images are two-dimensional and illegal factory construction may also lead to height increases, drone images can only detect illegal expansion and demolition on the horizontal plane of the factory, and it is difficult to detect changes in the height of the factory. Therefore, it is necessary to further add height information to the drone images to construct a 2.5D model of the factory. Compared with a three-dimensional model, this requires less calculation and faster processing speed.

[0097] Specifically, the digital elevation model of the factory location corresponding to the two drone images at different periods is obtained, and the coordinate system of the two sets of digital elevation models at different periods is converted to the coordinate system of the drone result image. The digital elevation model and the drone result image are unified into the same coordinate system, and the height data can be integrated into the drone result image to obtain the converted digital elevation model.

[0098] The two drone images taken at different times are registered with the corresponding converted digital elevation models to determine the position correspondence between the two images. According to the position correspondence between the two drone images taken at different times and the corresponding converted digital elevation models, the elevation values ​​are extracted from the converted digital elevation models to obtain the elevation values ​​corresponding to the two drone images taken at different times.

[0099] The elevation values ​​corresponding to the two UAV images taken at different periods are superimposed on the two UAV result images taken at different periods to obtain the 2.5D models of the factory in the two periods.

[0100] S4, calculating the factory difference data based on the factory 2.5D model of the two periods;

[0101] Since the processing volume of detecting the factory 2.5D model data of both periods is large, and the difference of the factory 2.5D model data of the two periods is the factory change information, the factory change can be determined by calculating the difference of the factory 2.5D model data of the two periods.

[0102] Specifically, the 2.5D factory models of the two periods are converted into raster data; the factory raster data of the two periods are superimposed to determine the factory area, and the factory raster data of the two periods are clipped according to the factory area to obtain the valid factory raster data of the two periods; the difference between the valid factory raster data of the latter period and the valid factory raster data of the previous period is calculated to obtain the factory difference data.

[0103] S5, inputting the factory difference data into the pre-trained factory target change detection model to obtain the factory target change result;

[0104] In this embodiment, before the factory target change detection model is used, it is necessary to first construct a factory target change sample data set and a factory target change detection model, and train the factory target change detection model.

[0105] Specifically, 100,000 pre-collected drone images from different periods and digital elevation models of the factory locations at the corresponding periods were obtained;

[0106] The coordinate system of the digital elevation models at different times is converted to the coordinate system of the drone image to obtain the converted digital elevation model;

[0107] Based on the position correspondence between 100,000 drone images from different periods and the corresponding converted digital elevation models, the converted digital elevation models are sampled to obtain the elevation values ​​corresponding to the 100,000 drone images from different periods.

[0108] The elevation values ​​corresponding to 100,000 drone images from different periods were superimposed on the drone images from the corresponding period to obtain 100,000 2.5D models of the factory from different periods;

[0109] Calculate factory difference data based on 100,000 factory 2.5D models at different times, and classify and label the factory difference data. The labeling information includes unchanged, expanded, demolished, raised, and lowered, and obtain a sample data set of factory target changes.

[0110] The factory target change detection model is built based on the U-net network. The U-net network consists of two parts, the encoder on the left and the decoder on the right. The encoder follows the typical convolutional network architecture, and the decoder is the deconvolution of the encoder's corresponding level. The input matrix size of the designed model is 512×512×6, and the output matrix size is 512×512×3. The input data is the factory target change sample data, and the output image of the model is the labeled image matrix 512×512×3. After the model is built, each sample is first forward propagated. The formula for the forward propagation operation is as follows:

[0111] net (l+1) =W l ·x l +b l

[0112] x (l+1) =f(net (l+1) )

[0113] Among them, net (l+1) is the weighted sum of the inputs of the l+1th layer, W l is the connection weight between layer l and layer l+1, x l is the node value of the lth layer, b l is the bias term of the lth layer, and f(·) is the activation function;

[0114] Using the above forward propagation formula, we can get the node values ​​of the second layer, the third layer and even the output layer. In order to find the parameters W and b that can minimize the cost function L(W,b), we first calculate the residual of the output layer:

[0115]

[0116] Among them, net nl is the weighted sum of the inputs to the output layer;

[0117] For l = n l -1,n l -2,n l For each layer of -3,...,2, their residuals are calculated according to the following formula:

[0118] δ (l) =((W (l) ) T δ (l+1) )·f(net (l) )

[0119] Then, calculate the partial derivative of the cost function for a single sample:

[0120]

[0121] Then calculate the sum of the partial derivatives of the cost function for all samples:

[0122]

[0123] Finally update the weight parameters:

[0124]

[0125] Among them, α is the learning rate and λ is the weight decay parameter.

[0126] These iterative steps are repeated to reduce the value of the cost function L(W, b) until the training of the factory target change detection model is completed.

[0127] Corresponding to the factory target change detection method based on drone images in the above embodiment, Figure 4 A schematic diagram of the structure of a factory target change detection device based on drone images provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 , the factory target change detection device based on drone images includes:

[0128] An acquisition unit 41 is used to acquire two drone images of the factory taken by the drone at different times;

[0129] The dodging processing unit 42 is used to perform dodging processing on two drone images of different periods using an improved Wallis dodging and color balancing algorithm to obtain two drone result images of different periods, wherein the improved Wallis dodging and color balancing algorithm is used to perform overlapping segmentation of the two drone images of different periods with a preset segmentation size to obtain all image blocks, and after dodging processing is performed on each image block, the preset segmentation size is changed until the dodging processing of the preset number of dodging processing times is completed;

[0130] The data fusion unit 43 is used to fuse the drone result images of two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods;

[0131] A factory difference data calculation unit 44 is used to calculate the factory difference data according to the factory 2.5D model of two periods;

[0132] The change detection unit 45 is used to input the factory difference data into a pre-trained factory target change detection model to obtain a factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, heightened area and lowered area.

[0133] Optionally, the light homogenization processing unit 42 is specifically used for:

[0134] The light index evaluation of multiple drone images collected in different periods is performed to determine a template image with the highest score, and the grayscale mean and standard deviation of the template image are calculated. The light index evaluation is a method of evaluating image quality by calculating parameter indicators that can reflect the color and brightness of the image.

[0135] The two drone images taken at different times are divided into blocks according to the preset segmentation size, the overlapping areas between the image blocks are set according to the preset ratio, and the grayscale mean and standard deviation of each image block are calculated;

[0136] The Wallis light and color uniformity algorithm is used to perform light uniformity processing on each image block, as follows:

[0137]

[0138] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0139] The image blocks after uniform light treatment are spliced ​​and the overlapping areas are smoothed to obtain two drone images taken at different times.

[0140] Change the preset segmentation size and repeat the previous two steps to perform homogenization on the two drone result images obtained from the last homogenization process at different times until the preset number of homogenization processes is reached.

[0141] Optionally, the data fusion unit 43 is specifically configured to:

[0142] Obtain the digital elevation model of the factory location at the time period corresponding to the two drone images at different times, and convert the coordinate systems of the two sets of digital elevation models at different times to the coordinate system of the drone result image to obtain the converted digital elevation model;

[0143] Based on the position correspondence between the two UAV images taken at different times and the corresponding converted digital elevation models, the converted digital elevation model is sampled to obtain the elevation values ​​corresponding to the two UAV images taken at different times.

[0144] The elevation values ​​corresponding to the two UAV images taken at different periods are superimposed on the two UAV result images taken at different periods to obtain the 2.5D models of the factory in the two periods.

[0145] Optionally, the factory difference data calculation unit 44 is specifically used to:

[0146] Convert the 2.5D factory models of the two periods into raster data;

[0147] The factory raster data of the two periods are superimposed and clipped to obtain the effective factory raster data of the two periods;

[0148] The difference between the effective factory grid data of the latter period and the effective factory grid data of the previous period is calculated to obtain the factory difference data.

[0149] Optionally, the device further comprises:

[0150] The training unit 46 is used to classify and label the factory difference data to obtain a factory target change sample data set; and use the factory target change sample data set to train the factory target change detection model to obtain the factory target change detection model.

[0151] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the electronic device 50 of this embodiment may include: a memory 51 and a processor 52.

[0152] A memory 51, used to store computer programs (such as application programs, functional modules, etc. for implementing the above-mentioned factory target change detection method based on drone images), computer instructions, etc.;

[0153] The above-mentioned computer programs, computer instructions, etc. may be stored in partitions in one or more memories 51 . And the above-mentioned computer programs, computer instructions, data, etc. may be called by the processor 52 .

[0154] The processor 52 is used to execute the computer program stored in the memory 51 to implement the various steps in the method involved in the above embodiment.

[0155] For details, please refer to the relevant description in the previous method embodiment.

[0156] The memory 51 and the processor 52 may be independent structures or integrated structures. When the memory 51 and the processor 52 are independent structures, the memory 51 and the processor 52 may be coupled and connected via a bus 53 .

[0157] An electronic device of this embodiment can execute Figure 2 For the technical solution in the method shown, its specific implementation process and technical principles can be found in Figure 2 The relevant descriptions in the method shown will not be repeated here.

[0158] Those skilled in the art will appreciate that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed.

[0159] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely the embodiment forms of implementing the claims.

Claims

1. A factory target change detection method based on drone images, characterized in that: include: S1, obtain two drone images of the factory taken by drone at different times; S2. Performing light balancing on two drone images taken at different times using an improved Wallis light balancing and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis light balancing and color balancing algorithm is used to perform overlapping segmentation of the two drone images taken at different times with a preset segmentation size to obtain all image blocks, and after performing light balancing on each image block, the preset segmentation size is changed until the light balancing is completed for a preset number of times; S3, fusing the drone image results of two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods; S4, calculating the factory difference data based on the factory 2.5D model of the two periods; S5. Input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, heightened area and lowered area.

2. The method for detecting factory target changes based on drone images according to claim 1 is characterized in that: The improved Wallis uniform light and color algorithm is used to process two drone images taken at different times to obtain two drone result images taken at different times, including: S201, performing light index evaluation on a plurality of pre-collected drone images at different times to determine a template image with the highest quality score, and calculating the grayscale mean and standard deviation of the template image, wherein the light index evaluation is used to perform parameter index calculation on any drone image to obtain a quality score corresponding to the drone image, and the parameter index includes an image color parameter and an image brightness parameter; S202, dividing two drone images taken at different times into blocks according to a preset segmentation size, setting overlapping areas between image blocks according to a preset ratio, and calculating the grayscale mean and standard deviation of each image block; S203, using the Wallis light and color equalization algorithm to perform light equalization processing on each image block, as follows: Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image; S204, stitching the image blocks after the homogenization processing, and smoothing the overlapping areas to obtain two drone result images of different periods; S205, changing the preset segmentation size, and repeatedly executing steps S202-S204 to perform light homogenization on two drone result images of different periods obtained by the last light homogenization process, until the preset number of light homogenization processes is reached.

3. The method for detecting factory target changes based on drone images according to claim 2 is characterized in that: The parameter indicators include mean, standard deviation, average gradient, and information entropy.

4. The method for detecting factory target changes based on drone images according to claim 2 is characterized in that: The preset number of homogenization processing times is 3 times, the preset segmentation size is 6×6 grids, the preset ratio is 1 / 3 to set the overlapping area, the preset segmentation size is changed, and steps S202-S204 are repeatedly performed to homogenize the two drone result images of different periods obtained by the previous homogenization processing until the preset number of homogenization processing times is reached, including: Change the preset segmentation size from 6×6 grid to 12×12 grid, divide the drone image obtained by the first homogenization process into blocks, set the overlapping area between image blocks at a ratio of 1 / 3, calculate the grayscale mean and standard deviation of each image block, use the Wallis homogenization and color balancing algorithm to homogenize each image block, splice the homogenized image blocks, and smooth the overlapping area; The preset segmentation size was changed from 12×12 grid to 36×36 grid, and the UAV image obtained by the second homogenization treatment was divided into blocks. The overlapping areas were set at a ratio of 1 / 3 between the image blocks, and the grayscale mean and standard deviation of each image block were calculated. The Wallis homogenization and color balancing algorithm was used to homogenize each image block, and the image blocks after homogenization were spliced, and the overlapping areas were smoothed.

5. The method for detecting factory target changes based on drone images according to claim 1 is characterized in that: The two drone images taken at different times are fused with the digital elevation model of the factory location at the corresponding time to obtain a 2.5D model of the factory at the two times, including: S301, obtaining a digital elevation model of the factory location at a time period corresponding to two drone images at different times, and converting the coordinate systems of the two sets of digital elevation models at different times to the coordinate system of the drone images to obtain a converted digital elevation model; S302, performing data sampling on the converted digital elevation model based on the position correspondence between the two drone images taken at different times and the corresponding converted digital elevation models, to obtain elevation values ​​corresponding to the two drone images taken at different times; S303, superimposing the elevation values ​​corresponding to the two drone images taken at different periods onto the two drone result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

6. The method for detecting factory target changes based on drone images according to claim 1 is characterized in that: The factory difference data is calculated based on the factory 2.5D model of the two periods, including: S401, converting the 2.5D factory models of the two periods into raster data; S402, superimposing and clipping the factory grid data of the two periods to obtain the valid factory grid data of the two periods; S403, calculating the difference between the effective factory grid data of the next period and the effective factory grid data of the previous period to obtain factory difference data.

7. The method for detecting factory target changes based on drone images according to claim 1 is characterized in that: The factory target change detection model is constructed based on the U-net network.

8. The method for detecting factory target changes based on drone images according to any one of claims 1 to 7, characterized in that: Before obtaining two drone images of the factory taken by the drone at different times, the method further includes: Obtain multiple pre-collected drone images from different periods and the digital elevation model of the factory location at the corresponding period; The coordinate system of the digital elevation models at different times is converted to the coordinate system of the drone image to obtain the converted digital elevation model; Based on the position correspondence between the drone images of different periods and the corresponding converted digital elevation models, the converted digital elevation models are sampled to obtain the elevation values ​​corresponding to the drone images of different periods; The elevation values ​​corresponding to the drone images of multiple different periods are superimposed on the drone images of the corresponding period to obtain the 2.5D model of the factory at multiple periods; Calculate factory difference data based on factory 2.5D models of multiple periods, and classify and label the factory difference data to obtain a sample data set of factory target changes; The factory target change detection model is trained using the factory target change sample data set to obtain the factory target change detection model.

9. A factory target change detection device based on drone images, characterized in that: include: An acquisition unit, used to acquire two drone images of the factory taken by the drone at different times; A dodging processing unit is used to perform dodging processing on two drone images taken at different times using an improved Wallis dodging and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis dodging and color balancing algorithm is used to perform overlapping segmentation of the two drone images taken at different times with a preset segmentation size to obtain all image blocks, and after dodging processing is performed on each image block, the preset segmentation size is changed until dodging processing for a preset number of dodging processing times is completed; The data fusion unit is used to fuse the drone image results of two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods; A factory difference data calculation unit, used to calculate factory difference data based on a 2.5D factory model of two periods; The change detection unit is used to input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, heightened area and lowered area.

10. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 8.

Images