Factory target change detection method, device and electronic equipment based on drone images

Abstract

The present invention relates to the field of intelligent image processing technology, and specifically provides a method, device, and electronic device for detecting factory target changes based on drone images. The method comprises: obtaining two drone images of a factory taken at different times by a drone; performing light grading on the two drone images at different times using an improved Wallis light grading and color grading algorithm; fusing the two drone images from different times after light grading with a digital elevation model of the factory location at the corresponding time to obtain 2.5D models of the factory at the two times; calculating factory difference data based on the 2.5D models of the factory at the two times; and inputting the factory difference data into a trained factory target change detection model to obtain factory target change results. The present invention solves the problems of uneven illumination and contrast in drone images, improves the quality of drone images, and implements detection of factory height information by a factory target change detection model based on a 2.5D model, thereby improving the accuracy of factory target change detection.

Description

Technical Field

[0001] The present invention relates to the field of intelligent image processing technology, and in particular to a factory target change detection method, device and electronic equipment based on drone images. Background Art

[0002] With the development of drone technology and the high resolution and portability of drone images, drone remote sensing technology has played an increasingly important role in various fields. Some factories have begun to use drone images for detection of illegal structures.

[0003] At present, most of the detection of illegal factory structures based on drone images is still done by manually comparing drone images from different periods to detect illegal structures. This has problems such as low efficiency and strong subjectivity. In recent years, a method of detecting illegal structures through image detection algorithms has emerged. Although this method improves detection efficiency, the detection results are affected by the color and brightness differences of drone images, and the accuracy is difficult to guarantee. Color uniformity of drone images can eliminate the color and brightness differences of images to a certain extent, but there are many types of existing color uniformity algorithms with different effects. In addition, drone images can only detect illegal expansion and demolition on the horizontal surface of the factory, and it is difficult to detect changes in the height of the factory.

[0004] Therefore, further research is needed on how to design a color uniformity method suitable for UAV images to eliminate the color and brightness differences between UAV images to improve detection accuracy, and how to detect the height changes of factories to avoid missed detections. Summary of the Invention

[0005] The present invention provides a method, device, and electronic equipment for detecting factory target changes based on drone images. The method improves on the traditional Wallis uniform illumination and color algorithm to solve the problems of uneven illumination and contrast in drone images. By integrating an elevation digital model with the drone images and based on a 2.5D model, the method realizes factory target change detection and improves detection accuracy.

[0006] In a first aspect, the present invention provides a method for detecting factory target changes based on drone images, comprising:

[0007] S1. Obtain two drone images of the factory taken at different times;

[0008] S2. Performing light balancing on the two drone images taken at different times using an improved Wallis light balancing and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis light balancing and color balancing algorithm is used to perform overlapping segmentation of the two drone images taken at different times at a preset segmentation size to obtain all image blocks, and after performing light balancing on each image block, the preset segmentation size is changed until a preset number of light balancing processes are completed;

[0009] S3, fusing the two drone images from different periods with the digital elevation models of the factory locations at the corresponding periods to obtain 2.5D models of the factory at the two periods;

[0010] S4. Calculate factory difference data based on the factory 2.5D model of the two periods;

[0011] S5. Input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, raised area and lowered area.

[0012] Optionally, the improved Wallis uniform light and color algorithm is used to process two drone images taken at different times to obtain two drone result images taken at different times, including:

[0013] S201, performing a light index evaluation on multiple pre-collected drone images from different time periods to determine a template image with the highest quality score, and calculating the grayscale mean and standard deviation of the template image, wherein the light index evaluation is used to calculate parameter indicators for any drone image to obtain a quality score corresponding to the drone image, wherein the parameter indicators include image color parameters and image brightness parameters;

[0014] S202, dividing the two drone images taken at different times into blocks according to a preset segmentation size, setting overlapping areas between the image blocks according to a preset ratio, and calculating the grayscale mean and standard deviation of each image block;

[0015] S203: Perform light balancing on each image block using the Wallis light balancing and color balancing algorithm, as follows:

[0016]

[0017] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and mg is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0018] S204: stitching the image blocks after the light uniforming process and smoothing the overlapping areas to obtain two drone images taken at different times;

[0019] S205 , changing the preset segmentation size, and repeating steps S202 - S204 to perform light dodging on the two drone result images obtained in the last light dodging process at different times, until the preset light dodging process times are reached.

[0020] Furthermore, the parameter indicators include mean, standard deviation, average gradient, and information entropy.

[0021] Furthermore, the preset number of dodging processes is 3, the preset segmentation size is a 6×6 grid, the preset ratio is 1 / 3 to set the overlapping area, and the preset segmentation size is changed, and steps S202-S204 are repeatedly performed to perform dodging processes on two drone result images obtained in different periods of time from the previous dodging process until the preset number of dodging processes is reached, including:

[0022] The preset segmentation size was changed from a 6×6 grid to a 12×12 grid. The drone imagery obtained from the first dodging process was divided into blocks. The overlapping areas between the image blocks were set at a ratio of 1 / 3. The grayscale mean and standard deviation of each image block were calculated. Each image block was dodged using the Wallis dodging algorithm. The dodging-processed image blocks were then spliced ​​and the overlapping areas were smoothed.

[0023] The preset segmentation size was changed from a 12×12 grid to a 36×36 grid. The drone image obtained by the second light-dodging process was divided into blocks. The overlapping areas between the image blocks were set at a ratio of 1 / 3. The grayscale mean and standard deviation of each image block were calculated. The Wallis light-dodging and color-dodging algorithm was used to dovetail each image block. The image blocks after light-dodging were spliced, and the overlapping areas were smoothed.

[0024] Optionally, the two drone images taken at different times are fused with the digital elevation model of the factory location at the corresponding time to obtain a 2.5D model of the factory at the two time periods, including:

[0025] S301, obtaining a digital elevation model of the factory location at a time period corresponding to two sets of drone images taken at different time periods, and converting the coordinate systems of the two sets of digital elevation models at different time periods to the coordinate system of the drone images to obtain a converted digital elevation model;

[0026] S302: performing data sampling on the converted digital elevation model based on the positional correspondence between the two drone images taken at different times and the corresponding converted digital elevation models to obtain elevation values ​​corresponding to the two drone images taken at different times;

[0027] S303 , superimposing the elevation values ​​corresponding to the two drone images taken at different periods onto the two drone result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

[0028] Optionally, the calculating of factory difference data based on the factory 2.5D models of the two periods includes:

[0029] S401, converting the 2.5D factory models of the two periods into raster data;

[0030] S402, superimposing and clipping the factory raster data of the two periods to obtain valid factory raster data of the two periods;

[0031] S403 , calculating the difference between the valid factory grid data of the next period and the valid factory grid data of the previous period to obtain factory difference data.

[0032] Optionally, the factory target change detection model is constructed based on a U-net network.

[0033] Optionally, before obtaining two drone images of the factory taken by the drone at different times, the method further includes:

[0034] Obtain multiple pre-collected drone images from different periods and the digital elevation model of the factory location at the corresponding period;

[0035] The coordinate systems of the digital elevation models at different times are converted to the coordinate system of the UAV images to obtain the converted digital elevation models;

[0036] Based on the position correspondence between the UAV images of different periods and the corresponding converted digital elevation models, the converted digital elevation models are sampled to obtain the elevation values ​​corresponding to the UAV images of different periods.

[0037] The elevation values ​​corresponding to the drone images from different periods were superimposed onto the drone images from the corresponding period to obtain the 2.5D model of the factory at multiple periods;

[0038] Calculate factory difference data based on the factory 2.5D model of multiple periods, and classify and label the factory difference data to obtain the factory target change sample data set;

[0039] The factory target change detection model is trained using the factory target change sample dataset to obtain the factory target change detection model.

[0040] In a second aspect, the present invention provides a factory target change detection device based on drone images, comprising:

[0041] an acquisition unit, used to acquire two drone images of the factory taken at different times;

[0042] a dodging processing unit, configured to perform dodging processing on two drone images taken at different times using an improved Wallis dodging and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis dodging and color balancing algorithm is configured to perform overlapping segmentation of the two drone images taken at different times at a preset segmentation size to obtain all image blocks, and after dodging processing is performed on each image block, the preset segmentation size is changed until the dodging processing is completed for a preset number of times;

[0043] The data fusion unit is used to fuse the drone images from two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods;

[0044] A factory difference data calculation unit is used to calculate factory difference data based on the factory 2.5D model of two periods;

[0045] The change detection unit is used to input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, raised area and lowered area.

[0046] Optionally, the light uniformity processing unit is specifically configured to:

[0047] Perform a light index evaluation on multiple pre-collected drone images from different periods to determine a template image with the highest quality score, and calculate the grayscale mean and standard deviation of the template image. The light index evaluation is used to calculate parameter indicators for any drone image to obtain the corresponding quality score of the drone image. The parameter indicators include image color parameters and image brightness parameters.

[0048] Two drone images taken at different times are divided into blocks according to a preset segmentation size. The overlapping areas between the image blocks are set according to a preset ratio. The grayscale mean and standard deviation of each image block are calculated.

[0049] The Wallis light and color uniformity algorithm is used to perform light uniformity processing on each image block, as follows:

[0050]

[0051] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0052] The image blocks after uniform light treatment are stitched together and the overlapping areas are smoothed to obtain two drone images taken at different times.

[0053] Change the preset segmentation size and repeat the previous two steps to perform light dodging on the two drone images obtained from the previous light dodging process at different times until the preset number of light dodging processes is reached.

[0054] Optionally, the data fusion unit is specifically configured to:

[0055] Obtain the digital elevation model of the factory location at the time period corresponding to the two drone images taken at different time periods, and convert the coordinate systems of the two sets of digital elevation models at different time periods to the coordinate system of the drone images to obtain the converted digital elevation model;

[0056] Based on the position correspondence between the two UAV images taken at different times and the corresponding converted digital elevation models, the converted digital elevation model is sampled to obtain the elevation values ​​corresponding to the two UAV images taken at different times.

[0057] The elevation values ​​corresponding to the two UAV images taken at different periods were superimposed on the two UAV result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

[0058] Optionally, the factory difference data calculation unit is specifically used to:

[0059] Convert the 2.5D factory models of the two periods into raster data;

[0060] The factory raster data of the two periods are superimposed and clipped to obtain the effective factory raster data of the two periods;

[0061] The difference between the effective factory grid data of the latter period and the effective factory grid data of the previous period is calculated to obtain the factory difference data.

[0062] Optionally, the device further includes:

[0063] The training unit is configured to classify and label the factory difference data to obtain a factory target change sample data set; and train a factory target change detection model using the factory target change sample data set to obtain a factory target change detection model. In a third aspect, the present invention provides an electronic device comprising: at least one processor and a memory;

[0064] The memory stores computer-executable instructions;

[0065] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the factory target change detection method described in any one of the first aspects.

[0066] The present invention provides a factory target change detection method, device and electronic equipment based on drone images. Compared with the existing technology, the solution of the present invention improves the traditional Wallis uniform light and color algorithm to perform uniform light processing on drone images, which not only solves the color and brightness difference problems between drone images, but also eliminates the brightness inconsistency problem between pixels within drone images through uniform light processing after multiple overlapping blocks, thereby improving the quality of drone images and further improving the accuracy of factory target change detection. The factory 2.5D model is obtained by fusing the elevation digital model with the drone image after uniform light processing. The factory target change detection model is trained based on the factory 2.5D model to detect factory changes. It can not only detect illegal expansion and demolition of the factory horizontal surface, but also detect changes in the height of the factory, avoiding missed detection of factory height changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 A schematic diagram of the scenario architecture provided by the present invention;

[0069] Figure 2A flow chart of a method for detecting factory target changes based on drone images provided by the present invention;

[0070] Figure 3 This is a schematic diagram of the UAV image segmentation;

[0071] Figure 4 This is a schematic diagram of the structure of a factory target change detection device based on drone images provided by the present invention;

[0072] Figure 5 This is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0074] Figure 1 The schematic diagram of the scenario architecture provided by the present invention is as follows: Figure 1 As shown, a scenario architecture based on which the present disclosure is based may include a factory target change detection device 1 and a drone 2.

[0075] Among them, the factory target change detection device 1 can be hardware, software or a combination of software and hardware, which can be used to execute the factory target change detection method based on drone images described in the following embodiments.

[0076] When the factory target change detection device 1 is hardware, it can be an electronic device with computing capabilities. When the factory target change detection device 1 is software, it can be installed in an electronic device with computing capabilities. Such electronic devices include, but are not limited to, servers, laptops, and desktop computers.

[0077] The factory target change detection device 1 can be a server integrated on the drone 2 and provide factory target change detection services for the drone 2. At the same time, the factory target change detection device 1 can also use the shooting function of the drone 2 to obtain factory images taken by the drone 2.

[0078] Of course, in other usage scenarios, the factory target change detection device 1 can also be integrated into a server for processing drone images, such as an image processing server. At this time, the drone 2 can communicate and exchange data with the aforementioned factory target change detection device 1 through the network. The drone 2 can transmit the drone images to be processed to the factory target change detection device 1, so that the factory target change detection device 1 processes the drone images in the manner shown below to obtain the factory target change detection results.

[0079] The following further describes the method, device, and electronic device for detecting factory target changes based on drone images provided by this application:

[0080] Figure 2 The following is a flow chart of a factory target change detection based on drone images provided by an embodiment of the present disclosure. Figure 2 As shown, the embodiment of the present disclosure provides a factory target change detection method based on drone images, including:

[0081] S1. Obtain two drone images of the factory taken at different times;

[0082] In this embodiment, in order to detect whether there are any illegal structures in the factory before and after two periods, a drone equipped with a high-definition camera takes drone images of the factory from above the factory at two different periods.

[0083] S2. Use the improved Wallis uniform light and color algorithm to perform uniform light processing on the two UAV images taken at different times to obtain two UAV result images taken at different times;

[0084] Since drone photography is affected by different shooting environments (such as solar altitude angle, clouds, etc.), terrain undulations (such as mountains, plains, etc.), camera characteristics (such as exposure, white balance, etc.), aerial scale and other factors, it is easy for the captured images and images to have problems such as uneven color and inconsistent brightness. If these differences are not processed, the images will have obvious color and brightness differences, which will bring corresponding difficulties to the subsequent image processing. The drone image uniformity algorithm is used to solve this problem. According to the processing ideas, it is mainly divided into three categories: histogram-based uniformity algorithm, statistical method-based uniformity algorithm and global-based uniformity algorithm. The traditional Wallis uniformity algorithm is a uniformity algorithm based on statistical methods. The mean and variance of the template image are used as the standard. By transforming the grayscale value of the image, its mean and variance are approximate to the mean and variance of the template image, thereby realizing image uniformity processing. However, the traditional Wallis uniformity algorithm is based on single pixel processing, so the brightness of different pixels will be inconsistent after uniformity, and dividing lines will appear. Therefore, based on the traditional Wallis uniformity algorithm, the drone image is segmented multiple times with different preset segmentation sizes and the image block overlapping area is set. The preset segmentation size and the proportion of the overlapping area can be determined according to the size of the drone image, and then uniformity processing is performed to solve the color and brightness differences between drone images and the brightness inconsistency between pixels within the drone image, thereby improving the overall quality of the drone image.

[0085] Specifically, 100 pre-collected drone images from different periods were evaluated for light indexes to determine a template image with the highest quality score. Light index evaluation is a method of evaluating image quality by calculating parameter indicators that can reflect image color and brightness. Since the mean can reflect image brightness changes, the standard deviation can reflect the differences between image pixels, the average gradient can reflect the contrast and texture changes of small details in the image, and also express the clarity of the image, and the information entropy can reflect the information richness of the image, the parameter indicators used are: mean, standard deviation, average gradient, and information entropy. The 100 drone images were scored using the evaluation indicators, and the drone image with the highest score was determined as the template image. The grayscale mean and standard deviation of the template image were calculated.

[0086] The two drone images from different periods are divided into 6×6 grids, and the overlapping areas between the image blocks are set at a ratio of 1 / 3. The overlapping modes include left-right overlap and top-bottom overlap, such as Figure 3As shown in the figure, the 6×6 red boundary image block and the 6×6 yellow boundary image block overlap left and right, and the 2×6 grid area where the two overlap represents the overlapping area. The 6×6 red boundary image block and the 6×6 blue boundary image block overlap top and bottom, and the 6×2 grid area where the two overlap represents the overlapping area. Calculate the grayscale mean and standard deviation of each image block;

[0087] The Wallis light and color uniformity algorithm is used to perform light uniformity processing on each image block, as follows:

[0088]

[0089] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0090] The image blocks after uniform light treatment are stitched together, and the overlapping areas are smoothed. The optimal smoothing algorithm can be determined by experimentally comparing multiple smoothing algorithms to obtain two drone images taken at different times.

[0091] The drone image obtained from the first dodging process was divided into 12×12 blocks. The overlapping areas between the image blocks were set at a ratio of 1 / 3. The grayscale mean and standard deviation of each image block were calculated. The Wallis dodging algorithm was used to dovetail each image block. The dodging-processed image blocks were then spliced ​​and the overlapping areas were smoothed.

[0092] The drone image obtained from the second dodging process was divided into 36×36 blocks. The overlapping areas between the image blocks were set at a ratio of 1 / 3. The grayscale mean and standard deviation of each image block were calculated. The Wallis dodging algorithm was used to dovetail each image block. The dodging-processed image blocks were then spliced, and the overlapping areas were smoothed.

[0093] The original UAV images, the UAV images processed by the traditional Wallis algorithm, and the UAV images processed by the proposed algorithm of the same period were respectively calculated to obtain parameter indicators that can reflect the image quality. Then, the parameter indicators were quantitatively analyzed to determine the image quality. Since the mean can reflect the change in image brightness, the standard deviation can reflect the difference between image pixels, the average gradient can reflect the contrast and texture change characteristics of small details in the image, and also express the clarity of the image, the information entropy can reflect the information richness of the image, so the parameter indicators used are: mean, standard deviation, average gradient, and information entropy.

[0094] The results show that the mean, standard deviation, average gradient and information entropy of the drone images obtained after processing by the algorithm in this paper are all ranked in the top two, indicating that the drone images obtained after the third uniform light treatment have good color preservation, rich image information, good details, clear images, relatively concentrated image grayscale distribution, and small differences between pixels.

[0095] S3, fusing the two drone images from different periods with the digital elevation models of the factory locations at the corresponding periods to obtain 2.5D models of the factory at the two periods;

[0096] Since drone images are two-dimensional, and illegal factory construction may also lead to height increases, drone images can only detect illegal expansion and demolition on the horizontal surface of the factory, and it is difficult to detect changes in the factory's height. Therefore, it is necessary to further add height information to the drone images to construct a 2.5D model of the factory. Compared with a three-dimensional model, this requires less calculation and faster processing speed.

[0097] Specifically, the digital elevation model of the factory location at the time period corresponding to the two drone images at different time periods is obtained, and the coordinate systems of the two sets of digital elevation models at different time periods are converted to the coordinate system of the drone result image. The digital elevation model and the drone result image are unified into the same coordinate system. The height data can be integrated into the drone result image to obtain the converted digital elevation model.

[0098] The two drone images taken at different times are registered with the corresponding converted digital elevation models to determine the positional correspondence between the two images. The elevation values ​​in the converted digital elevation models are extracted based on the positional correspondence between the two drone images taken at different times and the corresponding converted digital elevation models to obtain the elevation values ​​corresponding to the two drone images taken at different times.

[0099] The elevation values ​​corresponding to the two UAV images taken at different periods were superimposed on the two UAV result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

[0100] S4. Calculate factory difference data based on the factory 2.5D model of the two periods;

[0101] Since the processing amount of detecting the factory 2.5D model data of both periods is large, and the difference between the factory 2.5D model data of the two periods is the factory change information, the factory change can be determined by calculating the difference between the factory 2.5D model data of the two periods.

[0102] Specifically, the 2.5D factory models of the two periods are converted into raster data; the factory raster data of the two periods are superimposed to determine the factory area, and the factory raster data of the two periods are clipped according to the factory area to obtain the valid factory raster data of the two periods; the difference between the valid factory raster data of the latter period and the valid factory raster data of the previous period is calculated to obtain the factory difference data.

[0103] S5. Input the factory difference data into a pre-trained factory target change detection model to obtain a factory target change result;

[0104] In this embodiment, before the factory target change detection model is used, it is necessary to first construct a factory target change sample data set and a factory target change detection model, and then train the factory target change detection model.

[0105] Specifically, 100,000 pre-collected drone images from different periods and the digital elevation model of the factory’s location at the corresponding period were obtained;

[0106] The coordinate systems of the digital elevation models at different times are converted to the coordinate system of the UAV images to obtain the converted digital elevation models;

[0107] Based on the position correspondence between 100,000 drone images from different periods and the corresponding converted digital elevation models, data sampling was performed on the converted digital elevation model to obtain the elevation values ​​corresponding to the 100,000 drone images from different periods.

[0108] The elevation values ​​corresponding to 100,000 drone images from different periods were superimposed onto the drone images from the corresponding period to obtain 100,000 2.5D models of the factory at different periods;

[0109] Calculate factory difference data based on 100,000 2.5D factory models from different periods and classify and label the factory difference data. The labeling information includes unchanged, expanded, demolished, raised, and lowered, thus obtaining a sample dataset of factory target changes.

[0110] The factory target change detection model is built based on the U-net network. The U-net network consists of two parts: the encoder on the left and the decoder on the right. The encoder follows the typical convolutional network architecture, and the decoder is the deconvolution of the encoder's corresponding layer. The input matrix size of the designed model is 512×512×6, and the output matrix size is 512×512×3. The input data is the factory target change sample data, and the output image of the model is a labeled image matrix of 512×512×3. After the model is built, the forward propagation operation is first performed on each sample. The formula for the forward propagation operation is as follows:

[0111] net (l+1) =W l ·x l +b l

[0112] x (l+1) =f(net (l+1) )

[0113] Among them, net (l+1) is the weighted sum of the inputs of the l+1th layer, W l is the connection weight between layer l and layer l+1, x l is the node value of the lth layer, b l is the bias term of layer l, and f(·) is the activation function;

[0114] Using the above forward propagation formula, we can get the node values ​​of the second layer, the third layer and even the output layer. In order to find the parameters W and b that can minimize the cost function L(W,b), we first calculate the residual of the output layer:

[0115]

[0116] Among them, net nl is the weighted sum of the inputs to the output layer;

[0117] For l=n l -1,n l -2,n l For each layer of -3,...,2, their residuals are calculated according to the following formula:

[0118] δ (l) =((W (l) ) T δ (l+1) )·f(net (l) )

[0119] Then, calculate the partial derivative of the cost function for a single sample:

[0120]

[0121] Then calculate the sum of the partial derivatives of the cost function for all samples:

[0122]

[0123] Finally update the weight parameters:

[0124]

[0125] Among them, α is the learning rate and λ is the weight decay parameter.

[0126] These iterative steps are repeated to reduce the value of the cost function L(W, b) until the training of the factory target change detection model is completed.

[0127] Corresponding to the factory target change detection method based on drone images in the above embodiment, Figure 4 This is a schematic diagram of the structure of a factory target change detection device based on drone images provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 , the factory target change detection device based on drone images includes:

[0128] An acquisition unit 41 is used to acquire two drone images of the factory taken at different times by the drone;

[0129] A dodging processing unit 42 is configured to perform dodging processing on two drone images taken at different times using an improved Wallis dodging and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis dodging and color balancing algorithm is configured to perform overlapping segmentation of the two drone images taken at different times at a preset segmentation size to obtain all image blocks, and after dodging processing is performed on each image block, the preset segmentation size is changed until a preset number of dodging processing operations are completed;

[0130] The data fusion unit 43 is used to fuse the two drone images taken at different times with the digital elevation models of the factory location at the corresponding times to obtain 2.5D models of the factory at the two times;

[0131] a factory difference data calculation unit 44 for calculating factory difference data based on the factory 2.5D model of the two periods;

[0132] The change detection unit 45 is used to input the factory difference data into a pre-trained factory target change detection model to obtain a factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label includes at least unchanged area, expanded area, demolished area, raised area and lowered area.

[0133] Optionally, the light uniformity processing unit 42 is specifically configured to:

[0134] Multiple drone images collected at different times were evaluated for light index to determine a template image with the highest score. The grayscale mean and standard deviation of the template image were calculated. Light index evaluation is a method of evaluating image quality by calculating parameter indicators that reflect image color and brightness.

[0135] Two drone images taken at different times are divided into blocks according to a preset segmentation size. The overlapping areas between the image blocks are set according to a preset ratio. The grayscale mean and standard deviation of each image block are calculated.

[0136] The Wallis light and color uniformity algorithm is used to perform light uniformity processing on each image block, as follows:

[0137]

[0138] Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image;

[0139] The image blocks after uniform light treatment are stitched together and the overlapping areas are smoothed to obtain two drone images taken at different times.

[0140] Change the preset segmentation size and repeat the previous two steps to perform light dodging on the two drone images obtained from the previous light dodging process at different times until the preset number of light dodging processes is reached.

[0141] Optionally, the data fusion unit 43 is specifically configured to:

[0142] Obtain the digital elevation model of the factory location at the time period corresponding to the two drone images taken at different times, and convert the coordinate systems of the two sets of digital elevation models at different times to the coordinate system of the drone result image to obtain the converted digital elevation model;

[0143] Based on the position correspondence between the two UAV images taken at different times and the corresponding converted digital elevation models, the converted digital elevation model is sampled to obtain the elevation values ​​corresponding to the two UAV images taken at different times.

[0144] The elevation values ​​corresponding to the two UAV images taken at different periods were superimposed on the two UAV result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

[0145] Optionally, the factory difference data calculation unit 44 is specifically configured to:

[0146] Convert the 2.5D factory models of the two periods into raster data;

[0147] The factory raster data of the two periods are superimposed and clipped to obtain the effective factory raster data of the two periods;

[0148] The difference between the effective factory grid data of the latter period and the effective factory grid data of the previous period is calculated to obtain the factory difference data.

[0149] Optionally, the device further includes:

[0150] The training unit 46 is used to classify and label the factory difference data to obtain a factory target change sample data set; and use the factory target change sample data set to train the factory target change detection model to obtain the factory target change detection model.

[0151] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the electronic device 50 of this embodiment may include: a memory 51 and a processor 52.

[0152] Memory 51, for storing computer programs (such as application programs and functional modules for implementing the above-mentioned factory target change detection method based on drone images), computer instructions, etc.;

[0153] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories 51 . Furthermore, the aforementioned computer programs, computer instructions, data, etc. may be called by the processor 52 .

[0154] The processor 52 is configured to execute the computer program stored in the memory 51 to implement the various steps in the method involved in the above embodiment.

[0155] For details, please refer to the relevant description in the previous method embodiment.

[0156] The memory 51 and the processor 52 may be independent structures or integrated structures. When the memory 51 and the processor 52 are independent structures, the memory 51 and the processor 52 may be coupled to each other via a bus 53 .

[0157] An electronic device of this embodiment can perform Figure 2 The technical solution in the method shown, its specific implementation process and technical principles can be found in Figure 2 The relevant descriptions in the method shown will not be repeated here.

[0158] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments.

[0159] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely exemplary forms of implementing the claims.

Claims

1. A factory target change detection method based on drone images, characterized in that: include: S1. Obtain two drone images of the factory taken at different times; S2. Performing light balancing on the two drone images taken at different times using an improved Wallis light balancing and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis light balancing and color balancing algorithm is used to perform overlapping segmentation of the two drone images taken at different times at a preset segmentation size to obtain all image blocks, and after performing light balancing on each image block, the preset segmentation size is changed until a preset number of light balancing processes are completed; S3, fusing the two drone images from different periods with the digital elevation models of the factory locations at the corresponding periods to obtain 2.5D models of the factory at the two periods; S4. Calculate factory difference data based on the factory 2.5D model of the two periods; S5. Inputting the plant difference data into a pre-trained plant target change detection model to obtain a plant target change result, wherein the pre-trained plant target change detection model is determined after training based on the plant difference data and a change result label corresponding to each plant difference data, wherein the change result label includes at least unchanged area, expanded area, demolished area, heightened area, and lowered area; The improved Wallis uniform light and color algorithm is used to process two drone images taken at different times to obtain two drone result images taken at different times, including: S201, performing a light index evaluation on multiple pre-collected drone images from different time periods to determine a template image with the highest quality score, and calculating the grayscale mean and standard deviation of the template image, wherein the light index evaluation is used to calculate parameter indicators for any drone image to obtain a quality score corresponding to the drone image, wherein the parameter indicators include image color parameters and image brightness parameters; S202, dividing the two drone images taken at different times into blocks according to a preset segmentation size, setting overlapping areas between the image blocks according to a preset ratio, and calculating the grayscale mean and standard deviation of each image block; S203: Perform light balancing on each image block using the Wallis light balancing and color balancing algorithm, as follows: Among them, g(x,y) is the gray value of the original image at (x,y), f(x,y) is the gray value of the image at (x,y) after being processed by the Wallis uniform light and color algorithm, and m g is the grayscale mean of the original image block, s g is the grayscale standard deviation of the original image block, m f is the grayscale mean of the template image, s f is the grayscale standard deviation of the template image, c∈[0,1] is the expansion constant of the image variance, and b∈[0,1] is the brightness coefficient of the image; S204: stitching the image blocks after the light uniforming process and smoothing the overlapping areas to obtain two drone images taken at different times; S205: Change the preset segmentation size and repeat steps S202-S204 to perform dodging processing on the two drone images obtained in the previous dodging processing at different times until the preset number of dodging processing times is reached; The two drone images taken at different times are fused with the digital elevation models of the factory locations at the corresponding times to obtain 2.5D factory models at the two times, including: S301, obtaining a digital elevation model of the factory location at a time period corresponding to two sets of drone images taken at different time periods, and converting the coordinate systems of the two sets of digital elevation models at different time periods to the coordinate system of the drone images to obtain a converted digital elevation model; S302: performing data sampling on the converted digital elevation model based on the positional correspondence between the two drone images taken at different times and the corresponding converted digital elevation models to obtain elevation values ​​corresponding to the two drone images taken at different times; S303 , superimposing the elevation values ​​corresponding to the two drone images taken at different periods onto the two drone result images taken at different periods to obtain the 2.5D models of the factory at the two periods.

2. The factory target change detection method based on drone images according to claim 1 is characterized in that: The parameter indicators include mean, standard deviation, average gradient, and information entropy.

3. The factory target change detection method based on drone images according to claim 1 is characterized in that: The preset number of dodging processes is 3, the preset segmentation size is a 6×6 grid, the preset ratio is 1 / 3 to set the overlapping area, the preset segmentation size is changed, and steps S202-S204 are repeatedly performed to perform dodging processes on two drone result images of different periods obtained in the previous dodging process until the preset number of dodging processes is reached, including: The preset segmentation size was changed from a 6×6 grid to a 12×12 grid. The drone imagery obtained from the first dodging process was divided into blocks. The overlapping areas between the image blocks were set at a ratio of 1 / 3. The grayscale mean and standard deviation of each image block were calculated. Each image block was dodged using the Wallis dodging algorithm. The dodging-processed image blocks were then spliced ​​and the overlapping areas were smoothed. The preset segmentation size was changed from a 12×12 grid to a 36×36 grid. The drone image obtained by the second light-dodging process was divided into blocks. The overlapping areas between the image blocks were set at a ratio of 1 / 3. The grayscale mean and standard deviation of each image block were calculated. The Wallis light-dodging and color-dodging algorithm was used to dovetail each image block. The image blocks after light-dodging were spliced, and the overlapping areas were smoothed.

4. The method for detecting factory target changes based on drone images according to claim 1 is characterized in that: The calculation of factory difference data based on the factory 2.5D model of the two periods includes: S401, converting the 2.5D factory models of the two periods into raster data; S402, superimposing and clipping the factory raster data of the two periods to obtain valid factory raster data of the two periods; S403 , calculating the difference between the valid factory grid data of the next period and the valid factory grid data of the previous period to obtain factory difference data.

5. The factory target change detection method based on drone images according to claim 1 is characterized in that: The factory target change detection model is constructed based on the U-net network.

6. The method for detecting factory target changes based on drone images according to any one of claims 1 to 5, characterized in that: Before obtaining two drone images of the factory taken by the drone at different times, the following steps are also included: Obtain multiple pre-collected drone images from different periods and the digital elevation model of the factory location at the corresponding period; The coordinate systems of the digital elevation models at different times are converted to the coordinate system of the UAV images to obtain the converted digital elevation models; Based on the position correspondence between the UAV images of different periods and the corresponding converted digital elevation models, the converted digital elevation models are sampled to obtain the elevation values ​​corresponding to the UAV images of different periods. The elevation values ​​corresponding to the drone images from different periods were superimposed onto the drone images from the corresponding period to obtain the 2.5D model of the factory at multiple periods; Calculate factory difference data based on the factory 2.5D model of multiple periods, and classify and label the factory difference data to obtain the factory target change sample data set; The factory target change detection model is trained using the factory target change sample dataset to obtain the factory target change detection model.

7. A factory target change detection device based on drone images, using the method according to any one of claims 1 to 6, characterized in that: include: an acquisition unit, used to acquire two drone images of the factory taken at different times; a dodging processing unit, configured to perform dodging processing on two drone images taken at different times using an improved Wallis dodging and color balancing algorithm to obtain two drone result images taken at different times, wherein the improved Wallis dodging and color balancing algorithm is configured to perform overlapping segmentation of the two drone images taken at different times at a preset segmentation size to obtain all image blocks, and after dodging processing is performed on each image block, the preset segmentation size is changed until the dodging processing is completed for a preset number of times; The data fusion unit is used to fuse the drone images from two different periods with the digital elevation model of the factory location at the corresponding period to obtain the 2.5D model of the factory at the two periods; A factory difference data calculation unit is used to calculate factory difference data based on the factory 2.5D model of two periods; The change detection unit is used to input the factory difference data into a pre-trained factory target change detection model to obtain the factory target change result, wherein the pre-trained factory target change detection model is determined after training based on the factory difference data and the change result label corresponding to each factory difference data, and the change result label at least includes unchanged area, expanded area, demolished area, raised area and lowered area.

8. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 6.

Images