Unmanned aerial vehicle photoelectric image change detection method and device, equipment and medium

Abstract

The embodiment of the invention provides an unmanned aerial vehicle photoelectric image change detection method and device, equipment and a medium, and the method comprises the following steps: training a target semantic segmentation model through a semantic segmentation training set, and training a sparse window macroscopic model through a sparse window training set; acquiring two unmanned aerial vehicle photoelectric images, registering the two unmanned aerial vehicle photoelectric images, and generating an image pair after registration and an inverse matrix of the transformation matrix during deregistration; inputting the registered image pair into a target semantic segmentation model, generating a pixel corresponding to the registered image pair and a first change image of a target, inputting the registered image into a sparse window macroscopic model, and generating a second change image block corresponding to a window area; and combining the first change image block and the second change image block to obtain change data on the photoelectric image of the unmanned aerial vehicle. According to the scheme, the missing detection targets in the semantic segmentation model are subjected to secondary screening through the sparse window macroscopic model, and the detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and medium for detecting photoelectric image changes of an unmanned aerial vehicle. Background Art

[0002] With the continuous development of ground detection and sensor technology and UAV technology, the relevant technologies in the field of aerial reconnaissance and detection for UAV platforms have attracted more and more attention in recent years, and the mode of UAVs carrying high-resolution optoelectronic payloads has become increasingly mature. Among them, change detection technology is an important means of performing reconnaissance tasks. It mainly obtains optoelectronic images of a certain reconnaissance area from drones at different times, analyzes these images at different times, and finds the change information in the area after this period, such as the addition, displacement or loss of important targets, the addition or removal of buildings, etc. This information is crucial to the field of aerial reconnaissance, and can help reconnaissance personnel quickly find valuable intelligence information, which is of great value in saving interpretation time and labor costs.

[0003] Nowadays, change detection methods are widely used in the field of satellite remote sensing image detection, and a series of progress has been made and several effective methods have been proposed, but research in the field of change detection in UAV photoelectric images is relatively rare. One of the main reasons is that compared with satellite remote sensing images, UAV photoelectric images often have certain differences in shooting angles and lighting conditions, which makes it difficult to obtain paired image samples for neural network model training; in addition, the changes detected in traditional satellite remote sensing images are often large-scale buildings, vegetation, etc., and are mostly used in urban planning, environmental monitoring and other fields, but there is a lack of research on target-level detection. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method for detecting changes in UAV optoelectronic images to solve the technical problems in the prior art of detecting changes in UAV optoelectronic images, such as differences in viewing angles, lack of training samples, and poor detection of target-level changes. The method includes:

[0005] Mark the target of the category of interest in the optoelectronic image of the UAV, generate optoelectronic image pairs, annotate the labels on the optoelectronic image pairs, generate a semantic segmentation training set, train the target semantic segmentation model through the semantic segmentation training set, annotate the labels on the optoelectronic image pairs, generate a sparse window training set, and train the sparse window macro model through the sparse window training set, wherein the optoelectronic image pairs include original images and changed images;

[0006] Acquire two UAV optoelectronic images in the same target area but at different phases, register the two UAV optoelectronic images using feature point detection and description methods, generate the registered image pair and the inverse matrix of the transformation matrix during deregistration;

[0007] Input the registered image pair into the target semantic segmentation model to generate pixels corresponding to the registered image pair and a first change image of the target; input the registered image into the sparse window macro model to generate a window region pair with changes corresponding to the registered image pair; perform differential processing on the window region pair with changes to generate a second change image block that the window region should correspond to;

[0008] The first change image and the second change image block are merged to generate the change result of the registered image pair, and the change result is restored to the UAV photoelectric image through the inverse matrix to obtain the change data on the UAV photoelectric image.

[0009] The embodiment of the present invention also provides a device for detecting changes in the optoelectronic image of a drone, so as to solve the technical problems in the prior art of detecting changes in the optoelectronic image of a drone, such as differences in viewing angles, lack of training samples, and poor detection of target-level changes. The device includes:

[0010] A model training module is used to mark the target of the interested category in the optoelectronic image of the UAV, generate optoelectronic image pairs, annotate labels on the optoelectronic image pairs, generate a semantic segmentation training set, train the target semantic segmentation model through the semantic segmentation training set, annotate labels on the optoelectronic image pairs, generate a sparse window training set, and train the sparse window macro model through the sparse window training set, wherein the optoelectronic image pairs include original images and changed images;

[0011] The image registration module is used to obtain two UAV optoelectronic images in the same target area but at different phases, register the two UAV optoelectronic images through feature point detection and description methods, generate the registered image pair and the inverse matrix of the transformation matrix during deregistration;

[0012] A change detection module is used to input the registered image pair into the target semantic segmentation model to generate target segmentation change result data, generate pixels corresponding to the registered image pair and a first change image of the target, input the registered image into the sparse window macro model to generate a change area of ​​the sparse window, generate a window area pair with changes corresponding to the registered image pair, perform differential processing on all window area pairs with changes, and generate a second change image block that the window area should correspond to;

[0013] The change data generation module is used to merge the first change image and the second change image block to generate the change result of the registered image pair, and restore the change result to the UAV photoelectric image through the inverse matrix to obtain the change data on the UAV photoelectric image.

[0014] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for detecting changes in the optoelectronic image of a drone, so as to solve the technical problems in the prior art of detecting changes in optoelectronic images of drones, such as differences in viewing angles, lack of training samples, and poor detection of target-level changes.

[0015] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned methods for detecting changes in drone optoelectronic images, so as to solve the technical problems in the prior art of change detection of drone optoelectronic images, such as perspective differences, lack of training samples, and poor target-level change detection effects.

[0016] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0017] Detecting target-level change areas in UAV optoelectronic images through target semantic segmentation model and sparse window macro model can effectively prevent missed detection and false alarm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a method for detecting photoelectric image changes of a drone provided by an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for detecting changes in the above-mentioned UAV optoelectronic image provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a target semantic segmentation model provided by an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of the structure of a sparse window macro model provided by an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of detection results of the target semantic segmentation model provided by an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of the detection results of the sparse window macro model provided by an embodiment of the present invention;

[0025] Figure 7 It is a schematic diagram of detection results obtained after post-processing the detection results of the sparse window macro model and integrating them with the detection results of the target semantic segmentation model provided by an embodiment of the present invention;

[0026] Figure 8 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0027] Fig. 9 The present invention provides a block diagram of a device for detecting photoelectric image changes of a UAV. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0030] Unmanned Aerial Vehicle Electro-Optical Imaging (UAV) refers to images captured using electro-optical sensors carried onboard an unmanned aerial vehicle (UAV).

[0031] In an embodiment of the present invention, a method for detecting changes in an unmanned aerial vehicle photoelectric image is provided. Figure 1 and Figure 2 As shown, the method includes:

[0032] Step S101: marking targets of the interested category in the optoelectronic image of the UAV, generating optoelectronic image pairs, labeling the optoelectronic image pairs, generating a semantic segmentation training set, training the target semantic segmentation model through the semantic segmentation training set, labeling the optoelectronic image pairs, generating a sparse window training set, and training the sparse window macro model through the sparse window training set, wherein the optoelectronic image pairs include original images and changed images;

[0033] Step S102: acquiring two UAV optoelectronic images in the same target area but at different time phases, registering the two UAV optoelectronic images by a feature point detection description method, generating a registered image pair and an inverse matrix of the transformation matrix during deregistration;

[0034] Step S103: input the registered image pair into the target semantic segmentation model to generate pixels corresponding to the registered image pair and a first change image of the target, input the registered image into the sparse window macro model to generate a window area pair with changes corresponding to the registered image pair, perform differential processing on the window area pair with changes, and generate a second change image block that the window area should correspond to;

[0035] Step S104: merge the first change image and the second change image block to generate a change result of the registered image pair, restore the change result to the UAV optoelectronic image through an inverse matrix, and obtain change data on the UAV optoelectronic image.

[0036] In the specific implementation, the following steps are performed to label the photoelectric image pairs and generate a semantic segmentation training set:

[0037] Label the original image, generate a first labeled image, and generate a corresponding first loss value; label the changed image, generate a second labeled image, and generate a corresponding second loss value; compare the original image and the changed image, obtain the area where the target has changed, binary-mark the area where the target has changed, generate a third labeled image, and generate a corresponding third loss value; use the first loss value, the second loss value, and the third loss value as training loss values ​​of the target semantic segmentation model; generate a semantic segmentation training set through the first labeled image, the second labeled image, and the third labeled image.

[0038] In specific implementation, the following steps are performed to label the photoelectric image pairs and generate a sparse window training set:

[0039] The first window size, the first step length, the second window size and the second step length are set, the original image is segmented according to the first window size and the first step length to generate a first segmented image, and the changed image is segmented according to the second window size and the second step length to generate a second segmented image; the first segmented image and the second segmented image are used as a segmented image pair, a marking reference is determined, the segmented image pair is labeled according to the marking reference, and the labeled segmented image pair is used as a sparse window training set.

[0040] Specifically, the training sets of the target semantic segmentation model and the sparse window macro model are labeled, and a certain number of UAV aerial photoelectric reconnaissance images (as UAV photoelectric images) are prepared, and each image is processed according to the following steps:

[0041] Step 1: copy the UAV optoelectronic image to generate two identical UAV optoelectronic images, mark the target of the category of interest contained in the first UAV optoelectronic image, and generate marking information, wherein the marking information includes the pixel range occupied by the target and the category number of the corresponding pixel;

[0042] Step 2: For the second UAV photoelectric image, deduct the target contained in it and fill it with the surrounding background, and randomly place the deducted target in the remaining positions of the second UAV photoelectric image to compare the disappearance / appearance of the target in the original image. This process can be implemented by writing a script after the marking in step 1 is completed, which is used to construct the target change. At the same time, all new target areas are marked in the same way as in step 1, and the two images are used as a photoelectric image sample pair (the first UAV photoelectric image is not processed in any way and is used as the original image, and the second UAV photoelectric image is processed with the target moved and is used as the changed image);

[0043] Step 3: The photoelectric image sample pairs are labeled with two twin segmentation networks as follows: the twin network output label of the first image in the photoelectric image sample is recorded as the first label, and the first labeled image is generated. Corresponding to the first loss, the target pixel area contained in the first image is marked as the target area of ​​the corresponding category. At the same time, the target pixel area contained in the second image but not in the first image is marked as the "unknown change" category in the first image, and the pixels in the remaining positions are marked as "unchanged". The processing result is used as the first label; similarly, the second image is labeled using the same method, and the processing result is used as the second label to generate the second labeled image; if there are three categories of the target of interest, the set of all labels is ["unchanged", "unknown change", "category 1", "category 2", "category 3"], a total of 5 labels. Among them, the number of categories can be determined according to actual conditions. For example, category 1 is a vehicle, category 2 is a building, and so on.

[0044] Step 4: Binarize and mark all the regions where the target changes in the first image (original image) and the second image (changed image) as "changed", and mark the remaining regions as "unchanged". The binary mark is recorded as the third label, and the loss corresponding to the third label is the third loss (i.e., the training loss of the PSPNet network);

[0045] Step 5: Divide the photoelectric image sample pairs at the corresponding positions of the original image and the changed image according to two sizes: window size 50×50, step size 35, window size 100×100, step size 75, and form new image block sample pairs. At the same time, mark the image block sample pairs as "unchanged" / "changed" according to the previously marked target pixel area, set the marking criterion (in this embodiment, the marking criterion is when the number of changed pixels is greater than 1 / 8 of the number of window pixels), and update the sample record to "changed" according to the marking criterion.

[0046] The target semantic segmentation model and the sparse window macro model are trained separately, and the hyperparameter settings in the model can be adjusted according to actual conditions.

[0047] Specifically, two UAV optoelectronic images in the same target area and different phases are obtained as UAV optoelectronic image pairs (including the first image and the second image). For a new set of UAV optoelectronic image pairs, image registration is first performed to register the second image to the viewing angle of the first image, and the registered image is cropped to the same area range and resolution as the first image to obtain two registered images (registered image pairs). Feature descriptors such as SIFT and SURF can be used, and the inverse matrix of the transformation matrix during registration can be solved at the same time.

[0048] In specific implementation, the following steps are performed to input the registered image pair into the target semantic segmentation model to generate pixels corresponding to the registered image pair and the first change image of the target:

[0049] The first image and the second image in the registered image pair are respectively input into two identical semantic segmentation models to generate a first target change image and a second target change image, wherein the first target change image and the second target change image include a high-dimensional feature representation of the target change region and the image; the first target change image and the second target change image are input into a change detection network to generate a first change image of pixels and targets.

[0050] Specifically, Figure 3 As shown in the figure, the target semantic segmentation model uses a twin network-based architecture to receive two image data im1 and im2 (i.e., registered image pairs) at different times, and uses two twin HR-Nets combined with PSPNet to simultaneously learn the network's semantic segmentation and change detection capabilities. Two weight-sharing twin HR-Net networks are used to learn the semantic segmentation-level target change regions of the two images, and high-dimensional feature representations of the images are extracted for PSPNet network input.

[0051] The target semantic segmentation model is first downsampled through two convolutional layers. The Bottlenectk structure is introduced here to reduce the number of parameters and use 1×1 convolution to expand the number of channels to obtain high-dimensional primary features after downsampling. Then, according to the HR-Net structure, the resolution of each stage is first expanded, a new resolution branch is added, and then all branches are fused with each other. After multi-resolution fusion of a total of 4 stages, three 1×1 convolution modules and a 4x upsampling are passed through, and finally a feature map of the same resolution size is output. The number of channels is the number of semantic segmentation categories plus 2, including one category of unchanged categories and one category of unknown changed categories. The two twin HR-Nets share weights, and Loss1 (first loss value) and Loss2 (second loss value) are calculated for the two input images respectively. The output loss of HR-Net is the multi-category cross entropy loss:

[0052] The image feature vector extracted by HR-Net is used for feature fusion and then input into PSPNet, which is used to improve the feature extraction capabilities of the two twin networks, while paying more attention to learning the changed areas themselves.

[0053] Here, the outputs of Stage 4 of the two HR-Nets are concat-operated as a fusion of the feature information of the two images and sent to PSPNet for two-stage processing. PSPNet uses ResNet101 with a dilated convolution strategy as the feature extraction backbone, and then sends the feature map to the pyramid pooling module. Here, four levels of pooling are used, and the pooling window sizes are 1×1, 2×2, 3×3, and 6×6 respectively. The pooled outputs are upsampled to the original feature map size by bilinear interpolation, and the four upsampled results are compared with the original feature map. Figure 1 The concat operation is then repeated, and the final output size is the same as the original image. The number of channels is 1, which indicates the probability value of the pixel belonging to the changed part. The loss of this part is recorded as Loss3 (the third loss value), and the binary cross entropy loss is used:

[0054]

[0055] The network loss is the sum of the losses of the two twin HR-Net and PSPNet, namely: Loss sum =Loss1+Loss2+Loss3.

[0056] The change detection through the target semantic segmentation model is as follows Figure 5 shown.

[0057] In specific implementation, the following steps are performed to input the registered image into the sparse window macro model, generate a pair of window regions with changes corresponding to the registered image pair, perform differential processing on the pair of window regions with changes, and generate a second changed image block corresponding to the window region:

[0058] The first image and the second image in the registered image pair are respectively input into two identical feature extraction networks to generate a first feature change image and a second feature change image; the first feature change image and the second feature change image are compared to obtain change features; the change features are input into a classifier to obtain window area pairs with changes, and differential processing is performed on all window area pairs with changes to generate a second change image block that the window area should correspond to.

[0059] Specifically, after the sparse window macro model is trained, change detection is performed on the registered untrained image pairs. The first-stage twin network part is the semantic segmentation output, and the second-stage network part is the change region discrimination output. According to actual needs, the first-stage output can be used as multi-category target segmentation change detection or the second-stage target change region detection.

[0060] like Figure 4 As shown in the figure, the network input of the sparse window macroscopic model is a local image block cropped from the first image in the registered image pair according to a certain window size and step size. The feature vectors of two image blocks (the first image and the second image in the registered image pair) are extracted simultaneously through the twin feature extraction network VGG16, and the difference vector between the two image blocks is generated by taking the difference and taking the absolute value of the two feature vectors. The difference vector is extended to one dimension and input into the fully connected layer, and finally output as the probability value of the change between the two image blocks through the sigmoid function. The binary cross entropy loss is used during training.

[0061] Change detection is performed on the registered image pairs. Models with different resolutions can be selected according to the target size in the actual application scenario. The image pairs are segmented according to the window size, and each pair of segmented image blocks is sent to the model to detect whether there is a change in the current block area.

[0062] The sparse window macroscopic model can make up for the shortcoming of the target semantic segmentation model that relies on the target categories marked during training, resulting in missed detection of unknown category targets. At the same time, it can avoid the distortion of the target caused by image registration, which makes the target semantic segmentation model unable to recognize the target, and can improve the versatility and robustness of the change detection method. The sparse window macroscopic model can detect the range of the area where the change occurs, but it cannot specifically locate the target pixel segmentation area where the change occurs. Therefore, it is necessary to merge the results obtained by the sparse window macroscopic model with the results obtained by the target semantic segmentation model.

[0063] The detection of changes through sparse window macroscopic model is as follows Figure 6 shown.

[0064] In specific implementation, the following steps are performed to merge the first change image and the second change image block to generate the change result of the registered image pair, and the change result is restored to the UAV photoelectric image through the inverse matrix to obtain the change data on the UAV photoelectric image:

[0065] A first change mask is generated through the first change image; the second change image block is post-processed to generate a second change mask; the first change mask and the second change mask are unioned to generate a merged change mask; according to the inverse matrix, the area of ​​the merged change mask is restored to the second image of the registered image pair to generate change data on the UAV optoelectronic image.

[0066] In a specific implementation, the second changed image block is post-processed to generate a second change mask through the following steps:

[0067] Obtain all second change image blocks, and perform the following processing on each second change image block until all second change image blocks are processed: if the second change image block is a color image, grayscale the second change image block to generate a grayscale change image block; perform difference processing on the grayscale change image block and take the absolute value to generate a difference result, perform binary segmentation on the difference result according to a set threshold value to generate a binary segmented image; set a first window size and a second window size of the filter; use a window of the first window size to perform median filtering on the binary segmented image to generate a first filtering result, and use a window of the second window size to perform median filtering on the first filtering result to generate a second filtering result; use a connected domain in the second filtering result that is different from the background as a second change mask.

[0068] The following method is provided to post-process the second changed image block to generate a second change mask:

[0069] 1. Grayscale the two image blocks detected to contain changes (if the input is a color TV image);

[0070] Second, perform differential processing on the image block and take the absolute value, and perform binary segmentation on the differential result according to the threshold;

[0071] 3. Use a 7×7 window (first window size) to perform median filtering on the segmentation results to filter out some erroneous differences;

[0072] Fourth, use a 5×5 window (second window size) to perform median filtering on the above results and filter out invalid differences again;

[0073] 5. The connected domain whose binarization result after filtering is different from the background is used as the segmentation mask of the target change (the second change mask).

[0074] like Figure 7 As shown in the figure, the output of the semantic segmentation model (the first change image) and the post-processing output of the sparse window macroscopic model (the second change mask) are combined pixel by pixel to obtain the change results of the two registered image pairs. The changed pixel area on the second image of the registered image pair is restored to the UAV photoelectric image (the second UAV photoelectric image) according to the inverse matrix of the transformation matrix during de-registration, and together with the detection result of the first image of the registered image pair, it constitutes the final output result of the entire change detection process.

[0075] In this embodiment, a computer device is provided, such as Figure 8 As shown, it includes a memory 801, a processor 802, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned methods for detecting changes in the optoelectronic image of a drone is implemented.

[0076] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0077] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned methods for detecting changes in the optoelectronic image of a drone.

[0078] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0079] Based on the same inventive concept, an embodiment of the present invention also provides a device for detecting photoelectric image changes of a drone, as described in the following embodiments. Since the principle of solving the problem by the device for detecting photoelectric image changes of a drone is similar to the method for detecting photoelectric image changes of a drone, the implementation of the device for detecting photoelectric image changes of a drone can refer to the implementation of the method for detecting photoelectric image changes of a drone, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0080] Fig. 9 is a structural block diagram of a device for detecting photoelectric image changes of a drone according to an embodiment of the present invention. Fig. 9 As shown, it includes: a model training module 901, an image registration module 902, a change detection module 903 and a change data generation module 904. The structure is described below.

[0081] The model training module 901 is used to mark the target of the interested category in the optoelectronic image of the UAV, generate an optoelectronic image pair, annotate the label on the optoelectronic image pair, generate a semantic segmentation training set, train the target semantic segmentation model through the semantic segmentation training set, annotate the label on the optoelectronic image pair, generate a sparse window training set, and train the sparse window macro model through the sparse window training set, wherein the optoelectronic image pair includes an original image and a changed image;

[0082] The image registration module 902 is used to obtain two UAV optoelectronic images in the same target area and different phases, register the two UAV optoelectronic images by using a feature point detection description method, and generate an image pair after registration and an inverse matrix of the transformation matrix during deregistration;

[0083] The change detection module 903 is used to input the registered image pair into the target semantic segmentation model to generate target segmentation change result data, generate pixels corresponding to the registered image pair and a first change image of the target, input the registered image into the sparse window macro model to generate a change area of ​​the sparse window, generate a window area pair with changes corresponding to the registered image pair, perform differential processing on all window area pairs with changes, and generate a second change image block that the window area should correspond to;

[0084] The change data generation module 904 is used to merge the first change image and the second change image block to generate the change result of the registered image pair, and restore the change result to the UAV optoelectronic image through the inverse matrix to obtain the change data on the UAV optoelectronic image.

[0085] In one embodiment, the model training module includes:

[0086] A first labeling unit, used to label the original image, generate a first labeled image, and generate a corresponding first loss value;

[0087] A second labeling unit, used to label the changed image, generate a second labeled image, and generate a corresponding second loss value;

[0088] A third annotation label unit is used to compare the original image and the changed image, obtain the area where the target has been transformed, perform binary marking on the area where the target has been transformed, generate a third annotated image, and generate a corresponding third loss value;

[0089] A loss value calculation unit, used for taking the first loss value, the second loss value and the third loss value as training loss values ​​of a target semantic segmentation model;

[0090] The training set generating unit is used to generate a semantic segmentation training set through the first annotated image, the second annotated image and the third annotated image.

[0091] In one embodiment, the model training module further includes:

[0092] An image segmentation unit is used to set a first window size, a first step length, a second window size, and a second step length, segment the original image according to the first window size and the first step length to generate a first segmented image, and segment the changed image according to the second window size and the second step length to generate a second segmented image;

[0093] The training set generating unit is used to take the first segmented image and the second segmented image as a segmented image pair, determine a marking reference, label the segmented image pair according to the marking reference, and take the labeled segmented image pair as a sparse window training set.

[0094] In one embodiment, the change detection module includes:

[0095] A semantic segmentation unit, used for inputting the first image and the second image in the registered image pair into two identical semantic segmentation models respectively, to generate a first target change image and a second target change image, wherein the first target change image and the second target change image include a target change region and a high-dimensional feature representation of the image;

[0096] The change detection unit is used to input the first target change image and the second target change image into the change detection network to generate a first change image of pixels and targets.

[0097] In one embodiment, the change detection module further includes:

[0098] A feature extraction unit, used for inputting the first image and the second image in the registered image pair into two identical feature extraction networks respectively to generate a first feature change image and a second feature change image;

[0099] A feature comparison unit, used for comparing the first feature change image and the second feature change image to obtain a change feature;

[0100] The change image generating unit is used to input the change feature into the classifier, obtain the window area pairs with changes, perform differential processing on all the window area pairs with changes, and generate a second change image block that the window area should correspond to.

[0101] In one embodiment, the change data generation module includes:

[0102] A first change mask generating unit, configured to generate a first change mask using a first change image;

[0103] A post-processing unit, configured to perform post-processing on the second changed image block to generate a second change mask;

[0104] a merge mask unit, configured to obtain a union of the first change mask and the second change mask to generate a merged change mask;

[0105] The change data generating unit is used to restore the area merged with the change mask to the second image of the registered image pair according to the inverse matrix, and generate the change data on the photoelectric image of the drone.

[0106] In one embodiment, a post-processing unit is used to obtain all second change image blocks, and perform the following processing on each second change image block until all second change image blocks are processed: if the second change image block is a color image, grayscale the second change image block to generate a grayscale change image block; perform differential processing on the grayscale change image block and take the absolute value to generate a differential result, perform binary segmentation on the differential result according to a set threshold value to generate a binary segmented image; set a first window size and a second window size of the filter; use a window of the first window size to perform median filtering on the binary segmented image to generate a first filtering result, and use a window of the second window size to perform median filtering on the first filtering result to generate a second filtering result; use a connected domain in the second filtering result that is different from the background as a second change mask.

[0107] The embodiments of the present invention achieve the following technical effects:

[0108] The target semantic segmentation model and the sparse window macro model are used to detect target-level change areas in UAV optoelectronic images, which can effectively prevent missed detections and false alarms. At the same time, a training set construction method corresponding to the two network models is given in the absence of a training set for the UAV optoelectronic image change detection task. Users can prepare training sets of different target categories and scales according to actual needs. The construction process is simple and feasible, which improves the feasibility of the change detection task of UAV optoelectronic reconnaissance images.

[0109] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting changes in the photoelectric image of an unmanned aerial vehicle, characterized in that: include: Marking targets of interest categories in the optoelectronic images of the UAV, generating optoelectronic image pairs, labeling the optoelectronic image pairs, generating semantic segmentation training sets, training the target semantic segmentation model through the semantic segmentation training sets, labeling the optoelectronic image pairs, generating sparse window training sets, and training the sparse window macro model through the sparse window training sets, wherein the optoelectronic image pairs include original images and changed images; Acquire two UAV photoelectric images in the same target area and at different phases, register the two UAV photoelectric images by a feature point detection and description method, and generate a registered image pair and an inverse matrix of the transformation matrix during deregistration; Input the registered image pair into the target semantic segmentation model to generate pixels corresponding to the registered image pair and a first change image of the target, input the registered image into a sparse window macro model to generate a window area pair with changes corresponding to the registered image pair, perform differential processing on the window area pair with changes, and generate a second change image block that the window area should correspond to; The first change image and the second change image blocks are merged to generate a change result of the registered image pair, and the change result is restored to the UAV photoelectric image through the inverse matrix to obtain change data on the UAV photoelectric image.

2. The method for detecting changes in the photoelectric image of a drone according to claim 1, characterized in that: The first change image and the second change image block are combined to generate a change result of the registered image pair, and the change result is restored to the UAV photoelectric image through the inverse matrix to obtain change data on the UAV photoelectric image, including: generating a first change mask using the first change image; post-processing the second changed image block to generate a second change mask; Taking a union of the first change mask and the second change mask to generate a merged change mask; According to the inverse matrix, the area of ​​the merged change mask is restored to the second image of the registered image pair to generate change data on the optoelectronic image of the drone.

3. The method for detecting changes in the photoelectric image of a drone as claimed in claim 2, characterized in that: Post-processing the second changed image block to generate a second change mask includes: Acquire all the second-changed image blocks, and perform the following processing on each of the second-changed image blocks until all the second-changed image blocks are processed: If the second changed image block is a color image, grayscale the second changed image block to generate a grayscale changed image block; Performing differential processing on the grayscale change image block and taking the absolute value to generate a differential result, performing binary segmentation on the differential result according to a set threshold value to generate a binary segmented image; Set the first window size and the second window size of the filter; Performing median filtering on the binary segmented image using a window of the first window size to generate a first filtering result, and performing median filtering on the first filtering result using a window of the second window size to generate a second filtering result; The connected domain in the second filtering result that is different from the background is used as a second change mask.

4. The method for detecting changes in the photoelectric image of a drone according to claim 1, characterized in that: Label the optoelectronic image pairs to generate a semantic segmentation training set, including: Labeling the original image to generate a first labeled image and a corresponding first loss value; Labeling the changed image to generate a second labeled image and a corresponding second loss value; Comparing the original image with the changed image, obtaining a region where the target has changed, performing binary marking on the region where the target has changed, generating a third annotated image, and generating a corresponding third loss value; Using the first loss value, the second loss value, and the third loss value as training loss values ​​of the target semantic segmentation model; A semantic segmentation training set is generated by using the first annotated image, the second annotated image, and the third annotated image.

5. The method for detecting changes in the photoelectric image of a drone as claimed in claim 1, characterized in that: Label the optoelectronic image pair to generate a sparse window training set, including: Setting a first window size, a first step length, a second window size, and a second step length, segmenting the original image according to the first window size and the first step length to generate a first segmented image, and segmenting the changed image according to the second window size and the second step length to generate a second segmented image; The first segmented image and the second segmented image are used as a segmented image pair, a marking reference is determined, a label is annotated on the segmented image pair according to the marking reference, and the annotated segmented image pair is used as a sparse window training set.

6. The method for detecting changes in the photoelectric image of a drone as claimed in claim 1, characterized in that: Inputting the registered image pair into a target semantic segmentation model to generate pixels corresponding to the registered image pair and a first change image of the target, including: Inputting the first image and the second image in the registered image pair into two identical semantic segmentation models respectively to generate a first target change image and a second target change image, wherein the first target change image and the second target change image include a target change region and a high-dimensional feature representation of the image; The first target change image and the second target change image are input into a change detection network to generate a first change image of pixels and targets.

7. The method for detecting changes in the photoelectric image of a drone as claimed in claim 1, characterized in that: Inputting the registered image into a sparse window macro model, generating a window region pair with changes corresponding to the registered image pair, performing differential processing on the window region pair with changes, and generating a second changed image block corresponding to the window region, including: Inputting the first image and the second image in the registered image pair into two identical feature extraction networks respectively to generate a first feature change image and a second feature change image; comparing the first characteristic change image and the second characteristic change image to obtain a change characteristic; The change feature is input into a classifier to obtain window area pairs with changes, and differential processing is performed on all the window area pairs with changes to generate a second change image block that the window area should correspond to.

8. A device for detecting changes in the photoelectric image of an unmanned aerial vehicle, characterized in that: include: A model training module is used to mark targets of interest in the optoelectronic image of the UAV, generate optoelectronic image pairs, annotate labels on the optoelectronic image pairs, generate semantic segmentation training sets, train the target semantic segmentation model through the semantic segmentation training sets, annotate labels on the optoelectronic image pairs, generate sparse window training sets, and train the sparse window macro model through the sparse window training sets, wherein the optoelectronic image pairs include original images and changed images; An image registration module is used to obtain two UAV optoelectronic images in the same target area and at different phases, register the two UAV optoelectronic images by a feature point detection description method, and generate an image pair after registration and an inverse matrix of the transformation matrix during deregistration; A change detection module, used for inputting the registered image pair into a target semantic segmentation model to generate target segmentation change result data, generating a first change image of pixels and targets corresponding to the registered image pair, inputting the registered image into a sparse window macro model to generate a change region of a sparse window, generating a pair of window regions with changes corresponding to the registered image pair, performing differential processing on all the pairs of window regions with changes, and generating a second change image block that the window region should correspond to; The change data generation module is used to merge the first change image and the second change image blocks to generate the change result of the registered image pair, and restore the change result to the UAV photoelectric image through the inverse matrix to obtain the change data on the UAV photoelectric image.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting changes in the photoelectric image of a drone according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method for detecting changes in the optoelectronic image of a drone according to any one of claims 1 to 7.

Images