Remote sensing image parallax change processing method for detecting garbage scattering area of unmanned aerial vehicle
By combining the feature pyramid module and optical flow alignment module of the end-to-end change detection network model with the progressive difference feature fusion module, the problem of image change processing with large viewpoint differences in the detection of drone debris scattered areas is solved, thereby improving detection accuracy and throughput.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUNNAN NORMAL UNIV
- Filing Date
- 2023-11-08
- Publication Date
- 2026-05-12
AI Technical Summary
In the detection of drone-borne debris in areas where drones are scattered, traditional parallax change detection models struggle to effectively handle image changes with significant differences in viewing angle, resulting in insufficient detection accuracy.
An end-to-end change detection network model based on a feature pyramid module, an optical flow alignment module, and a progressive difference feature fusion module is adopted. By downsampling, feature alignment, and difference mapping, duplicate features are eliminated, thereby improving the accuracy of viewpoint difference detection.
It improves the performance of UAV systems in scenarios with large parallax variations in remote sensing images, reduces accumulated errors and redundant feature extraction, and increases detection throughput.
Smart Images
Figure CN117474885B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method for processing parallax variation in remote sensing images for detecting litter-scattered areas by unmanned aerial vehicles (UAVs). Background Technology
[0002] Small unmanned aerial vehicles (UAVs) play an important role in Earth remote sensing observation, offering advantages such as collaborative technology, ease of operation, high-resolution image acquisition, and flexibility. However, when small UAVs capture images from the same location at different times, factors such as inaccurate GPS (Global Positioning System) signals, changes in flight attitude, wind speed and direction, and complex geographical environments inevitably lead to viewing angle differences in the acquired images.
[0003] To address the problem of change detection in images with viewpoint differences, relevant technical solutions employ weakly supervised semantic scene change detection models. The main approach involves dividing the image into grid cells, extracting features from these grid cells using a convolutional neural network (CNN), calculating the Euclidean distance between the features of each grid cell, and then using the Euclidean distance to determine whether a change has occurred within the grid region.
[0004] However, during the conception and implementation of this application, the inventors discovered that when drones are used for detecting litter areas, and traditional parallax change detection models are employed, image changes with viewing angle differences can only be accurately detected when corresponding pixels are divided into grid cells at the same location. Furthermore, since its ability to process images with viewing angle differences depends on the size of the grid cells, increasing the grid cell size to handle images with large viewing angle differences introduces errors into the change detection area. Therefore, this method is difficult to apply to processing image changes with large viewing angle differences, exhibiting insufficient applicability.
[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0006] The main objective of this invention is to provide a method for processing parallax variation in remote sensing images for detecting garbage dumps by drones, aiming to solve the problem of how to process parallax variation in drone remote sensing images with different viewing angles.
[0007] To achieve the above objectives, the present invention provides a method for processing parallax variation in remote sensing images for detecting garbage dispersion areas using unmanned aerial vehicles (UAVs), the method comprising:
[0008] Based on the feature pyramid module, at least two UAV remote sensing images collected by the UAV from the garbage-scattered area are downsampled to obtain a multi-level feature map set corresponding to each UAV remote sensing image.
[0009] Based on the optical flow alignment module, feature alignment mapping is performed on the multi-level feature maps corresponding to the two matched UAV remote sensing images to obtain a pair of feature-aligned UAV remote sensing image combinations. The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature maps corresponding to the two matched UAV remote sensing images.
[0010] Based on the progressive difference feature fusion and detection module, the feature difference mapping processing of the UAV remote sensing image combination is performed to eliminate the duplicate features in the UAV remote sensing image combination and obtain a multi-level feature difference map after parallax change processing.
[0011] Optionally, the feature pyramid module is built based on the VGG16 network module. The step of downsampling at least two UAV remote sensing images collected by the UAV from the garbage-scattered area based on the feature pyramid module to obtain a multi-level feature map set corresponding to each UAV remote sensing image includes:
[0012] Feature extraction is performed on the UAV remote sensing image to obtain multiple feature maps at different levels, which constitute the multi-level feature map set corresponding to the UAV remote sensing image;
[0013] The number of channels in the feature map located in the next lower layer is twice the number of channels in the feature map located in the adjacent upper layer.
[0014] Optionally, the two matched UAV remote sensing images include a first UAV remote sensing image and a second UAV remote sensing image. The multi-level feature set includes a first multi-level feature set corresponding to the first UAV remote sensing image and a second multi-level feature set corresponding to the second UAV remote sensing image. The step of performing feature alignment mapping on the multi-level feature sets corresponding to the two matched UAV remote sensing images based on the optical flow alignment module to obtain a pair of feature-aligned UAV remote sensing image combinations includes:
[0015] Calculate the local correlation mapping and global correlation mapping between the first multi-level feature map set and the second multi-level feature map set;
[0016] Call the optical flow estimator;
[0017] Based on the global correlation mapping and the optical flow estimator, the global optical flow estimation result of the lowest resolution feature map in the multi-level feature map set is determined, and based on the local correlation mapping and the optical flow estimator, the local optical flow estimation result of other feature maps in the multi-level feature map set other than the lowest resolution feature map is determined. The multi-level feature map set includes the first multi-level feature map set and the second multi-level feature map set.
[0018] Based on the global optical flow estimation results and the local optical flow estimation results, the pixels in the first multi-level feature map set and the second multi-level feature map set are moved by the warping layer to obtain a coarsely aligned first UAV remote sensing image and a coarsely aligned second UAV remote sensing image.
[0019] The coarsely aligned first UAV remote sensing image and the coarsely aligned second UAV remote sensing image are determined as the UAV remote sensing image combination.
[0020] Optionally, the warping layer includes a first type of warping layer and a second type of warping layer. The step of moving the pixels in the first multi-level feature map set and the second multi-level feature map set through the warping layer according to the global optical flow estimation result and the local optical flow estimation result to obtain a coarsely aligned first multi-level feature map set and a coarsely aligned second multi-level feature map set includes:
[0021] By using the first type of warping layer, the feature map of the second multi-level feature map set is distorted to align with the feature map of the first multi-level feature map set, resulting in the coarsely aligned first multi-level feature map set and the coarsely aligned second multi-level feature map set; and / or
[0022] By using the second type of warping layer, the optical flow estimation result of the previous layer is used to warp the feature map of the current layer so that the feature maps of each layer of the second multi-level feature map set are aligned with the feature maps of the corresponding layers in the first multi-level feature map set, thereby obtaining the coarsely aligned first multi-level feature map set and the coarsely aligned second multi-level feature map set. The optical flow estimation result includes the global optical flow estimation result and the local optical flow estimation result.
[0023] The second type of warped layer is applied to the feature maps of the multi-level feature map set, except for the bottom layer feature map.
[0024] Optionally, the UAV remote sensing image assemblage includes a coarsely aligned first UAV remote sensing image and a coarsely aligned second UAV remote sensing image. The step of performing feature difference mapping processing on the UAV remote sensing image assemblage based on the progressive difference feature fusion and detection module to eliminate duplicate features in the UAV remote sensing image assemblage and obtain a multi-level feature difference map after parallax transformation processing includes:
[0025] Determine the feature map between the feature map in the coarsely aligned second UAV remote sensing image and the feature map in the coarsely aligned first UAV remote sensing image, and calculate the absolute value of the feature map difference.
[0026] Upsampling is performed on the feature difference map generated at the previous level to obtain an upsampled feature difference map;
[0027] The absolute value of the difference map and the upsampled feature difference map are fused by an attention mechanism and multi-layer convolution processing to obtain the multi-level feature difference map.
[0028] Optionally, before the step of downsampling at least two UAV remote sensing images collected by the UAV from the garbage-scattered area based on the feature pyramid module to obtain the multi-level feature map set corresponding to each UAV remote sensing image, the method further includes:
[0029] Based on the endpoint error loss function, the optical flow error loss between the feature-aligned UAV remote sensing image combination and the initially acquired UAV remote sensing image is constrained.
[0030] Based on the binary cross-entropy loss function, the feature error between the deep features of the multi-level feature difference map and the input UAV remote sensing image is constrained.
[0031] Optionally, before the step of downsampling at least two UAV remote sensing images collected by the UAV from the garbage-scattered area based on the feature pyramid module to obtain the multi-level feature map set corresponding to each UAV remote sensing image, the method further includes:
[0032] A remote sensing image training dataset is obtained, and the remote sensing image training dataset is enhanced by random flipping to obtain a preprocessed remote sensing image training dataset;
[0033] The garbage-scattered areas in the preprocessed remote sensing image training dataset are identified, and random affine transformations are performed on image pairs in the garbage-scattered areas to enhance viewpoint differences in the preprocessed remote sensing image training dataset.
[0034] The preprocessed remote sensing image training dataset with enhanced viewpoint differences is used to train a preset neural network in the UAV.
[0035] The remote sensing image training dataset includes a change map synthesized from garbage distribution images with viewpoint differences and remote sensing images collected by UAVs, as well as optical flow information corresponding to the garbage distribution images, the remote sensing images, and the change map.
[0036] Furthermore, to achieve the above objectives, the present invention also provides an end-to-end change detection network model, the end-to-end change detection network model comprising:
[0037] The feature pyramid module is used to downsample at least two UAV remote sensing images collected by the UAV from the garbage-scattered area to obtain a multi-level feature map set corresponding to each UAV remote sensing image.
[0038] An optical flow alignment module is used to perform feature alignment mapping on the multi-level feature maps corresponding to two matched UAV remote sensing images to obtain a pair of feature-aligned UAV remote sensing image combinations. The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature maps corresponding to the two matched UAV remote sensing images.
[0039] The progressive difference feature fusion and detection module is used to perform feature difference mapping processing on the combination of UAV remote sensing images to eliminate duplicate features in the combination of UAV remote sensing images and obtain a multi-level feature difference map after parallax transformation processing.
[0040] Furthermore, to achieve the above objectives, the present invention also provides a UAV parallax variation processing system, the UAV parallax variation processing system comprising: a memory, a processor, and a remote sensing image parallax variation processing program for detecting UAV litter areas stored in the memory and executable on the processor, wherein when the remote sensing image parallax variation processing program for detecting UAV litter areas is executed by the processor, the steps of the remote sensing image parallax variation processing method for detecting UAV litter areas as described above are implemented.
[0041] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a parallax variation processing program for remote sensing images of drone-generated debris-scattering areas. When executed by a processor, the parallax variation processing program for remote sensing images of drone-generated debris-scattering areas implements the steps of the parallax variation processing method for remote sensing images of drone-generated debris-scattering areas as described above.
[0042] This invention provides a method, apparatus, and storage medium for processing parallax changes in remote sensing images for detecting litter areas by unmanned aerial vehicles (UAVs). By proposing an end-to-end change detection network model, aligning feature maps using optical flow estimation, and simultaneously performing change detection, the network can directly process images with viewpoint differences. This effectively improves the performance of UAV systems for scenes with accumulated errors or large parallax changes in remote sensing images, and increases throughput by eliminating redundant feature extraction. Attached Figure Description
[0043] Figure 1 This is a schematic diagram of the hardware operating environment of the UAV parallax change processing system according to an embodiment of the present invention;
[0044] Figure 2 This is a flowchart illustrating the first embodiment of the remote sensing image parallax change processing method for detecting garbage scattered areas by unmanned aerial vehicles according to the present invention.
[0045] Figure 3 This is a schematic flowchart illustrating the process involved in the second embodiment of the remote sensing image parallax change processing method for detecting garbage dispersion areas by unmanned aerial vehicles (UAVs) according to the present invention.
[0046] Figure 4 This is another flowchart illustrating a second embodiment of the remote sensing image parallax change processing method for detecting garbage scattered areas by unmanned aerial vehicles (UAVs) according to the present invention.
[0047] Figure 5 This is a schematic flowchart illustrating the process involved in the third embodiment of the remote sensing image parallax change processing method for detecting garbage scattering areas by unmanned aerial vehicles (UAVs) according to the present invention.
[0048] Figure 6 This is a schematic diagram of the architecture of the end-to-end change detection network model involved in the remote sensing image parallax change processing method for detecting garbage scattering areas by drones according to the present invention.
[0049] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0050] This application proposes an end-to-end change detection network model for a UAV parallax change processing system. The end-to-end change detection network model preprocesses at least two UAV remote sensing images with viewpoint differences using a feature pyramid module, improving the network model's ability to handle viewpoint differences in UAV remote sensing images. Then, it aligns feature maps using optical flow estimation, accurately detecting viewpoint differences even if corresponding pixels in multiple UAV remote sensing images are not assigned to the same location. Upon detecting viewpoint differences, it processes images with viewpoint differences through a progressive difference feature fusion and detection module, improving the performance of the UAV system for scenarios with accumulated errors or large parallax changes in remote sensing images. Furthermore, it increases throughput by eliminating redundant feature extraction, avoiding errors introduced by the change detection region and eliminating the dependence on the size of the model's grid cells in traditional visual difference change detection models, thus enabling the processing of image changes with large viewpoint differences.
[0051] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0052] As one implementation scheme, Figure 1 This is a schematic diagram of the hardware operating environment of the UAV parallax change processing system involved in the embodiments of the present invention.
[0053] like Figure 1 As shown, the UAV parallax variation processing system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0054] Those skilled in the art will understand that Figure 1The architecture of the UAV parallax change processing system shown in the figure does not constitute a limitation on the UAV parallax change processing system. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0055] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a remote sensing image parallax change processing program for detecting drone debris dispersion areas. The operating system is a program that manages and controls the hardware and software resources of the drone parallax change processing system, the remote sensing image parallax change processing program for detecting drone debris dispersion areas, and the operation of other software or programs.
[0056] exist Figure 1 In the UAV parallax variation processing system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; the processor 1001 can be used to call the remote sensing image parallax variation processing program for UAV garbage dispersion area detection stored in the memory 1005.
[0057] In this embodiment, the UAV parallax variation processing system includes: a memory 1005, a processor 1001, and a remote sensing image parallax variation processing program for detecting UAV litter dispersion areas, stored in the memory and executable on the processor, wherein:
[0058] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0059] Based on the feature pyramid module, at least two UAV remote sensing images collected by the UAV from the garbage-scattered area are downsampled to obtain a multi-level feature map set corresponding to each UAV remote sensing image.
[0060] Based on the optical flow alignment module, feature alignment mapping is performed on the multi-level feature maps corresponding to the two matched UAV remote sensing images to obtain a pair of feature-aligned UAV remote sensing image combinations. The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature maps corresponding to the two matched UAV remote sensing images.
[0061] Based on the progressive difference feature fusion and detection module, the feature difference mapping processing of the UAV remote sensing image combination is performed to eliminate the duplicate features in the UAV remote sensing image combination and obtain a multi-level feature difference map after parallax change processing.
[0062] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0063] Feature extraction is performed on the UAV remote sensing image to obtain multiple feature maps at different levels, which constitute the multi-level feature map set corresponding to the UAV remote sensing image;
[0064] The number of channels in the feature map located in the next lower layer is twice the number of channels in the feature map located in the adjacent upper layer.
[0065] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0066] Calculate the local correlation mapping and global correlation mapping between the first multi-level feature map set and the second multi-level feature map set;
[0067] Call the optical flow estimator;
[0068] Based on the global correlation mapping and the optical flow estimator, the global optical flow estimation result of the lowest resolution feature map in the multi-level feature map set is determined, and based on the local correlation mapping and the optical flow estimator, the local optical flow estimation result of other feature maps in the multi-level feature map set other than the lowest resolution feature map is determined. The multi-level feature map set includes the first multi-level feature map set and the second multi-level feature map set.
[0069] Based on the global optical flow estimation results and the local optical flow estimation results, the pixels in the first multi-level feature map set and the second multi-level feature map set are moved by the warping layer to obtain a coarsely aligned first UAV remote sensing image and a coarsely aligned second UAV remote sensing image.
[0070] The coarsely aligned first UAV remote sensing image and the coarsely aligned second UAV remote sensing image are determined as the UAV remote sensing image combination.
[0071] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0072] By using the first type of warping layer, the feature map of the second multi-level feature map set is distorted to align with the feature map of the first multi-level feature map set, resulting in the coarsely aligned first multi-level feature map set and the coarsely aligned second multi-level feature map set; and / or
[0073] By using the second type of warping layer, the optical flow estimation result of the previous layer is used to warp the feature map of the current layer so that the feature maps of each layer of the second multi-level feature map set are aligned with the feature maps of the corresponding layers in the first multi-level feature map set, thereby obtaining the coarsely aligned first multi-level feature map set and the coarsely aligned second multi-level feature map set. The optical flow estimation result includes the global optical flow estimation result and the local optical flow estimation result.
[0074] The second type of warped layer is applied to the feature maps of the multi-level feature map set, except for the bottom layer feature map.
[0075] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0076] Determine the feature map between the feature map in the coarsely aligned second UAV remote sensing image and the feature map in the coarsely aligned first UAV remote sensing image, and calculate the absolute value of the feature map difference.
[0077] Upsampling is performed on the feature difference map generated at the previous level to obtain an upsampled feature difference map;
[0078] The absolute value of the difference map and the upsampled feature difference map are fused by an attention mechanism and multi-layer convolution processing to obtain the multi-level feature difference map.
[0079] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0080] Based on the endpoint error loss function, the optical flow error loss between the feature-aligned UAV remote sensing image combination and the initially acquired UAV remote sensing image is constrained.
[0081] Based on the binary cross-entropy loss function, the feature error between the deep features of the multi-level feature difference map and the input UAV remote sensing image is constrained.
[0082] When processor 1001 calls the parallax change processing program for remote sensing images of drone debris dispersion areas stored in memory 1005, it performs the following operations:
[0083] A remote sensing image training dataset is obtained, and the remote sensing image training dataset is enhanced by random flipping to obtain a preprocessed remote sensing image training dataset;
[0084] Determine the litter distribution areas in the preprocessed remote sensing image training dataset, and perform random affine transformation on the image pairs in the litter distribution areas to enhance the viewpoint differences in the preprocessed remote sensing image training dataset.
[0085] The preprocessed remote sensing image training dataset with enhanced viewpoint differences is used to train a preset neural network in the UAV.
[0086] The remote sensing image training dataset includes a change map synthesized from garbage distribution images with viewpoint differences and remote sensing images collected by UAVs, as well as optical flow information corresponding to the garbage distribution images, the remote sensing images, and the change map.
[0087] Based on the hardware architecture of the above-mentioned UAV parallax variation processing system based on computer vision technology, an embodiment of the remote sensing image parallax variation processing method for UAV garbage dispersion area detection of the present invention is proposed.
[0088] First Embodiment
[0089] In this embodiment, the UAV parallax change processing system is equipped with the end-to-end change detection network model proposed in this embodiment. The end-to-end change detection network model mainly includes three modules: feature pyramid module, optical flow alignment module, and progressive difference feature fusion and detection module.
[0090] The main purpose of the feature pyramid module is to downsample UAV remote sensing images to obtain feature maps with richer semantic information, while improving the processing speed of the end-to-end change detection network model, making it easier for subsequent modules to identify changes.
[0091] The main purpose of the optical flow alignment module is to align the multi-level feature maps obtained after processing by the feature pyramid module, thereby avoiding performance degradation of the entire UAV parallax change processing system due to accumulated errors during UAV image acquisition, and avoiding the extraction of too many redundant features that would reduce system throughput.
[0092] Since the image frames in the UAV remote sensing image combination obtained after alignment by the optical flow alignment module are multi-level feature maps with overlapping features between different layers, the progressive difference feature fusion and detection module proposed in this embodiment is used to eliminate the overlapping features between image frames within the combination, thereby generating a multi-level feature difference map after parallax transformation processing, thus completing the parallax transformation processing of the UAV remote sensing image.
[0093] Reference Figure 2 In this embodiment, the method for processing parallax changes in remote sensing images of drone-generated garbage dispersion areas includes the following steps:
[0094] Step S10: Based on the feature pyramid module, downsample at least two UAV remote sensing images collected by the UAV from the garbage-scattered area to obtain a multi-level feature map set corresponding to each UAV remote sensing image.
[0095] In this embodiment, during low-altitude flight of the drone above the litter-scattered area, the monocular camera mounted on the drone captures remote sensing images of the litter-scattered area below, i.e., drone remote sensing images. These drone remote sensing images are then input into the drone's built-in parallax variation processing system to perform parallax variation processing. Since parallax variation only occurs based on two consecutive image frames captured during the drone's flight, we first need to acquire at least two drone remote sensing images.
[0096] In this step, the feature pyramid module downsamples at least two input UAV remote sensing images to obtain multiple multi-level feature maps corresponding to each UAV remote sensing image, thus forming a multi-level feature map set.
[0097] As an optional implementation method, downsampling is specifically as follows:
[0098] First, feature extraction is performed on the UAV remote sensing image to obtain multiple feature maps at different levels, which constitute the multi-level feature map set corresponding to the UAV remote sensing image. Specifically, in the extracted feature maps at different levels, the number of channels in the next lower level feature map is twice the number of channels in the adjacent upper level feature map.
[0099] Optionally, the feature pyramid module can be built using a VGG16 network, which is a VGG (Visual Geometry Group) network with 16 convolutional layers, consisting of 16 convolutional layers and several pooling layers stacked alternately, and finally using a fully connected layer for classification.
[0100] For example, suppose the extracted multi-level feature map has 4 layers, and the initial UAV remote sensing image contains 512 channels. First, in order to reduce the number of network parameters, the number of channels in the initial UAV remote sensing image is adjusted to 256, which is half of the original. Then, 4 different levels of feature maps are extracted from the input image frame. These feature maps encapsulate different semantic information. The sizes of the 4 feature maps are 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image size, respectively, and the number of channels in each feature map are 64, 128, 256 and 256, respectively.
[0101] The multi-level feature maps extracted by the feature pyramid module contain both deep semantic information and shallow, precise location details. The fusion of these deep and shallow feature maps can contain rich semantic information and accurate location information.
[0102] Step S20: Based on the optical flow alignment module, feature alignment mapping is performed on the multi-level feature maps corresponding to the two matched UAV remote sensing images to obtain a pair of feature-aligned UAV remote sensing image combinations. The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature maps corresponding to the two matched UAV remote sensing images.
[0103] In this embodiment, after obtaining the multi-level feature map set, feature alignment mapping is performed on the multi-level feature map set based on the optical flow alignment module. Optical flow is an effective method for establishing a dense correspondence between a pair of images, and the alignment of image feature maps is achieved by establishing a dense correspondence between a pair of image feature maps.
[0104] The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature sets corresponding to the two matched UAV remote sensing images.
[0105] The optical flow estimator consists of a multi-layer convolutional neural network used to estimate pixel displacements between consecutive frames. Its goal is to find the displacement vector of each pixel in adjacent image frames to describe the direction and velocity of an object's movement in the image. In this embodiment, it is used to progressively reduce the number of channels in the input, ultimately generating optical flow information with two channels, representing the horizontal and vertical displacement of the pixel in coordinates, respectively. Optionally, the optical flow estimator can consist of six convolutional layers designed for optical flow calculations, all using 3x3 kernels.
[0106] The warping layer refers to a layer in the end-to-end change detection network model used to perform warping operations. It is responsible for deforming the feature maps of two matching images so that image B, which is the object to be matched, matches image A, which is the object to be matched, thereby achieving alignment between the two feature maps.
[0107] For example, the definition of a warped layer is as follows:
[0108] F bω (x)=F b (x+of(x))
[0109] Where x is the F of the image B being matched. bω The coordinates in the graph are given, and of(x) represents the optical flow estimation result.
[0110] Optionally, in the end-to-end change detection network model proposed in this embodiment, the warping layer can be further divided into two types. The first type uses the estimated optical flow level within the same range to warp the feature map of image B, which is the object to be matched, and aligns it with the feature map of image A, which is the object to be matched. The goal is to enhance the alignment of the feature maps, thereby improving the accuracy of change detection. The second type uses the optical flow estimated by the previous layer to warp the feature map of image B on the current layer, so that the feature map of image B is coarsely aligned with the feature map of image A. This type of warping layer is applicable to all layers except the lowest layer. Its purpose is to achieve coarse alignment of the feature maps, aiming to improve the accuracy of optical flow estimation at the current level. This type uses the optical flow information estimated from the previous level for coarse alignment, obtaining accurate optical flow while reducing computational workload and computation time.
[0111] As an optional implementation, in this embodiment, the feature alignment mapping includes two types of mapping: local correlation mapping and global correlation mapping. Pixel-level correspondences between feature maps are constructed using these two mappings with different ranges. Then, a pre-set optical flow estimator in the optical flow alignment module is invoked, and the optical flow estimation result is determined based on the optical flow estimator and the two mappings. The optical flow estimation result also includes the global optical flow estimation result and the local optical flow estimation result. Feature alignment of the multi-level feature map set is performed using a pre-set warp layer in the optical flow alignment module, along with the global and local optical flow estimation results.
[0112] Since feature alignment is an action between two matched UAV remote sensing images, for ease of description, the two matched UAV remote sensing images are referred to as the first UAV remote sensing image and the second UAV remote sensing image. The multi-level feature set corresponding to the first UAV remote sensing image is called the first multi-level feature set, and the multi-level feature set corresponding to the second UAV remote sensing image is called the second multi-level feature set.
[0113] The process will be explained in detail below:
[0114] First, calculate the local correlation mapping and global correlation mapping between the first multi-level feature map set and the second multi-level feature map set.
[0115] In this process, the difference between local correlation mapping and global correlation mapping lies in the different calculation ranges. For example, local correlation mapping calculates the correspondence within a specified distance 'd' around the x-coordinate, while global correlation mapping calculates the correspondence between pixel "x" and all pixels in the same image.
[0116] For example, the relevant mapping is defined as follows:
[0117] c(x1, x2) = F a (x1) T Fb (x2)
[0118] Where x1 represents the first multi-level feature map set F a In the coordinates, x2 represents the second multi-level feature map set F. b The coordinates in the diagram.
[0119] In this context, the local correlation mapping is |x1-x2|≤d, where d is a specified distance. Optionally, d can be set to 1. The global correlation mapping, however, does not specify a distance.
[0120] Second, the optical flow estimator is invoked. Then, based on the global correlation mapping and the optical flow estimator, the global optical flow estimation result of the lowest resolution feature map in the multi-level feature map set is determined, and the local optical flow estimation result of the other feature maps in the multi-level feature map set, excluding the lowest resolution feature map, is determined based on the local correlation mapping and the optical flow estimator. Note that the multi-level feature map set here includes the first multi-level feature map set and the second multi-level feature map set.
[0121] It should be noted that, in this process, estimating optical flow using global correlation mapping on high-resolution feature maps is a computationally intensive and time-consuming process. Therefore, we only utilize global correlation to estimate optical flow on the lowest-resolution feature maps. For the alignment of feature maps at other levels, local correlation mapping is used to estimate optical flow.
[0122] It should also be noted that the determination of the optical flow estimation result is no different from the conventional method of using an optical flow estimator for optical flow estimation. The difference is that the optical flow estimation result in this embodiment is divided into different ranges (i.e., global or local), and different optical flow estimation methods are used for feature maps of different resolutions in the multi-level feature map set. This ensures the accuracy of the estimation result as much as possible while reducing the computational cost of the optical flow estimation module in the system.
[0123] Third, based on the global optical flow estimation results and the local optical flow estimation results, the pixels in the first multi-level feature map set and the second multi-level feature map set are moved by the warping layer to obtain the coarsely aligned first UAV remote sensing image and the coarsely aligned second UAV remote sensing image.
[0124] In this process, global optical flow estimation is used to estimate pixel displacement between image frames, thereby determining the flight direction and speed of objects in the UAV remote sensing image. Local optical flow estimation is used to estimate more detailed displacement information in local areas of the image, thus capturing subtle changes and relative motion between objects. A warping layer is used to move pixels in two image frames based on the two optical flow estimation results to achieve coarse alignment.
[0125] Fourth, the coarsely aligned first UAV remote sensing image and the coarsely aligned second UAV remote sensing image are determined as the UAV remote sensing image combination.
[0126] Two coarsely aligned UAV remote sensing images, namely the first coarsely aligned UAV remote sensing image and the second coarsely aligned UAV remote sensing image, are combined to form a UAV remote sensing image ensemble. In subsequent steps, the parallax transformation processing is carried out using the UAV remote sensing image ensemble as the unit.
[0127] Step S30: Based on the progressive difference feature fusion and detection module, the feature difference mapping processing of the UAV remote sensing image combination is performed to eliminate the duplicate features in the UAV remote sensing image combination and obtain a multi-level feature difference map after parallax change processing.
[0128] In this embodiment, after obtaining the combination of UAV remote sensing images, the UAV remote sensing image combination feature difference mapping is processed based on the progressive difference feature fusion and detection module to eliminate the duplicate features in the UAV remote sensing image combination and obtain a multi-level feature difference map after parallax change processing.
[0129] As an optional implementation, the feature difference mapping process is performed as follows: First, the feature map in the coarsely aligned second UAV remote sensing image is determined, and the feature difference map between it and the feature map in the coarsely aligned first UAV remote sensing image is calculated. The feature difference map can be obtained by subtracting the feature maps of corresponding levels between the coarsely aligned second UAV remote sensing image and the coarsely aligned first UAV remote sensing image. The absolute value of the feature difference map is then obtained.
[0130] Then, the feature difference map generated at the previous level is upsampled to obtain an upsampled feature difference map. The purpose of this step is to improve the resolution of the feature difference map. Optionally, the upsampling process can be performed using nearest neighbor interpolation or bilinear interpolation.
[0131] Finally, the absolute value of the difference map and the upsampled feature difference map are fused through attention mechanism and multi-layer convolution processing to obtain a multi-level feature difference map.
[0132] It should be noted that the role of attention mechanisms is to guide the model's focus towards locally important information. Attention mechanisms include channel attention mechanisms and spatial attention mechanisms.
[0133] Channel attention mechanism refers to learning a channel attention graph to enhance or suppress different channels, enabling the network model to combine different channels and improve the scalability of the end-to-end change detection network model.
[0134] It is important to note that feature difference maps generated at different levels contain different information. Shallow layers produce feature difference maps with fine location details and object-specific features, making them particularly effective for detecting small objects. Conversely, feature difference maps from deeper layers have rich semantic information, making them more suitable for detecting larger targets. To leverage the advantages of these different levels, this embodiment uses a channel attention mechanism to fuse their feature difference maps, which helps reduce differences in semantic and location information across different channels.
[0135] For example, the channel attention graph is defined as follows:
[0136] M c =σ(MLP(Avgpool(F))+MLP(Maxpool(F))
[0137] Where σ represents the sigmoid function, F represents the fused multi-level feature difference map, and MLP represents the multilayer perceptron used to learn channel weights to determine the importance of each channel in the channel attention mechanism.
[0138] Spatial attention mechanisms target individual pixels in a feature image. By learning a spatial attention map, specific pixels in the feature map can be enhanced or suppressed, allowing end-to-end change detection networks to focus on key pixels of interest. However, these constantly changing key pixels typically constitute only a small portion of the entire spatial attention map. Therefore, this embodiment employs a spatial attention mechanism to learn the spatial attention map, enabling the model to allocate more attention to pixels that have undergone changes, thus improving the model's sensitivity to the most important pixels in the change detection task.
[0139] For example, the spatial attention diagram is defined as follows:
[0140] M s =σ(f 7*7 ([Avgpool(F);MLP(Maxpool(F)]))
[0141] Among them, f 7*7 This represents a convolution operator with a filter size of 7x7. F represents the fused multi-level feature difference map.
[0142] In the technical solution provided in this embodiment, an end-to-end change detection network model is proposed and integrated into a UAV parallax change processing system. This model preprocesses at least two UAV remote sensing images with viewing angle differences using a feature pyramid module, enhancing the network model's ability to handle these differences. Then, optical flow estimation is used to align feature maps, accurately detecting viewing angle differences even if corresponding pixels in multiple UAV remote sensing images are not assigned to the same location. Upon detecting viewing angle differences, a progressive difference feature fusion and detection module processes the images with these differences, improving the performance of the UAV system for scenarios with accumulated errors or large parallax changes in remote sensing images. Furthermore, by eliminating redundant feature extraction, throughput is increased, avoiding errors introduced by the change detection region and eliminating the dependence on the size of the model's grid cells in traditional visual difference change detection models. This enables the processing of image changes with significant viewing angle differences.
[0143] Second Embodiment
[0144] Reference Figure 3 Based on the first embodiment, before step S10, the method further includes:
[0145] Step S40: Based on the endpoint error loss function, constrain the optical flow error loss between the feature-aligned UAV remote sensing image combination and the initially acquired UAV remote sensing image.
[0146] As an optional embodiment, in order to improve the end-to-end change detection network model's ability to process images with large viewpoint differences, the endpoint error loss function (EPE) is used as a monitoring mechanism to constrain the optical flow error loss between the feature-aligned UAV remote sensing image combination and the initially acquired UAV remote sensing image, ensuring that the end-to-end change detection network model can accurately predict optical flow.
[0147] In this embodiment, the endpoint error loss function is used to calculate the Euclidean distance between the predicted optical flow and the actual ground optical flow. For example, the endpoint error loss function is defined as follows:
[0148]
[0149] in, This represents the l-th level predicted optical flow predicted by the end-to-end change detection network model. This is the actual ground optical flow, which serves as a reference value. The endpoint error loss weighting parameter.
[0150] Reference Figure 4 Based on any of the foregoing embodiments, before step S10, the method further includes:
[0151] Step S50: Based on the binary cross-entropy loss function, constrain the feature error between the deep features of the multi-level feature difference map and the input UAV remote sensing image.
[0152] Furthermore, in this embodiment, the final change map is obtained by combining multi-level feature difference maps generated by change detection networks at different levels. The quality of the feature difference maps at each level directly affects the overall performance of the final change map, and the effectiveness of the feature difference map at the next level is also affected by the feature difference map at the previous level. Therefore, errors occurring at a certain level in the deep feature difference map will amplify errors in subsequent levels of feature difference maps, leading to the gradient vanishing problem during backpropagation, which poses a challenge to the training process. To avoid poor performance due to accumulated errors during long-term operation of the end-to-end change detection network model, this embodiment implements a deep supervision strategy based on the binary cross-entropy (BCE) loss function within the network. At each level of the change detection network, the generated feature difference maps are processed to obtain the final change map.
[0153] For example, the binary cross-entropy loss function is defined as follows:
[0154]
[0155] Among them, GT l (x) refers to the label of the l-th pixel x, p l (x) refers to the network's prediction of the l-th pixel x. The weight parameters for the binary cross-entropy loss are denoted as .
[0156] Finally, in this embodiment, the total loss of the entire training process is defined as the sum of the endpoint error loss and the binary cross-entropy loss:
[0157] L 总 =L epe +L bce
[0158] In the technical solution provided in this embodiment, in order to improve the processing capability of the end-to-end change detection network model for images with large viewpoint differences, and to avoid poor performance of the end-to-end change detection network model due to accumulated errors during long-term operation, the endpoint error loss function (EPE) is used as a monitoring mechanism to constrain the optical flow error loss between the feature-aligned UAV remote sensing image combination and the initially acquired UAV remote sensing image. Furthermore, the binary cross-entropy loss function is used as a deep supervision strategy to constrain the feature error between the deep features of the multi-level feature difference map and the input UAV remote sensing image.
[0159] Third Embodiment
[0160] Reference Figure 5 Based on any embodiment, before step S10, the method further includes:
[0161] Step S60: Obtain the remote sensing image training dataset, and enhance the remote sensing image training dataset by random flipping to obtain a preprocessed remote sensing image training dataset;
[0162] Step S70: Determine the litter distribution area in the preprocessed remote sensing image training dataset, and perform random affine transformation on the image pairs in the litter distribution area to enhance the viewpoint differences in the preprocessed remote sensing image training dataset.
[0163] Step S80: Based on the preprocessed remote sensing image training dataset with enhanced viewpoint differences, train the end-to-end change detection network model in the UAV.
[0164] As an optional embodiment, to improve the prediction accuracy of the end-to-end change detection network model, it is necessary to train the change detection network model with a large number of training samples. However, currently, no training dataset for change detection of viewpoint differences is publicly available. Therefore, this embodiment proposes a training dataset that can train the model's viewpoint difference detection capability and optical flow supervision capability.
[0165] Most publicly available change detection datasets typically contain an image A as the matching object, an image B as the object being matched, and a fused change map of images A and B. However, when image A or image B undergoes affine transformations such as rotation or translation, the accuracy of the fused change map may be affected because the positions of objects in the images change. This can negatively impact model performance. This limitation makes most publicly available change detection datasets unsuitable for creating change detection datasets with viewpoint differences.
[0166] In this embodiment, the remote sensing image training dataset includes a change map synthesized from a garbage distribution image with viewpoint differences and a remote sensing image collected by a drone, as well as optical flow information corresponding to the garbage distribution image, the remote sensing image, and the change map.
[0167] Specifically, a large number of remote sensing images collected by drones are first gathered to form a remote sensing image training dataset. This dataset is then enhanced by random flipping to obtain a preprocessed remote sensing image training dataset. The purpose of random flipping is to enhance the diversity and robustness of the data.
[0168] For example,
[0169] Then, the litter distribution regions in the preprocessed remote sensing image training dataset are determined. The litter distribution region refers to the area in the image where the litter of interest is located; this region can be determined using common convolutional neural network algorithms.
[0170] In some specific implementation scenarios, drones can be used for detecting littered areas. This involves using drones to collect remote sensing images during low-altitude flight and then identifying littered areas within those images. When applying drones to detect littered areas, a litter distribution image is generated by randomly combining images of various materials (including but not limited to plastics, metals, and glass).
[0171] It should be noted that in real-world scenarios, a pair of images taken by a drone at the same location but at different times will naturally show a difference in viewpoint. To replicate this viewpoint difference in the simulated dataset, a random affine transformation is applied to the image portion corresponding to the litter-scattered area to enhance the viewpoint differences between the images in the dataset.
[0172] Furthermore, to improve the optical flow alignment capability of the optical flow detection module in the end-to-end change detection network model, optical flow data required for aligning these paired images was also included in the dataset. That is, the remote sensing image training dataset also needs to include a change map synthesized from a garbage distribution image with viewpoint differences and a remote sensing image acquired by a drone, as well as the optical flow information corresponding to the garbage distribution image, the remote sensing image, and the change map.
[0173] Finally, based on the preprocessed remote sensing image training dataset enhanced with viewpoint differences, an end-to-end change detection network model for UAVs was trained.
[0174] In the technical solution provided in this embodiment, a training dataset is proposed that can train the model's viewpoint difference detection capability and optical flow supervision capability, and the construction process of the training dataset is given, thereby improving the prediction accuracy of the end-to-end change detection network model.
[0175] In addition, refer to Figure 6 This embodiment also proposes an end-to-end change detection network model, which includes:
[0176] The feature pyramid module 100 is used to downsample at least two UAV remote sensing images collected by the UAV from the garbage-scattered area to obtain a multi-level feature map set corresponding to each UAV remote sensing image.
[0177] The optical flow alignment module 200 is used to perform feature alignment mapping on the multi-level feature maps corresponding to two matched UAV remote sensing images to obtain a pair of feature-aligned UAV remote sensing image combinations. The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature maps corresponding to the two matched UAV remote sensing images.
[0178] The progressive difference feature fusion and detection module 300 is used to perform feature difference mapping processing on the combination of UAV remote sensing images to eliminate duplicate features in the combination of UAV remote sensing images and obtain a multi-level feature difference map after parallax change processing.
[0179] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the UAV parallax change processing system to implement the process steps of the embodiments of the above methods.
[0180] Therefore, the present invention also provides a computer-readable storage medium storing a parallax variation processing program for remote sensing images of drone-generated debris-scattering areas. When executed by a processor, the parallax variation processing program for remote sensing images of drone-generated debris-scattering areas implements the various steps of the parallax variation processing method for remote sensing images of drone-generated debris-scattering areas as described in the above embodiments.
[0181] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0182] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.
[0183] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0184] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0187] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0188] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0189] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for processing parallax variation in remote sensing images used for detecting litter dispersion areas by unmanned aerial vehicles (UAVs), characterized in that, An application is made to a parallax variation processing system for unmanned aerial vehicles (UAVs). This system is equipped with an end-to-end variation detection network model. The method for processing parallax variation in remote sensing images of UAV-detected litter areas includes the following steps: Based on the feature pyramid module, at least two UAV remote sensing images collected by the UAV from the garbage-scattered area are downsampled to obtain a multi-level feature map set corresponding to each UAV remote sensing image. Based on the optical flow alignment module, feature alignment mapping is performed on the multi-level feature maps corresponding to the two matched UAV remote sensing images to obtain a pair of feature-aligned UAV remote sensing image combinations. The optical flow alignment module includes an optical flow estimator for optical flow estimation and a warping layer for feature alignment between the multi-level feature maps corresponding to the two matched UAV remote sensing images. The two matched UAV remote sensing images include a first UAV remote sensing image and a second UAV remote sensing image. The multi-level feature set includes a first multi-level feature set corresponding to the first UAV remote sensing image and a second multi-level feature set corresponding to the second UAV remote sensing image. The step of performing feature alignment mapping on the multi-level feature sets corresponding to the two matched UAV remote sensing images based on the optical flow alignment module to obtain a pair of feature-aligned UAV remote sensing image combinations includes: Calculate the local correlation mapping and global correlation mapping between the first multi-level feature map set and the second multi-level feature map set; Call the optical flow estimator; Based on the global correlation mapping and the optical flow estimator, the global optical flow estimation result of the lowest resolution feature map in the multi-level feature map set is determined, and based on the local correlation mapping and the optical flow estimator, the local optical flow estimation result of other feature maps in the multi-level feature map set other than the lowest resolution feature map is determined. The multi-level feature map set includes the first multi-level feature map set and the second multi-level feature map set. Based on the global optical flow estimation results and the local optical flow estimation results, the pixels in the first multi-level feature map set and the second multi-level feature map set are moved by the warping layer to obtain a coarsely aligned first UAV remote sensing image and a coarsely aligned second UAV remote sensing image. The coarsely aligned first UAV remote sensing image and the coarsely aligned second UAV remote sensing image are determined as the UAV remote sensing image combination; Based on the progressive difference feature fusion and detection module, the feature difference mapping processing of the UAV remote sensing image combination is performed to eliminate the duplicate features in the UAV remote sensing image combination and obtain a multi-level feature difference map after parallax change processing.
2. The method as described in claim 1, characterized in that, The feature pyramid module is built based on the VGG16 network module. The step of downsampling at least two UAV remote sensing images collected by the UAV from the garbage-scattered area based on the feature pyramid module to obtain the multi-level feature map set corresponding to each UAV remote sensing image includes: Feature extraction is performed on the UAV remote sensing image to obtain multiple feature maps at different levels, which constitute the multi-level feature map set corresponding to the UAV remote sensing image; The number of channels in the feature map located in the next lower layer is twice the number of channels in the feature map located in the adjacent upper layer.
3. The method as described in claim 1, characterized in that, The warping layer includes a first type of warping layer and a second type of warping layer. The step of moving the pixels in the first multi-level feature map set and the second multi-level feature map set through the warping layer according to the global optical flow estimation result and the local optical flow estimation result to obtain a coarsely aligned first multi-level feature map set and a coarsely aligned second multi-level feature map set includes: By using the first type of warping layer, the feature map of the second multi-level feature map set is distorted to align with the feature map of the first multi-level feature map set, resulting in the coarsely aligned first multi-level feature map set and the coarsely aligned second multi-level feature map set; and / or By using the second type of warping layer, the optical flow estimation result of the previous layer is used to warp the feature map of the current layer so that the feature maps of each layer of the second multi-level feature map set are aligned with the feature maps of the corresponding layers in the first multi-level feature map set, thereby obtaining the coarsely aligned first multi-level feature map set and the coarsely aligned second multi-level feature map set. The optical flow estimation result includes the global optical flow estimation result and the local optical flow estimation result. The second type of warped layer is applied to the feature maps of the multi-level feature map set, except for the bottom layer feature map.
4. The method as described in claim 1, characterized in that, The UAV remote sensing image assemblage includes a coarsely aligned first UAV remote sensing image and a coarsely aligned second UAV remote sensing image. The step of performing feature difference mapping processing on the UAV remote sensing image assemblage based on the progressive difference feature fusion and detection module to eliminate duplicate features in the UAV remote sensing image assemblage and obtain a multi-level feature difference map after disparity transformation processing includes: Determine the feature map between the feature map in the coarsely aligned second UAV remote sensing image and the feature map in the coarsely aligned first UAV remote sensing image, and calculate the absolute value of the feature map difference. Upsampling is performed on the feature difference map generated at the previous level to obtain an upsampled feature difference map; The absolute value of the difference map and the upsampled feature difference map are fused by an attention mechanism and multi-layer convolution processing to obtain the multi-level feature difference map.
5. The method as described in claim 1, characterized in that, Before the step of downsampling at least two UAV remote sensing images collected by the UAV from the garbage-scattered area based on the feature pyramid module to obtain the multi-level feature map sets corresponding to each UAV remote sensing image, the method further includes: Based on the endpoint error loss function, the optical flow error loss between the feature-aligned UAV remote sensing image combination and the initially acquired UAV remote sensing image is constrained. Based on the binary cross-entropy loss function, the feature error between the deep features of the multi-level feature difference map and the input UAV remote sensing image is constrained.
6. The method as described in claim 1, characterized in that, Before the step of downsampling at least two UAV remote sensing images collected by the UAV from the garbage-scattered area based on the feature pyramid module to obtain the multi-level feature map sets corresponding to each UAV remote sensing image, the method further includes: A remote sensing image training dataset is obtained, and the remote sensing image training dataset is enhanced by random flipping to obtain a preprocessed remote sensing image training dataset; Determine the litter distribution areas in the preprocessed remote sensing image training dataset, and perform random affine transformation on the image pairs in the litter distribution areas to enhance the viewpoint differences in the preprocessed remote sensing image training dataset. The preprocessed remote sensing image training dataset with enhanced viewpoint differences is used to train a preset neural network in the UAV. The remote sensing image training dataset includes a change map synthesized from garbage distribution images with viewpoint differences and remote sensing images collected by UAVs, as well as optical flow information corresponding to the garbage distribution images, the remote sensing images, and the change map.
7. A parallax variation processing system for unmanned aerial vehicles (UAVs), characterized in that, The UAV parallax change processing system includes: a memory, a processor, and a remote sensing image parallax change processing program for detecting UAV litter areas stored in the memory and executable on the processor. When the remote sensing image parallax change processing program for detecting UAV litter areas is executed by the processor, it implements the steps of the remote sensing image parallax change processing method for detecting UAV litter areas as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a parallax variation processing program for remote sensing images used for detecting litter areas by drones. When the remote sensing image parallax variation processing program for detecting litter areas by drones is executed by a processor, it implements the steps of the remote sensing image parallax variation processing method for detecting litter areas by drones as described in any one of claims 1 to 6.