Remote sensing image processing method and device

Abstract

Embodiments of the present application provide a remote sensing image processing method and device. The method comprises: inputting a training remote sensing image into a generative adversarial network (GAN) to output a plurality of generated remote sensing images corresponding to the training remote sensing image; inputting the training remote sensing image and the plurality of generated remote sensing images into an initial YOLO model for training to obtain a target detection model; inputting target information of a training set detected from the initial YOLO model into a time fusion transformer (TFT) model as a time series data set for training to obtain a future time series prediction model; inputting a to-be-detected remote sensing image into the target detection model for target detection to output target information of the to-be-detected remote sensing image; and inputting the target information of the to-be-detected remote sensing image into the future time series prediction model for prediction to output a target state prediction of a future time step of the to-be-detected remote sensing image. The method and device of the embodiments of the present application can optimize the current detection result and reduce the problem of missed detection and false detection.

Description

Technical Field

[0001] This application relates to the field of remote sensing technology, and more particularly to a remote sensing image processing method and apparatus. Background Technology

[0002] With the rapid development of remote sensing technology, remote sensing image data has experienced explosive growth, playing a vital role in many fields such as land and resources monitoring, urban planning, military reconnaissance, agricultural assessment, and environmental monitoring.

[0003] In the application of remote sensing images, target detection has become a crucial task. Traditionally, target detection in remote sensing images relies primarily on manual visual interpretation, where professionals identify and label targets by directly observing the images based on their experience and knowledge. This method was adequate in the early stages when data volume was relatively small, target features were relatively obvious, and timeliness requirements were not high. However, with the continuous advancement of remote sensing technology, the resolution of acquired images has increased significantly, the coverage area has expanded broadly, and the data volume has grown exponentially. Faced with massive amounts of remote sensing image data, manual visual interpretation faces enormous challenges.

[0004] In recent years, object detection algorithms based on computer vision technology have been widely researched and applied, bringing new opportunities for object detection in remote sensing images. Deep learning algorithms, such as Convolutional Neural Networks (CNNs) and their variants, have achieved remarkable results in the field of object detection in natural images. They can automatically learn feature representations in images and have powerful feature extraction and classification capabilities. However, directly applying these object detection algorithms suitable for natural images to remote sensing images still faces many problems. For example, existing remote sensing image object detection technologies mainly rely on single-frame images for object recognition, failing to fully utilize the temporal dimension information of time-series images. This results in inaccurate capture of dynamic changes in targets within continuous image sequences, easily leading to missed detections or false detections. Summary of the Invention

[0005] In view of this, embodiments of this application provide a remote sensing image processing method and apparatus, which, on the one hand, can overcome the problems of scarce remote sensing data and insufficient utilization of time series information by single-frame detection methods. On the other hand, it can also integrate the time patterns and trends in historical detection results to generate accurate predictions of the future state of the target, thereby optimizing the current detection results and reducing the problems of missed detections and false detections.

[0006] In a first aspect, a remote sensing image processing method is provided, comprising: inputting a training remote sensing image into a generative adversarial network (GAN) to output a multi-frame generated remote sensing image corresponding to the training remote sensing image; inputting the training remote sensing image and the multi-frame generated remote sensing image as a training set into an initial YOLO model for training to obtain a target detection model; inputting the target information of the training set detected from the initial YOLO model as a time series dataset into a time fusion converter (TFT) model for training to obtain a future time series prediction model; inputting a remote sensing image to be detected into the target detection model for target detection and outputting the target information of the remote sensing image to be detected; inputting the target information of the remote sensing image to be detected into the future time series prediction model for prediction and outputting the target state prediction of the remote sensing image at a future time step.

[0007] In one possible implementation, the step of inputting the remote sensing image to be detected into the target detection model for target detection and outputting the target information of the remote sensing image to be detected includes: processing the remote sensing image to be detected into an input image suitable for the target detection model; extracting features from the input image to obtain multiple feature vectors; extracting time series information from the multiple feature vectors; adjusting the detection result of the target detection model based on the time series information, and then outputting the target information of the remote sensing image to be detected.

[0008] In one possible implementation, processing the temporal features of the multiple feature vectors to obtain time series information includes: using a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model to process the temporal features of the multiple feature vectors to obtain the time series information.

[0009] In one possible implementation, the target state prediction for the future time step includes the target category, location, and trend of change for the future time step.

[0010] In one possible implementation, the Generative Adversarial Network (GAN), the YOLO model, and the Time Fusion Converter (TFT) model are integrated into one unit.

[0011] Secondly, a remote sensing image processing apparatus is provided, comprising: a generative adversarial network (GAN) for receiving training remote sensing images and outputting multi-frame generated remote sensing images corresponding to the training remote sensing images; a YOLO model for receiving the training remote sensing images and the multi-frame generated remote sensing images as a training set for training to obtain a target detection model, wherein the target detection model receives a remote sensing image to be detected for target detection and outputs target information of the remote sensing image to be detected; and a time fusion converter (TFT) model for receiving the target information of the training set detected from the initial YOLO model as a time series dataset for training to obtain a future time series prediction model, wherein the future time series prediction model receives the target information of the remote sensing image to be detected for prediction and outputs a target state prediction for the future time step of the remote sensing image to be detected.

[0012] In one possible implementation, the target detection model is specifically used to: process the remote sensing image to be detected into an input image suitable for the target detection model; extract features from the input image to obtain multiple feature vectors; extract time series information from the multiple feature vectors; adjust the detection result of the target detection model based on the time series information, and then output the target information of the remote sensing image to be detected.

[0013] In one possible implementation, the target detection model includes a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model, used to process the temporal features of the multiple feature vectors to obtain the time series information.

[0014] In one possible implementation, the target state prediction for the future time step includes the target category, location, and trend of change for the future time step.

[0015] In one possible implementation, the Generative Adversarial Network (GAN), the YOLO model, and the Time Fusion Converter (TFT) model are integrated into one unit.

[0016] Thirdly, a remote sensing image processing apparatus is provided, comprising: a processor, and a memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute the remote sensing image processing method of the first aspect and any possible implementation thereof by calling the program instructions.

[0017] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform a remote sensing image processing method as described in the first aspect and any possible implementation thereof.

[0018] Based on the above technical solutions, high-quality time-series remote sensing data with temporal consistency is generated through Generative Adversarial Networks (GANs), significantly expanding the size of the original training dataset, effectively alleviating the data scarcity problem, and improving the model's training performance and generalization ability in dynamic scenes. Secondly, in addition to utilizing the YOLO (You Only Look Once) model for efficient real-time target detection, a Temporal Fusion Transformer (TFT) model is used to deeply analyze the time-series features of the detection results, predicting the potential future trends of the target. The TFT model can integrate the temporal patterns and trends in historical detection results to generate accurate predictions of the target's future state, thereby optimizing the current detection results and reducing missed and false detections. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0020] Figure 1 A schematic block diagram of a remote sensing image processing method according to an embodiment of this application is shown.

[0021] Figure 2 A schematic diagram of a generative adversarial network model according to an embodiment of this application is shown.

[0022] Figure 3 A schematic diagram of the YOLO model according to an embodiment of this application is shown.

[0023] Figure 4 A schematic block diagram of another remote sensing image processing method according to an embodiment of this application is shown.

[0024] Figure 5 A schematic diagram of a TFT model according to an embodiment of this application is shown.

[0025] Figure 6 A schematic block diagram of a remote sensing image processing apparatus according to an embodiment of this application is shown. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] With the rapid development of remote sensing technology, remote sensing image data has experienced explosive growth, playing a vital role in many fields such as land and resources monitoring, urban planning, military reconnaissance, agricultural assessment, and environmental monitoring.

[0028] In the application of remote sensing images, target detection has become a crucial task. Traditionally, target detection in remote sensing images relies primarily on manual visual interpretation, where professionals identify and label targets by directly observing the images based on their experience and knowledge. This method was adequate in the early stages when data volume was relatively small, target features were relatively obvious, and timeliness requirements were not high. However, with the continuous advancement of remote sensing technology, the resolution of acquired images has increased significantly, the coverage area has expanded broadly, and the data volume has grown exponentially. Faced with massive amounts of remote sensing image data, manual visual interpretation faces enormous challenges.

[0029] In recent years, object detection algorithms based on computer vision technology have been widely researched and applied, bringing new opportunities for object detection in remote sensing images. Existing technologies for object detection in remote sensing images are mainly based on the CNN framework, identifying and locating targets by extracting spatial features from the image. Mainstream methods include models such as Faster R-CNN, Mask R-CNN, and YOLO, which typically follow a process of feature extraction, target localization, and classification. First, the image is input into a pre-trained CNN to extract multi-scale features. Subsequently, a Region Proposal Network (RPN) or direct regression (such as YOLO) is used to detect the bounding box and category of the target. Faster R-CNN employs a two-stage detection process, first generating candidate regions and then performing classification and regression, while models like YOLO use a single-stage detection process, predicting the target category and location simultaneously through a single forward propagation, offering higher real-time performance and efficiency. In detection tasks, data augmentation techniques are often combined with image preprocessing, such as scaling, flipping, and cropping, to improve the robustness of the model. The model is trained using a pre-labeled image dataset, with annotations including target category labels and bounding box locations. During training, cross-entropy loss and bounding box regression loss are used to optimize the model's predictive performance, ensuring accurate target identification and localization during detection. The trained model can be applied to target detection in remote sensing image data for fields such as environmental monitoring, disaster management, and resource exploration.

[0030] However, existing remote sensing image target detection technologies primarily rely on single-frame images for target recognition, failing to fully utilize the temporal dimension information of time-series images. This results in inaccurate capture of dynamic changes in targets within continuous image sequences, easily leading to missed or false detections. Furthermore, the acquisition cost of remote sensing data is high, especially within specific time periods and regions, where data scarcity is significant, limiting the training effectiveness and generalization ability of models. Additionally, existing temporal analysis models suffer from information loss when handling long-term dependencies, failing to effectively model long-term temporal dependencies in remote sensing images. These technical shortcomings lead to suboptimal performance of detection systems in complex scenarios, making it difficult to meet the requirements for high accuracy and robustness. Moreover, these existing background technologies often only detect the current target and cannot predict future trends of the target.

[0031] Therefore, this application provides a remote sensing image processing method that generates high-quality time-series remote sensing data with temporal consistency through a Generative Adversarial Network (GAN), significantly expanding the size of the original training dataset, effectively alleviating the data scarcity problem, and improving the model's training performance and generalization ability in dynamic scenes. Secondly, it not only utilizes the YOLO model to achieve efficient real-time target detection, but also uses a TFT model to perform in-depth analysis of the time-series features of the detection results, predicting the possible future trends of the target. The TFT model can integrate the temporal patterns and trends in historical detection results to generate accurate predictions of the target's future state, thereby optimizing the current detection results and reducing missed and false detections.

[0032] Figure 1 A schematic block diagram illustrating a remote sensing image processing method according to an embodiment of this application is shown. Optionally, this remote sensing image processing method can be executed by a processor, such as any one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or Neural Processing Unit (NPU). Figure 1 As shown, the remote sensing image processing method 100 includes some or all of the following contents.

[0033] S110, the training remote sensing images are input into the generative adversarial network (GAN), and the multi-frame generated remote sensing images corresponding to the training remote sensing images are output.

[0034] S120, the training remote sensing image and the multi-frame generated remote sensing image are used as training set inputs into the initial YOLO model for training to obtain the target detection model.

[0035] S130, the target information of the training set detected from the initial YOLO model is used as a time series dataset and input into the time fusion converter TFT model for training to obtain a future time series prediction model.

[0036] S140, the remote sensing image to be detected is input into the target detection model for target detection, and the target information of the remote sensing image to be detected is output.

[0037] S150, the target information of the remote sensing image to be detected is input into the future time series prediction model for prediction, and the target state prediction of the remote sensing image to be detected at the future time step is output.

[0038] First, the remote sensing image processing method of this application includes a training phase and a detection phase. The training phase includes steps S110 to S130, while the detection phase includes steps S140 and S150.

[0039] Secondly, both the training images used in the training phase and the detection images used in the detection phase can be remote sensing images obtained from real data sources. For example, in the training phase, a DJI Phantom 4 RTK drone equipped with a high-resolution camera can be used to take aerial photos of urban areas, ensuring a flight altitude of 100-150 meters and setting flight path parameters to ensure a forward overlap of 80% and a lateral overlap of 70%, covering the entire target area. The drone performs multiple aerial photos according to the set flight plan, with each photo taken 3 hours apart, to obtain multiple frames of remote sensing images with time-series characteristics, which can be directly input into the Generative Adversarial Network (GAN) as training images. In the detection phase, a drone with the same configuration can be used to take aerial photos of urban areas to obtain remote sensing images at a specific time point, which can be directly input into the trained target detection model as the remote sensing images to be detected in this application.

[0040] In other embodiments, remote sensing data acquired from real data sources can be preprocessed before being input into a Generative Adversarial Network (GAN) or a trained object detection model. For example, images acquired by a drone are first stitched and orthorectified to combine multiple images into a seamless orthophoto, eliminating distortions caused by camera tilt or terrain differences. Then, the stitched images are cropped to focus on the target urban area, and images are resampled to ensure uniform resolution across all images. Finally, the Scale-Invariant Feature Transform (SIFT) algorithm is used to match feature points in images from different time periods, achieving precise spatial registration and ensuring temporal consistency, laying the foundation for subsequent time-series analysis. Next, image preprocessing is performed. First, the processed images are denoised using OpenCV's median filter library to remove noise and improve image clarity. Then, the contrast enhancement technique CLAHE is used to further enhance image details, especially key areas such as buildings and roads in the city. Finally, the images are annotated to standardize pixel values, ensuring that subsequent deep learning models can efficiently process this image data. Finally, LabelImg was used to annotate the image targets, including buildings, vehicles, and other targets.

[0041] After acquiring the training remote sensing images, they are input into the Generative Adversarial Network (GAN). These training images can be a single frame or multiple frames; if multiple frames, they should possess time-series characteristics. The following describes this step in detail using the example of training images consisting of multiple frames with time-series characteristics. For instance, representative image segments are selected from the original multi-frame training images and cropped to ensure uniform size and spatiotemporal consistency. For time-series data generation, the training data is ensured to contain consecutive time points, forming an image sequence. The Pix2Pix GAN model is used for time-series data generation. This model consists of a generator and a discriminator. The generator uses a U-shaped convolutional network (UNET) structure to take the image from the previous time period as input and generate the predicted image for the next time period. The discriminator uses PatchGAN to distinguish the generated images from real images, improving the realism of the generated images. The generator and discriminator settings can be as follows: Figure 2 As shown.

[0042] (1) The generator uses U-Net as its main architecture. It combines low-level and high-level features through skip connections, which is suitable for preserving detailed information and thus generating more accurate images. U-Net is an encoder-decoder architecture where the encoder extracts high-level features of the image, the decoder gradually reconstructs the image, and retains the original detailed information through skip connections. The generator's input can be a remote sensing image (RGB image) from the previous time point with a size of 256*256*3. The output is the generated image from the next time point, also with a size of 256*256*3. The generator consists of four layers, for example:

[0043] The encoder consists of 7 convolutional layers (Conv2D), each using a 3x3 kernel with a stride of 2x2. The output feature maps have 64, 128, 256, 512, 512, 512, and 512 channels respectively. Each convolutional layer is followed by batch normalization using LeakyReLU as the activation function with a slope of 0.2.

[0044] Convolutional layer: 4*4 kernel, 512 channels, using ReLU as the activation function to extract the highest level features.

[0045] The decoder consists of 7 deconvolutional (Conv2DTranspose) layers with a 4x4 kernel size and a 2x2 stride, producing feature maps with 512, 512, 512, 512, 256, 128, and 64 channels respectively. Each deconvolutional layer is followed by BatchNormalization and ReLU activation. Skip connections are used in the output of each decoder layer to connect features from the encoder to the decoder, preserving low-level details.

[0046] Output layer: The last deconvolutional layer uses tanh as the activation function to normalize the output value to [-1,1], generating an image of the same size as the input (256*x256*3).

[0047] (2) The discriminator uses PatchGAN with a patch size of 64*64. Unlike traditional image-level classifiers, PatchGAN judges local regions of the image and outputs whether each region is part of the real image or a generated image. This method can capture the local texture of the image more finely, and is especially suitable for pixel-level image generation tasks. Real and generated images are used as input, each image being 256*256*3 in size. The output is a 4*4 matrix, where each value represents whether a local region of the image is a real image. The discriminator contains only one convolutional layer, for example:

[0048] Convolutional layers: 5 convolutional (Conv2D) operations, with a kernel size of 4x4 and a stride of 2x2. The output channels of the convolutional layers are 64, 128, 256, 512, and 1 respectively. Each layer uses the LeakyReLU activation function with a slope of 0.2. Only the last layer outputs 1, indicating that the region is either a real image or a generated image.

[0049] The GAN network is trained using remote sensing images inputted at multiple consecutive time points. The goal is to generate the image for the next time point and then evaluate whether the generated image matches the real image using a discriminator. A weighted combination of L1 loss and adversarial loss is used as the loss function to optimize the quality and realism of the generated images. The Adam optimizer is used for training with a learning rate of 0.0002 for 200 epochs.

[0050] In S120, after generating multiple frames of remote sensing images using GAN, these generated images, along with the training images, can be used as a training set to input into the YOLO model for training, thereby obtaining the object detection model. The YOLO model is a real-time object detection algorithm capable of quickly detecting multiple objects in a single image, while simultaneously predicting the object category and bounding box.

[0051] In some embodiments, before training, these remote sensing images can be resized to fit the input requirements of the YOLO model. For example, the image size can be adjusted to a fixed width and height (e.g., 416*416 pixels or other suitable dimensions). Secondly, the pixel values ​​of the images are normalized, typically by converting the pixel value range from [0,255] to the [0,1] or [-1,1] interval, which helps the model converge better. For example, for RGB images, the pixel value of each channel can be normalized by dividing by 255. Next, a suitable YOLO version, such as YOLOv3, YOLOv4, or YOLOv5, is selected based on the task requirements and data characteristics. Different versions differ in network structure, performance, and other aspects.

[0052] The following will combine Figure 3 YOLOv5 was chosen as the base model for object detection. For example, the entire model can include three basic layers: Input layer: The input image size is set to 256*256. Feature extraction layer: A Cross-Stage Partial Network (CSPNet) is used as the backbone network to enhance feature extraction capabilities, reduce computational cost, and improve model stability. Multi-scale feature extraction capabilities are enhanced through feature pyramids, especially when handling small objects. Output layer: Outputs the class, bounding box coordinates, and confidence score for each detected object. The model will return information in the following format: {"class_id":"0","confidence":0.95,"bbox":[x_center,y_center,width,height]}. The model batch size is set to 16, the initial learning rate is set to 0.001 and dynamically adjusted based on training progress, and the Adam optimizer is used for optimization. Training is accelerated using pre-trained weights from the COCO dataset, and then fine-tuned using remote sensing images generated by GAN to ensure the model adapts to specific scenarios. The model is trained for 200 epochs, and its performance is evaluated using mean average precision (mAP) and intersection over union (IoU), with parameters adjusted based on the evaluation results.

[0053] Optionally, such as Figure 4 As shown, S120 may specifically include some or all of the following contents.

[0054] S121, the remote sensing image to be detected is processed into an input image suitable for the target detection model.

[0055] S122, Perform feature extraction on the input image to obtain multiple feature vectors.

[0056] S123, Extract the time series information of the multiple feature vectors.

[0057] S124, Based on the time series information, adjust the detection results of the target detection model, and then output the target information of the remote sensing image to be detected.

[0058] In other words, a temporal feature enhancement layer can be added after the feature extraction layer to capture temporal dependencies, enhance the model's ability to understand temporal information, and process temporal features. For example, this temporal feature enhancement layer can be an LSTM model or a GPU model, used to capture temporal dependencies and enhance the model's ability to understand temporal information. This layer receives the output from the feature extraction layer and processes the temporal features. In some embodiments, this temporal feature enhancement layer can be a Bi-LSTM model.

[0059] Furthermore, in the output layer, the output of the temporal feature enhancement layer can be combined to adjust the detection results based on time series information, thereby improving the accuracy of moving targets.

[0060] In S130, a TFT model is used to perform in-depth analysis of the time-series data obtained after target detection by the YOLO model. The purpose of this step is to extract the temporal features of the targets and perform dynamic prediction and analysis. Specifically, firstly, the target information detected by the YOLO model is organized into a time-series dataset suitable for the TFT input format. Each data point in this time-series dataset can include the following: timestamp – time information corresponding to each frame of the image; target information – target category, bounding box coordinates (center point x and y, width and height); and confidence score. Then, a TFT model is constructed. The TFT consists of multiple components, each contributing to capturing the complexity of the time-series data. The TFT model used in this scheme mainly contains four layers, for example, such as... Figure 5 As shown, the TFT model includes:

[0061] Input layer: Receives the processed time series data, with the time series length set to 5. Each time step contains the target information and other features shown in 1) above.

[0062] Temporal Residual Encoder (GRNk): Used to perform nonlinear transformations and selections on features at each time step, enhancing the model's expressive power. It also employs a self-attention mechanism to capture long-short-term dependencies in the time series.

[0063] Context extractor: Extracts temporal context information, allowing the model to dynamically adjust the importance of input data.

[0064] Prediction layer: Outputs the target state prediction for the future time step, including the target's category, bounding box, and motion trajectory.

[0065] The model batch size was set to 32, the learning rate was initially set to 0.001 and dynamically adjusted according to the training results, the Adam optimizer was used for optimization, and the mean squared error loss function was used.

[0066] After generating enough training remote sensing images using GAN and training the YOLO and TFT models, the resulting target detection model and future time series prediction model can be used to detect and predict targets in subsequently acquired remote sensing images.

[0067] Specifically, in S140, the acquired remote sensing image to be detected can be input into the trained YOLO model, i.e., the object detection model, to perform object detection. The model returns the object category, bounding box and its confidence score, in the same format as in the training phase.

[0068] In S150, the target information of the remote sensing image to be detected by the target detection model is passed to the trained TFT model, i.e., the future time series prediction model, to predict future time series. That is, inputting historical time series data, the model will output the target state at several future time steps, i.e., the future information of the target, including the target category, location and its changing trend or movement trajectory at future time steps.

[0069] A specific embodiment will be described below.

[0070] Suppose a drone is used to take aerial photographs of a city area every two hours, obtaining remote sensing images at five time points throughout the day: 8:00, 10:00, 12:00, 14:00, and 16:00. These images are then sequentially input into a GAN (Generative Advancement Array). For each time point, four generated remote sensing images are obtained, resulting in five sets of data. Each set corresponds to a single time point and includes the original remote sensing image and its four generated images. The five images from each set are then stitched together. The stitched images are then fed into a YOLO model for training. Once all five stitched images have been used as training data, a target detection model is obtained. Finally, the output data from training the YOLO model is fed into a TFT (Thinking Theory Model) model for training. Once all five stitched images have been used as training data, a future time-series prediction model is obtained.

[0071] In some embodiments, the Generative Adversarial Network (GAN), the YOLO model, and the Temporal Fusion Converter (TFT) model are integrated into one unit. The collaborative work of these three components achieves end-to-end integration from data generation to target detection and temporal optimization, significantly improving detection accuracy and system robustness. This multi-model integration design provides a novel solution for the detection and analysis of temporal remote sensing images.

[0072] After obtaining the trained object detection model and future time series prediction model, their performance can be evaluated using a test set (e.g., other remote sensing images acquired from UAVs). For the object detection model, four metrics are primarily used to evaluate its performance in detecting different targets: mean precision, accuracy, recall, and precision. For the future time series prediction model, two metrics are used: root mean square error and mean absolute error to evaluate the accuracy of the model's predictions of time series data. Based on the evaluation results, the model can be optimized again. The optimized model needs to be validated on a new dataset or in a real-world scenario to ensure the effectiveness of the optimization. Real-time testing of the object detection model and future time series prediction model is then conducted again using remote sensing images acquired from UAVs to observe the model's performance in a real-world environment. Validation includes the accuracy of target detection, the number of false negatives and false positives, and the accuracy of time series data prediction, ensuring that the system maintains stable performance in changing environments.

[0073] Next, another remote sensing image at a given time point is acquired from the drone and sequentially input into the target detection model and the future time series prediction model to obtain the target state of the remote sensing image at that time point and the target state of the remote sensing image at the next time point.

[0074] It should be noted that the remote sensing images obtained for training and the remote sensing images obtained for detection can be multiple time-series remote sensing images acquired by the UAV, with one part serving as the training set and the other part as the detection set.

[0075] This application also provides a remote sensing image processing apparatus. For example... Figure 6 As shown, the remote sensing image processing device 200 includes some or all of the following components.

[0076] Generative Adversarial Network (GAN 210) is used to receive training remote sensing images and output multi-frame generated remote sensing images corresponding to the training remote sensing images.

[0077] YOLO model 220 is used to receive the training remote sensing image and the multi-frame generated remote sensing image as a training set for training to obtain target detection model 240. The target detection model is used to receive the remote sensing image to be detected for target detection and output the target information of the remote sensing image to be detected.

[0078] The time fusion converter TFT model 230 is used to receive the target information of the training set detected from the initial YOLO model as a time series dataset for training to obtain a future time series prediction model 250. The future time series prediction model is used to receive the target information of the remote sensing image to be detected for prediction and output the target state prediction of the remote sensing image at the future time step.

[0079] In some embodiments, the target detection model is specifically used for:

[0080] The remote sensing image to be detected is processed into an input image suitable for the target detection model;

[0081] Feature extraction is performed on the input image to obtain multiple feature vectors;

[0082] Extract the time series information of the multiple feature vectors;

[0083] Based on the time series information, the detection results of the target detection model are adjusted, and then the target information of the remote sensing image to be detected is output.

[0084] In some embodiments, the target detection model includes a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model, used to process the temporal features of the multiple feature vectors to obtain the time series information.

[0085] In some embodiments, the target state prediction for the future time step includes the target category, location, and trend of change for the future time step.

[0086] In some embodiments, the Generative Adversarial Network (GAN), the YOLO model, and the Time Fusion Converter (TFT) model are integrated into one unit.

[0087] Based on the same idea, this application also provides another remote sensing image processing apparatus. This remote sensing image processing apparatus includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the remote sensing image processing apparatus performs the various method embodiments described above.

[0088] It should be noted that the details of the device-side embodiment can be found in the method-side embodiment, and for the sake of brevity, they will not be repeated here.

[0089] Based on the same idea, embodiments of this application also provide a computer-readable storage medium, which includes a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute the various method embodiments described above.

[0090] Although this application has been described with reference to preferred embodiments, various modifications can be made thereto and components can be replaced with equivalents without departing from the scope of this application. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no structural conflict. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A remote sensing image processing method, characterized in that, include: The training remote sensing images are input into the generative adversarial network (GAN), and the network outputs multi-frame generated remote sensing images corresponding to the training remote sensing images. The training remote sensing images are multi-frame remote sensing images with time-series features. The training remote sensing images and the multi-frame generated remote sensing images are used as training sets and input into the initial YOLO model for training to obtain the target detection model. Feature extraction is performed on the remote sensing images in the training set to obtain multiple first feature vectors; The temporal features of the multiple first feature vectors are processed using a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model to obtain the first time series information of the multiple first feature vectors; Based on the first time series information, the detection results of the initial YOLO model are adjusted, and the target information of the training set is output. The target information of the training set detected from the initial YOLO model is used as a time series dataset and input into the time fusion converter TFT model for training to obtain a future time series prediction model; The remote sensing image to be detected is input into the target detection model for target detection, and the target information of the remote sensing image to be detected is output. The target information of the remote sensing image to be detected is input into the future time series prediction model for prediction, and the target state prediction of the remote sensing image to be detected at the future time step is output. The target state prediction at the future time step includes the target category, location and change trend at the future time step.

2. The remote sensing image processing method according to claim 1, characterized in that, The step of inputting the remote sensing image to be detected into the target detection model for target detection and outputting the target information of the remote sensing image to be detected includes: The remote sensing image to be detected is processed into an input image suitable for the target detection model; Feature extraction is performed on the input image to obtain multiple second feature vectors; Extract the second time series information from the multiple second feature vectors; Based on the second time series information, the detection results of the target detection model are adjusted, and then the target information of the remote sensing image to be detected is output.

3. The remote sensing image processing method according to claim 2, characterized in that, The extraction of the second time series information from the plurality of second feature vectors includes: The temporal features of the multiple second feature vectors are processed using a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model to obtain the second time series information.

4. The remote sensing image processing method according to any one of claims 1 to 3, characterized in that, The Generative Adversarial Network (GAN), the YOLO model, and the Time Fusion Converter (TFT) model are integrated into one unit.

5. A remote sensing image processing device, characterized in that, include: Generative Adversarial Network (GAN) is used to receive training remote sensing images and output multi-frame generated remote sensing images corresponding to the training remote sensing images. The training remote sensing images are multi-frame remote sensing images with time-series features. The YOLO model is used to receive the training remote sensing images and the multi-frame generated remote sensing images as training sets for training to obtain a target detection model. The target detection model is used to receive the remote sensing images to be detected, perform target detection, and output the target information of the remote sensing images to be detected. A Time Fusion Transformer (TFT) model is used to receive target information from the training set detected by the YOLO model as a time-series dataset for training, thereby obtaining a future time-series prediction model. This future time-series prediction model receives target information from the remote sensing image to be detected, makes predictions, and outputs a target state prediction for the next time step of the remote sensing image. The target state prediction for the next time step includes the target category, location, and change trend for that future time step. The YOLO model is also used for: Feature extraction is performed on the remote sensing images in the training set to obtain multiple first feature vectors; The temporal features of the multiple first feature vectors are processed using a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model to obtain the first time series information of the multiple first feature vectors; Based on the first time series information, the detection results of the YOLO model are adjusted, and the target information of the training set is output.

6. The remote sensing image processing apparatus according to claim 5, characterized in that, The target detection model is specifically used for: The remote sensing image to be detected is processed into an input image suitable for the target detection model; Feature extraction is performed on the input image to obtain multiple second feature vectors; Extract the second time series information from the multiple second feature vectors; Based on the second time series information, the detection results of the target detection model are adjusted, and then the target information of the remote sensing image to be detected is output.

7. The remote sensing image processing apparatus according to claim 6, characterized in that, The target detection model includes a Long Short-Term Memory (LSTM) network model or a Gated Recurrent Unit (GRU) model, which is used to process the temporal features of the multiple second feature vectors to obtain the second time series information.

8. The remote sensing image processing apparatus according to any one of claims 5 to 7, characterized in that, The Generative Adversarial Network (GAN), the YOLO model, and the Time Fusion Converter (TFT) model are integrated into one unit.

Images