Data set adaptive construction method and device for remote sensing image few-sample classification

Abstract

The invention discloses a data set adaptive construction method and device for remote sensing image few-sample classification, and relates to the field of remote sensing image classification. A prompt pool including various prompt strategies is constructed, and a real remote sensing data set is generated; randomly selecting prompt texts from the prompt pool, and constructing a plurality of prompt sets; a three-dimensional list is constructed to represent the prompt sets, the first dimension of the three-dimensional list represents the number of the prompt sets, the second dimension of the three-dimensional list represents the number of the task categories, and the third dimension of the three-dimensional list represents the number of the prompt texts selected in each task category; using the generative large model to input prompt texts in the prompt set, and generating a corresponding generation data set; training a deep network classifier adaptive to a downstream classification task on the synthetic data set; and evaluating the deep network classifier by using the real remote sensing data set to obtain an optimized prompt set so as to generate an optimal data set. The problem of how to construct the most suitable prompt under the condition of giving the task and find the data set most suitable for the task is solved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing image classification, and in particular to a method and device for adaptively constructing a dataset for remote sensing image few-sample classification. Background Art

[0002] In recent years, remote sensing image classification has played a vital role in areas such as land cover monitoring, urban planning, and disaster assessment. While the widespread application of deep learning techniques has significantly improved classification accuracy, the reliance on large amounts of labeled data for high-performance models remains a key bottleneck. Limited by the cost of acquiring remote sensing data, the difficulty of labeling, and privacy concerns, available data resources are often limited, making it difficult to train deep models.

[0003] To alleviate the problem of data scarcity, generative large models have demonstrated strong data generation capabilities in the field of image synthesis. Especially in computer vision tasks, generative large models (Generative Foundation Models) have been widely used to synthesize high-quality samples and enhance the generalization ability of the model. However, in remote sensing image classification tasks, the potential of generative large models has not yet been fully explored. Because existing large-scale text-to-image generation models often construct completely different datasets when facing different text inputs (prompts), which greatly affects the training of downstream deep models, how to design a data generation process suitable for remote sensing scenarios and ensure that the generated samples are truly adapted to specific remote sensing image classification tasks is an important challenge in current research.

[0004] Existing prompt construction techniques often rely on manual construction or leveraging existing models. However, manually constructed prompts require researchers to possess highly specialized knowledge of the downstream task. Furthermore, they are often tailored to a specific style and lack content diversity. Model-based prompts are often influenced by the style of the model itself, and can even contain many low-quality prompts. Furthermore, because the existing data generation process is decoupled from the downstream task training process, it is impossible to find prompts that are truly suitable for the task. This often results in different approaches being preferred for different tasks, severely impacting practical applications.

[0005] Therefore, how to adaptively construct the most appropriate prompts for a given task and find the most suitable dataset for the task has become an urgent problem to be solved. Summary of the Invention

[0006] In an embodiment of the present application, a method for adaptively constructing a dataset for few-sample classification of remote sensing images is provided to solve the problem of how to adaptively construct the most appropriate prompts under a given task and find the dataset that best suits the task.

[0007] In a first aspect, an embodiment of the present application provides a method for adaptively constructing a dataset for few-sample classification of remote sensing images, characterized in that it includes: constructing a prompt pool including diverse prompt strategies and generating a real remote sensing dataset; wherein the diverse prompt strategies include multiple prompt templates; randomly selecting prompt texts from the prompt pool to construct multiple prompt sets; wherein each prompt set includes multiple prompt texts, and each prompt set corresponds to a dataset generation strategy; constructing a three-dimensional list to represent the prompt set, wherein the first dimension of the three-dimensional list represents the number of prompt sets, the second dimension represents the number of task categories, and the third dimension represents the number of prompt texts selected in each task category; using a generative large model to input the prompt texts in the prompt set to generate a corresponding generated dataset; training a deep network classifier adapted for downstream classification tasks on the synthetic dataset; wherein the synthetic dataset includes multiple generated datasets, each generated dataset corresponds to a prompt set in the prompt pool, and includes multiple images generated by the prompt texts and their corresponding labels; using the real remote sensing dataset to evaluate the deep network classifier and perform prompt set optimization to obtain the optimized prompt set; generating an optimal dataset from the optimized prompt set; wherein the optimal dataset is the dataset that is most suitable for the current task.

[0008] In one possible implementation, the construction includes a prompt pool of diverse prompt strategies and generates a real remote sensing dataset, including: selecting a first preset number of task categories in the remote sensing image dataset as classification tasks; randomly selecting a second preset number of images in each task category to construct a real remote sensing dataset to simulate a few-sample setting; wherein the real remote sensing dataset includes real images; based on prior knowledge of the classification task, constructing a first prompt template; wherein the prompt content of the first prompt template includes coverage domain, style and camera perspective; using a large language model to construct a second prompt template; wherein the second prompt template is used to input a category name and output multiple prompt texts including the category name; using an image annotation model to construct a third prompt template; wherein the third prompt template is used to input a real image and output a prompt text describing the real image; integrating the prompt texts generated by the first prompt template, the second prompt template and the third prompt template into a prompt pool, and storing the prompt pool in the form of a two-dimensional list; wherein the first dimension of the two-dimensional list represents the number of task categories, and the second dimension represents a list of all prompt texts under each task category.

[0009] In one possible implementation, the synthetic dataset includes multiple generated datasets, each generated dataset corresponds to a prompt set in a prompt pool, and includes multiple images generated by prompt texts and their corresponding labels, including: the generated dataset includes multiple elements, each element corresponds to an image generated by a prompt text; according to the task category to which each prompt text in the prompt set belongs, a corresponding label is assigned to each image in the generated dataset; a two-dimensional list is constructed to represent the synthetic dataset to determine the task category to which each image in the synthetic dataset belongs and the prompt text from which it comes; wherein the first dimension of the two-dimensional list represents the number of task categories, and the second dimension represents the total number of images generated by all prompt texts under each category.

[0010] In one possible implementation, the use of a real remote sensing dataset to evaluate a deep network classifier includes: training a deep network classifier using a generated dataset; evaluating the trained deep network classifier on a real remote sensing dataset; obtaining a task matching score for each prompt set based on the classification loss of the deep network classifier on the real remote sensing dataset; wherein a smaller task matching score indicates a better performance of the deep network classifier on the prompt set and a higher task matching score; and associating each prompt set with its corresponding task matching score to form a prompt set-task matching score pair.

[0011] In one possible implementation, the prompt set optimization to obtain the optimized prompt set includes: providing a batch of initial prompt sets as the initial prompt set population of the evolutionary algorithm; iteratively executing the optimization steps until a maximum number of iterations is met; and outputting the optimized prompt set; wherein the optimized prompt set has a high task matching score on a real remote sensing dataset.

[0012] In one possible implementation, the optimization step includes: taking the current prompt set population as input; using the prompt set-task matching score pair to select a better prompt set as the next generation prompt set population according to the search strategy of the evolutionary algorithm; and using the selected next generation prompt set population as the input of the next generation to update the prompt set population.

[0013] In one possible implementation, generating an optimal data set from the optimized prompt set includes: setting generation parameters; setting the generation parameters includes: setting the number of optimal data sets to be generated, and setting the ratio of the generated image for each prompt text; selecting the optimal prompt set from the optimized prompt set; and generating the optimal data set using the generative large model and the optimal prompt set according to the set generation parameters.

[0014] In the second aspect, an embodiment of the present application provides a device for constructing a small-sample adaptive dataset for classification tasks, the device comprising: a construction module for constructing a prompt pool including diverse prompt strategies and generating a real remote sensing dataset; wherein the diverse prompt strategies include multiple prompt templates; a selection module for randomly selecting prompt texts from the prompt pool to construct multiple prompt sets; wherein each prompt set includes multiple prompt texts, and each prompt set corresponds to a dataset generation strategy; a representation module for constructing a three-dimensional list to represent the prompt set, the first dimension of the three-dimensional list represents the number of prompt sets, the second dimension represents the number of task categories, and the third dimension represents the number of prompt texts selected in each task category. ; Input module, used to use the generative large model to input the prompt text in the prompt set to generate the corresponding generated data set; Training module, used to train a deep network classifier adapted to the downstream classification task on the synthetic data set; wherein, the synthetic data set includes multiple generated data sets, each generated data set corresponds to a prompt set in the prompt pool, and includes multiple images generated by the prompt text and their corresponding labels; Acquisition module, used to evaluate the deep network classifier using a real remote sensing data set, and optimize the prompt set to obtain the optimized prompt set; Generation module, used to generate the optimal data set from the optimized prompt set; wherein, the optimal data set is the data set that is most suitable for the current task.

[0015] In a third aspect, an embodiment of the present application provides a server for constructing a small-sample adaptive dataset for classification tasks, comprising a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the method described in the first aspect or any possible implementation method of the first aspect.

[0017] One or more technical solutions provided in the embodiments of this application have at least the following technical effects:

[0018] The present invention provides a method for adaptively constructing datasets for few-shot classification of remote sensing images. By constructing a cue pool containing diverse cue strategies and generating real remote sensing datasets, this method effectively addresses the problem of existing data generation algorithms only targeting certain styles. The diverse cue strategies in the cue pool cover different content, styles, and scenarios, ensuring the diversity of the generated dataset and providing richer training material for classification tasks. By evaluating a deep network classifier using a real remote sensing dataset and optimizing the cue set, an optimized cue set is obtained, thereby generating a dataset that is optimally suited to the task at hand. This process overcomes the shortcomings of existing methods, which are unable to find cue sets suitable for specific tasks, significantly improving classification performance. The introduction of an optimization method for cue set-task matching score pairs avoids the tedious and expensive double-layer optimization gradient calculation. Furthermore, this method provides significant freedom for inner-layer optimization, allowing for the free selection of appropriate training architectures and even compatibility with various irreversible operations. This feature greatly enhances the scalability of the task and enables the method to adapt to a wider range of application scenarios. Generating a dataset optimally suited to the task using the optimized cue set demonstrates strong generalization. On the one hand, the size of the dataset can be infinitely expanded to meet the needs of tasks of different scales and complexities; on the other hand, this application is only targeted at datasets, so it has the potential to be model-independent and can be freely generalized to various network architectures and training methods. This makes this method more flexible and versatile in practical applications. This application solves the defect that existing data generation algorithms cannot perceive downstream tasks, and effectively improves the accuracy of classification tasks in few-sample settings. It solves the problem of how to adaptively construct the most appropriate hints for a given task and find the dataset that best suits the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a method for adaptively constructing a dataset for small-sample classification of remote sensing images provided in an embodiment of the present application;

[0021] Figure 2 A diagram of the specific implementation process provided for the embodiments of this application;

[0022] Figure 3 Some samples of the dataset generated on the EuroSAT dataset provided in the embodiments of this application;

[0023] Figure 4 The accuracy of the algorithm generated by the embodiment of this application on the EuroSAT dataset compared with other datasets;

[0024] Figure 5 The performance of the method of the present application on different network architectures on the EuroSAT dataset provided in the embodiments of the present application;

[0025] Figure 6 Ablation experiments on the effect of prompt selection on the EuroSAT dataset provided in the embodiments of this application;

[0026] Figure 7 A schematic diagram of a data set adaptive construction device for remote sensing image few-sample classification provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a dataset adaptive construction server for small-sample classification of remote sensing images provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0030] The present invention provides a method for adaptively constructing a dataset for remote sensing image classification using a small number of samples. Figure 1 As shown, the method includes steps S101 to S107. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of a dataset adaptive construction method for remote sensing image few-sample classification. Figure 1 The steps shown may be performed in parallel or reversed.

[0031] S101: Constructing a prompt pool including diverse prompt strategies and generating a real remote sensing dataset, wherein the diverse prompt strategies include multiple prompt templates.

[0032] Constructing a prompt pool including diverse prompt strategies and generating a real remote sensing dataset includes the following steps.

[0033] A first preset number of task categories are selected from the image dataset as classification tasks.

[0034] A second preset number of images are randomly selected in each task category to construct a real remote sensing dataset to simulate a few-shot setting, wherein the real remote sensing dataset includes real images.

[0035] Based on prior knowledge of the classification task, a first prompt template is constructed. The prompt content of the first prompt template includes the covered domain, style, and camera perspective. Of course, the prompt content of the first prompt template may also include other content, and this application is not limited to the above prompt content.

[0036] A second prompt template is constructed using the large language model, wherein the second prompt template is used to input a category name and output multiple prompt texts including the category name.

[0037] The image annotation model is used to construct a third prompt template, wherein the third prompt template is used to input a real image and output a prompt text describing the real image.

[0038] The prompt texts generated by the first prompt template, the second prompt template, and the third prompt template are integrated into a prompt pool, and the prompt pool is stored in the form of a two-dimensional list, wherein the first dimension of the two-dimensional list represents the number of task categories, and the second dimension represents a list of all prompt texts under each task category.

[0039] Specifically, this application uses the EuroSAT classification task as an example to illustrate the given task. The image dataset in this application can be EuroSAT. The first preset number can be 10. The second preset number can also be 10. Specifically, 10 images are randomly selected from each class in the EuroSAT dataset to construct a real remote sensing dataset to simulate a few-shot setting. Based on prior knowledge of the classification task, a first prompt template can be manually constructed. The prompt content of the first prompt template can include information covering domain, style, and camera perspective. For example, "a satellite photo of {}" or "An overhead satellite view of {}," where {} represents the category name of the classification task, which can be obtained from the dataset's official website. A second prompt template is constructed using a large language model, such as ChatGPT or T5. The large language model accepts a category name as input and outputs multiple sentences containing the category name, i.e., multiple prompt texts. A third prompt template is constructed using an image annotation model, such as the BLIP2 model, which accepts a real image as input. Assume that 256 prompts are constructed for each class in the EuroSAT dataset. All prompt texts are integrated into a prompt pool and stored in the form of a two-dimensional list. The first dimension of this two-dimensional list represents the number of task categories, i.e., the 10 categories in EuroSAT, and the second dimension represents the list of all prompt texts under each task category, i.e., the 256 prompts in each category.

[0040] S102: randomly selecting prompt texts from the prompt pool to construct multiple prompt sets, wherein each prompt set includes multiple prompt texts, and each prompt set corresponds to a data set generation strategy.

[0041] S103: Construct a three-dimensional list to represent the prompt set. The first dimension of the three-dimensional list represents the number of prompt sets, the second dimension represents the number of task categories, and the third dimension represents the number of prompt texts selected in each task category.

[0042] Specifically, when constructing a prompt set, prompt texts are randomly selected from the prompt pool to form a batch of prompt sets. Each prompt set includes multiple prompt texts, each of which is sampled from the prompt pool. Because an evolutionary algorithm is used for optimization, a batch of prompt sets is required as the initial population. Each prompt set corresponds to a dataset generation strategy, i.e., a generated dataset. Ultimately, after completing the entire method process, each prompt set will output a corresponding task matching score.

[0043] Specifically, suppose we construct 70 prompt sets. In each prompt set, 10 prompts are selected for each task category (for example, in EuroSAT, 10 prompts are selected for each task category). Thus, these 70 prompt sets can be constructed as a Python list of size [70, 10, 10]. The first dimension represents the number of prompt sets (i.e., 70 prompt sets), the second dimension represents the number of task categories (i.e., 10 task categories in EuroSAT), and the third dimension represents the number of prompt texts selected in each task category (i.e., 10 prompts selected in each category). Because the subsequent optimization process requires numerical values, this list can be converted from an index to an integer list. In the specific example of EuroSAT, each element of the list is an integer between 0 and 256, representing the index of the prompt in the prompt pool. This index allows a one-to-one mapping of the integer to a specific prompt text. This three-dimensional list allows convenient access and manipulation of the prompt texts in each prompt set.

[0044] S104: Use the generative large model to input the prompt text in the prompt set to generate a corresponding generated data set.

[0045] Specifically, the generative model accepts a prompt set as input and outputs a corresponding dataset. Since each prompt text in the prompt set has been clearly assigned its task category when constructing the prompt pool, the labels of the synthetic dataset are naturally determined.

[0046] Furthermore, the Stable Diffusion XL-Turbo model can be used as a generative model. This model accepts text as input and outputs semantically diverse images with varying content based on a random seed. For each prompt set, 20 images are generated for each prompt. This outputs a Python list of size [10, 10*20] for each prompt set. Each element of the list is an image, and the first dimension of the list represents the number of categories (i.e., 10 task categories), while the second dimension represents the total number of 20 images generated for each of the 10 prompts in each category.

[0047] S105: Training a deep network classifier adapted for downstream classification tasks on the synthetic dataset, wherein the synthetic dataset includes multiple generated datasets, each generated dataset corresponding to a prompt set in the prompt pool and including multiple images generated by the prompt text and their corresponding labels.

[0048] The synthetic dataset includes multiple generated datasets, each of which corresponds to a prompt set in a prompt pool and includes multiple images generated by prompt texts and their corresponding labels, including: the generated dataset includes multiple elements, each element corresponds to an image generated by a prompt text. According to the task category to which each prompt text in the prompt set belongs, a corresponding label is assigned to each image in the generated dataset. A two-dimensional list is constructed to represent the synthetic dataset to determine the task category to which each image in the synthetic dataset belongs and the prompt text from which it originates. The first dimension of the two-dimensional list represents the number of task categories, and the second dimension represents the total number of images generated by all prompt texts under each category.

[0049] Specifically, for each list of images generated by the prompt set, each image can be assigned a corresponding label based on the task category to which the prompt text belongs. In this way, each synthetic dataset is a supervised dataset that can be used for training classification tasks.

[0050] Furthermore, after generating the synthetic dataset, a deep network classifier adapted to the downstream classification task is trained on the synthetic dataset. The training process can be flexibly selected. The following is an example of a training scheme. The pre-trained CLIP network is used as the initial model of the deep network classifier. The use of pre-trained models helps to reduce computational overhead and speed up the training process. The optimization algorithm selects stochastic gradient descent (SGD). SGD is a widely used optimization algorithm suitable for training large-scale datasets. Set the momentum of SGD to 0.9. The momentum parameter helps to speed up the training process and reduce oscillations near the optimal solution. The weight decay coefficient is set to 110 -4 . Weight decay helps prevent the model from overfitting and improves generalization ability. The initial learning rate is set to 0.001. The choice of the initial learning rate is crucial for model training. Too high may cause training instability or divergence, and too low may cause the training speed to be too slow or fail to converge to a good solution. A total of 30 epochs are trained. The choice of the number of training rounds depends on the size of the dataset and the complexity of the model. The learning rate adopts a step-by-step decay strategy. The learning rate decays once every 5 epochs, with a decay coefficient of 0.5. This strategy helps the model adjust parameters more finely in the later stage of training and improve the training effect. The loss function adopts cross entropy loss, which is a commonly used loss function in classification tasks. In order to further prevent overfitting, a 1e-4 two-norm regularization is adopted. The regularization term helps to limit the scale of model parameters and improve the generalization ability of the model. It should be noted that the above parameter settings are only exemplary and are not limitations of this application. In actual applications, the parameter settings can be flexibly adjusted according to the characteristics of specific tasks and datasets.

[0051] S106: Use a real remote sensing dataset to evaluate the deep network classifier and optimize the prompt set to obtain the optimized prompt set.

[0052] Evaluating a deep network classifier using a real remote sensing dataset includes: training a deep network classifier using a generated dataset; evaluating the trained deep network classifier on the real remote sensing dataset; and obtaining a task matching score for each prompt set based on the classification loss of the deep network classifier on the real remote sensing dataset. A smaller task matching score indicates better performance of the deep network classifier on that prompt set, and a higher task matching score. Each prompt set is associated with its corresponding task matching score to form a prompt set-task matching score pair.

[0053] Performing prompt set optimization to obtain an optimized prompt set includes: providing a batch of initial prompt sets as the initial prompt set population for the evolutionary algorithm; iteratively executing the optimization steps until a maximum number of iterations is met; and outputting the optimized prompt set. The optimized prompt set has a high task matching score on a real remote sensing dataset.

[0054] Specifically, during prompt set optimization, real remote sensing datasets are used to evaluate the suitability of the generated datasets for each prompt set for the task at hand. Real remote sensing datasets can be used to represent specific tasks and contain key features and patterns for that task. A trained deep network classifier, representing the synthetic dataset, is then evaluated on a real remote sensing dataset. Since the deep network classifier is trained on the synthetic dataset generated from the prompt set, its performance on the real remote sensing dataset directly reflects the suitability of the generated dataset for the task. Specifically, the lower the classification loss of the deep network classifier on the real remote sensing dataset, the more suitable the generated dataset is for the task at hand, indicating a higher quality dataset generated by that prompt set. This establishes a connection between the data generation process and the downstream task. By iteratively optimizing the prompt set and evaluating the performance of the generated datasets on real remote sensing datasets, the most suitable data generation method for the task can be gradually identified.

[0055] The optimization process involves taking the current prompt set population as input, using the prompt set-task matching score pairs, and selecting a better prompt set as the next prompt set population according to the evolutionary algorithm's search strategy. The selected next prompt set population is used as input for the next generation, and the prompt set population is updated.

[0056] Specifically, the evolutionary algorithm can be a genetic algorithm (GA). When using a GA, the optimization steps are described in detail below. The current prompt set population serves as input to the GA. This population contains multiple prompt sets, each corresponding to a dataset generation strategy. The GA evaluates the quality of each prompt set based on its task-matching score on a real remote sensing dataset. Prompt sets with lower scores indicate that the datasets they generate are more suitable for the current task. Based on the task-matching scores, the algorithm selects a subset of the better prompt sets as parents to generate the next generation of prompt sets. Based on the existing solutions, or parent prompt sets, the GA performs a cross-compilation operation. This is a core step in the GA. By exchanging some elements from the parent prompt sets, a new set of prompt sets is generated as offspring. These new prompt sets inherit some of the good properties of the parent set, while also potentially introducing new mutations, thereby increasing the diversity of the population. The resulting offspring prompt sets are added to the current population, replacing some of the poorer ones, thereby updating the prompt set population. This process is repeated until the maximum number of iterations is met or another stopping condition is reached. This optimization process is inherently a discrete optimization process, as the prompt set consists of discrete prompt texts. The search space is the entire prompt pool, meaning the algorithm can construct a new prompt set from any combination of prompt texts in the prompt pool. Through these steps, the genetic algorithm gradually finds data generation methods that are more suitable for the task at hand, resulting in optimized prompt sets. These prompt sets achieve high task-matching scores on real remote sensing datasets, enabling the generation of higher-quality datasets for training downstream classification tasks.

[0057] The above content is the result given in this application using the EuroSAT classification task as an example. For other datasets, it is only necessary to change the specific form of the prompt pool construction. In addition, the specific parameters in the above content are not within the scope of this application and should not be used as a limitation. In actual applications, these parameters can be flexibly adjusted according to the characteristics of the specific task and dataset.

[0058] S107: Generate an optimal dataset from the optimized prompt set, wherein the optimal dataset is the dataset that is most suitable for the current task.

[0059] Generating an optimal dataset from the optimized prompt set includes setting generation parameters. Setting the generation parameters includes setting the number of optimal datasets to be generated and setting the ratio of images generated for each prompt text. Selecting an optimal prompt set from the optimized prompt set. Using the generative large model and the optimal prompt set, the optimal dataset is generated according to the set generation parameters.

[0060] Specifically, the generation parameters can be freely set according to actual needs to meet different application scenarios and data requirements. The optimal prompt set is selected from the optimized prompt set. This optimal prompt set is obtained by evolutionary algorithm optimization in the previous step and has the highest task matching score, indicating that the generated dataset is most suitable for the current task. The generative large model can generate corresponding images based on the prompt text, and the optimal prompt set ensures that the generated image is highly relevant to the current task. The generated optimal dataset can be used as the dataset that is most suitable for the task and applied to the training of the task. Since the generated optimal dataset is highly relevant to the classification task and the generation process does not depend on a specific training method or model architecture, it can be used in a variety of training environments and model architectures to improve the training effect and generalization ability of the task.

[0061] Figure 2 The specific implementation process diagram provided for the embodiment of this application. For the existing prompt set, first use the large-scale image generation model to generate the corresponding generated dataset, then train a downstream classification model, i.e., a deep network classifier for the downstream classification task, on the dataset, and then evaluate it on a real few-sample dataset, i.e., a real remote sensing dataset, to obtain the meta-loss L meta , that is, obtaining the prompt set-task matching score pair, and finally relying on the evolutionary algorithm to search for the optimal prompt set in the prompt pool.

[0062] Figure 3 This is a sample of a dataset generated on the EuroSAT dataset provided in the embodiments of this application. It can be seen that the generated images have good diversity while maintaining high image quality, which is beneficial to classification tasks.

[0063] Figure 4 The accuracy of the algorithm generated on the EuroSAT dataset provided by the embodiment of the present application is compared with other datasets. Figure 4 The results of classification accuracy on the EuroSAT dataset are provided. All experiments are performed under the condition of having access to only 10 samples per category. The experiments are conducted using the CLIP-RN50 model. Zero-shot represents the results without any synthetic samples. Vanilla represents the use of a single prompt "a satellite photo of {}", and the multi method represents the use of multiple domain prompts constructed manually. The LE method represents the use of a label-to-sentence generation model for prompt construction, accepting the category name as input and outputting a sentence containing the word. The CiP method represents the use of the image annotation model BLIP2 for prompt construction, accepting a real image as input and outputting a description of the image. DeCap represents the method described above.

[0064] Figure 5The embodiments of the present application provide the performance of the method of the present application on different network architectures using the EuroSAT dataset. Figure 5 A comparison of the effects of using different training methods on the EuroSAT dataset on the optimized generated dataset and the original generated dataset is provided. The results show that the method of this application is independent of the choice of model architecture and has strong generalization performance.

[0065] Figure 6 This is an ablation experiment on the effect of prompt selection on the EuroSAT dataset provided in the embodiments of the present application. Figure 6 The provided ablation experiment demonstrates the importance of task-guided prompt selection. It compares the effects of randomly selecting prompts, selecting prompts from the entire prompt pool, and randomly replacing half of the optimized prompts. The results show that prompt selection can significantly improve classification results and truly achieve the optimal prompt set.

[0066] The embodiment of the present application also provides a data set adaptive construction device 700 for remote sensing image few-sample classification, such as Figure 7 As shown, the device includes: a construction module 701, a selection module 702, a representation module 703, an input module 704, a training module 705, an acquisition module 706 and a generation module 707.

[0067] The construction module 701 is used to construct a prompt pool including various prompt strategies and generate a real remote sensing data set. The various prompt strategies include multiple prompt templates.

[0068] The selection module 702 is used to randomly select prompt texts from the prompt pool to construct multiple prompt sets, wherein each prompt set includes multiple prompt texts, and each prompt set corresponds to a data set generation strategy.

[0069] The representation module 703 is used to construct a three-dimensional list to represent the prompt set. The first dimension of the three-dimensional list represents the number of prompt sets, the second dimension represents the number of task categories, and the third dimension represents the number of prompt texts selected in each task category.

[0070] The input module 704 is used to use the generative large model to input the prompt text in the prompt set to generate a corresponding generated data set.

[0071] The training module 705 is used to train a deep network classifier adapted for downstream classification tasks on a synthetic dataset, wherein the synthetic dataset includes multiple generated datasets, each of which corresponds to a prompt set in the prompt pool and includes multiple images generated from the prompt text and their corresponding labels.

[0072] The acquisition module 706 is used to evaluate the deep network classifier using a real remote sensing data set, and to optimize the prompt set to obtain the optimized prompt set.

[0073] The generation module 707 is used to generate an optimal data set from the optimized prompt set, wherein the optimal data set is the data set that is most suitable for the current task.

[0074] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0075] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0076] The methods, devices, or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to implement the same function of the controller in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0077] like Figure 8 As shown, an embodiment of the present application also provides a dataset adaptive construction server for remote sensing image few-sample classification, including a memory 801 and a processor 802; the memory 801 is used to store computer-executable instructions; the processor 802 is used to execute computer-executable instructions to implement the above-described dataset adaptive construction method for remote sensing image few-sample classification in the embodiment of the present application.

[0078] An embodiment of the present application also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the method for adaptively constructing a dataset for few-sample classification of remote sensing images described above in the embodiment of the present application.

[0079] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, or it can be embodied in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiment of the present application.

[0080] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in detail. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations.

[0081] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A method for adaptively constructing a dataset for remote sensing image few-sample classification, characterized by: include: Constructing a prompt pool including diverse prompt strategies and generating a real remote sensing dataset; wherein the diverse prompt strategies include multiple prompt templates; Prompt texts are randomly selected from the prompt pool to construct multiple prompt sets; each prompt set includes multiple prompt texts, and each prompt set corresponds to a data set generation strategy; Construct a three-dimensional list to represent the prompt set, wherein the first dimension of the three-dimensional list represents the number of prompt sets, the second dimension represents the number of task categories, and the third dimension represents the number of prompt texts selected in each task category; Use the generative large model to input the prompt text in the prompt set to generate the corresponding generated dataset; Training a deep network classifier adapted for a downstream classification task on a synthetic dataset; wherein the synthetic dataset includes multiple generated datasets, each generated dataset corresponds to a prompt set in a prompt pool and includes multiple images generated from the prompt text and their corresponding labels; Use real remote sensing datasets to evaluate deep network classifiers and optimize the prompt set to obtain the optimized prompt set; Generate an optimal dataset from the optimized prompt set; where the optimal dataset is the dataset that is most suitable for the current task.

2. The method for adaptively constructing a dataset for remote sensing image few-sample classification according to claim 1, characterized in that: The method constructs a prompt pool including diverse prompt strategies and generates a real remote sensing dataset, including: Selecting a first preset number of task categories from the remote sensing image dataset as classification tasks; Randomly selecting a second preset number of images in each task category to construct a real remote sensing dataset to simulate a few-shot setting; wherein the real remote sensing dataset includes real images; Based on prior knowledge of the classification task, a first prompt template is constructed; wherein the prompt content of the first prompt template includes the covered domain, style, and camera perspective; A second prompt template is constructed using the large language model; wherein the second prompt template is used to input a category name and output multiple prompt texts including the category name; A third prompt template is constructed using the image annotation model; wherein the third prompt template is used to input a real image and output a prompt text describing the real image; The prompt texts generated by the first prompt template, the second prompt template and the third prompt template are integrated into a prompt pool, and the prompt pool is stored in the form of a two-dimensional list; wherein the first dimension of the two-dimensional list represents the number of task categories, and the second dimension represents a list of all prompt texts under each task category.

3. The method for adaptively constructing a dataset for remote sensing image few-sample classification according to claim 1, characterized in that: The synthetic dataset includes multiple generated datasets, each generated dataset corresponds to a prompt set in the prompt pool and includes multiple images generated by the prompt text and their corresponding labels, including: The generated dataset includes multiple elements, each element corresponds to an image generated by a prompt text; Assign a corresponding label to each image in the generated dataset based on the task category to which each prompt text in the prompt set belongs; A two-dimensional list is constructed to represent the synthetic dataset to determine the task category to which each image in the synthetic dataset belongs and the prompt text from which it originates. The first dimension of the two-dimensional list represents the number of task categories, and the second dimension represents the total number of images generated by all prompt texts under each category.

4. The method for adaptively constructing a dataset for remote sensing image few-sample classification according to claim 1, characterized in that: The method of evaluating a deep network classifier using a real remote sensing dataset includes: Use the generated dataset to train a deep network classifier; Evaluate the trained deep network classifier on real remote sensing datasets; According to the classification loss of the deep network classifier on the real remote sensing dataset, the task matching score of each prompt set is obtained; the smaller the task matching score, the better the performance of the deep network classifier on the prompt set, and the higher the task matching score; Associate each cue set with its corresponding task matching score to form a cue set-task matching score pair.

5. The method for adaptively constructing a dataset for remote sensing image few-sample classification according to claim 4, characterized in that: Optimizing the prompt set and obtaining the optimized prompt set includes: Given a batch of initial prompt sets as the initial prompt set population of the evolutionary algorithm; Iterate the optimization steps until the maximum number of iterations is met; Output an optimized prompt set; wherein the optimized prompt set has a high task matching score on a real remote sensing dataset.

6. The method for adaptively constructing a dataset for remote sensing image few-sample classification according to claim 5, characterized in that: The optimization step comprises: Take the current prompt set population as input; Using the cue set-task matching score pairs, the better cue set is selected as the next generation cue set population according to the search strategy of the evolutionary algorithm; The selected next generation prompt set population is used as the input of the next generation to update the prompt set population.

7. The method for adaptively constructing a dataset for remote sensing image few-sample classification according to claim 6, characterized in that: Generating an optimal data set from the optimized prompt set includes: Set generation parameters; Setting the generation parameters includes: setting the number of optimal data sets generated and setting the ratio of the generated images for each prompt text; Select the optimal prompt set from the optimized prompt set; Using the generative large model and the optimal prompt set, the optimal dataset is generated according to the set generation parameters.

8. A data set adaptive construction device for remote sensing image few-sample classification, characterized by: include: A construction module is used to construct a prompt pool including diverse prompt strategies and generate a real remote sensing dataset; wherein the diverse prompt strategies include multiple prompt templates; A selection module is used to randomly select prompt texts from the prompt pool to construct multiple prompt sets; each prompt set includes multiple prompt texts, and each prompt set corresponds to a data set generation strategy; A representation module is used to construct a three-dimensional list to represent the prompt set, wherein the first dimension of the three-dimensional list represents the number of prompt sets, the second dimension represents the number of task categories, and the third dimension represents the number of prompt texts selected in each task category; The input module is used to input the prompt text in the prompt set using the generative large model to generate the corresponding generated dataset; A training module for training a deep network classifier adapted for downstream classification tasks on a synthetic dataset; wherein the synthetic dataset includes multiple generated datasets, each generated dataset corresponding to a prompt set in the prompt pool and including multiple images generated from the prompt text and their corresponding labels; The acquisition module is used to evaluate the deep network classifier using real remote sensing datasets, optimize the prompt set, and obtain the optimized prompt set; The generation module is used to generate an optimal dataset from the optimized prompt set; wherein the optimal dataset is the dataset that is most suitable for the current task.

9. A dataset adaptive construction server for remote sensing image few-sample classification, characterized by: including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method according to any one of claims 1 to 7 can be implemented.

Images