Remote sensing image processing method and system based on space-based remote sensing model, electronic equipment and medium

Abstract

The invention discloses a remote sensing image processing method and system based on a space-based remote sensing model, electronic equipment and a medium, and the method comprises the steps: obtaining and marking remote sensing image data; setting a task description instruction for a remote sensing image processing task, designing a cue word template, filling the task description instruction into the cue word template, and inputting the cue word template into the large language model to realize expansion of the task description instruction to obtain a task description instruction set; setting an output form instruction for a remote sensing image processing task, filling the output form instruction into a cue word template, and inputting the cue word template into the large language model to realize expansion of the output form instruction to obtain an output form instruction set; constructing a multi-modal dialogue data set based on the labeled remote sensing image data, the task description instruction set and the output form instruction set; training a space-based remote sensing model by using the multi-modal dialogue data set; the trained space-based remote sensing model is used for outputting an answer according to the input remote sensing image to be processed, the task description instruction and the output form instruction.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and in particular relates to a remote sensing image processing method, system, electronic equipment and medium based on a space-based remote sensing model. Background Art

[0002] With the rapid development of space technology and remote sensing technology, satellite remote sensing has become the core means of obtaining information on the earth's surface. Modern remote sensing satellites have the ability to capture high-resolution, multi-spectral image data, which are widely used in environmental monitoring, urban planning, agricultural management, disaster warning and other fields. The richness and complexity of remote sensing data have put forward higher requirements for data processing and analysis, especially in large-scale, multi-task and multi-modal data environments.

[0003] Currently, many remote sensing image processing systems and models are mainly optimized for a single task, such as image classification, image description, object detection, or semantic segmentation. Although these methods perform well in their respective application scenarios, they have significant limitations when processing multi-task and multi-modal data. Specific problems include:

[0004] (1) Insufficient multi-task processing capabilities: Existing remote sensing data processing methods usually focus on processing specific tasks, such as image classification or target detection. For example, some models may focus on classifying buildings in remote sensing images or detecting specific targets in images. However, in practical applications, remote sensing images often need to complete multiple tasks at the same time, such as detecting buildings, extracting road information, and analyzing environmental changes in a city remote sensing image. Existing models can usually only handle one of these tasks and cannot effectively and comprehensively handle multiple tasks.

[0005] (2) Weak response to multimodal data: The multimodal data involved in remote sensing tasks include images, text descriptions, geographic information, etc., which have different characteristics and formats. Existing models often face the challenge of information fusion when processing these multimodal data. For example, combining the visual information of remote sensing images with the corresponding geographic text descriptions to provide a comprehensive environmental analysis is still a complex task for existing models. When models effectively integrate information from different modalities, they often find it difficult to realize their potential, which affects the overall task processing effect.

[0006] (3) Insufficient command response capability: In practical applications, users often require models to perform a variety of tasks based on different commands. For example, users may want the model to complete a specific task based on the command of "detecting all buildings in the city" or "analyzing the environment around a water body." However, existing multimodal models generally lack sufficient flexibility and accuracy when processing these complex commands. Many models perform poorly in responding to specific commands, resulting in poor results when processing a variety of tasks, which limits the scope of application and practical application value of the models.

[0007] In summary, the existing remote sensing data processing methods show obvious deficiencies when facing complex, multi-task and multi-modal data. Therefore, it is necessary to develop a new method that can process multi-task and multi-modal data at the same time and has flexible command response capabilities. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a remote sensing image processing method, system, electronic equipment, and medium based on a space-based remote sensing model.

[0009] In a first aspect, an embodiment of the present invention provides a remote sensing image processing method based on a space-based remote sensing model, the method comprising:

[0010] Acquire remote sensing image data and annotate the remote sensing image data;

[0011] For remote sensing image processing tasks, task description instructions are set, prompt word templates are designed, task description instructions are filled into the prompt word templates and then input into the large language model to achieve the expansion of task description instructions and obtain task description instruction sets;

[0012] Output form instructions are set for remote sensing image processing tasks, the output form instructions are filled into the prompt word template and then input into the large language model to achieve the expansion of the output form instructions and obtain the output form instruction set;

[0013] Construct a multimodal dialogue dataset based on annotated remote sensing image data, task description instruction set, and output form instruction set;

[0014] A space-based remote sensing model is trained using a multimodal dialogue dataset; wherein the trained space-based remote sensing model is used to output answers according to input remote sensing images to be processed, task description instructions, and output form instructions.

[0015] In a second aspect, an embodiment of the present invention provides a remote sensing image processing system based on a space-based remote sensing model, which is used to implement the above-mentioned remote sensing image processing method based on a space-based remote sensing model, and the system includes:

[0016] A remote sensing image data acquisition and annotation module is used to acquire remote sensing image data and annotate the remote sensing image data;

[0017] The task description instruction set building module is used to set the task description instruction for the remote sensing image processing task, design the prompt word template, fill the task description instruction into the prompt word template and then input it into the large language model to achieve the expansion of the task description instruction and obtain the task description instruction set;

[0018] The output form instruction set needle construction module is used to set the output form instruction for the remote sensing image processing task, fill the output form instruction into the prompt word template and then input it into the large language model to achieve the expansion of the output form instruction and obtain the output form instruction set;

[0019] A multimodal dialogue dataset construction module is used to construct a multimodal dialogue dataset based on annotated remote sensing image data, task description instruction set, and output form instruction set;

[0020] Space-based remote sensing model training module, used to train space-based remote sensing models using multimodal dialogue datasets;

[0021] The remote sensing image processing module is used to output the answer after remote sensing image processing based on the input remote sensing image to be processed, task description instructions, and output form instructions through the trained space-based remote sensing model.

[0022] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned remote sensing image processing method based on the space-based remote sensing model.

[0023] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned remote sensing image processing method based on the space-based remote sensing model.

[0024] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned remote sensing image processing method based on the space-based remote sensing model.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention provides a remote sensing image processing method based on a space-based remote sensing model, by annotating the acquired remote sensing image data, setting a task description instruction and an output form instruction for each remote sensing image processing task, constructing a multimodal dialogue data set based on the annotated remote sensing image data, the task description instruction set, and the output form instruction set, and training the space-based remote sensing model using the multimodal dialogue data set; the trained space-based remote sensing model is used to output an answer according to the input remote sensing image to be processed, the task description instruction, and the output form instruction. Thus, the trained space-based remote sensing model can effectively and comprehensively process multiple tasks, improve the responsiveness of multimodal data, and improve the responsiveness of instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of a process flow of a remote sensing image processing method based on a space-based remote sensing model provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a remote sensing image preprocessing process provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a remote sensing image processing system based on a space-based remote sensing model provided by an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0034] like Figure 1 As shown, an embodiment of the present invention provides a remote sensing image processing method based on a space-based remote sensing model, and the method includes the following steps:

[0035] Step S1, acquiring remote sensing image data and performing preprocessing.

[0036] Specifically, Figure 2 As shown, the step S1 specifically includes:

[0037] Step S101, collect remote sensing image data from multiple remote sensing data sources, the data sources include but are not limited to satellites, drones, ground stations and other channels. The remote sensing image data include but are not limited to high-resolution optical images, synthetic aperture radar images, multispectral / hyperspectral images and geographic information system data.

[0038] Step S102, preprocessing the acquired remote sensing image data, wherein the processing steps include: format unification, data cleaning, noise filtering, image segmentation and standardization processing.

[0039] Among them, format conversion: convert the format of the collected remote sensing image data to ensure that the format of all remote sensing image data is consistent for subsequent processing. For example, convert all remote sensing image data into a format suitable for reading, such as JPEG or TIFF, and unify the size and resolution of remote sensing image data to ensure that the classification model can process these remote sensing image data.

[0040] Data cleaning: Clean the remote sensing image data to remove invalid or erroneous remote sensing image data and improve data quality. For example, remove blurry remote sensing images caused by cloud cover or remove remote sensing images with significant noise to ensure that the classification model receives high-quality input.

[0041] Image slicing. For remote sensing image files that are too large, split them into smaller parts to meet the needs of the classification task. For example, a large image can be cut into multiple sub-images of a fixed size, and each sub-image can be used as an independent classification input, thereby improving the accuracy and efficiency of the classification model.

[0042] Step S2: labeling the preprocessed remote sensing image data.

[0043] In this example, the objects of interest to be labeled are divided into four categories. The first category is civil and industrial buildings, such as houses, storage tanks, dams, transmission towers, etc.; the second category is transportation vehicles and parking facilities, such as ships, cars, airplanes, roads, ports, etc.; the third category is public services and entertainment facilities, such as cemeteries, swimming pools, basketball courts, football fields, parks, etc.; the fourth category is man-made and natural resources, such as cultivated land, paddy fields, mining areas, forests, rivers, lakes, etc.

[0044] Specifically, for scene-level remote sensing tasks, target-level remote sensing tasks and pixel-level remote sensing tasks, classification labels, description labels and target location labels are set for civil and industrial buildings, transportation vehicles and parking facilities, public services and entertainment facilities, man-made and natural resources in remote sensing image data to complete labeling.

[0045] Furthermore, the scene-level remote sensing tasks include: remote sensing image classification tasks, remote sensing image description tasks, counting tasks, and visual question answering tasks.

[0046] Among them, the remote sensing image classification task is to identify and classify the main types of objects in remote sensing images. For example, distinguish different types of land cover such as houses, roads, rivers, and forests.

[0047] Among them, the remote sensing image description task is: to give a brief or detailed text description of the main elements in the remote sensing image. For example, to give a brief description of the main elements in the remote sensing image: there is a wide river next to a dense forest, and there are a few houses and farmland on the river bank; to provide more details and context for the main elements in the remote sensing image, usually including no less than 60 words or more than 3 sentences to explain the features in the image for detailed text description: there is a large forest in the lower left of the image. This forest is composed of a variety of tree species with high density, and some open spaces can be seen from the air. In the center of the image is a wide stream of water, and there are a few town buildings on the upper right bank of the river. The color of the houses is mainly light gray, and some roofs are decorated with red or blue tiles. In addition, there are some farmlands on the edge of the town, showing regular geometric shapes, suggesting the existence of agricultural production activities.

[0048] The counting task is to count the number of objects of a certain category in a remote sensing image. For example:

[0049] Question: "Calculate the total number of houses in an area and express it as a number."

[0050] Answer: "14."

[0051] Among them, the Visual Question Answering (VQA) task is to answer questions about the attributes of remote sensing images, such as the color, shape, and distribution of the main elements in the remote sensing images. For example:

[0052] Ask: "What color are the houses in this area?"

[0053] A: "The houses in this area are mostly light grey, with some having red or blue tiles on the roofs."

[0054] Question: “What are the characteristics of the building’s appearance?”

[0055] A: “The buildings are mainly rectangular, but there are also some circular or polygonal designs.”

[0056] Question: “How are trees distributed in a forest?”

[0057] Answer: "The trees in the forest are unevenly distributed, with sparse trees in the north and dense forests in the south."

[0058] Furthermore, the target-level remote sensing tasks include: rectangular area classification tasks, rectangular area description tasks, target detection tasks, and visual positioning tasks.

[0059] The target-level task focuses on the object of interest, and uses rectangular boxes and text to mark the location and category of the object respectively. Its form consists of text and location coordinates, including four subtasks: rectangular area classification, rectangular area description, target detection, and visual positioning. It is worth mentioning that the rectangular box includes two types: horizontal box and rotation box. The horizontal box only marks the coordinates of the upper left corner and the lower right corner, while the rotation box starts from the upper left corner and marks the coordinates of the four vertices in a counterclockwise direction. Among them, the coordinates in the image include horizontal and vertical coordinates. The origin of the coordinate system is the upper left corner of the image, the right is the positive direction of the X axis, and the downward is the positive direction of the Y axis.

[0060] Among them, the rectangular region classification task is to set a rectangular region for the remote sensing image to be processed, and identify and classify objects such as vehicles and buildings in the rectangular region. For example, use a rectangular frame to frame each object in the remote sensing image and mark its category, such as car, ship, house, etc.

[0061] The rectangular region description task is to give a brief text description of the object in the rectangular box. For example, given a rectangular box, output a text like "a red car parked on the roadside" or "a three-story white house" to describe the object in the rectangular box.

[0062] Among them, the target detection task is to identify and locate multiple objects in remote sensing images. For example, detect all vehicles in the remote sensing image and mark the position coordinates of all vehicles in the remote sensing image in turn.

[0063] Among them, the visual localization task is to use the known object location information to determine the exact location of a region or object in the image. For example, through text like "a red car parked on the side of the road" or "a three-story white building", the specific coordinates of the object described in the text in the remote sensing image are determined and represented by a rectangular box.

[0064] Furthermore, the pixel-level remote sensing task requires the target area to be represented in the form of a polygon, which is more flexible than a rectangular box and suitable for processing irregularly shaped objects to provide a more detailed understanding of remote sensing images; the pixel-level remote sensing task includes: polygonal area classification task, semantic segmentation task, and change detection task. It is worth mentioning that the polygonal box will start from the upper left corner and mark the coordinates of all vertices in a counterclockwise direction. Among them, the coordinates in the image include horizontal and vertical coordinates. The origin of the coordinate system is the upper left corner of the image, the right is the positive direction of the X axis, and the downward is the positive direction of the Y axis.

[0065] The polygonal region classification task is to identify and classify the categories of objects in the polygonal region selected in the remote sensing image to be processed. For example, farmland, water bodies, etc. in a polygonal region are classified. Specifically, in the annotation process, each object is framed with a polygonal frame and its category is marked, such as farmland, river, lake, etc.

[0066] The semantic segmentation task is to assign each pixel in the remote sensing image to a corresponding category. The semantic segmentation task is mainly used to finely divide the components of the image. For example, the task of segmenting all river areas in a remote sensing image requires marking the position coordinates of the polygon vertices that surround all rivers in the remote sensing image in sequence.

[0067] The change detection task is to compare the differences between two or more remote sensing images to find the changes. For example, compare satellite images at two points in time to detect whether construction or destruction has occurred in a certain area, and mark the changed area with a polygon.

[0068] Step S3, setting task description instructions for remote sensing image processing tasks, designing prompt word templates, filling the task description instructions into the prompt word templates and then inputting them into the large language model to achieve expansion of the task description instructions and obtain a task description instruction set.

[0069] For example, for an image classification task, the task description instruction can be designed as: determine the category of the image content; for an image description task, the task description instruction can be designed as: describe the content in the image, describe the content in the image in detail; for a counting task, the task description instruction can be designed as: count the number of cells in the image. <class>For visual question answering tasks, the task description instructions can be designed as follows: <class>shape;

[0070] For example, for the rectangular area classification task, the task description instruction can be designed as: <region>Which class does the object in belong to? For the rectangular area description task, the task description instruction can be designed as: description <region>For the target detection task, the task description instruction can be designed as: detect the objects in the image <class>;For the visual localization task, the task description instruction can be designed as: find the area in the image to match the corresponding description:.

[0071] For example, for the polygonal area classification task, the task description instruction can be designed as: <region>Which class does the object in the image belong to? For the semantic segmentation task, the task description instruction can be designed as: <class>;For the change detection task, the task description instruction can be designed as: find the changed area in two images.

[0072] It should be noted that in actual use, <class>Will be replaced by words that represent the category; will be replaced by a sentence or paragraph of description; <region>Will be replaced by a special set of tokens representing horizontal bounding rectangles, rotated bounding rectangles, and polygon vertex coordinates.

[0073] Among them, the horizontal box rectangle will be represented as " <box> <x1> <y1> <x2> <y2>< / y2> < / x2> < / y1> < / x1> < / box> ",For example" <box> <554><821><592><846>< / box> ", (X1, Y1) and (X2, Y2) are the coordinates of the upper left corner and lower right corner of the rectangle respectively, and their values ​​are expressed in thousandths.

[0074] The rotation box rectangle will be represented as " <quad> <x1> <y1> <x2> <y2> <x3> <y3> <x4> <y4>< / y4> < / x4> < / y3> < / x3> < / y2> < / x2> < / y1> < / x1> < / quad> ",For example" <quad> <703><605><703><397><431><397><431><605>< / quad> ”, where (X1, Y1), (X2, Y2), (X3, Y3) and (X4, Y4) are the coordinates of the four vertices of the rectangle, and their values ​​are expressed in thousandths.

[0075] Polygons will be represented as " <poly> <x1> <y1> … <xn> <yn>< / yn> < / xn> < / y1> < / x1> < / poly> ",For example" <poly> <934><1000><783><901><618><836><549><820><501><1000><687><1000><703><974><741><1000>< / poly> ", where (Xn, Yn) represents the coordinates of the polygon vertices, and its value is expressed in thousandths. In addition, the special word " <box> ”、" <quad>"and" <poly> "Represents the beginning of the shape, and the special word"< / poly> < / quad> < / box> ”, "” and "” represent the end of the shape.

[0076] Furthermore, the prompt word template is set as: "Please follow the example below and give X sentences of instructions with similar meanings. [Task description instruction example]".

[0077] For each task, the task description instructions are expanded. For example, taking the image description task as an example, the task description instructions are filled into the prompt word template, and the prompt words obtained are "Please follow the example below and give X sentences with similar meanings. [Describe the content in the image in detail]". The prompt words are input into the large language model to obtain the expanded task description instructions. For example, describe the main features and details in the image in detail, and summarize the main content and scenes shown in the image.

[0078] Step S4, setting output form instructions for the remote sensing image processing task, filling the output form instructions into the prompt word template and then inputting them into the large language model to achieve the expansion of the output form instructions and obtain an output form instruction set.

[0079] Specifically, for image classification, rectangular region classification, and polygonal region classification tasks, the output form instruction can be designed as: answer with one or a few words.

[0080] For target detection and visual positioning tasks, the output form instructions can be designed as: represented by a horizontal box or represented by a rotating box.

[0081] For each output form, the output form instruction is expanded. For example, taking the horizontal box output as an example, the output form instruction is filled into the prompt word template, and the prompt word obtained is: "Please follow the example below and give X sentences with similar meanings. [Represented by a horizontal box]". The prompt word is input into the large language model to obtain the expanded output form instruction, for example: Please use a rectangular box to represent this area in the image and use a horizontal bounding box to describe it.

[0082] Step S5, constructing a multimodal dialogue dataset based on the annotated remote sensing image data, the task description instruction set, and the output form instruction set.

[0083] Specifically, step S5 includes:

[0084] Integrate each remote sensing image data with its corresponding classification label, description label, and target location label to form a comprehensive remote sensing image sample;

[0085] For a certain remote sensing image processing task, one or more task description instructions are selected from the task description instruction set for a certain remote sensing image comprehensive sample, and one output form instruction set is selected from the output form instruction set;

[0086] The selected task description instructions and output form instruction sets are spliced ​​together as questions in a question-answer pair; the label of the remote sensing image comprehensive sample corresponding to the question is used as the answer in the question-answer pair, thereby obtaining a question-answer pair;

[0087] Traverse each comprehensive remote sensing image sample to build a question-answer pair for each remote sensing image processing task, and use all question-answer pairs as a multimodal dialogue dataset.

[0088] For example, for an image classification task, a question-answer pair is formed; for example:

[0089] Question: "Determine the category of the image content and answer with one or a few words."

[0090] Answer: "The forest."

[0091] For image description tasks, we combine them into question-answer pairs, for example:

[0092] Q: "Describe what is in the image."

[0093] A: "There is a wide river next to a dense forest, with a few houses and farmland along the river bank."

[0094] For the image description task, we combine them into question-answer pairs, for example:

[0095] Q: "Describe in detail what is in the image."

[0096] Answer: "There is a large forest in the lower left corner of the image. This forest is composed of a variety of tree species with a high density, and some open spaces can be seen from the air. In the center of the image is a wide stream, and there are a few town buildings on the river bank in the upper right corner. The color of the houses is mainly light gray, and some roofs are decorated with red or blue tiles. In addition, there are some farmlands on the edge of the town, showing regular geometric shapes, suggesting the existence of agricultural production activities."

[0097] For counting tasks, we combine them into question-answer pairs, for example:

[0098] Question: "Count the number of houses in an image."

[0099] Answer: "14."

[0100] For visual question answering tasks, there is no need to combine them, as they are question-answer pairs themselves, for example:

[0101] Question: "What color are the houses in this area?"

[0102] A: "The houses in this area are mostly light grey, with some having red or blue tiles on the roofs."

[0103] Q: "What are the characteristics of the building's appearance?"

[0104] A: "The buildings are mainly rectangular, but there are also some circular or polygonal designs."

[0105] Question: "How are trees distributed in a forest?"

[0106] A: "The trees in the forest are unevenly distributed, with sparse trees in the north and dense forests in the south."

[0107] For the rectangular area classification task, we combine them into question-answer pairs, for example:

[0108] ask:" <box> <509><491><556><509>< / box> To which class does the object in the picture belong? Answer with one or a few words. "

[0109] Answer: "Car."

[0110] ask:" <quad> <417><766><417><781><478><781><478><763>< / quad> To which class does the object in the picture belong? Answer with one or a few words. "

[0111] Answer: "Plane."

[0112] For the rectangular area description task, we combine them into question-answer pairs, for example:

[0113] Q: "Description <box> <486><850><531><899>< / box> objects in it."

[0114] A: "A swimming pool with an area of ​​approximately 457 square meters, a pond and a basketball court nearby."

[0115] For target detection tasks, we combine them into question-answer pairs, for example:

[0116] Question: "Detect cars in an image. Represent them as horizontal boxes."

[0117] answer:" <box> <853><565><942><795>< / box> <box> <795><587><884><817>< / box> ”

[0118] Question: "Detect cars in an image. Represent them as rotating boxes."

[0119] answer: " <quad> <853><576><895><795><942><779><897><565>< / quad> <quad> <795><600><837><817><884><799><835><587>< / quad> ”

[0120] For visual localization tasks, we combine them into question-answer pairs, for example:

[0121] Question: "Find the area in the image that matches the description: a swimming pool of approximately 457 square meters with a pond and a basketball court nearby. Represent it with a horizontal box."

[0122] answer:" <box> <486><850><531><899>< / box> ”

[0123] For polygonal area classification tasks, we combine them into question-answer pairs, for example:

[0124] ask: " <poly> <263><921><346><916><345><904><340><906><334><906><349><900><347><871><310><845><259><847><251><882><238><886><238><913><243><925><247><906><262><902><247><906>< / poly> To which class does the object in the picture belong? Answer with one or a few words. "

[0125] Answer: "An intersection."

[0126] For semantic segmentation tasks, we combine them into question-answer pairs, for example:

[0127] Question: "Segment houses in an image."

[0128] answer: " <poly> <49><0><0><98><0><257><20><265><31><235><54><244><62><235><92><244><108><237><109><220><73><205><106><163><106><138><144><62><169><38><164><18><173><0>< / poly> <poly> <183><0><175><13><199><23><210><0>< / poly> <poly> <221><0><182><80><246><110><289><22><245><0>< / poly> <poly> <335><142><356><147><401><67><321><26><282><118><302><127><301><140><330><151>< / poly> ”

[0129] For the change detection task, we combine them into question-answer pairs. For example:

[0130] Question: "Find the changing regions in two images"

[0131] answer: " <poly> <124><500><169><438><165><432><138><423><95><493>< / poly> <poly> <289><500><347><458><321><425><210><497><199><488><179><500>< / poly> ”

[0132] Step S6, using the multimodal dialogue data set to train the space-based remote sensing model; wherein the trained space-based remote sensing model is used to output an answer according to the input remote sensing image to be processed, the task description instruction, and the output form instruction.

[0133] Specifically, during the training process, this example improves the performance of the space-based remote sensing model by introducing attention mechanisms and cross-modal fusion technology.

[0134] At the same time, the space-based remote sensing model is fine-tuned based on the multimodal dialogue dataset designed in S4 to ensure that the space-based remote sensing model can respond to complex questioning methods and enhance the flexibility and adaptability of the model. For example, the instructions in the classification task can have many forms, including "please classify the image", "select the appropriate category for the image", "perform the image classification task", etc. Further fine-tuning can enable the space-based remote sensing model to understand that the above instructions are all used to perform image classification tasks, thereby improving the human-computer interaction experience.

[0135] Furthermore, the fine-tuned space-based remote sensing model is comprehensively evaluated using test sets (such as EuroSAT, RSD46-WHU, Hurricane Damage, AID, RSITMD, MAR, DOTA, DIOR, HRSC2016, RSICD, DIOR-RSVG, HRBEN, LRBEN, DLRSD, LoveDA, GEONRW, etc.). The evaluation indicators should cover multiple aspects such as precision and recall, and test different tasks and data modalities separately. The following are some of the evaluation indicators.

[0136] Table 1 Image classification task evaluation indicators

[0137]

[0138] Table 2 Image description task evaluation indicators

[0139]

[0140] Table 3 Evaluation indicators of image detailed description task

[0141]

[0142] Table 4 Evaluation indicators of visual question answering tasks

[0143]

[0144] Table 5 Visual positioning (horizontal box) task evaluation indicators

[0145]

[0146] Table 6 Object detection (horizontal box) task evaluation indicators

[0147]

[0148] Table 7 Semantic segmentation task evaluation indicators

[0149]

[0150] It can be seen from Tables 1-7 above that the space-based remote sensing model trained by the present invention performs excellently in image classification tasks, image description tasks, image detailed description tasks, visual question answering tasks, visual positioning (horizontal box) tasks, target detection (horizontal box) tasks, and semantic segmentation tasks.

[0151] On the other hand, Figure 3 As shown, an embodiment of the present invention provides a remote sensing image processing system based on a space-based remote sensing model, which is used to implement the above-mentioned remote sensing image processing method based on a space-based remote sensing model. The system includes:

[0152] A remote sensing image data acquisition and annotation module is used to acquire remote sensing image data and annotate the remote sensing image data;

[0153] The task description instruction set building module is used to set the task description instruction for the remote sensing image processing task, design the prompt word template, fill the task description instruction into the prompt word template and then input it into the large language model to achieve the expansion of the task description instruction and obtain the task description instruction set;

[0154] The output form instruction set needle construction module is used to set the output form instruction for the remote sensing image processing task, fill the output form instruction into the prompt word template and then input it into the large language model to achieve the expansion of the output form instruction and obtain the output form instruction set;

[0155] A multimodal dialogue dataset construction module is used to construct a multimodal dialogue dataset based on annotated remote sensing image data, task description instruction set, and output form instruction set;

[0156] Space-based remote sensing model training module, used to train space-based remote sensing models using multimodal dialogue datasets;

[0157] The remote sensing image processing module is used to output the answer after remote sensing image processing based on the input remote sensing image to be processed, task description instructions, and output form instructions through the trained space-based remote sensing model.

[0158] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0159] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0160] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the remote sensing image processing method based on the space-based remote sensing model as described above. Figure 4 As shown in FIG. 1 , a hardware structure diagram of any device with data processing capability in which the remote sensing image processing method based on the space-based remote sensing model provided by the embodiment of the present invention is located, except Figure 4 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0161] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the remote sensing image processing method based on the space-based remote sensing model as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0162] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0163] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.< / region> < / class> < / class> < / region> < / class> < / region> < / region> < / class> < / class>

Claims

1. A remote sensing image processing method based on a space-based remote sensing model, characterized in that: The method comprises: Acquire remote sensing image data and annotate the remote sensing image data; For remote sensing image processing tasks, task description instructions are set, prompt word templates are designed, task description instructions are filled into the prompt word templates and then input into the large language model to achieve the expansion of task description instructions and obtain task description instruction sets; Output form instructions are set for remote sensing image processing tasks, the output form instructions are filled into the prompt word template and then input into the large language model to achieve the expansion of the output form instructions and obtain the output form instruction set; Construct a multimodal dialogue dataset based on annotated remote sensing image data, task description instruction set, and output form instruction set; A space-based remote sensing model is trained using a multimodal dialogue dataset; wherein the trained space-based remote sensing model is used to output answers according to input remote sensing images to be processed, task description instructions, and output form instructions.

2. The remote sensing image processing method based on the space-based remote sensing model according to claim 1, characterized in that: The process of labeling remote sensing image data includes: For scene-level remote sensing tasks, target-level remote sensing tasks and pixel-level remote sensing tasks, classification labels, description labels and target location labels are set for civil and industrial buildings, transportation vehicles and parking facilities, public services and entertainment facilities, man-made and natural resources in remote sensing image data to complete labeling.

3. The remote sensing image processing method based on the space-based remote sensing model according to claim 2 is characterized in that: The target position label includes a horizontal box, a rotation box, and a polygonal box; The coordinates of the upper left corner point and the lower right corner point are marked in the horizontal frame; The coordinates of the four vertices in the rotation frame are marked in sequence from the upper left corner point in a counterclockwise direction; The coordinates of all vertices in the polygonal frame are marked in sequence starting from the upper left corner and in a counterclockwise direction.

4. The remote sensing image processing method based on a space-based remote sensing model according to claim 2, characterized in that: The civil and industrial buildings include: houses, storage tanks, dams, transmission towers; The transportation vehicles and parking facilities include: ships, cars, airplanes, roads, and ports; The public services and recreational facilities include: cemeteries, swimming pools, basketball courts, football fields, and parks; The man-made and natural resources include: cultivated land, paddy fields, mining areas, forests, rivers and lakes.

5. The remote sensing image processing method based on a space-based remote sensing model according to claim 2, characterized in that: The scene-level remote sensing tasks include: remote sensing image classification tasks, remote sensing image description tasks, counting tasks, and visual question answering tasks; The target-level remote sensing tasks include: rectangular area classification tasks, rectangular area description tasks, target detection tasks, and visual positioning tasks; The pixel-level remote sensing tasks include: polygon area classification tasks, semantic segmentation tasks, and change detection tasks.

6. The remote sensing image processing method based on a space-based remote sensing model according to claim 1, characterized in that: The process of constructing a multimodal dialogue dataset based on annotated remote sensing image data, task description instruction set, and output form instruction set includes: Integrate each remote sensing image data with its corresponding classification label, description label, and target location label to form a comprehensive remote sensing image sample; For a certain remote sensing image processing task, one or more task description instructions are selected from the task description instruction set for a certain remote sensing image comprehensive sample, and one output form instruction set is selected from the output form instruction set; The selected task description instructions and output form instruction sets are spliced ​​together as questions in a question-answer pair; the label of the remote sensing image comprehensive sample corresponding to the question is used as the answer in the question-answer pair, thereby obtaining a question-answer pair; Traverse each comprehensive remote sensing image sample to build a question-answer pair for each remote sensing image processing task, and use all question-answer pairs as a multimodal dialogue dataset.

7. A remote sensing image processing system based on a space-based remote sensing model, characterized in that: A remote sensing image processing method based on a space-based remote sensing model for implementing any one of claims 1 to 6, the system comprising: A remote sensing image data acquisition and annotation module is used to acquire remote sensing image data and annotate the remote sensing image data; The task description instruction set building module is used to set the task description instruction for the remote sensing image processing task, design the prompt word template, fill the task description instruction into the prompt word template and then input it into the large language model to achieve the expansion of the task description instruction and obtain the task description instruction set; The output form instruction set needle construction module is used to set the output form instruction for the remote sensing image processing task, fill the output form instruction into the prompt word template and then input it into the large language model to achieve the expansion of the output form instruction and obtain the output form instruction set; A multimodal dialogue dataset construction module is used to construct a multimodal dialogue dataset based on annotated remote sensing image data, task description instruction set, and output form instruction set; Space-based remote sensing model training module, used to train space-based remote sensing models using multimodal dialogue datasets; The remote sensing image processing module is used to output the answer after remote sensing image processing based on the input remote sensing image to be processed, task description instructions, and output form instructions through the trained space-based remote sensing model.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the remote sensing image processing method based on the space-based remote sensing model as described in any one of claims 1-6 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the remote sensing image processing method based on the space-based remote sensing model as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the remote sensing image processing method based on the space-based remote sensing model described in any one of claims 1 to 6 is implemented.

Images