Remote sensing multi-task decision conversion method based on anaphora segmentation

Abstract

The invention provides a remote sensing multi-task decision conversion method based on anaphora segmentation, and the method comprises the steps: 1) employing a remote sensing anaphora segmentation model as a basic model, constructing a large-scale generalized anaphora segmentation data set, and carrying out the pre-training of the remote sensing anaphora segmentation model; 2) converting the output of the anaphora segmentation task into the output of a semantic segmentation task, generating the output of a pixel-level task, and achieving the decision conversion of the pixel-level task; (3) pixel-level task output is converted into region-level task output, and decision conversion of region-level tasks is achieved; 4) based on the pixel-level task output and the area-level task output, deriving image-level task output, and realizing decision conversion of the image-level task; according to the method, a multi-task unified remote sensing visual basic model is constructed, the simplification and robustness of a model architecture are effectively improved, and an efficient and unified solution for remote sensing data processing and analysis is provided.

Description

Technical Field

[0001] The present invention relates to a remote sensing multi-task decision transformation method based on reference segmentation, and belongs to the technical field of remote sensing image processing and computer vision. Background Art

[0002] Remote sensing technology has developed rapidly over the past few decades and has become an indispensable tool for Earth observation and practical applications. Remote sensing images acquired through platforms such as satellites and drones provide high-resolution information on the Earth's surface and are widely used in a variety of fields, including urban planning, environmental monitoring, agricultural management, and disaster response. In urban planning, remote sensing data can be used to help analyze urban expansion and land use changes and optimize urban layout. In environmental monitoring, remote sensing technology supports long-term observation of forest cover and vegetation health, providing a scientific basis for ecological protection. Agricultural management uses remote sensing images for crop monitoring and yield prediction, improving the efficiency and accuracy of agricultural production. At the same time, remote sensing technology can provide key geographic information in the rapid response after natural disasters, supporting disaster relief and recovery efforts. Therefore, the organic combination of remote sensing technology and the field of artificial intelligence to enhance the ability to process and analyze remote sensing data has become a hot topic and focus in the field of remote sensing research.

[0003] Currently, the common remote sensing multi-task basic model methods are generally divided into two categories: one method configures a specific decoder for each downstream task, such as application publication number CN119007031A (Multimodal unified representation learning method for remote sensing images based on mask image modeling), which sets decoders for semantic segmentation and scene classification tasks respectively and performs task-specific fine-tuning, which not only separates the shared knowledge between tasks, but also increases the model size and training cost; the other method, such as application publication number CN119152373A (Remote sensing image processing method, system, electronic equipment, and medium based on space-based remote sensing model), unifies the results of each task into a natural language format under the visual language model, but is not suitable for pixel-intensive tasks such as reference segmentation and semantic segmentation. Summary of the Invention

[0004] The present invention proposes a remote sensing multi-task decision transformation method based on reference segmentation, which aims to solve the problems of low efficiency and limited applicability of current remote sensing multi-task basic model methods.

[0005] The technical solution of the present invention is a remote sensing multi-task decision transformation method based on reference segmentation, which includes:

[0006] 1) Using the remote sensing referent segmentation model as the base model, we build a large-scale generalized referent segmentation dataset and pre-train the remote sensing referent segmentation model;

[0007] 2) Convert the output of the referential segmentation task into the output of the semantic segmentation task, generate pixel-level task output, and realize the decision transformation of pixel-level tasks;

[0008] 3) Convert pixel-level task outputs into region-level task outputs to achieve decision-making transformation of region-level tasks;

[0009] 4) Based on the pixel-level task output and region-level task output, the image-level task output is derived to achieve decision transformation of the image-level task.

[0010] Furthermore, the remote sensing multi-task decision transformation method based on reference segmentation further includes:

[0011] 5) Summarize the decision transformation of pixel-level tasks, region-level tasks, and image-level tasks to complete the construction of the remote sensing vision basic model.

[0012] Furthermore, the remote sensing referent segmentation model is used as a basic model to construct a large-scale generalized referent segmentation dataset and pre-train the remote sensing referent segmentation model, specifically including:

[0013] 1-1) Construct a large-scale generalized referent segmentation dataset;

[0014] 1-2) Input data and decode:

[0015] The remote sensing reference segmentation model is pre-trained using the constructed large-scale generalized reference segmentation dataset. The remote sensing image I and the reference text prompt T are input into the visual encoder and text encoder respectively to generate the reference segmentation task output. H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (1):

[0016]

[0017] In formula (1) Refers to the output of the segmentation task, F v Represents the encoding by the visual encoder, F t It means encoding by text encoder, and Q means decoding by decoder;

[0018] 1-3) Calculate the loss and update the parameters:

[0019] Calculate the output of the referent segmentation task and the true mask label M label Segmentation loss Update the parameters of each layer of neurons and iterate several times until the model parameters converge to the optimal value. The optimal goal is to minimize

[0020] Furthermore, the construction of a large-scale generalized referential segmentation dataset specifically includes:

[0021] 1-1-1) One-to-one strategy: A single referential text hint corresponds to a referential segmentation mask of a single object instance, integrating the remote sensing image I, the referential text hint T, and the referential segmentation mask M, and storing them as a (I, T, M) triplet;

[0022] 1-1-2) One-to-many strategy: A single referential text hint corresponds to the referential segmentation mask of all object instances under a single object category. Design the referential text hint T = "{c} in the image", where c represents the object category. Combine the remote sensing image I and the referential segmentation mask M corresponding to all object instances of object category c and store it as a (I, T, M) triplet;

[0023] 1-1-3) One-to-zero strategy: A single referential text prompt corresponds to an empty referential segmentation mask, and a referential text prompt T of a category that does not exist in the remote sensing image I is constructed. The referential segmentation mask M is an all-zero matrix and is stored as a (I, T, M) triplet.

[0024] Furthermore, the output of the referential segmentation task is converted into the output of the semantic segmentation task, generating a pixel-level task output and realizing the decision conversion of the pixel-level task, specifically including:

[0025] For the object category set C of the semantic segmentation task seg The specific category name category seg ,category seg ∈C seg , construct semantic segmentation text prompt T seg =”All{category seg}in the image", input remote sensing reference segmentation model to get category seg The corresponding mask output Then integrate C seg The mask output of all categories in the semantic segmentation task is obtained

[0026] Furthermore, the pixel-level task output is converted into a region-level task output to achieve the decision conversion of the region-level task, specifically including:

[0027] 3-1) Visual localization task:

[0028] First, the visual localization task is regarded as a pixel-level referent segmentation task. The text prompt is input into the remote sensing referent segmentation model to obtain the referent segmentation task output. The output of the referential segmentation task is then converted into the object bounding box of the visual localization task As shown in formula (2):

[0029]

[0030] Where (x min ,y min ) and (x max ,y max ) are the upper left corner and lower right corner of the object bounding box, F M2B It is a conversion strategy from the referential segmentation mask to the object bounding box, that is, by finding the connected domains in the referential segmentation mask, each connected domain determines a corresponding object bounding box by the minimum and maximum values of its sub-pixels on the x-axis and y-axis.

[0031] Furthermore, the conversion of pixel-level task outputs into region-level task outputs to achieve decision conversion of region-level tasks specifically includes:

[0032] 3-2) Object detection task:

[0033] For the object category set C of the target detection task det The specific category name category det ,category det ∈C det , constructing structured text prompts T for target detection det =”All{category det}in the image", input remote sensing reference segmentation model to get category det The corresponding mask output Then, the remote sensing image I is input into the edge detection model F c , get the edge mask H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (3):

[0034]

[0035] use Refine category det The corresponding mask output The boundary of Prevent the generation of erroneous object bounding boxes containing multiple object instances due to mask overlap, as shown in the following formula (4):

[0036]

[0037] Then through F M2B The mask Converted to a single-category object bounding box Then integrate C det All categories in Output, get the object bounding box of target detection

[0038] Furthermore, based on the pixel-level task output and the region-level task output, the image-level task output is derived to realize the decision transformation of the image-level task, specifically including: 4-1) multi-label classification task; 4-2) image classification task; 4-3) image description task; 4-4) target counting task.

[0039] Furthermore, the multi-label classification task specifically includes: for the object category set C of the multi-label classification task MLC The specific category name category MLC ,category MLC ∈C MLC , construct multi-label classification text prompt T MLC =”All{category MLC}in the image", input remote sensing reference segmentation model to get category MLC Probability distribution matrix corresponding to mask output Category is obtained by weighted summation of global average pooling and global maximum pooling MLC The confidence score S c , as shown in formula (5):

[0040]

[0041] Where λ is the balance coefficient of global average pooling and global maximum pooling, H and W represent the height and width of the remote sensing image respectively, and P c (i,j) represents the probability distribution matrix The value of the element at row i and column j, Represents global average pooling, max i,j P c (i,j) represents the global maximum pooling; take C MLC Classes with confidence scores greater than 0.5 are output as multi-label classification tasks.

[0042] Furthermore, the image classification task specifically includes: for the object category set C of the image classification task CLS , use the strategy in the multi-label classification task to obtain the confidence score S corresponding to each category c , then select C CLS The class with the highest confidence score is output as the image classification task;

[0043] The image description task specifically includes: for the potential object category set C in the remote sensing image IC, using the strategy in the target detection task, obtain the bounding box of the target detected object, and then adopt a rule-based image description method: according to the number of the bounding boxes of the target detected object, use the preset template to describe the existence information of the object in the remote sensing image; according to the position of the center of the bounding box of the target detected object in the remote sensing image, use the preset template to describe the position information of the object in the remote sensing image;

[0044] The target counting task specifically includes: object category category for the target counting task CNT , the strategy in the target detection task is used to obtain the single-category object bounding box corresponding to the category, and the number of single-category object bounding boxes is counted as the output of the target counting task.

[0045] Beneficial effects of the present invention:

[0046] 1) This paper conducts multi-task decision transformation based on the coreference segmentation task, fully leveraging the enormous potential of the coreference segmentation model, filling the exploration gap centered on coreference segmentation in the multi-task unified paradigm, and demonstrating strong robustness and applicability in different application scenarios.

[0047] 2) Based on the referent segmentation model, this paper achieves unified processing of multiple tasks, avoiding the complexity of independent decoders for multiple tasks, significantly reducing model size and training costs. Furthermore, through pre-training on a large-scale generalized referent segmentation dataset, it effectively improves pixel-level task accuracy and demonstrates high accuracy in multiple downstream tasks.

[0048] 3) The present invention constructs a multi-task unified remote sensing vision basic model by effectively converting the dense pixel output of referential segmentation into various remote sensing downstream task results at the pixel level, region level, and image level, effectively improving the simplicity and robustness of the model architecture, and providing an efficient and unified solution for remote sensing data processing and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Attachment Figure 1 Schematic diagram of the flow of the remote sensing multi-task decision transformation method based on reference segmentation according to the present invention.

[0050] Attachment Figure 2 This is a visualization of the output of the multi-label classification task in Example 1.

[0051] Attachment Figure 3 This is a visualization of the output of the image classification task in Example 1.

[0052] Attachment Figure 4 This is a visualization of the output of the target counting task in Example 1. DETAILED DESCRIPTION

[0053] A remote sensing multi-task decision transformation method based on reference segmentation, the method comprising:

[0054] 1) Using the remote sensing referent segmentation model as the base model, we build a large-scale generalized referent segmentation dataset and pre-train the remote sensing referent segmentation model;

[0055] 2) Convert the output of the referential segmentation task into the output of the semantic segmentation task, generate pixel-level task output, and realize the decision transformation of pixel-level tasks;

[0056] 3) Convert pixel-level task outputs into region-level task outputs to achieve decision-making transformation of region-level tasks;

[0057] 4) Based on the pixel-level task output and region-level task output, the image-level task output is derived to achieve decision transformation of the image-level task.

[0058] A remote sensing multi-task decision transformation method based on reference segmentation, the method further comprising:

[0059] 5) Summarize the decision transformation of pixel-level tasks, region-level tasks, and image-level tasks to complete the construction of the remote sensing vision basic model.

[0060] The remote sensing referent segmentation model is used as a basic model to construct a large-scale generalized referent segmentation dataset and pre-train the remote sensing referent segmentation model, specifically including:

[0061] 1-1) Construct a large-scale generalized referential segmentation dataset, specifically including:

[0062] 1-1-1) One-to-one strategy: A single referential text hint corresponds to a referential segmentation mask of a single object instance, integrating the remote sensing image I, the referential text hint T, and the referential segmentation mask M, and storing them as a (I, T, M) triplet;

[0063] 1-1-2) One-to-many strategy: A single referential text hint corresponds to the referential segmentation mask of all object instances under a single object category. Design the referential text hint T = "{c} in the image", where c represents the object category. Combine the remote sensing image I and the referential segmentation mask M corresponding to all object instances of object category c and store it as a (I, T, M) triplet;

[0064] 1-1-3) One-to-zero strategy: A single referential text hint corresponds to an empty referential segmentation mask, and a referential text hint T of a category that does not exist in the remote sensing image I is constructed. The referential segmentation mask M is an all-zero matrix and is stored as a (I, T, M) triplet;

[0065] 1-2) Input data and decode:

[0066] The remote sensing reference segmentation model is pre-trained using the constructed large-scale generalized reference segmentation dataset. The remote sensing image I and the reference text prompt T are input into the visual encoder and text encoder respectively to generate the reference segmentation task output. H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (1):

[0067]

[0068] In formula (1) Refers to the output of the segmentation task, F v Represents the encoding by the visual encoder, F t It means encoding by text encoder, and Q means decoding by decoder;

[0069] 1-3) Calculate the loss and update the parameters:

[0070] Calculate the output of the referent segmentation task and the true mask label M label Segmentation loss Update the parameters of each layer of neurons and iterate several times until the model parameters converge to the optimal value. The optimal goal is to minimize

[0071] The output of the referential segmentation task is converted into the output of the semantic segmentation task, generating a pixel-level task output and realizing the decision conversion of the pixel-level task, specifically including:

[0072] For the object category set C of the semantic segmentation task seg The specific category name category seg ,category seg ∈C seg , construct semantic segmentation text prompt T seg =”All{category seg}in the image", input remote sensing reference segmentation model to get category seg The corresponding mask output Then integrate C seg The mask output of all categories in the semantic segmentation task is obtained

[0073] The pixel-level task output is converted into a region-level task output to achieve the decision conversion of the region-level task, specifically including:

[0074] 3-1) Visual localization task:

[0075] First, the visual localization task is regarded as a pixel-level referent segmentation task. The text prompt is input into the remote sensing referent segmentation model to obtain the referent segmentation task output. The output of the referential segmentation task is then converted into the object bounding box of the visual localization task As shown in formula (2):

[0076]

[0077] Where (x min ,y min ) and (x max ,y max ) are the upper left corner and lower right corner of the object bounding box, F M2B The conversion strategy from the referential segmentation mask to the object bounding box is to find the connected domains in the referential segmentation mask. Each connected domain determines a corresponding object bounding box by the minimum and maximum values of its sub-pixels on the x-axis and y-axis.

[0078] 3-2) Object detection task:

[0079] For the object category set C of the target detection task det The specific category name category det ,category det ∈C det , constructing structured text prompts T for target detection det =”All{category det} in the image", input remote sensing reference segmentation model to get category det The corresponding mask output Then, the remote sensing image I is input into the edge detection model F c , get the edge mask H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (3):

[0080]

[0081] use Refine category det The corresponding mask output The boundary of Prevent the generation of erroneous object bounding boxes containing multiple object instances due to mask overlap, as shown in the following formula (4):

[0082]

[0083] Then through F M2B The mask Converted to a single-category object bounding box Then integrate C det All categories in Output, get the object bounding box of target detection

[0084] The image-level task output is derived based on the pixel-level task output and the region-level task output, thereby realizing the decision transformation of the image-level task. Specifically, the following steps are involved:

[0085] 4-1) Multi-label classification task:

[0086] For the multi-label classification task, the object category set C MLC The specific category name category MLC ,category MLC ∈C MLC , construct multi-label classification text prompt T MLC =”All{category MLC}in the image", input remote sensing reference segmentation model to get category MLC Probability distribution matrix corresponding to mask output The category is obtained by weighted summation of global average pooling and global maximum pooling. MLC The confidence score S c , as shown in formula (5):

[0087]

[0088] Where λ is the balance coefficient of global average pooling and global maximum pooling, H and W represent the height and width of the remote sensing image respectively, and P c (i,j) represents the probability distribution matrix The value of the element at row i and column j, Represents global average pooling, max i,j P c (i,j) represents global maximum pooling;

[0089] C is preferred MLC Classes with confidence scores greater than 0.5 are output as multi-label classification tasks;

[0090] 4-2) Image classification task:

[0091] For the object category set C of the image classification task CLS , use the strategy in the multi-label classification task to obtain the confidence score S corresponding to each category c , then select C CLS The class with the highest confidence score is output as the image classification task;

[0092] 4-3) Image description task:

[0093] For the potential object category set C in remote sensing images IC, using the strategy in the target detection task, obtain the bounding box of the target detected object, and then adopt a rule-based image description method: according to the number of the bounding boxes of the target detected object, use the preset template to describe the existence information of the object in the remote sensing image; according to the position of the center of the bounding box of the target detected object in the remote sensing image, use the preset template to describe the position information of the object in the remote sensing image;

[0094] For example, for a remote sensing image with a resolution of 800*800, it is known that the output of the target detection task is "playground: [(461,502),(610,647)]". The specific process of constructing the output of the image description task is:

[0095] First, there is a playground object bounding box, so the filled template is "There is {1 playground} in the image"; then, the length and width of the remote sensing image are divided into three equal parts to form nine rectangular areas. The center coordinates of the playground object bounding box (531, 575) fall in the area in the lower right corner, so the filled template is "There is {1 playground} in the {lower right corner} of the image"; finally, all the text is spliced together to obtain the image description task output "There is a playground in the image, and there is a playground in the lower right corner of the image";

[0096] 4-4) Target counting task:

[0097] Object category for target counting tasks CNT , the strategy in the target detection task is used to obtain the single-category object bounding box corresponding to the category, and the number of single-category object bounding boxes is counted as the output of the target counting task.

[0098] Example 1

[0099] The following will be combined with the Figures 1 to 4 The method of the present invention is described in detail with reference to the specific embodiments; the implementation methods described below with reference to the drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0100] like Figure 1 As shown, this embodiment proposes a remote sensing multi-task decision transformation method based on reference segmentation, which includes the following steps:

[0101] S1: Using the remote sensing referent segmentation model as the base model, a large-scale generalized referent segmentation dataset is constructed to pre-train the remote sensing referent segmentation model;

[0102] The specific process is:

[0103] S1-1: Construct a large-scale generalized referent segmentation dataset, specifically including:

[0104] S1-1-1: One-to-one strategy: a single referential text hint corresponds to the referential segmentation mask of a single object instance, integrating the remote sensing image I, the referential text hint T, and the referential segmentation mask M, and storing them as a (I, T, M) triplet;

[0105] S1-1-2: One-to-many strategy: A single referential text prompt corresponds to the referential segmentation mask of all object instances under a single object category. Design a referential text prompt T = "{c} in the image", where c represents the object category. Integrate the remote sensing image I and the referential segmentation mask M corresponding to all object instances of c and store it as a (I, T, M) triplet;

[0106] S1-1-3: One-to-zero strategy: A single referential text hint corresponds to an empty referential segmentation mask, and a referential text hint T of a category that does not exist in the remote sensing image I is constructed. The referential segmentation mask M is an all-zero matrix and is stored as a (I, T, M) triplet;

[0107] S1-2: Input data and decode:

[0108] The remote sensing reference segmentation model is pre-trained using the constructed large-scale generalized reference segmentation dataset. The remote sensing image I and the reference text prompt T are input into the visual encoder and text encoder respectively to generate the reference segmentation task output. H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (1):

[0109]

[0110] In formula (1) Refers to the output of the segmentation task, F v Represents the encoding by the visual encoder, F t It means encoding by text encoder, and Q means decoding by decoder;

[0111] S1-3: Calculate loss and update parameters:

[0112] Calculate the output of the referent segmentation task and the true mask label M label Segmentation loss Update the parameters of each layer of neurons and iterate several times until the model parameters converge to the optimal value. The optimal goal is to minimize

[0113] S2: Convert the output of the referential segmentation task into the output of the semantic segmentation task, generate pixel-level task output, and realize the decision transformation of the pixel-level task:

[0114] For the object category set C of the semantic segmentation task seg The specific category name category seg ,categoryseg ∈C seg , construct semantic segmentation text prompt T seg =”All{category seg}in the image", input remote sensing reference segmentation model to get category seg The corresponding mask output Then integrate C seg The mask output of all categories in the semantic segmentation task is obtained

[0115] S3: Convert pixel-level task outputs into region-level task outputs to achieve decision transformation of region-level tasks;

[0116] The specific process is:

[0117] S3-1: Visual localization task:

[0118] First, the visual localization task is regarded as a pixel-level referent segmentation task. The text prompt is input into the remote sensing referent segmentation model to obtain the referent segmentation task output. The output of the referential segmentation task is then converted into the object bounding box of the visual localization task As shown in formula (2):

[0119]

[0120] Where (x min ,y min ) and (x max ,y max ) are the upper left corner and lower right corner of the object bounding box, F M2B The conversion strategy from the referential segmentation mask to the object bounding box is to find the connected domains in the referential segmentation mask. Each connected domain determines a corresponding object bounding box by the minimum and maximum values of its sub-pixels on the x-axis and y-axis.

[0121] S3-2: Target detection task:

[0122] For the object category set C of the target detection task det The specific category name category det ,category det ∈C det , constructing structured text prompts T for target detection det =”ALl{category det}in the image", input remote sensing reference segmentation model to get category det The corresponding mask output Then, the remote sensing image I is input into the edge detection model Fc , get the edge mask H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (3):

[0123]

[0124] use Refine category det The corresponding mask output The boundary of Prevent the generation of erroneous object bounding boxes containing multiple object instances due to mask overlap, as shown in the following formula (4):

[0125]

[0126] Then through F M2B The mask Converted to a single-category object bounding box Then integrate C det All categories in Output, get the object bounding box of target detection

[0127] S4: Based on the pixel-level task output and region-level task output, the image-level task output is derived to achieve decision transformation of the image-level task;

[0128] The specific process is:

[0129] S4-1: Multi-label classification task:

[0130] For the multi-label classification task, the object category set C MLC The specific category name category MLC ,category MLC ∈C MLC , construct multi-label classification text prompt T MLC =”All{category MLC}in the image", input remote sensing reference segmentation model to get category MLC Probability distribution matrix corresponding to mask output The category is obtained by weighted summation of global average pooling and global maximum pooling. MLC The confidence score S c , as shown in formula (5):

[0131]

[0132] Where λ is the balance coefficient between global average pooling and global maximum pooling, H and W represent the height and width of the image respectively, and P c(i,j) represents the probability distribution matrix The value of the element at row i and column j, Represents global average pooling, max i,j P c (i,j) represents global maximum pooling;

[0133] C is preferred MLC The categories with confidence scores greater than 0.5 are output as multi-label classification tasks, and the results are visualized as follows Figure 2 As shown;

[0134] S4-2: Image classification task:

[0135] For the object category set C of the image classification task CLS , use the strategy in the multi-label classification task to obtain the confidence score S corresponding to each category c , then select C CLS The category with the highest confidence score is output as the image classification task, and the result visualization is as follows Figure 3 As shown;

[0136] S4-3: Image description task:

[0137] For the potential object category set C in remote sensing images IC , using the strategy in the target detection task, obtain the bounding box of the target detected object, and then adopt a rule-based image description method: according to the number of the bounding boxes of the target detected object, use the preset template to describe the existence information of the object in the remote sensing image; according to the position of the center of the bounding box of the target detected object in the remote sensing image, use the preset template to describe the position information of the object in the remote sensing image;

[0138] For example, for a remote sensing image with a resolution of 800*800, it is known that the output of the target detection task is "playground: [(461,502),(610,647)]". The specific process of constructing the output of the image description task is:

[0139] First, there is a playground object bounding box, so the filled template is "There is {1 playground} in the image"; then, the length and width of the remote sensing image are divided into three equal parts to form nine rectangular areas. The center coordinates of the playground object bounding box (531, 575) fall in the area in the lower right corner, so the filled template is "There is {1 playground} in the {lower right corner} of the image"; finally, all the text is spliced together to obtain the image description task output "There is a playground in the image, and there is a playground in the lower right corner of the image";

[0140] S4-4: Target counting task:

[0141] Object category for target counting tasks CNT, the strategy in the target detection task is used to obtain the single-category object bounding box corresponding to the category, and the number of single-category object bounding boxes is counted as the output of the target counting task.

[0142] S5: Summarize the decision transformation of pixel-level tasks, region-level tasks, and image-level tasks to complete the construction of the remote sensing vision basic model.

Claims

1. A remote sensing multi-task decision transformation method based on reference segmentation, characterized by include: 1) Using the remote sensing referent segmentation model as the base model, we build a large-scale generalized referent segmentation dataset and pre-train the remote sensing referent segmentation model; 2) Convert the output of the referential segmentation task into the output of the semantic segmentation task, generate pixel-level task output, and realize the decision transformation of pixel-level tasks; 3) Convert pixel-level task outputs into region-level task outputs to achieve decision-making transformation of region-level tasks; 4) Based on the pixel-level task output and region-level task output, the image-level task output is derived to achieve decision transformation of the image-level task.

2. The remote sensing multi-task decision transformation method based on reference segmentation according to claim 1 is characterized by Also includes: 5) Summarize the decision transformation of pixel-level tasks, region-level tasks, and image-level tasks to complete the construction of the remote sensing vision basic model.

3. A remote sensing multi-task decision transformation method based on reference segmentation according to claim 1 or 2, characterized in that The remote sensing referent segmentation model is used as a basic model to construct a large-scale generalized referent segmentation dataset and pre-train the remote sensing referent segmentation model, specifically including: 1-1) Construct a large-scale generalized referent segmentation dataset; 1-2) Input data and decode: The remote sensing reference segmentation model is pre-trained using the constructed large-scale generalized reference segmentation dataset. The remote sensing image I and the reference text prompt T are input into the visual encoder and text encoder respectively to generate the reference segmentation task output. H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (1): In formula (1) Refers to the output of the segmentation task, F v Represents the encoding by the visual encoder, F t It means encoding by text encoder, and Q means decoding by decoder; 1-3) Calculate the loss and update the parameters: Calculate the output of the referent segmentation task and the true mask label M label Segmentation loss Update the parameters of each layer of neurons and iterate several times until the model parameters converge to the optimal value. The optimal goal is to minimize 4. The remote sensing multi-task decision transformation method based on reference segmentation according to claim 3 is characterized by The construction of a large-scale generalized referential segmentation dataset specifically includes: 1-1-1) One-to-one strategy: A single referential text hint corresponds to a referential segmentation mask of a single object instance, integrating the remote sensing image I, the referential text hint T, and the referential segmentation mask M, and storing them as a (I, T, M) triplet; 1-1-2) One-to-many strategy: A single referential text hint corresponds to the referential segmentation mask of all object instances under a single object category. Design the referential text hint T = "{c} in the image", where c represents the object category. Combine the remote sensing image I and the referential segmentation mask M corresponding to all object instances of object category c and store it as a (I, T, M) triplet; 1-1-3) One-to-zero strategy: A single referential text prompt corresponds to an empty referential segmentation mask, and a referential text prompt T of a category that does not exist in the remote sensing image I is constructed. The referential segmentation mask M is an all-zero matrix and is stored as a (I, T, M) triplet.

5. A remote sensing multi-task decision transformation method based on reference segmentation according to claim 1 or 2, characterized in that The output of the referential segmentation task is converted into the output of the semantic segmentation task, generating a pixel-level task output and realizing the decision conversion of the pixel-level task, specifically including: For the object category set C of the semantic segmentation task seg The specific category name category seg ,category seg ∈C seg , construct semantic segmentation text prompt T seg =”All{category seg }in the image", input remote sensing reference segmentation model to get category seg The corresponding mask output Then integrate C seg The mask output of all categories in the semantic segmentation task is obtained 6. A remote sensing multi-task decision transformation method based on reference segmentation according to claim 1 or 2, characterized in that The pixel-level task output is converted into a region-level task output to achieve the decision conversion of the region-level task, specifically including: 3-1) Visual localization task: First, the visual localization task is regarded as a pixel-level referent segmentation task. The text prompt is input into the remote sensing referent segmentation model to obtain the referent segmentation task output. The output of the referential segmentation task is then converted into the object bounding box of the visual localization task As shown in formula (2): Where (x min ,y min ) and (x max ,y max ) are the upper left corner and lower right corner of the object bounding box, F M2B It is a conversion strategy from the referential segmentation mask to the object bounding box, that is, by finding the connected domains in the referential segmentation mask, each connected domain determines a corresponding object bounding box by the minimum and maximum values of its sub-pixels on the x-axis and y-axis.

7. The remote sensing multi-task decision transformation method based on reference segmentation according to claim 6 is characterized by The pixel-level task output is converted into a region-level task output to achieve the decision conversion of the region-level task, which specifically includes: 3-2) Object detection task: For the object category set C of the target detection task det The specific category name category det ,category det ∈C det , constructing structured text prompts T for target detection det =”All{category det } in the image", input remote sensing reference segmentation model to get category det The corresponding mask output Then, the remote sensing image I is input into the edge detection model F c , get the edge mask H and W are the height and width of the remote sensing image, respectively, as shown in the following formula (3): use Refine category det The corresponding mask output The boundary of Prevent the generation of erroneous object bounding boxes containing multiple object instances due to mask overlap, as shown in the following formula (4): Then through F M2B The mask Converted to a single-category object bounding box Then integrate C det All categories in Output, get the object bounding box of target detection 8. A remote sensing multi-task decision transformation method based on reference segmentation according to claim 1 or 2, characterized in that Based on the pixel-level task output and the region-level task output, the image-level task output is derived to realize the decision transformation of the image-level task, which specifically includes: 4-1) multi-label classification task; 4-2) image classification task; 4-3) image description task; 4-4) target counting task.

9. The remote sensing multi-task decision transformation method based on reference segmentation according to claim 8 is characterized by The multi-label classification task specifically includes: for the object category set C of the multi-label classification task MLC The specific category name category MLC ,category MLC ∈C MLC , construct multi-label classification text prompt T MLC =”All{category MLC }in theimage", input remote sensing reference segmentation model to get category MLC Probability distribution matrix corresponding to mask output Category is obtained by weighted summation of global average pooling and global maximum pooling MLC The confidence score S c , as shown in formula (5): Where λ is the balance coefficient of global average pooling and global maximum pooling, H and W represent the height and width of the remote sensing image respectively, and P c (i,j) represents the probability distribution matrix The value of the element at row i and column j, Represents global average pooling, max i,j P c (i,j) represents the global maximum pooling; take C MLC Classes with confidence scores greater than 0.5 are output as multi-label classification tasks.

10. The remote sensing multi-task decision transformation method based on reference segmentation according to claim 8 is characterized by The image classification task specifically includes: for the object category set C of the image classification task CLS , use the strategy in the multi-label classification task to obtain the confidence score S corresponding to each category c , then select C CLS The class with the highest confidence score is output as the image classification task; The image description task specifically includes: for the potential object category set C in the remote sensing image IC , using the strategy in the target detection task, obtain the bounding box of the target detected object, and then adopt a rule-based image description method: according to the number of the bounding boxes of the target detected object, use the preset template to describe the existence information of the object in the remote sensing image; according to the position of the center of the bounding box of the target detected object in the remote sensing image, use the preset template to describe the position information of the object in the remote sensing image; The target counting task specifically includes: object category category for the target counting task CNT , the strategy in the target detection task is used to obtain the single-category object bounding box corresponding to the category, and the number of single-category object bounding boxes is counted as the output of the target counting task.

Images