A self-optimization map spot monitoring method and system based on a multi-modal large model

Abstract

The application provides a self-optimization map spot monitoring method and system based on a multi-modal large model, inputs a remote sensing picture with a map spot and loaded recall total prompt words into a multi-modal large model, recalls M categories from N categories based on the multi-modal large model, inputs the remote sensing picture with the map spot and loaded classification total prompt words into the multi-modal large model, selects one category from the recalled M categories based on the multi-modal large model, obtains the category of the map spot in the remote sensing picture, optimizes the recall total prompt words based on a failed map spot sample of recall, and optimizes the classification total prompt words based on a failed map spot sample of classification. The application realizes the automation and intelligentization of map spot monitoring by analyzing and classifying the map spot in the picture based on the understanding ability of the multi-modal large model to the remote sensing picture.

Description

Technical Field

[0001] This invention relates to the field of natural resource monitoring technology, and more specifically, to a self-optimizing patch monitoring method and system based on a multimodal large model. Background Technology

[0002] In the field of natural resource monitoring, monitoring land use change through satellite remote sensing images is a crucial part of natural resource supervision. The traditional approach involves manually observing specific areas (i.e., image patches) in the images and analyzing them based on professional knowledge to determine if there have been any land changes, such as a change from forest to hardened land.

[0003] However, this method, requiring careful manual observation and professional analysis, is not only relatively inefficient but also demands a high level of expertise from personnel. Improving the automation and intelligence of this map-based monitoring process is one of the key challenges in natural resource regulation. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a self-optimizing patch monitoring method and system based on a multimodal large model, thereby achieving automated patch monitoring and overcoming the deficiencies in the prior art.

[0005] According to a first aspect of the present invention, a self-optimizing patch monitoring method based on a multimodal large model is provided, comprising:

[0006] Step 1: Input the remote sensing image with patches and the loaded total recall prompts into the multimodal large model. Based on the multimodal large model, recall M categories from N categories, where N > M;

[0007] Step 2: Input the remote sensing image with patches and the loaded classification prompt words into the multimodal large model. Based on the multimodal large model, select 1 category from the recalled M categories to obtain the category of the patches in the remote sensing image.

[0008] Step 3: Optimize the total recall prompts based on the image samples that failed to be recalled, and optimize the total classification prompts based on the image samples that failed to be classified.

[0009] Based on the above technical solution, the present invention can also be improved as follows.

[0010] Optionally, the structure of the total recall prompt includes "Category and Category Features" and "Analysis Method". The "Category and Category Features" includes N category features. The "Analysis Method" prompts the use of a recall analysis method and indicates on the interface that only M items can be selected from the above N categories, and returns them in the format of "Category 1; Category 2; ...; Category M".

[0011] Optionally, step 1, which involves inputting the remote sensing image with patches and the loaded total recall prompts into a multimodal large model, and recalling M categories from N categories based on the multimodal large model, includes:

[0012] The remote sensing image with the patch and the total recall prompt are input into the multimodal large model. Based on the output of the multimodal large model, the recall category is parsed out.

[0013] If the output of the multimodal large model does not conform to the specifications, reload the total recall keywords and parse them.

[0014] Set the maximum number of recall attempts (max_recall). If the number of recall attempts exceeds max_recall, output "Recall Failed".

[0015] Optionally, the structure of the classification prompt includes "Category and Category Features", "Analysis Method", and "Special Notes". "Category and Category Features" includes the M category features to be recalled. "Analysis Method" indicates the classification analysis method used. "Special Notes" indicates the specific matters that need attention. The interface prompts that only one category can be selected from the above M categories and returns in the format of "Select Category, Reason for Selection". The selected category is the selected category, and the reason for selection is the reason for selecting the category.

[0016] Optionally, in step 2, the remote sensing image with patches and the loaded classification prompt words are input into the multimodal large model. Based on the multimodal large model, one category is selected from M categories to obtain the patch category, including:

[0017] The remote sensing image with the patch and the total classification prompt are input into the multimodal large model. Based on the output of the multimodal large model, the classification category is parsed out.

[0018] If the output of the multimodal large model does not conform to the specifications, reload the total classification prompts and parse them.

[0019] Set the maximum number of classification attempts (max_classify). If the number of classification attempts exceeds max_classify, output "Classification failed".

[0020] Optionally, step 3, based on the failed image sample recall, optimizes the total recall prompts, including:

[0021] The correct category is d, and the recall categories are d1, d2, ..., d. M h failed recall samples were used as recall optimization samples;

[0022] Based on each recalled and optimized sample, the category features and recall analysis methods of category d are optimized to obtain h optimized category features and optimized recall analysis methods for category d, which respectively constitute the optimized category feature set and optimized recall analysis method set for category d;

[0023] The optimized category feature set and optimized recall analysis method set of category d are input into the large model for summarization, so as to obtain the final optimized category features and final optimized recall analysis method of category d;

[0024] Iterate through all N categories to obtain the final optimized category features and the final optimized recall analysis method for each of the N categories;

[0025] Based on the final optimized category features and the final optimized recall analysis method for each category, the total recall prompts are updated and optimized.

[0026] Optionally, based on each recalled optimized sample, the category features and recall analysis method of category d are optimized to obtain h optimized category features and optimized recall analysis methods for category d, including:

[0027] For each recalled optimized sample, its correct class is d, and its recalled classes are d1, d2, ..., dn. m For categories d1, d2, ..., d M Two photos are randomly selected from each category, and the categories are d1, d2, ..., d... M The photos of the samples and the photos of the recalled and optimized samples were uniformly numbered;

[0028] Construct a total recall optimization prompt, the structure of which includes "comparison photos", "category features", "analysis methods", and "reflections", wherein "comparison photos" includes recall categories d1, d2, ..., d M The photos' "category features" include categories d1, d2, ..., d M The category characteristics of d, the "Analysis Method" suggests using recall analysis, and the "Reflection" suggests that "photos of category d are predicted to be {category d1}, {category d2}, ..., {category d...}". M "One of the categories is {Category d}, but it is actually {Category d}. Please reflect on the above information and update the category features and recall analysis method of category d". Return the results in the format of "Category Features: Output the optimized category features of {Category d}, Recall Analysis Method: Output the optimized recall analysis method".

[0029] The comparison photos, the recall optimization sample photos, and the total recall optimization prompts are input into the multimodal large model to obtain the optimized category features of category d and the optimized recall analysis method.

[0030] Optionally, in step 3, the total classification prompt words are optimized based on the image samples that failed to be classified, including:

[0031] Obtain p failed classification samples with the correct category d and the classification category a as classification optimization samples;

[0032] Based on each classification optimization sample, the classification analysis method between category d and category a is optimized to obtain p optimized classification analysis methods between category d and category a, which constitute a set of optimized classification analysis methods between category d and category a;

[0033] The set of optimized classification analysis methods between category d and category a is input into a large model for summarization, resulting in the final optimized classification analysis method between category d and category a.

[0034] Traverse the two category pairs to obtain the final optimized classification analysis method between each category pair. Based on the final optimized classification analysis method between each category pair, update and optimize the total classification prompt words.

[0035] Optionally, the optimization of the classification analysis method between category d and category a based on each classification optimization sample yields p optimized classification analysis methods between category d and category a, including:

[0036] For each classification optimization sample with the correct category d and the classification category a, obtain four photos of category a and four photos of category d respectively, and number them.

[0037] Construct a general suggestion term for classification optimization. The structure of the general suggestion term includes "Comparison Photos", "Category Features", "Analysis Method", "Precautions", and "Reflection". Among them, "Comparison Photos" includes photos of category a and category d, "Category Features" includes the category features of category a and category d, "Analysis Method" indicates the classification analysis method used, "Precautions" indicates specific matters that need attention, and "Reflection" indicates "The photo is classified as {category a}, but the correct category is {category d}. Please reflect on the above information and update the classification analysis method". Return the suggestion term in the format of "Classification Analysis Method: Output Optimized Classification Analysis Method, Precautions: Output Optimized Precautions".

[0038] Input the comparison photos, classification optimization samples, and the total classification optimization prompts into the multimodal large model, and output the optimized classification analysis method between category d and category a.

[0039] Based on a second aspect of the present invention, a self-optimizing patch monitoring system based on a multimodal large model is provided. The system includes a patch monitoring module and a prompt word optimization module. The patch monitoring module includes a recall submodule and a classification submodule.

[0040] The recall submodule is used to input the remote sensing image with patches and the loaded total recall prompts into the multimodal large model, and recall M categories from N categories based on the multimodal large model, where N > M;

[0041] The classification submodule is used to input the remote sensing image with patches and the loaded classification prompt words into the multimodal large model, and select one category from the recalled M categories based on the multimodal large model to obtain the category of the patches in the remote sensing image;

[0042] The prompt word optimization module is used to optimize the total recall prompt words based on the image patch samples that failed to be recalled, and to optimize the total classification prompt words based on the image patch samples that failed to be classified.

[0043] This invention provides a self-optimizing patch monitoring method and system based on a multimodal large model. The task of classifying patches in remote sensing images is divided into two stages: a recall stage and a classification stage. By combining these two stages with a large model, the patch classification task is progressively refined. Furthermore, based on samples that fail to be recalled or classified, the recall and classification prompts are optimized to further improve the accuracy of patch classification. This invention leverages the understanding capabilities of a multimodal large model of remote sensing images to analyze and classify patches, achieving automated and intelligent patch monitoring. Attached Figure Description

[0044] Figure 1 The following is a flowchart of a self-optimizing patch monitoring method based on a multimodal large model, provided in one embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the classification of remote sensing images according to an embodiment of the present invention;

[0046] Figure 3 This is a flowchart illustrating the optimization process for recall prompts according to an embodiment of the present invention.

[0047] Figure 4 This is a flowchart of a classification prompt word optimization process according to an embodiment of the present invention;

[0048] Figure 5 A block diagram of a self-optimizing patch monitoring system based on a multimodal large model provided in one embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram illustrating the working principle of a self-optimizing patch monitoring system based on a multimodal large model. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0051] Traditional remote sensing image patch monitoring relies primarily on manual image observation combined with professional knowledge to determine whether there is any illegal activity in the patch area. This method requires a high level of expertise and is extremely inefficient.

[0052] With the development of computer technology, image processing models have gradually been applied in remote sensing monitoring. Examples include monitoring methods combining spectral feature changes and polarization feature maps, and compliance identification based on convolutional neural networks in field surveys. These methods use computer algorithms or models to automate part of the patch monitoring process. However, these methods have drawbacks. Because the algorithms are designed for specific scenarios or use a limited number of model parameters, they only perform well in single-category or simple scenarios: for example, they can only detect construction projects. In complex backgrounds (i.e., input photos with numerous regional categories), their performance is very poor.

[0053] With the development of artificial intelligence technology, existing multimodal large models possess very strong image and language processing capabilities. Based on this, this invention utilizes the powerful processing capabilities of multimodal large models to construct a method for patch detection. Compared to existing methods, this method can better identify multiple significantly different categories in complex backgrounds; furthermore, it can continuously improve and optimize the method based on input information, thus making the method more adaptable.

[0054] Figure 1 The flowchart of a self-optimizing patch monitoring method based on a multimodal large model provided by the present invention is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0055] Step 1: Input the remote sensing image with patches and the loaded total recall prompts into the multimodal large model. Based on the multimodal large model, recall M categories from N categories, where N > M;

[0056] Step 2: Input the remote sensing image with patches and the loaded classification prompt words into the multimodal large model. Based on the multimodal large model, select 1 category from the recalled M categories to obtain the category of the patches in the remote sensing image.

[0057] Step 3: Optimize the total recall prompts based on the image samples that failed to be recalled, and optimize the total classification prompts based on the image samples that failed to be classified.

[0058] Understandably, the task of classifying patches in remote sensing images can be accomplished in two stages: the first stage is the recall stage, and the second stage is the classification stage, which together complete the classification of patches in remote sensing images.

[0059] The recall part involves selecting M most likely categories from N categories. In this embodiment of the invention, 3 most likely categories are selected from N categories.

[0060] Recall method: Input a remote sensing image with patches, load recall keywords, input the remote sensing image and recall keywords simultaneously into a multimodal large model, and the large model returns the 3 most likely categories.

[0061] Category section: Select one category from three recall categories.

[0062] Method: Input a remote sensing image with patches, load classification hints, input the remote sensing image and classification hints simultaneously into a multimodal large model, the large model returns one final category, and the patch classification is completed.

[0063] In the process of classifying image patches, multiple samples can be used to recall and classify the images, along with existing recall and classification tips. Samples that fail to be recalled or classified are then sent to a multimodal large model for analysis and optimization, and optimized recall and classification tips are generated.

[0064] Figure 2 The flowchart of the classification task for remote sensing images with patches is shown. The task is to first recall and then classify, that is, input a remote sensing image with patches, first predict the three most likely categories, and then predict the most likely category from these three categories.

[0065] For the recall portion, see [link / reference]. Figure 3 The processing steps are as follows: input a remote sensing image with patches, and load the total recall keywords. The format structure of the total recall keywords is as follows:

[0066] You are a remote sensing analyst. Based on the following information, analyze the remote sensing images and select the three most likely categories.

[0067] "Category and Category Features": {Category Features};

[0068] "Analysis Methods": {Recall Analysis Methods};

[0069] You can only select 3 from the above categories, and the returned format must be: Category 1; Category 2; Category 3.

[0070] Load category feature suggestions and recall analysis method suggestions into the total recall suggestions to instantiate the total recall suggestions.

[0071] For example, category feature prompts:

[0072] Category 1: Features of Category 1; Category 2: Features of Category 2; ...; Category 10: Features of Category 10.

[0073] Analysis method prompts:

[0074] First, look at the whole map; second, observe the changes in the patches; third, determine the category.

[0075] Then, the category feature suggestion term replaces {category feature} in the total recall suggestion term; the analysis method suggestion term replaces {recall analysis method} in the total suggestion term, forming the instantiated total recall suggestion term, which suggests selecting 3 possible categories from 10 categories.

[0076] The instantiated total recall keywords and remote sensing images with patches are input into a multimodal large model. Based on the output of the multimodal large model, the three recall categories are parsed out. If the model's return does not conform to the specifications, the total recall keywords are reconstructed, and the recall is restarted. A maximum recall count (max_recall) is set; if the recall count exceeds max_recall, a recall failure is output. For example:

[0077] If the large model returns: Category 1; Category 3; Category 5; then directly parse the results and output them.

[0078] The large model returns: the possible categories are 1, 4, 4; if they do not conform to the specification of "category 1; category 2; category 3", the recall process is re-executed.

[0079] After recalling three categories from all N categories, a final category is determined from the three recalled categories through a classification phase.

[0080] In the classification phase, see Figure 4 Input a remote sensing image with patch data, and load the overall classification suggestions. The format structure of the overall classification suggestions is as follows:

[0081] You are a remote sensing analyst. Based on the following information, analyze the remote sensing images and select the most likely category.

[0082] "Category and Category Features": {Category Features};

[0083] "Analysis Methods": {Classification Analysis Methods};

[0084] "Special Note": {Precautions};

[0085] You can only select one from the above categories, and the return format must be: Selected Category: Category; Reason for Selection: Reason for your selection.

[0086] Load category feature suggestions, classification analysis methods, and precautions suggestions into the overall classification suggestion word, and instantiate the overall classification suggestion word. The category feature suggestions only load features from the recalled category. For example, if three category features are recalled during the recall phase, then the category feature suggestions in the overall classification suggestion word will only load these three category features to determine one category from the three categories. For example:

[0087] Category feature hints:

[0088] Category 1: Features of Category 1; Category 2: Features of Category 2; ...; Category 10: Features of Category 10.

[0089] If the recall categories are 1, 3, and 5, then only the categories 1, 3, and 5 and their characteristics will be extracted to replace "category and category characteristics" in the overall classification prompt. Classification analysis method prompt: The main difference between category 1 and category 3 is as follows: Category 1 "Rural Roads" exhibits heterogeneous traffic compaction marks, showing ruts of varying depths, tire tracks, localized gravel, or naturally compacted layers, with a mottled color transition over use; Category 3 "Linear Engineering" has a long, continuous, highly uniform surface of flat bare soil or backfill, appearing as a homogeneous "band-shaped color block" after mechanical leveling, strictly lacking details of vehicle wear. The main difference between Category 3 and Category 5 is as follows: Category 3 "linear projects" have the attribute of "through traffic", and nodes such as intersections and ramps are allowed, but each branch must be radial and point to the external road network, residential area or functional area, with a clear traffic access logic; Category 5 "with construction activities" have a "plot cohesion" or "closed operation" structure, and the internal road network is mostly ring / grid-shaped serving a single plot. The endpoints are often closed or only anchored to internal functional areas, with no cross-regional connectivity.

[0090] Replace {Classification Analysis Method} in the overall classification prompt with the classification analysis method prompt.

[0091] Note: If a feature A is present in a patch, it is directly classified as category 1; if a feature B is present, it can only be category 2 or 3. Replace "Note" in the overall classification note with the note.

[0092] The instantiated classification prompts and remote sensing images with polygons are input into a multimodal large-scale model. Based on the output of the multimodal large-scale model, the final category and selection rationale are parsed. If the model's return does not conform to the specifications, the classification process is re-executed and parsed; a maximum number of classification attempts (max_classify) is set. If the number of classification attempts exceeds max_classify, the classification fails. For example:

[0093] The large model returns: Select category: Category 1; Select reason: Reason 1; then directly parse the result and output it.

[0094] The large model returns: Category 1, the reason for selection is this; does not meet the criteria; Select Category: Category; Reason for selection: your reason for selecting the category; if it does not meet this specification, then re-execute the classification process.

[0095] Replace {Notes} in the category's main prompts with the "Notes" prompt.

[0096] To improve the accuracy of image classification, the total recall keywords and the total classification keywords in the recall and classification stages were optimized. The optimization process involved first recalling or classifying the samples, then filtering out the recall failures or misclassifications, and inputting the failed recall or misclassification samples into the multimodal large model to reflect on the errors and regenerate new recall or classification keywords.

[0097] The optimization process for recall prompts is as follows:

[0098] Input samples, recall samples, and randomly select recall_fail_num samples from the failed recall samples as candidate recall optimization samples.

[0099] For example, given a batch of 100 samples, we need to recall the image patches within the samples. Assuming that 23 samples failed to be recalled, and recall_fail_num is set to 10, we will randomly select 10 failed samples from the 23 failed samples as the recall optimization samples.

[0100] The definition of recall failure is: for a sample whose correct category is category 1, if category 1 is not found in the three output categories or the recall fails directly, then it is defined as recall failure.

[0101] Input the recall_fail_num samples into the multimodal large model, and the multimodal large model will perform the following tasks respectively.

[0102] First, collect, for example, multiple correct category h and correct category d, but recalled as a, b, and c. For each recalled sample with correct category d but recalled as a, b, and c, select two photos for each of the recall categories a, b, and c to form a comparison photo. Number both the comparison photo and the recalled sample photo. Based on the numbered photos, construct the overall recall optimization suggestion. The structure of the overall recall optimization suggestion is as follows:

[0103] "Comparison Photos": Photos 1-2 are photos of {Category a}, Photos 3-4 are photos of {Category b}, and Photos 5-6 are photos of {Category c};

[0104] "Category Features": {Category features for categories a, b, c, d};

[0105] "Analysis Methods": {Recall Analysis Methods};

[0106] "Reflection": Photo 7 was predicted to be one of {Category a}, {Category b}, or {Category c}, but it actually belongs to {Category d}. Please reflect on the above information and update the category features and recall analysis methods.

[0107] The output format is:

[0108] Category Features: Output the optimized category features for {category d}; Recall Analysis Method: Output the optimized recall analysis method.

[0109] Load category features and existing recall analysis methods into the overall recall optimization keywords to instantiate the overall recall optimization keywords. For example:

[0110] For a sample that failed to be recalled, the recall results are category 1, category 2, and category 3; however, its correct category is actually category 4. First, extract category 1 from the category feature prompts and replace all {category a} in the total prompts. Similarly, replace {category b} with category 2, {category c} with category 3, and {category d} with category 4. Next, extract the relevant features of categories 1, 2, 3, and 4 (for example, if there are 10 category features, only 4 are needed here) and replace the {category features} in the total prompts. Finally, replace {recall analysis method} with the analysis method prompt.

[0111] Then, from the recalled categories 1, 2, and 3, two photos are randomly selected from each category and numbered.

[0112] For example, if a sample fails to be recalled and the recall results are category 1, category 2, and category 3, then 2 photos are randomly selected from each category 1 to 3, for a total of 6 photos, and numbered 1 to 6 according to category 1, 2, and 3. The photo that failed to be recalled is numbered 7.

[0113] The photos of each category with completed numbering and the corresponding category features are loaded into the total recall optimization prompt, and the total recall optimization prompt is instantiated.

[0114] Input the numbered photos and the instantiated recall optimization total prompts into the multimodal large model to obtain the optimized category features of category d and the optimized recall analysis method.

[0115] For h correctly classified samples of type d, but recalled as a, b, or c, the optimized recall analysis method is used to output h optimized category features of type d through a multimodal large model.

[0116] Construct recall optimization keywords, such as:

[0117] The following are the category characteristics and recall analysis methods for several failed recall samples after reflection:

[0118] "Category Features": {Optimized set of category features};

[0119] "Recall Analysis Methods": {Optimize the set of recall analysis methods};

[0120] Based on the information above, please output a concise category feature and recall analysis method.

[0121] The return format is as follows:

[0122] Category Features: Output the optimized category features for {category d}; Recall Analysis Method: Output the optimized recall analysis method.

[0123] Replace the above {set of optimized category features} and {set of optimized recall analysis methods} with the optimized category features and optimized recall analysis methods of the obtained h categories d, respectively, and instantiate the recall optimization prompts.

[0124] The instantiated recall optimization tips are input into the multimodal large model, and the final optimized category features and final optimized recall analysis method for category d are summarized based on the multimodal large model.

[0125] For each of the N categories, the category features and recall analysis method are optimized to obtain the final optimized category features and final optimized recall analysis method for each category. The final optimized category features of each category are then used to overwrite the original category features in the total recall keywords, and the final optimized recall analysis method replaces the original recall analysis method in the total recall keywords, resulting in the total recall keywords. For example:

[0126] The generated category features are assumed to be: Category 1: optimized category feature; Category 4: optimized category feature; then the optimized category features are used to replace the original category features of Category 1 and Category 4 respectively; other category features remain unchanged, and the recall analysis method is similar.

[0127] You can repeat the optimization of recall keywords multiple times and choose the version with the best performance. For example:

[0128] The process was repeated three times. After the first optimization, the recall rate was 0.8; after the second optimization, the recall rate was 0.9; and after the third optimization, the recall rate was 0.85. Therefore, the suggestion word from the second optimization was selected, retained, and output.

[0129] The optimization process for category suggestions is similar to that for recall suggestions. The optimization process for category suggestions is as follows:

[0130] Input samples, perform recall, select successfully recalled samples and classify the successful samples, and randomly select classify_fail_num samples from the failed samples as classification optimization samples.

[0131] Input the classify_fail_num classification optimization samples into the classify_fail_num multimodal large models respectively. Each multimodal large model performs the following tasks for a single classification optimization sample.

[0132] In the process of optimizing the classification prompts, we first collect g incorrectly classified samples, such as those whose correct category is d but are classified as a, as classification optimization samples. Based on the g classification optimization samples, we optimize the classification analysis method between category d and category a.

[0133] Load category-optimized overall suggestion keywords. For example:

[0134] "Comparison Photos": Photos 1-4 are photos of {Category a}, and photos 5-8 are photos of {Category d};

[0135] "Category Features": {Category Features};

[0136] "Analysis Methods": {Classification Analysis Methods};

[0137] "Important Notes": {Important Notes};

[0138] "Reflection": Photo 9 was predicted to be in category a, but it was actually in category d. Please reflect on this information and update your classification analysis method.

[0139] The output format is:

[0140] Classification analysis method: Output optimized classification analysis method; Notes: Output optimized notes.

[0141] Load the categories and corresponding features, classification analysis methods, and precautions, and merge them into the overall classification optimization suggestion words. Instantiate the overall classification optimization suggestion words. For example:

[0142] For a classification optimization sample, the classification result is category 1; however, its correct category is actually category 4. First, extract category 1 from the category feature hint words and replace all {category a} in the total classification optimization hint words, and replace {category d} with category 4.

[0143] Next, extract the relevant features of categories 1 and 4 (for example, if all category features include 10, only 2 are needed here) and replace the {category features} in the total suggestions for category optimization.

[0144] Then, replace the classification analysis method prompt with {classification analysis method} and the precaution prompt with {precautions}.

[0145] For each of the predicted and correct categories, randomly select four photos and number them. For example:

[0146] For a sample that is misclassified, the classification result is category 1 and the correct category is category 4. Then, 4 photos are randomly selected from category 1 and category 4 respectively, for a total of 8 photos, and numbered 1 to 8 according to category 1 and 4. The misclassified sample photo is numbered 9.

[0147] Input the numbered photos and the instantiated classification optimization key words into the multimodal large model to obtain the classification analysis method between category d and category a.

[0148] For g correctly classified samples of category d and classified as a, we can obtain the classification analysis method and precautions between g categories d and a.

[0149] Classification analysis methods and precautions between g categories d and category a, constructing overall classification optimization prompts, such as:

[0150] The following are classification analysis methods and precautions for several failed classification samples, after reflection:

[0151] "Classification Analysis Methods": {A set of classification analysis methods};

[0152] "Notes": {A set of notes};

[0153] Based on the information above, please provide a concise classification analysis method and points to note.

[0154] The return format is as follows:

[0155] Classification Analysis Method: Optimize the classification analysis method between category d and category a; Notes: Optimization considerations.

[0156] Replace the {set of classification analysis methods} and {set of precautions} in the overall classification optimization prompt with the optimized classification analysis methods and precautions for the obtained g categories d and category a, respectively.

[0157] The instantiated classification optimization key words are input into the large model. Based on the large model, the {set of classification analysis methods} and {set of considerations} are summarized to obtain the final optimized classification analysis method and optimization considerations between category d and category a.

[0158] The classification analysis optimization method described above is used to optimize the classification analysis method between category pairs formed by two categories, resulting in a new classification analysis method between category pairs. The optimized classification analysis method then overrides the original classification analysis method, and the optimized precautions replace the original precautions.

[0159] The classification analysis method was optimized multiple times, and the version of the classification suggestion words that performed best was selected.

[0160] See Figure 5 This paper illustrates a self-optimizing patch monitoring system based on a multimodal large model according to the present invention. The system includes a patch monitoring module 51 and a prompt word optimization module 52. The patch monitoring module 51 includes a recall submodule 511 and a classification submodule 512.

[0161] The recall submodule 511 is used to input the remote sensing image with patches and the loaded total recall prompt words into the multimodal large model, and recall M categories from N categories based on the multimodal large model, where N > M;

[0162] The classification submodule 512 is used to input the remote sensing image with patches and the loaded classification prompt words into the multimodal large model, and select one category from M categories based on the multimodal large model to obtain the patch category;

[0163] The prompt word optimization module 52 is used to optimize the total recall prompt words based on the image patch samples that failed to be recalled, and to optimize the total classification prompt words based on the image patch samples that failed to be classified.

[0164] Combination Figure 5 and Figure 6 As shown, remote sensing images of patches are input into the patch monitoring module, and recall and classification results are output. Based on the samples that failed to be recalled and the samples that were misclassified, the recall and classification suggestions are optimized.

[0165] It is understood that the self-optimizing patch monitoring system based on a multimodal large model provided by this invention corresponds to the self-optimizing patch monitoring method based on a multimodal large model provided in the foregoing embodiments. The relevant technical features of the self-optimizing patch monitoring system based on a multimodal large model can be referred to the relevant technical features of the self-optimizing patch monitoring method based on a multimodal large model, and will not be repeated here.

[0166] The self-optimizing patch monitoring method and system based on a multimodal large model provided in this invention has the following advantages:

[0167] (1) Multimodal large models have a strong ability to describe complex environments and content. When performing analysis, based on the powerful capabilities of multimodal large models, the content and changes of photos can be accurately analyzed, thus accurately describing the content of the images. At the same time, combined with the feature description of categories, the content of the image patches can be effectively classified.

[0168] (2) Since the input remote sensing images may change over time and the features of different categories may change for various reasons, the self-optimization method proposed in this invention can update the features and recognition methods of different categories in a timely manner, thereby improving the adaptability of the patch monitoring method and continuously improving the classification ability of the method.

[0169] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0170] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0175] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A self-optimizing patch monitoring method based on a multimodal large model, characterized in that, include: Step 1: Input the remote sensing image with patches and the loaded total recall prompts into the multimodal large model. Based on the multimodal large model, recall M categories from N categories, where N > M; Step 2: Input the remote sensing image with patches and the loaded classification prompt words into the multimodal large model. Based on the multimodal large model, select 1 category from the recalled M categories to obtain the category of the patches in the remote sensing image. Step 3: Optimize the total recall prompts based on the image samples that failed to be recalled, and optimize the total classification prompts based on the image samples that failed to be classified.

2. The self-optimizing patch monitoring method based on a multimodal large model according to claim 1, characterized in that, The structure of the total recall prompt includes "Category and Category Features" and "Analysis Method". The "Category and Category Features" includes N category features. The "Analysis Method" indicates the recall analysis method used and prompts on the interface that only M items can be selected from the above N categories. The results are returned in the format "Category 1; Category 2; ...; Category M".

3. The self-optimizing patch monitoring method based on a multimodal large model according to claim 1 or 2, characterized in that, Step 1, which involves inputting the remote sensing image with patches and the loaded total recall keywords into a multimodal large model, and recalling M categories from N categories based on the multimodal large model, includes: The remote sensing image with the patch and the total recall prompt are input into the multimodal large model. Based on the output of the multimodal large model, the recall category is parsed out. If the output of the multimodal large model does not conform to the specifications, reload the total recall keywords and parse them. Set the maximum number of recall attempts (max_recall). If the number of recall attempts exceeds max_recall, output "Recall Failed".

4. The self-optimizing patch monitoring method based on a multimodal large model according to claim 3, characterized in that, The structure of the classification prompt includes "Category and Category Features", "Analysis Method", and "Special Notes". "Category and Category Features" includes the M category features to be recalled. "Analysis Method" indicates the classification analysis method used. "Special Notes" indicates specific matters that need attention. The interface prompts that only one category can be selected from the above M categories and returns in the format of "Select Category, Reason for Selection". The selected category is the selected category, and the reason for selection is the reason for selecting the category.

5. The self-optimizing patch monitoring method based on a multimodal large model according to claim 1 or 4, characterized in that, In step 2, the remote sensing image with the patch and the loaded classification prompt words are input into the multimodal large model. Based on the multimodal large model, one category is selected from M categories to obtain the patch category, including: The remote sensing image with the patch and the total classification prompt are input into the multimodal large model. Based on the output of the multimodal large model, the classification category is parsed out. If the output of the multimodal large model does not conform to the specifications, reload the total classification prompts and parse them. Set the maximum number of classification attempts (max_classify). If the number of classification attempts exceeds max_classify, output "Classification failed".

6. The self-optimizing patch monitoring method based on a multimodal large model according to claim 1, characterized in that, Step 3, based on the failed image samples, optimizes the total recall prompts, including: The correct category is d, and the recall categories are d1, d2, ..., d. M h failed recall samples were used as recall optimization samples; Based on each recalled and optimized sample, the category features and recall analysis methods of category d are optimized to obtain h optimized category features and optimized recall analysis methods for category d, which respectively constitute the optimized category feature set and optimized recall analysis method set for category d; The optimized category feature set and optimized recall analysis method set of category d are input into the large model for summarization, so as to obtain the final optimized category features and final optimized recall analysis method of category d; Iterate through all N categories to obtain the final optimized category features and the final optimized recall analysis method for each of the N categories; Based on the final optimized category features and the final optimized recall analysis method for each category, the total recall prompts are updated and optimized.

7. The self-optimizing patch monitoring method based on a multimodal large model according to claim 6, characterized in that, The process involves optimizing the category features and recall analysis method for each recalled and optimized sample to obtain h optimized category features and optimized recall analysis methods for category d, including: For each recalled optimized sample, its correct class is d, and its recalled classes are d1, d2, ..., dn. M For categories d1, d2, ..., d M Two photos are randomly selected from each category, and the categories are d1, d2, ..., d... M The photos of the samples and the photos of the recalled and optimized samples were uniformly numbered; Construct a total recall optimization prompt, the structure of which includes "comparison photos", "category features", "analysis method", and "reflection", wherein "comparison photos" includes recall categories d1, d2, ..., d M The photos, with "category features" including categories d1, d2, ..., d M The category characteristics of d, the "Analysis Method" suggests the recall analysis method used, and the "Reflection" suggests that "photos of category d are predicted to be {category d1}, {category d2}, ..., {category d...}". M "One of the categories in the list, but actually it is {category d}. Please reflect on the above information and update the category features and recall analysis method of category d". Return the results in the format "Category features: output the optimized category features of {category d}, Recall analysis method: output the optimized recall analysis method". The comparison photos, the recall optimization sample photos, and the total recall optimization prompts are input into the multimodal large model to obtain the optimized category features of category d and the optimized recall analysis method.

8. The self-optimizing patch monitoring method based on a multimodal large model according to claim 1, characterized in that, In step 3, based on the image samples that failed to be classified, the total classification prompt words are optimized, including: Obtain p failed classification samples with the correct category d and the classification category a as classification optimization samples; Based on each classification optimization sample, the classification analysis method between category d and category a is optimized to obtain p optimized classification analysis methods between category d and category a, which constitute a set of optimized classification analysis methods between category d and category a; The set of optimized classification analysis methods between category d and category a is input into a large model for summarization, resulting in the final optimized classification analysis method between category d and category a. Traverse the two category pairs to obtain the final optimized classification analysis method between each category pair. Based on the final optimized classification analysis method between each category pair, update and optimize the total classification prompt words.

9. The self-optimizing patch monitoring method based on a multimodal large model according to claim 8, characterized in that, The method of optimizing the classification analysis between category d and category a based on each classification optimization sample yields p optimized classification analysis methods between category d and category a, including: For each classification optimization sample with the correct category d and the classification category a, obtain four photos of category a and four photos of category d respectively, and number them. Construct a general suggestion term for classification optimization. The structure of the general suggestion term includes "Comparison Photos", "Category Features", "Analysis Method", "Precautions", and "Reflection". Among them, "Comparison Photos" includes photos of category a and category d; "Category Features" includes the category features of category a and category d; "Analysis Method" indicates the classification analysis method used; "Precautions" indicates specific matters that need attention; and "Reflection" indicates that "the photo is classified as {category a}, but the correct category is {category d}. Please reflect on the above information and update the classification analysis method." The suggestion term is returned in the format of "Classification Analysis Method: Output Optimized Classification Analysis Method, Precautions: Output Optimized Precautions". Input the comparison photos, classification optimization samples, and the total classification optimization prompts into the multimodal large model, and output the optimized classification analysis method between category d and category a.

10. A self-optimizing patch monitoring system based on a multimodal large model, characterized in that, It includes a patch monitoring module and a prompt word optimization module, wherein the patch monitoring module includes a recall submodule and a classification submodule; The recall submodule is used to input the remote sensing image with patches and the loaded total recall prompts into the multimodal large model, and recall M categories from N categories based on the multimodal large model, where N > M; The classification submodule is used to input the remote sensing image with patches and the loaded classification prompt words into the multimodal large model, and select one category from the recalled M categories based on the multimodal large model to obtain the category of the patches in the remote sensing image; The prompt word optimization module is used to optimize the total recall prompt words based on the image patch samples that failed to be recalled, and to optimize the total classification prompt words based on the image patch samples that failed to be classified.

Images