A multimodal large model picture identification prompt word optimization method and device

Abstract

This invention provides an optimization method and apparatus for multimodal large-scale image recognition prompt words, applied in the field of data processing technology. This application first extracts preprocessed initial basic prompt words and a massive number of images to be recognized, generating an initial prompt word set and standardized image data; the model initially identifies and filters target images and generates scene matching JSON information, then manually performs secondary verification to separate correct and incorrect sample sets; using the incorrect sample set for error analysis, new prompt words and difference data are generated, and the scene prompt word weights are dynamically adjusted based on the sample size; for scenes with excessively low weights, new initial weights are created and configured, and the top 9 high-weight prompt words are selected for the next round of recognition, forming a loop to adapt to spatiotemporal changes and improve recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an optimization method and apparatus for recognizing prompt words in multimodal large-scale images. Background Technology

[0002] With the development of multimodal large models, they have begun to be applied to various production fields. Among them, the demand for using multimodal large models to help identify massive amounts of images and select target images is increasing. However, since multimodal large models are different from traditional small models for specific scenarios, multimodal large models mainly rely on prompt words as the only variable to identify different scenarios.

[0003] For end users who use multimodal large models to identify specific images, their understanding of multimodal large models is not as good as that of artificial intelligence professionals. Therefore, the suggested words may not be suitable for multimodal large models to find target images. Furthermore, depending on the scene and time and space, the suggested words required for the target scene identified by multimodal large models will be affected to a certain extent. Therefore, a suggested word that is quite effective in the early stage of application may lose its effectiveness after a certain period of time.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0006] According to one aspect of this application, an optimization method for multimodal large-scale model image recognition prompt words is provided, comprising: extracting and preprocessing initial basic prompt words and a massive number of images to be recognized to generate an initial prompt word set for model recognition and standardized image data to be recognized; performing initial recognition on the standardized image data using a multimodal large-scale model, filtering target images confirmed by the model, and synchronously generating scene matching JSON information corresponding to each image, which includes scene number and associated description; performing a second manual verification on the target images confirmed by the model and the JSON information, separating the manually confirmed correct images, Manually excluded misidentified images and their corresponding JSON information are used to generate a set of correctly matched and incorrectly identified prompt words. The incorrectly identified sample set is input into a multimodal large model for error analysis, generating new prompt words for the corresponding scenario and data on the difference between new and old prompt words. Combined with the statistical results of the number of correct and incorrectly identified sample sets, the weights of prompt words for each scenario are dynamically adjusted. When the weight of a scenario is less than 1 / 10 of the average weight, a new scenario is created based on the new prompt words accumulated for that scenario and an initial weight is configured. Finally, the top 9 prompt words with the highest weights are selected as prompt words for the next round of recognition, forming an autoregressive update loop.

[0007] Another aspect of this application discloses an optimization device for multimodal large-scale image recognition prompt words, comprising: a data feature extraction and preprocessing unit, used to extract and preprocess initial basic prompt words and a large number of images to be recognized, generating an initial prompt word set for model recognition and standardized image data to be recognized; a data processing and optimization unit, used to perform initial multimodal large-scale recognition on the standardized image data to be recognized, filter target images confirmed by the model, and synchronously generate scene matching JSON information corresponding to each image, including scene number and associated description; and to perform manual secondary verification of the target images confirmed by the model and the JSON information. The system separates manually confirmed correct images and manually excluded misidentified images along with their corresponding JSON information, generating a correct sample set and a misjudged sample set for prompt word matching. The misjudged sample set is input into a multimodal large model for error analysis, generating new prompt words for the corresponding scenario and data on the difference between new and old prompt words. Based on the statistical results of the number of correct and misjudged sample sets, the weights of prompt words for each scenario are dynamically adjusted. When the scenario weight is lower than 1 / 10 of the average weight, a new scenario is created based on the new prompt words accumulated for that scenario, and an initial weight is configured. Finally, the top 9 prompt words with the highest weights are selected as prompt words for the next round of recognition, forming an autoregressive update loop.

[0008] According to another aspect of this application, an electronic device includes: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the above-described optimization method for multimodal large model image recognition prompts by executing the executable instructions.

[0009] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a second processor, implements the above-described optimization method for multimodal large model image recognition prompts.

[0010] This application provides an optimization method and apparatus for multimodal large-scale image recognition prompt words. The method involves extracting preprocessed initial basic prompt words and the image to be recognized, generating standardized data and an initial prompt word set; initially identifying and filtering target images using the model and generating scene matching JSON information; then manually verifying and separating correct and incorrect sample sets; using the incorrect sample set analysis to generate new prompt words and difference data; dynamically adjusting the prompt word weights based on the sample size; creating new scenes for low-weight scenes and configuring initial weights; and selecting the top 9 high-weight prompt words to enter the next round of iteration, thus achieving dynamic optimization of prompt words.

[0011] This application improves recognition accuracy by precisely optimizing prompt words and reducing false recognition through misjudgment sample analysis and dynamic weight adjustment. It adapts to spatiotemporal changes, with an autoregressive update loop ensuring continuous iteration of prompt words and preventing them from becoming ineffective due to scene changes. Furthermore, it optimizes resource allocation by selecting high-weight prompt words for recognition, improving both model recognition efficiency and resource utilization efficiency.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] Figure 1 The flowchart illustrates an optimization method for recognizing prompt words in multimodal large model images according to an embodiment of this application;

[0014] Figure 2 This illustration shows a schematic diagram of the structure of an optimization device for recognizing prompts in multimodal large model images according to an embodiment of this application. Detailed Implementation

[0015] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0016] The following is combined with Figure 1 This application describes an optimized method for recognizing prompts in multimodal large model images according to exemplary embodiments of the present application. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in any way. Rather, the embodiments of the present application are applicable to any suitable scenario.

[0017] In one implementation, Figure 1A schematic flowchart of an optimization method for recognizing prompt words in multimodal large model images according to an embodiment of this application is shown.

[0018] S101 extracts and preprocesses the initial basic prompt words and massive images to be recognized, generating an initial prompt word set and standardized image data to be recognized for model recognition.

[0019] In one implementation, basic prompts provided by end users during the initial application launch are collected. These prompts need to be formulated around the target scene requirements of image recognition, clearly defining the core recognition task and the scope of scenes to be matched. The basic prompts are standardized and organized according to a preset format to form an initial prompt set. Each prompt must include three core parts: first, a task description, clearly requiring the multimodal large model to sequentially identify whether the target scene exists in the image and output the results according to the constrained format; second, a scene list, listing the specific scenes to be recognized and clearly defining the recognition scope of each scene; and third, output constraints, stipulating that the results must be presented in JSON format and include two fields: "Scene" and "Description." The "Scene" field outputs the scene number that best matches the image (outputting 0 if no matching scene exists), and the "Description" field outputs the specific content in the image related to the matching scene. Content in the basic prompts that is semantically ambiguous, logically contradictory, or irrelevant to the image recognition task is removed to ensure that each prompt is accurate, unambiguous, and can be accurately understood by the multimodal large model, providing clear guidance for subsequent recognition tasks.

[0020] A massive amount of images to be recognized are extracted from the target application scenario, ensuring that the image data covers samples under different spatiotemporal conditions and environmental characteristics within that scenario. This guarantees data diversity and representativeness, avoiding limitations in model recognition performance due to a single sample. The collected images undergo a unified format conversion, adjusting images of different resolutions and formats to the standard format and size supported by the multimodal large-scale model, eliminating the impact of format differences on recognition results. Simultaneously, redundant information in the images (such as irrelevant watermarks and borders) is removed to reduce interference factors and improve model recognition efficiency. Invalid images that cannot meet recognition requirements, such as blurry, damaged, or missing information, are discarded, retaining images with clear image quality and containing effective visual information to form standardized image data to be recognized, ensuring that the data input to the multimodal large-scale model has a good recognition foundation.

[0021] After the above extraction and preprocessing, two types of core data are finally generated: one is the initial prompt word set for model recognition, which has a unified structure and clear semantics and can be directly used for the initial image recognition task of multimodal large models; the other is the standardized image data to be recognized, which has a unified format and meets the quality standards, and can provide reliable input support for model recognition, ensuring the smooth progress of the initial recognition process.

[0022] S102 performs initial multimodal large-scale model recognition on standardized image data to be identified, filters target images confirmed by the model, and synchronously generates scene matching JSON information corresponding to each image, including scene number and associated description.

[0023] In one implementation, standardized image data to be identified is matched and analyzed with initial basic prompts to generate target images confirmed by the model and corresponding scene matching JSON information. The initial basic prompts include task instructions with scene descriptions, a scene list, and JSON format output constraints. The standardized image data to be identified consists of a large number of preprocessed images. The core data input to the multimodal large model includes two types: first, standardized image data to be identified, which is a large number of preprocessed images with uniform format, clear image quality, and effective visual information, covering samples with different spatiotemporal and environmental characteristics under the target application scenario; second, the initial basic prompts, which contain three core components: task instructions with scene descriptions (explicitly requiring the model to sequentially identify whether the target scene exists in the images and output results according to constraints), a specific scene list (defining the scope of scenes to be identified), and JSON format output constraints (specifying the fields and format requirements to be included in the results).

[0024] After receiving standardized image data to be recognized and initial basic prompts, the multimodal large model performs feature comparison and matching analysis on each image against each scene in the scene list, according to the task instructions in the prompts. By recognizing visual elements and scene features in the image, the model determines whether it is associated with a certain scene, and thus preliminarily determines whether the image is the target image and associates it with the corresponding scene information.

[0025] After model matching and recognition, two types of associated data are output: first, the target images initially confirmed by the model (i.e., images that are determined to be matched with a certain scene in the prompt words); and second, preliminary scene matching information corresponding to each target image, providing basic data support for subsequent structured processing.

[0026] The matching and recognition results are filtered, retaining the target images identified by the model and discarding non-target images. Target images must meet the condition of matching with any scene in the initial basic prompts. Clear filtering criteria are set, namely, the core condition that the target image must meet the condition of "matching with any scene in the initial basic prompts." This condition is based on the model's matching and recognition results, focusing on the feature fit between the image and the scenes in the scene list, without adding any other irrelevant filtering dimensions.

[0027] Each result after model matching and recognition is verified. If the model determines that an image matches any of the initial basic prompts, it is retained as the target image. If the model determines that an image does not match any of the scenes, or cannot be clearly associated with any scene, it is determined to be a non-target image and discarded, ensuring that subsequent processing only focuses on valid target images.

[0028] The filtered target images undergo structured processing for scene matching information, generating JSON information containing scene and description fields. The scene field outputs the scene number that best matches the image, and the description field outputs related content from the image that relates to the matched scene. The process strictly adheres to the JSON format constraints outlined in the initial basic prompts, ensuring that the structured data includes two fixed fields: "scene" and "description," with the meanings of these fields consistent with the output requirements. This guarantees a unified data format that can be directly parsed and used in subsequent stages.

[0029] For the scene field, based on the model matching and recognition results, output the scene number that best matches the current image; if the image is identified as the target image, but after further verification it is found that the matching degree with all scenes does not reach the optimal standard, or there is a fuzzy matching situation, then output 0 according to the rules.

[0030] For the description field, key visual information and feature details related to the matching scene are extracted from the image and described in an accurate and concise text. This ensures that the description clearly reflects the association between the image and the corresponding scene, providing a useful reference for subsequent manual verification and prompt word updates.

[0031] S103, perform a second manual verification of the target images and JSON information confirmed by the model, separate the manually confirmed correct images and the manually excluded misidentified images and their corresponding JSON information, and generate a set of correct and incorrect sample sets for prompt word matching.

[0032] In one implementation, the target image confirmed by the model and its corresponding scene matching JSON information undergo manual verification feature extraction processing to generate manual verification result features, image association information features, and JSON structured data features. The manual verification result features include the image matching correctness conclusion data determined by the end user; the image association information features include the original data of the target image confirmed by the model and the scene matching association identifier; and the JSON structured data features include scene matching details data containing scene and description fields. Using the target image confirmed by the model and its corresponding scene matching JSON information as the extraction objects, focusing on the key information dimensions required for manual verification, and by parsing the core content of the data one by one, three independent but related feature data are separated and extracted, providing a foundation for subsequent verification analysis.

[0033] Regarding the characteristics of the manually confirmed results, the core is the judgment conclusion data of the end user. By viewing the target image and the corresponding scene matching JSON information, the end user can clearly determine whether the image matches the prompt word scene correctly, forming a binary conclusion data of "matching correctly" or "matching incorrectly", which directly reflects the human's calibration opinion on the model recognition results.

[0034] Image association information features include two core types of data: first, the original data of the target image confirmed by the model, that is, the image itself without any additional processing, which retains complete visual information; and second, scene matching association identifiers, which are used to associate the image with the scene number matched by the model during the initial recognition, as well as the matching confidence level and related identifier information, to ensure the traceability and association between the image and the corresponding recognition result.

[0035] Based on the characteristics of JSON structured data, it is entirely based on the extraction of JSON information for scene matching, covering the complete content of the two core fields of "scene" and "description". The scene number in the "scene" field and the associated description details in the "description" field are fully preserved, accurately reflecting the scene matching details initially identified by the model.

[0036] Feature extraction processing is performed on the judgment logic and sample classification standards in the manual verification process to generate correct sample judgment features, incorrect sample judgment features, and data separation rule features. Correct sample judgment features include manual confirmation data consistent with the prompt word scenario; incorrect sample judgment features include manual exclusion data inconsistent with the prompt word scenario; and data separation rule features include classification execution standard data for splitting samples according to confirmation results. Focusing on the core rule dimensions of the manual verification process, feature data to guide sample separation is extracted from the actual verification judgment logic and sample classification standards, ensuring that subsequent sample classification has clear and unified rule basis.

[0037] The core of the correct sample determination feature is the manual confirmation data that matches the prompt word scene. It clarifies the core standard for end users to determine "correct match" - that is, the description content in the image visual features and scene matching JSON information completely matches the description of the corresponding scene in the initial basic prompt word, without contradiction or deviation. This feature clarifies the determination boundary of the correct sample.

[0038] The core of the feature for identifying misjudged samples is the data used for manual exclusion based on the scenario that does not match the prompt word. It clarifies the core situations in which end users judge "matching error" - including the lack of correlation between the visual features of the image and the scenario description of the prompt word, and the inconsistency between the description in the JSON information and the scenario features. This feature defines the identification criteria for misjudged samples.

[0039] Regarding the characteristics of data separation rules, the core is to split the sample classification standard data according to the confirmation result. It is clearly stipulated that: images that are manually judged to be "correctly matched" and their corresponding JSON information are classified into one type of sample; images that are manually judged to be "incorrectly matched" and their corresponding JSON information are classified into another type of sample. The two types of samples must be associated with the corresponding manual confirmation conclusions to ensure the consistency of classification execution.

[0040] Based on the core requirement of manual calibration, this study analyzes and processes features including those of manual confirmation results, image association information, JSON structured data, correct sample judgment features, incorrect sample judgment features, and data separation rule features. During sample separation, the correlation between the manual judgment results and the image and JSON information is verified, generating a correct sample set and a incorrect sample set for prompt word matching. The correct sample set represents the image and associated data features that effectively match the prompt words, while the incorrect sample set represents the image and associated data features that mismatch the prompt words, resulting in a sample classification result that includes the manual confirmation results, image data, and JSON information. Guided by the core requirement of manual calibration (to correct model misidentification and select valid samples to support prompt word updates), the previously extracted basic feature data and rule feature data are correlated and fused for analysis. The focus is on verifying the logical consistency between the manual judgment conclusions and the original image data and JSON structured information to avoid sample classification errors due to data association bias.

[0041] Cases that are manually confirmed as "correctly matched" and meet the criteria for correct samples are selected. The original target image data, scene matching JSON information, and manual confirmation conclusions are then linked and integrated to form a correct prompt word matching sample set. This sample set is used to characterize the image and associated data features that effectively match the initial basic prompt words, providing data support for subsequent prompt word weight increments.

[0042] Cases that were manually confirmed as "matching incorrectly" and met the criteria for misjudged samples were selected. The corresponding original image data, JSON information, and manual confirmation conclusions were then linked and integrated to form a sample set of mismatched prompt words. This sample set is used to characterize the image and associated data features of initial basic prompt word mismatches, providing a core data source for subsequent error analysis and prompt word optimization.

[0043] Both sample sets fully contain the three core elements of manual confirmation results, image data, and JSON information, forming a unified and logically clear sample classification result, providing direct input for the subsequent error analysis and weight adjustment of the prompt word updater.

[0044] S104. Input the misjudged sample set into the multimodal large model for error analysis, generate new prompt words for the corresponding scenario and the difference between the new and old prompt words, and dynamically adjust the weight of prompt words for each scenario by combining the statistical results of the number of correct sample sets and misjudged sample sets.

[0045] In one implementation, images from the misjudged sample set, corresponding JSON information, and associated scene prompts are input into a multimodal large-scale model. Error localization analysis is performed to address the mismatch between image descriptions and scene prompts, generating optimized new prompts adapted to the specific scene. Three types of associated data are input into the multimodal large-scale model: the original image data from the misjudged sample set, the scene matching JSON information corresponding to the image, and the scene prompts associated with the model during its initial identification. These three types of data correspond one-to-one, ensuring the model can accurately trace the complete information chain of the misjudged scene.

[0046] After receiving the data, the model focuses on the core problem of "the image description does not match the scene description of the prompt word". It first parses the content of the "description" field in the JSON information and the scene description of the prompt word corresponding to the "scene" field. Then, it combines the visual features of the image to locate the root cause of the misjudgment, including problems such as the fuzzy scene description of the prompt word, incomplete definition of scene features, and mismatch with the actual visual information of the image.

[0047] The rules for generating new prompt words are as follows: Based on the error localization results, the model generates optimized new prompt words that are adapted to the scenario. The new prompt words need to make up for the defects of the original prompt words, clarify the core features of the scenario, refine the scope, ensure that similar misjudgment situations can be effectively avoided, and maintain the same structural framework as the original prompt words, which includes task description, scenario description, and output constraints.

[0048] A quantitative evaluation is conducted based on the semantic differences and scene matching dimension differences between the new and original prompt words, generating difference score data ranging from 0 to 10, where 10 represents the highest difference score and 0 represents the lowest. The number of correct sample sets (A) and the number of incorrect sample sets (B) are counted. Combined with the difference score data (D), a weight adjustment formula is constructed, and the prompt words that correctly match the scene are calculated with increasing weights (Wn=Wn+A / (A+B)). (0.75). The new prompts are evaluated against the original prompts based on two core dimensions: semantic difference and scene matching difference. Semantic difference focuses on the differences in accuracy and clarity of expression, while scene matching difference focuses on the differences in feature coverage and definition precision. The final result is a difference score ranging from 0 to 10, where 10 represents the highest difference score and 0 represents the lowest.

[0049] The number of correct samples in the current identification period is counted and denoted as A; the number of incorrectly identified samples is counted and denoted as B. This ensures that the parameter statistics cover all manually verified samples in the current period, providing data support for weight adjustment.

[0050] By combining the difference data D and parameters A and B, two types of weight adjustment formulas are constructed. The first is a weight-increasing formula for correctly matched scenarios, used to reward prompt words with good recognition performance; the second is a weight-decreasing formula for misjudged scenarios, used to penalize prompt words with poor recognition performance. The formula logic is directly linked to the scene recognition performance to ensure the rationality of the weight adjustment.

[0051] For prompts in incorrectly associated scenarios, a weight reduction calculation is performed (Wn=Wn-1 / (A+B)D / 100.8), and when D>5, the new prompt is stored in the corresponding scenario's database new field. For prompts in scenarios associated with the correct sample set, an adjustment value is calculated using a weight increase formula, and this adjustment value is added to the original scenario weight Wn to achieve a positive weight update and strengthen the priority of valid prompts.

[0052] The process of reducing the weight of misjudged scenarios is as follows: For the scenario prompt words associated with the misjudged sample set, the adjustment value is calculated using the weight reduction formula. This adjustment value is then subtracted from the original scenario weight Wn to achieve a negative weight update and reduce the priority of invalid prompt words.

[0053] The storage rules for new prompt words are as follows: determine whether the difference data D is greater than 5. If the condition is met, it means that the new prompt word is significantly different from the original prompt word and the optimization effect is obvious. The new prompt word is stored in the database dedicated field of the corresponding scene to accumulate materials for subsequent scene splitting and prompt word iteration. If D≤5, it is not stored for the time being, and only the weight adjustment is performed.

[0054] By integrating error analysis results with weight calculation rules, dynamic optimization data is generated, including new prompt word content, difference degree values, and scene weight update results. This optimization data characterizes the iterative optimization features of prompt word adaptation to different scenarios. The previously generated new prompt word content, difference degree values, and updated scene weight results are correlated and integrated to ensure a one-to-one mapping between the three types of data and their corresponding scenarios, with no correlation bias.

[0055] The final result is dynamic optimization data, which includes three core elements: details of new prompt words, quantification results of differences, and scene weight update records. This data is used to intuitively represent the iterative optimization characteristics of prompt words adapting to different scenarios, providing direct data support for subsequent scene segmentation and optimal prompt word selection, and ensuring the continuity and traceability of the prompt word update process.

[0056] S105, when the scene weight is less than 1 / 10 of the average weight, a new scene is created based on the new prompt words accumulated in the scene and the initial weight is configured. Finally, the top 9 prompt words with the highest weights are selected as prompt words for the next round of recognition, forming an autoregressive update loop.

[0057] In one implementation, the average weight W of all scenarios is calculated in real time. A threshold check is performed on the weights Wn of each scenario to determine whether the scenario splitting condition of Wn < W / 10 is met. The average weight calculation logic is as follows: the weight data of all current scenarios is aggregated in real time, and the average weight of all scenarios is calculated by taking the arithmetic mean, denoted as W. The average weight W is updated in real time with each adjustment of the scenario weights to ensure that it reflects the overall effectiveness of all current scenarios.

[0058] The threshold verification rules are as follows: for each scene's weight Wn, a threshold verification is performed one by one. The core judgment condition is whether the scene weight Wn is less than 1 / 10 of the average weight W (i.e., Wn < W / 10). This condition is used to filter out the scenes corresponding to prompt words with extremely poor recognition performance that can no longer adapt to the current scene requirements, providing a basis for subsequent scene segmentation.

[0059] After verification, two types of results are output: one is a list of scenarios that meet the splitting conditions (scenarios with too low weights), and the other is a list of scenarios that do not meet the splitting conditions (scenarios with weights within a reasonable range), thus clarifying the target scenarios that need to be processed next.

[0060] When the splitting condition is triggered, all new prompt words accumulated in the `new` field of the corresponding scenario database are extracted. These are then semantically integrated and optimized using a multimodal large-scale model to generate standardized prompt words adapted to the new scenario. For scenarios that meet the splitting condition, all accumulated new prompt words are extracted from the scenario-specific database fields. These new prompt words are all optimized prompt words previously generated based on misjudgment sample analysis, possessing the characteristic of compensating for the deficiencies of the original prompt words.

[0061] All extracted new prompt words are input into a multimodal large model. The model performs semantic analysis, feature fusion and logical sorting on these new prompt words, removes duplicate expressions and integrates core optimization points to ensure that the newly generated prompt words are semantically coherent, accurately define the scene and are free of logical contradictions.

[0062] Generate standardized prompts that adapt to new scenarios. These prompts must strictly follow the preset structural framework, including three core parts: task description, scenario description, and output constraints, to ensure that they can be directly used in the next round of image recognition tasks.

[0063] Create a new scenario and complete the data configuration. Write the integrated new prompt words into the scenario description field and set the initial weight to half of the current average weight W. Based on the scenario requirements triggered by the splitting condition, create a new independent scenario and assign it a unique scenario identifier to ensure it is distinguishable from existing scenarios and to guarantee the orderliness of scenario management.

[0064] The standardized prompt words, which have been integrated and optimized by the multimodal large model, are written into the scene description field of the new scene to clarify the recognition range and core features of the new scene. At the same time, the initial weight of the new scene is set to half of the current average weight W, which not only gives the new scene a certain initial priority, but also reserves space for adjusting the weight in the future based on the recognition effect.

[0065] All scenes are sorted from highest to lowest weight, and the top 9 scene prompts with the highest weights are selected as the core instructions for the next round of image recognition. The weight data of all scenes (including original scenes that have not been split and newly created scenes) are summarized and sorted from highest to lowest weight. The higher the weight, the better the recognition effect and the stronger the adaptability of the prompts for that scene.

[0066] From the sorted list of scenes, the top nine scenes with the highest weights are selected as the core instructions for the next round of image recognition. This selection rule focuses on recognition performance, ensuring that the optimal prompts are used first in the next round of recognition to improve accuracy.

[0067] By combining scene segmentation, new prompt word generation, and weighted selection logic into a closed loop, an autoregressive update result is generated, containing a new set of available prompt words and scene weight distribution. This update result characterizes the dynamic optimization features of prompt words adapting to spatiotemporal changes. The logic of the three core stages—scene segmentation (eliminating inefficient scenes), new prompt word generation (optimizing scene descriptions), and weighted selection (selecting the best efficient prompt words)—is integrated to form an autoregressive closed loop of "weight verification - scene optimization - prompt word selection - next round of recognition," ensuring that prompt words are continuously iterated and updated based on actual usage performance.

[0068] The final autoregressive update result contains two core data categories: a new set of available prompt words (i.e., the top 9 high-weight prompt words selected) and the weight distribution of all current scenes. This update result characterizes the dynamic optimization features of prompt words adapting to spatiotemporal changes, providing direct instruction support for the next round of image recognition, while also preserving traceability data for subsequent weight adjustments and scene optimization.

[0069] In one implementation, such as Figure 2 As shown, this application also provides an optimization device for recognizing prompt words in multimodal large model images, including:

[0070] The data feature extraction and preprocessing unit 201 is used to extract and preprocess the initial basic prompt words and massive images to be recognized, and generate the initial prompt word set and standardized image data to be recognized for model recognition;

[0071] The data processing and optimization unit 202 is used to perform initial multimodal large-scale model recognition on standardized image data to be recognized, filter target images confirmed by the model, and synchronously generate scene matching JSON information corresponding to each image, including scene number and associated description; perform secondary manual verification on the target images and JSON information confirmed by the model, separate manually confirmed correct images and manually excluded misidentified images and their corresponding JSON information, and generate a correct sample set and a misjudged sample set of prompt words; input the misjudged sample set into the multimodal large-scale model for error analysis, generate new prompt words for the corresponding scene and data on the difference between new and old prompt words, and dynamically adjust the weight of prompt words for each scene based on the statistical results of the number of correct sample sets and misjudged sample sets; when the scene weight is lower than 1 / 10 of the average weight, a new scene is created based on the new prompt words accumulated for that scene and the initial weight is configured, and finally the top 9 prompt words with the highest weights are selected as prompt words for the next round of recognition, forming an autoregressive update loop.

[0072] The computer-readable storage medium provided in the above embodiments of this application and the optimization method for multimodal large model image recognition prompts provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0073] The various embodiments in this application are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of the method, system, electronic device, and readable storage medium for evaluating the optimization of multimodal large model image recognition prompts are basically similar to the embodiments of the multimodal large model image recognition prompt optimization method described above, and therefore the descriptions are relatively simple. Relevant parts can be referred to the descriptions of the embodiments of the multimodal large model image recognition prompt optimization method described above.

Claims

1. An optimization method for multi-modal large model picture recognition prompt words, characterized in that, include: The initial basic prompt words and massive images to be recognized are extracted and preprocessed to generate the initial prompt word set and standardized image data to be recognized for model recognition. The standardized image data to be identified is initially identified using a multimodal large model. The target images confirmed by the model are then selected, and scene matching JSON information corresponding to each image is generated synchronously, including scene number and associated description. The target images and JSON information confirmed by the model are manually verified a second time. The manually confirmed correct images and the manually excluded misidentified images and their corresponding JSON information are separated to generate a set of correct and incorrect sample sets of prompt words. The misjudged sample set is input into a multimodal large model for error analysis. Three types of related data are input into the multimodal large model: the original image data in the misjudged sample set, the scene matching JSON information corresponding to the image, and the scene prompt words associated with the model during the initial recognition. The three types of related data are one-to-one, generating new prompt words for the corresponding scene and the difference between the new and old prompt words. Combining the statistical results of the number of correct sample sets and misjudged sample sets, the weights of each scene prompt word are dynamically adjusted. This includes inputting the images, corresponding JSON information, and associated scene prompt words in the misjudged sample set into the multimodal large model, performing error localization analysis on the problem of the image description and prompt word scene not matching, and generating optimized new prompt words adapted to the scene. A quantitative evaluation is conducted based on the semantic differences and scene matching dimension differences between the new prompt words and the original prompt words, generating difference data with a value range of 0-10, where 10 represents the highest difference and 0 represents the lowest difference. Count the number of correct samples (A) and the number of incorrect samples (B). Combine this with the difference data (D) to construct a weight adjustment formula. Perform weight increment calculation on the prompt words of the correct matching scenario (Wn=Wn+A / (A+B)). 0.75); For prompt words in misjudged scenarios, the weight is calculated by decreasing (Wn=Wn-1 / (A+B)D / 100.8), and when D>5, the new prompt word is stored in the new field of the corresponding scenario database; The error analysis results and weight calculation rules are integrated and processed to generate dynamic optimization data containing new prompt word content, difference value, and scenario weight update results, where the optimization data is used to characterize the iterative optimization features of prompt word adaptation scenarios; When the scene weight is less than 1 / 10 of the average weight, a new scene is created based on the new prompt words accumulated in the scene and the initial weight is configured. Finally, the top 9 prompt words with the highest weights are selected as prompt words for the next round of recognition, forming an autoregressive update loop, including real-time calculation of the average weight W of all scenes, threshold verification of the weight Wn of each scene, and determination of whether the scene splitting condition of Wn < W / 10 is met. When the splitting condition is triggered, all new prompt words accumulated in the new field of the corresponding scenario database are extracted, and semantic integration and optimization are performed through a multimodal big model to generate standardized prompt words adapted to the new scenario. Create a new scene and complete the data configuration. Write the integrated new prompt words into the scene description field and set the initial weight to half of the current average weight W. Sort all scenes from high to low weight and select the top 9 scene prompt words with the highest weight as the core instructions for the next round of image recognition. By combining scene segmentation, new prompt word generation, and weight selection logic to form a closed loop, an autoregressive update result is generated, which includes a new set of available prompt words and scene weight distribution. The update result is used to characterize the dynamic optimization features of prompt word adaptation to spatiotemporal changes.

2. The method as described in claim 1, characterized in that, The standardized image data to be identified is initially identified using a multimodal large-scale model. Target images confirmed by the model are then selected, and scene matching JSON information corresponding to each image is generated synchronously. This information includes a scene number and associated description, including: The standardized image data to be identified is matched and analyzed with the initial basic prompt words to generate the target image confirmed by the model and the corresponding scene matching JSON information. The initial basic prompt words include task instructions with scene description, scene list and JSON format output constraints. The standardized image data to be identified is a large number of images to be identified after preprocessing. The matching and recognition results are filtered, retaining the target images determined by the model and discarding non-target images. The target images must meet the judgment condition that they are matched with any scene in the initial basic prompt words. The filtered target images undergo scene matching information structuring processing to generate JSON information containing scene and description fields. The scene field outputs the scene number that best matches the image, and the description field outputs the related content in the image that is related to the matched scene.

3. The method as described in claim 2, characterized in that, The target images and JSON information confirmed by the model undergo secondary manual verification. This involves separating manually confirmed correct images and manually excluded misidentified images and their corresponding JSON information, generating a correct sample set and a misidentified sample set for prompt word matching, including: The target image confirmed by the model and the corresponding scene matching JSON information are manually verified and feature extraction is performed to generate manual verification result features, image association information features, and JSON structured data features. Among them, the manual verification result features include the image matching correctness conclusion data judged by the end user; the image association information features include the original data of the target image confirmed by the model and the scene matching association identifier; the JSON structured data features include scene matching details data containing scene and description fields. The judgment logic and sample classification criteria in the manual verification process are processed by feature extraction to generate correct sample judgment features, incorrect sample judgment features, and data separation rule features. Among them, the correct sample judgment features include manual confirmation basis data that matches the prompt word scenario; the incorrect sample judgment features include manual exclusion basis data that does not match the prompt word scenario; and the data separation rule features include classification execution standard data for splitting samples according to the confirmation result. Based on the core requirement of manual calibration, this study analyzes and processes the characteristics of manual confirmation results, image association information, JSON structured data, as well as the characteristics of correct sample judgment, incorrect sample judgment, and data separation rules. During sample separation, the correlation between the manual judgment results and the image and JSON information is verified, generating a correct sample set and an incorrect sample set for prompt word matching. The correct sample set is used to represent the image and associated data features that are effectively matched by prompt words, while the incorrect sample set is used to represent the image and associated data features that are mismatched by prompt words, forming a sample classification result that includes the manual confirmation results, image data, and JSON information.

4. An optimization device for recognizing prompt words in multimodal large-scale images, used to implement the method of claim 1, characterized in that, The device includes: The data feature extraction and preprocessing unit is used to extract and preprocess the initial basic prompt words and massive images to be identified, and generate the initial prompt word set and standardized image data to be identified for model recognition; The data processing and optimization unit performs initial multimodal large-scale model recognition on standardized image data to be recognized, filters target images confirmed by the model, and synchronously generates scene matching JSON information for each image, including scene number and associated description. It then performs secondary manual verification of the target images and JSON information confirmed by the model, separating manually confirmed correct images and manually excluded misidentified images and their corresponding JSON information, generating a correct sample set and a misjudged sample set for prompt word matching. The misjudged sample set is input into the multimodal large-scale model for error analysis, generating new prompt words for the corresponding scene and data on the difference between new and old prompt words. Based on the statistical results of the number of correct and misjudged sample sets, the weights of prompt words for each scene are dynamically adjusted. When the scene weight is lower than 1 / 10 of the average weight, a new scene is created based on the new prompt words accumulated for that scene, and an initial weight is configured. Finally, the top 9 prompt words with the highest weights are selected as prompt words for the next round of recognition, forming an autoregressive update loop.

5. An electronic device, characterized in that, include: First processor; and memory for storing executable instructions of the first processor; The first processor is configured to execute the optimization method for multimodal large model image recognition prompts as described in any one of claims 1 to 3 by executing the executable instructions.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the second processor, it implements the optimization method for recognizing prompt words for multimodal large model images as described in any one of claims 1 to 3.

Images