Prompt word optimization method and system based on approximate submodular function and continuous learning

Abstract

The disclosure provides a prompt word optimization method and system based on an approximate submodular function and continuous learning, relates to the technical field of artificial intelligence multi-modal perception, and comprises the following steps: constructing a candidate prompt word set, designing a combination objective function based on the properties of the approximate submodular function, solving the combination objective function by using a greedy selection algorithm combined with random disturbance, multi-round iteration and a task adaptive mechanism, realizing the optimization selection of the candidate prompt word set, gradually selecting prompt words with the maximum gain from the candidate prompt word set to join an optimized subset, and performing local optimization once a new prompt word is selected; and when all target prompt words are selected, all prompt words are jointly optimized in the continuous space by using an alternating optimization strategy, and iteration is continuously performed until the optimized prompt words are obtained. The disclosure realizes efficient adaptive updating of language model prompt words, so as to improve the generalization performance and robustness of the model in zero-shot, few-shot and concept drift scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence multimodal perception technology, and in particular to a prompt word optimization method and system based on approximate submodular functions and continuous learning. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence (AI), its application in complex environments is becoming increasingly widespread, particularly in the fields of multimodal perception and multi-task learning. Existing methods based on vision-language models (such as CLIP) have limitations in cue word selection and are unable to effectively and automatically select the optimal cue word. This severely hinders classification performance when faced with complex, open-ended recognition tasks. Therefore, a more effective cue word optimization method is urgently needed to enhance the model's recognition capabilities in diverse, dynamic, and uncertain environments, and improve its generalization and robustness.

[0004] When performing recognition tasks under complex and open conditions, data sources often have significant differences in modality, resolution, acquisition time, and spatial information. The fusion and processing of data from different modalities often face huge challenges. Especially in dynamic environments, the quality and stability of data are often affected by multiple factors, such as environmental changes, equipment performance, and uncertainty in acquisition conditions. In addition, the processing of these multimodal data usually requires a large amount of annotated data. However, the cost of obtaining high-quality annotated data is high, and the annotation process is often time-consuming and labor-intensive, making the acquisition of large-scale annotated data difficult.

[0005] Existing supervised methods rely heavily on high-quality and sufficient annotated data. However, in practical applications, especially when dealing with complex multimodal data, it is often difficult to obtain sufficient annotated data and achieve effective spatial and temporal alignment. Furthermore, existing multimodal data fusion methods suffer from poor generalization and real-time performance in the face of volatile environments and unstable data quality, making them difficult to meet the demands of complex tasks. Summary of the Invention

[0006] To address the above-mentioned issues, this paper proposes a prompt word optimization method and system based on approximate submodular functions and continuous learning. In a multimodal and multi-task environment, this method constructs a set of candidate prompt words and applies an approximate submodular utility function to evaluate their discriminability and diversity. The optimal subset is greedily selected, and then combined with continuous optimization to achieve efficient adaptive updating of the prompt words of the pre-trained language model, thereby improving the generalization performance and robustness of the model in zero-shot, few-shot, and concept drift scenarios.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] The prompt word optimization method based on approximate submodular functions and continuous learning includes:

[0009] Obtain image data to be classified;

[0010] The cue words corresponding to the image are optimized using an approximate submodular function and a continuous learning joint optimization strategy. The image data and the optimized cue words are input into a multimodal classification model for image-text matching, and the image classification result is output.

[0011] Among them, the process of implementing the joint optimization strategy of approximate submodular function and continuous learning for prompt words includes: constructing a set of candidate prompt words, designing a combined objective function based on the properties of approximate submodular function, and solving the combined objective function using a greedy selection algorithm that combines random perturbation, multiple rounds of iteration and task adaptation mechanism to achieve optimal selection of the candidate prompt word set, gradually selecting the prompt word with the largest gain from the candidate prompt word set and adding it to the optimized subset. Each time a new prompt word is selected, a local optimization is performed. When all target prompt words are selected, all prompt words are jointly optimized in the continuous space using an alternating optimization strategy, and iterate continuously until the optimized prompt word is obtained.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] The prompt word optimization system based on approximate submodular functions and continuous learning includes:

[0014] A data acquisition module, used to acquire image data to be classified;

[0015] The image-text matching classification module is used to optimize the prompt words corresponding to the image using an approximate submodular function and a continuous learning joint optimization strategy; the image data and the optimized prompt words are input into the multimodal classification model for image-text matching, and the image classification result is output;

[0016] Among them, the process of implementing the joint optimization strategy of approximate submodular function and continuous learning for prompt words includes: constructing a set of candidate prompt words, designing a combined objective function based on the properties of approximate submodular function, and solving the combined objective function using a greedy selection algorithm that combines random perturbation, multiple rounds of iteration and task adaptation mechanism to achieve optimal selection of the candidate prompt word set, gradually selecting the prompt word with the largest gain from the candidate prompt word set and adding it to the optimized subset. Each time a new prompt word is selected, a local optimization is performed. When all target prompt words are selected, all prompt words are jointly optimized in the continuous space using an alternating optimization strategy, and iterate continuously until the optimized prompt word is obtained.

[0017] According to some embodiments, the present disclosure adopts the technical solutions as follows:

[0018] A computer program product comprising a computer program which, when executed by a processor, implements the prompt word optimization method based on the approximate submodular function and continuous learning.

[0019] According to some embodiments, the present disclosure adopts the technical solutions as follows:

[0020] A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the prompt word optimization method based on the approximate submodular function and continuous learning.

[0021] According to some embodiments, the present disclosure adopts the technical solutions as follows:

[0022] An electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the prompt word optimization method based on the approximate submodular function and continuous learning.

[0023] Compared with the prior art, the present disclosure has the beneficial effects that:

[0024] The prompt word optimization method based on the approximate submodular function and continuous learning of the present disclosure significantly improves the performance of the CLIP model in the multi-modal image classification task through intelligent prompt word selection and stage-wise optimization training. It combines the optimization characteristics of the approximate submodular function, the efficiency of the greedy selection algorithm, and the advantages of discrete and continuous learning methods, thereby achieving better classification performance in complex and open environments and effectively addressing typical problems such as uneven data distribution, high label noise, and limited computing resources.

[0025] The prompt word optimization method based on the approximate submodular function and continuous learning of the present disclosure proposes a prompt word optimization technology combining the approximate submodular function and continuous learning. This method can efficiently improve the optimization efficiency and quality of the prompt word set, thereby improving the generalization ability and robustness of the model in complex recognition tasks. By adaptively updating the prompt words and performing real-time optimization, this technology can adapt to complex and dynamic environmental changes without relying on a large amount of labeled data, and is particularly suitable for scenarios where data quality is unstable or it is difficult to obtain a large amount of labeled data. This method not only significantly improves the accuracy and efficiency of multi-modal data processing, but also has a wide range of application prospects and is suitable for recognition tasks under various complex and open conditions.

[0026] The prompt optimization method based on the approximate submodular function and continuous learning of the disclosure uses the function as an evaluation standard for prompt selection, ensures that the selected prompt has representativeness and information coverage, and maximizes the expression ability of the model in multiple categories and multiple scenes. To solve the problem of "long-tail category" recognition difficulty in an open environment, the invention introduces a category adaptability weighting mechanism in the submodular function, dynamically adjusts the coverage evaluation value of the prompt according to the category frequency, makes the prompt selection process pay more attention to rare categories and edge scenes, and significantly improves the classification performance of the model on the tail categories and the overall generalization ability. In addition, the approximate submodular function has a significant advantage in theoretical calculation efficiency, so that the prompt screening process can be completed under limited computing resources even when facing thousands of candidate words.

[0027] The prompt optimization method based on the approximate submodular function and continuous learning of the disclosure adopts a greedy selection strategy to efficiently implement the prompt screening process, and gradually selects words with the largest marginal revenue from the candidate set to construct an optimal subset. The greedy algorithm has been proven to have theoretical performance guarantees when dealing with submodular function optimization problems, and can approach the global optimal solution with an approximation ratio of (about 63%) is particularly suitable for large-scale discrete selection tasks. Compared with traditional brute-force search or gradient-based global optimization methods, the greedy strategy is more scalable and computationally efficient, and can obtain high-quality solutions within an acceptable time.

[0028] The prompt optimization method based on the approximate submodular function and continuous learning of the disclosure further introduces a multi-round iteration and perturbation mechanism based on the standard greedy framework, guiding the algorithm to start from multiple initialization states to construct a subset in each iteration, effectively improving the diversity and coverage of the search space, and reducing the risk of falling into a local optimum. At the same time, the class distribution, task context and historical selection path are considered in the prompt selection process, and the word priority is dynamically adjusted to guide the model to pay more attention to difficult-to-classify categories and edge samples in the task, thereby enhancing the adaptability of the model in complex environments and the global representativeness of the prompt set.

[0029] The prompt optimization method based on the approximate submodular function and continuous learning of the disclosure proposes a discrete-continuous fusion learning framework, which first selects high-quality prompts from the discrete submodular function, and then fine-tunes end-to-end through the learnable vector in the CLIP framework. The training process adopts a phased scheduling mechanism, and the prompt set is fixed in each phase. The performance is evaluated after small batch training and the prompt pool is dynamically updated. This method effectively maintains the stability of the prompt semantic structure while improving the training efficiency and final classification accuracy. In actual deployment, this strategy can flexibly adapt to different data volumes and different scene training needs, and has strong universality and portability.

[0030] The prompt word optimization method based on the approximate submodular function and continuous learning of the present disclosure proposes an efficient and intelligent prompt word learning algorithm by effectively integrating the approximate submodular function, the optimized greedy selection algorithm and the improved discrete-continuous optimization learning method. The specific advantages include: (1) significantly reducing the dependence on labeled data, saving labor costs, and being particularly suitable for problems lacking accurate labels in open scenarios; (2) improving the explainability and generalization performance of the prompt word, and enhancing the processing capability of the model for new categories and unseen samples; (3) suitable for real-time target monitoring applications, supporting efficient deployment on edge computing devices, and meeting the high-frequency and low-latency task requirements.

[0031] The present disclosure not only can be widely applied to ecological environment monitoring, marine pollution tracing, offshore activity supervision and other task scenarios, but also has the potential to expand applications in multi-modal recognition fields such as remote sensing image analysis, urban security, and agricultural monitoring. In the future, with the further development of large models, few-sample learning and deployment requirements, the present invention has good industrial transformation value and research and promotion space, and will play an important role in promoting the development of environmental protection intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which form a part of this disclosure, are intended to provide further understanding of the present disclosure and are incorporated herein for illustrative purposes. The schematic embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0033] Figure 1 Flow chart of the approximate submodular function and continuous learning joint optimization strategy of the embodiments of the present disclosure;

[0034] Figure 2 Specific flowchart of the method applied to the offshore pollution image classification method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present disclosure belongs.

[0037] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they refer to the presence of a feature, step, operation, device, component and / or combination thereof.

[0038] Example 1

[0039] In one embodiment of the present disclosure, a prompt word optimization method based on approximate submodular functions and continuous learning is provided. The method is applied to image classification tasks. The prompt word optimization can be better applied to image classification and image-text matching, making the image classification results more accurate. The specific method steps include:

[0040] Step 1: Obtain image data to be classified;

[0041] Step 2: Optimize the prompt word corresponding to the image using an approximate submodular function and a continuous learning joint optimization strategy; input the image data and the optimized prompt word into a multimodal classification model for image-text matching, and output the image classification result;

[0042] Among them, the process of implementing the joint optimization strategy of approximate submodular function and continuous learning for prompt words includes: constructing a set of candidate prompt words, designing a combined objective function based on the properties of approximate submodular function, and solving the combined objective function using a greedy selection algorithm that combines random perturbation, multiple rounds of iteration and task adaptation mechanism to achieve optimal selection of the candidate prompt word set, gradually selecting the prompt word with the largest gain from the candidate prompt word set and adding it to the optimized subset. Each time a new prompt word is selected, a local optimization is performed. When all target prompt words are selected, all prompt words are jointly optimized in the continuous space using an alternating optimization strategy, and iterate continuously until the optimized prompt word is obtained.

[0043] As an embodiment, the present disclosure proposes a prompt word optimization method based on approximate submodular functions and continuous learning. By effectively integrating approximate submodular functions, an optimized greedy selection algorithm, and an improved discrete-continuous optimization learning method, an efficient and intelligent prompt word learning optimization algorithm is proposed. The prompt word optimization process is described in detail below:

[0044] Step 1: Construct a set of candidate prompt words;

[0045] In this paper, to achieve high-quality, interpretable prompt word selection, we first construct a set of candidate prompt words with good semantic coverage from the English vocabulary. The candidate prompt word set is constructed based on the NLTK standard vocabulary and is screened using a variety of linguistic and model adaptability rules to ensure that the candidate words have semantic integrity, model compatibility, and applicable frequency. The specific steps are as follows:

[0046] Step 1.1: Obtain candidate prompt words and conduct preliminary screening;

[0047] This example is based on the English corpus provided by NLTK (Natural Language Toolkit), and the words are traversed one by one, and only the words that meet the following conditions are retained:

[0048] (1) Alphabetic: Eliminate non-natural language words containing symbols, numbers or special characters;

[0049] (2) Semantic validity (WordNet coverage): By calling the WordNet interface, retain legal words that can be found in its vocabulary;

[0050] (3) Frequency limit (Zipf frequency ≥ 3.5): Zipf frequency score defined in the wordfreq library is used to measure the commonality of words, and low-frequency words that are extremely rare or domain-specific are removed to improve generalization ability.

[0051] Step 1.2: Model compatibility screening;

[0052] Further, to ensure the usability of the candidate prompt words in the visual-linguistic model (such as CLIP), further use the BPE tokenizer used by CLIP to encode and screen the candidate words, and only keep those words encoded as a single token. This strategy avoids the interference of incomplete syntax or cross-token prompt words in the embedding space, which is conducive to improving the stability and consistency of subsequent prompt word representation.

[0053] Finally, the remaining prompt words constitute the candidate prompt word set, denoted as:

[0054]

[0055] Where each is a semantic valid, frequency moderate (Zipf frequency ≥ 3.5) and directly processable by CLIP atomic word item through the above preliminary multiple screening. The candidate prompt word set constructed by this disclosure constitutes the input space of the prompt word optimization process in this disclosure, which is used for subsequent approximate submodular function evaluation and greedy selection operations.

[0056] Through the above processing, the candidate prompt word set constructed by this disclosure has good semantic breadth, model compatibility and interpretability, which can provide a solid foundation for subsequent prompt word selection.

[0057] Step 2: Design a combination objective function based on the properties of approximate submodular functions;

[0058] To efficiently select a set of semantically representative, discriminative, and complementary cue words from a large set of candidate cue words, this paper designs a combined objective function that combines task loss and semantic diversity. This function exhibits properties similar to submodular functions in practical optimization, providing a theoretical foundation and practical feasibility for subsequent greedy algorithms.

[0059] Step 2.1: Construct the combined objective function;

[0060] In the CLIP model, prompt words directly affect the image-text matching score, so choosing the right prompt words is crucial to improving classification accuracy. However, there are currently the following problems and requirements:

[0061] (1) Minimizing only the classification loss will cause the selected cue words to be overly concentrated in the semantic space, thereby reducing generalization ability;

[0062] (2) If there is no regularization constraint, the selected prompt words may be highly redundant and unable to cover the diverse image content;

[0063] (3) In open target monitoring tasks, the scenes are varied and the categories are complex, so the prompt words need to have both discriminative and semantic coverage.

[0064] Therefore, when constructing the objective function, the present disclosure takes into account both classification performance and semantic diversity, and constructs the following combined objective function:

[0065]

[0066] in, It is a CLIP-based contrast loss, used to measure the improvement of the selected cue words on the image discrimination ability; Indicates the semantic redundancy within the prompt word subset, using the average cosine similarity measurement; To control the strength of the diversity regularization term, adjust the trade-off between task-driven and semantic divergence; It is a set of selected category labels and candidate prompt words.

[0067] Furthermore, the combined objective function based on the properties of the approximate submodular function is composed of a classification performance term and a semantic diversity regularization term. It includes:

[0068] (1) Classification performance items Calculation method:

[0069] Given a category label and a set of candidate prompt words , construct the text prompt as: prompt="aphotoofa[CLASS]withemphasison: " "," ",…," Input the sentence into the CLIP text encoder and the image data into the image encoder to calculate the image-text matching similarity. , and then use cross entropy loss:

[0070]

[0071] in, For samples i CLIP image features; For the real category Corresponding text features; is the learnable temperature parameter; is the total number of categories. This loss term encourages the subset of prompt words to maximize the category discrimination ability.

[0072] (2) Diversity regularization term Calculation method:

[0073] To prevent the selected prompt words from clustering redundancy in the semantic space and reducing generalization performance, the present disclosure designs the following diversity regularization term:

[0074]

[0075] in, represents the embedding vector of the candidate prompt word, This represents the embedding vector of the selected cue word. A larger similarity value indicates closer semantics. Minimizing this value promotes a more even distribution of selected cue words in the semantic space, thereby improving the model's ability to perceive diverse image content.

[0076] As an embodiment, the word vector of the prompt word is extracted by the tokenizer + embedding layer of CLIP to ensure consistency with the model reasoning process and high expressiveness.

[0077] Step 2.2 Mathematical analysis and application of properties of approximate submodular functions

[0078] Although the combined loss function of the present disclosure is not a submodular function in the strict sense (i.e., it does not necessarily satisfy the diminishing marginal returns defined by all submodular functions), it exhibits approximate submodularity in practical applications, namely:

[0079] (1) With the candidate prompt word set Extension, newly added prompt word pairs The improvement effect shows a marginal decreasing trend;

[0080] (2) That is: there is a , for smaller sets Gain Higher than for larger sets Gain :

[0081]

[0082] This property lays the foundation for the greedy algorithm employed in this paper: greedy solutions to approximate submodular functions are guaranteed to achieve theoretical near-optimality. Furthermore, this objective is highly stable in practical tasks, possesses clear physical interpretability, and is task-driven, significantly improving the classification performance of the CLIP model in complex scenarios.

[0083] Step 3: A greedy selection algorithm that combines random perturbations, multiple iterations, and task adaptation is used to solve the combined objective function.

[0084] In order to efficiently solve the defined approximate submodular objective function, this paper designs a prompt word selection algorithm based on a greedy strategy, combining random perturbation, multiple rounds of iteration and task adaptation mechanism to achieve automatic construction and global optimization of prompt word subsets.

[0085] Step 3.1: Basic process of greedy selection algorithm;

[0086] Suppose the candidate prompt word set is The optimization goal is to not exceed the set length Under the premise of , so that the joint loss function Minimum. The algorithm steps are as follows:

[0087] 1. Initialize the optimized subset of prompt words to an empty set:

[0088]

[0089] 2. In each iteration, traverse all candidate prompt words , calculate the loss reduction after adding it to the optimized subset:

[0090]

[0091] 3. Select the candidate prompt word that brings the greatest loss reduction , add to the current collection:

[0092]

[0093] 4. Repeat steps 2-3 until the maximum number of words is reached or the loss drops below the preset threshold, stop the iteration, and finally get the optimized subset under the current greedy iteration, and finally output the prompt word set This is the optimal subset under the current greedy iteration.

[0094] Since the constructed combined objective function has an approximate submodular property, the greedy strategy adopted in this disclosure can obtain the following theoretical guarantees:

[0095] (1) In the standard submodular function minimization scenario, the greedy algorithm can obtain The approximate optimal solution of

[0096] (2) Although It does not fully satisfy the submodule definition, but its marginal improvement shows a decreasing trend, so the greedy strategy still shows high reliability in practice;

[0097] (3) The algorithm selection results disclosed in this paper are consistently superior to random sampling and continuous optimization in multiple tasks.

[0098] Step 3.2: Greedy selection algorithm enhancement mechanism;

[0099] To prevent the greedy algorithm from falling into local optimality and improve the coverage of the search space, this paper designs the following two enhancement mechanisms:

[0100] (1) Implementation process of multiple rounds of initialization combined with perturbation mechanism:

[0101] This mechanism is mainly used to improve the diversity and robustness of greedy search in large-scale prompt word candidate sets and prevent it from falling into local optimality. The specific implementation is as follows:

[0102] 1) Multiple rounds of parallel initialization:

[0103] Set up several search channels (e.g. 3-5), each starting from a different initial prompt word;

[0104] There are many strategies for selecting the initial cue words: such as random sampling, selecting high-frequency words, selecting words with high TF-IDF scores, or selecting words that are semantically close to the initial category.

[0105] 2) Injection of perturbation mechanism:

[0106] Before each round of greedy selection begins, the current candidate word set is randomly shuffled;

[0107] When selecting the word with the largest marginal benefit, if there are multiple candidate words with similar marginal benefits, the perturbed order will guide the search into different paths, thereby exploring more potential solution spaces;

[0108] Stability and exploration can be balanced by setting random seeds and controlling the perturbation amplitude.

[0109] 3) Result fusion and selection:

[0110] After multiple rounds of search, evaluate the set of prompt words obtained from each search path separately (e.g., calculate the classification accuracy or target loss on the validation set);

[0111] Finally, a set of prompt words with the best performance is selected as the output of the current stage.

[0112] (2) Implementation process of task adaptive weight adjustment mechanism:

[0113] This mechanism addresses the problem of tip word selection being biased towards the head class under imbalanced category distribution (especially long-tail distribution), with the goal of improving the model's ability to recognize the tail class (low-frequency class). The specific implementation is as follows:

[0114] 1) Category statistical modeling:

[0115] Before starting prompt word selection, count each category in the training set The number of samples ;

[0116] Calculate the weight of each class , for example, the inverse weight can be used ,in A small constant to prevent division by zero.

[0117] 2) Marginal gain adjustment:

[0118] In the greedy algorithm, each candidate clue word is calculated Marginal gain When , the semantic relevance of the word to each category is considered (which can be measured by the similarity between CLIP text embedding and category name);

[0119] Weight the marginal gain according to the weight of its associated category to obtain the adjusted marginal benefit:

[0120]

[0121] in, Indicates prompt words and categories semantic relevance.

[0122] 3) Dynamic adjustment mechanism:

[0123] As the training process progresses, the category distribution statistics can be dynamically updated and the weights can be adjusted in real time;

[0124] In the multi-stage cue word selection, the initial stage can focus on global coverage, while the later stage can strengthen the tail class adaptability.

[0125] The computational complexity of the greedy selection algorithm disclosed in this paper is approximately , under the premise of fixed candidate word set and limited category number, has good scalability. Compared with the traditional continuous optimization method based on gradient descent, the disclosure does not depend on random initialization, the result is stable and controllable, has clear semantic interpretability, and the prompt word is a readable natural language word; without additional back propagation stage, it is convenient to deploy on resource limited platform (such as unmanned aerial vehicle device); therefore, the greedy selection algorithm not only has theoretical advantages, but also meets the engineering needs of high performance and low cost prompt word optimization.

[0126] Step 4: local optimization and joint optimization are carried out in stages;

[0127] In order to effectively fuse the discrete prompt word selection and continuous prompt word optimization process, the disclosure designs a stage type alternating optimization mechanism, which improves the expression ability of the model while enhancing the matching and generalization between the prompt word and the image semantics.

[0128] Step 4.1: the whole prompt word learning process is divided into two stages:

[0129] Stage one: step-by-step selection and local optimization

[0130] In this stage, the model gradually selects semantic effective words from the candidate prompt word set. In each round, a prompt word is selected and inserted into the fixed position of the pre-defined template, and the word remains unchanged in the subsequent training and does not participate in gradient update. At the same time, only the continuous representation of context token is optimized to adapt to the semantic information introduced by the newly added semantic word. Through the alternating process of "word selection - freezing - optimization", the model can gradually adapt to the combination of semantic words, reduce the misleading of early selection bias, and improve the alignment degree of the final semantics and visual space.

[0131] Stage two: joint fine-tuning of fixed semantic words

[0132] After the word selection stage is completed, the positions of all selected semantic prompt words are fixed and no longer updated. Then the model enters the joint fine-tuning stage to continue optimizing the parameters of context token to improve the expression ability and classification discrimination of the whole embedding space. In this stage, all context tokens participate in training together, while the semantic prompt words remain frozen.

[0133] Optimization objective function

[0134] During the whole training process, the classification loss consistent with the sub-function evaluation stage is used as the optimization objective. Specifically, the standard contrastive cross entropy loss of CLIP (synchronous step 2.1) is used to measure the matching degree between image features and target category text features .

[0135] The two-stage optimization strategy effectively balances semantic interpretability and model expressiveness. By fixing semantic words, controllability and interpretability are achieved, while continuous learnable tokens are used to improve model adaptability and robustness. This method is particularly suitable for monitoring tasks with complex semantics, scarce annotations, or ambiguous class boundaries, and has significant advantages in stability and generalization ability.

[0136] Step 4.2: Training of prompt word fine-tuning in continuous space;

[0137] To more effectively integrate the discrete selected prompt words into the embedding space of the CLIP model, the disclosure adopts an alternating optimization strategy. In the prompt word screening and training process, the model does not select all prompt words at once and then optimize them uniformly, but introduces and jointly trains the prompt words in stages, gradually optimizing the prompt word representation in each iteration to enhance semantic consistency and model adaptability.

[0138] Specifically, the disclosure sets the size of the final prompt set to 2-3 prompt words on each dataset. During training, a new prompt word is greedily selected to join the current set each time, and the model is trained for 5 to 10 epochs to allow the model to fully adapt to the newly introduced semantic features. This process is repeated until all prompt words are selected. After that, the model uses the remaining training rounds to jointly fine-tune all prompt words to further improve overall performance.

[0139] The alternating strategy has the following advantages:

[0140] 1. Mitigate early selection bias: Gradual training allows the model to fully adjust its internal representation after introducing new prompt words, reducing the negative impact of initial prompt words dominating model training.

[0141] 2. Improve semantic and representation alignment: Each newly introduced prompt word is aligned through model training after being added, allowing it to form a good synergy with existing prompt words and enhancing overall prompt expression ability.

[0142] 3. Improve convergence efficiency and generalization ability: The phased optimization strategy avoids overfitting problems caused by one-time learning, achieving efficient convergence within a limited number of training rounds.

[0143] A unified classification loss is used throughout the training process As the only optimization objective, this loss measures the matching degree of images and their true class text prompts in the embedding space. Through the gradual strengthening of this optimization objective, the prompt words can eventually be highly consistent with the image classes in terms of semantics, greatly improving the performance of multi-modal classification in complex and open scenarios.

[0144] Step 5: Deployment of the application process;

[0145] The approximate submodular function and continuous learning joint optimization strategy proposed by the present disclosure can be integrated into existing multi-modal visual recognition systems to improve image classification performance in environmental monitoring scenarios, and is particularly suitable for environments where multi-modal data is difficult to annotate, class distribution is uneven, and scene changes are drastic in open environmental monitoring.

[0146] Embodiment 2

[0147] In an embodiment of the present disclosure, the approximate submodular function and continuous learning joint optimization strategy proposed by the present disclosure are deployed in the "image semantic recognition engine" in the intelligent monitoring system in a modular manner. The main process includes:

[0148] (1) Input: Receive video frames, high-resolution images from satellite remote sensing and unmanned aerial vehicle photography; combine metadata such as collection time, location, and lighting information.

[0149] (2) Multi-modal feature extraction module in CLIP model: use pre-trained CLIP to extract image features and text features corresponding to prompt words; support floating-point precision and quantization deployment, adapt to edge computing devices.

[0150] (3) Prompt word generation and optimization module: run greedy submodular selection and continuous fine-tuning processes in the offline stage to generate task-specific prompt words; load the prompt word list and concatenate the template for classification reasoning in the online stage.

[0151] (4) Image classification and decision-making module: perform softmax normalization on the image-text similarity results; select the most relevant class and output the pollution type, risk level, or alarm signal.

[0152] (5) Backend analysis and visualization module: integrate model recognition results with GIS systems to display classification results and trends in real time on maps or monitoring platforms.

[0153] As an embodiment, the application scenario of the present disclosure is offshore pollution identification, and the operation process of the above system modules is as follows:

[0154] Step 1: Image acquisition, the monitoring terminal device periodically collects image data and uploads it to the central server;

[0155] Step 2: Semantic recognition, the system loads the optimized prompt words and outputs the pollution type (such as red tide, oil pollution, plastic floating objects, etc.) based on image-text comparison;

[0156] Step 3: Event judgment and response, if the identification result is a high-risk category, the system triggers the early warning mechanism and notifies the operation and maintenance department;

[0157] Step 4: Result recording and optimization feedback. The model recognition results are automatically archived and fine-tuned with feedback combined with manual corrections, which can be used for subsequent prompt word optimization and migration.

[0158] The application system design disclosed in this disclosure has the following practical advantages:

[0159] (1) High deployment flexibility: compatible with GPU cloud deployment and lightweight edge deployment (such as Jetson, FPGA);

[0160] (2) Strong data adaptability: It can process multiple sources of input, including high-resolution remote sensing images, nearshore video frames, and shipborne images;

[0161] (3) Good human-computer interaction: The optimized prompt words are language interpretable, making it easier for experts to quickly understand the model behavior;

[0162] (4) Rapid portability: The prompt word optimization process can be migrated and reused in other scenarios (such as inland water quality monitoring and meteorological disaster warning).

[0163] In summary, the present disclosure not only theoretically establishes a prompt word learning framework that combines approximate submodule optimization and continuous fine-tuning, but also realizes the complete implementation from method to system in the typical multimodal weak supervision scenario of environmental monitoring, with good engineering feasibility and social application value.

[0164] Example 3

[0165] In one embodiment of the present disclosure, a prompt word optimization system based on approximate submodular functions and continuous learning is provided, comprising:

[0166] Obtain image data to be classified;

[0167] The cue words corresponding to the image are optimized using an approximate submodular function and a continuous learning joint optimization strategy. The image data and the optimized cue words are input into a multimodal classification model for image-text matching, and the image classification result is output.

[0168] Among them, the process of implementing the joint optimization strategy of approximate submodular function and continuous learning for prompt words includes: constructing a set of candidate prompt words, designing a combined objective function based on the properties of approximate submodular function, and solving the combined objective function using a greedy selection algorithm that combines random perturbation, multiple rounds of iteration and task adaptation mechanism to achieve optimal selection of the candidate prompt word set, gradually selecting the prompt word with the largest gain from the candidate prompt word set and adding it to the optimized subset. Each time a new prompt word is selected, a local optimization is performed. When all target prompt words are selected, all prompt words are jointly optimized in the continuous space using an alternating optimization strategy, and iterate continuously until the optimized prompt word is obtained.

[0169] Embodiment 4

[0170] In an embodiment of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the prompt word optimization method based on the approximate submodular function and continuous learning.

[0171] Embodiment 5

[0172] In an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, for storing computer instructions which, when executed by a processor, implement the prompt word optimization method based on the approximate submodular function and continuous learning.

[0173] Embodiment 6

[0174] In an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the prompt word optimization method based on the approximate submodular function and continuous learning.

[0175] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flowcharts and / or block diagrams.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed by the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 a step that performs the functions specified in the flowcharts and / or block diagrams.

[0177] The specific embodiments of the present disclosure are described above with reference to the accompanying drawings, but are not intended to limit the protection scope of the present disclosure, and those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present disclosure without creative labor are still within the protection scope of the present disclosure.

Claims

1. A prompt word optimization method based on approximate submodular functions and continuous learning, characterized in that: include: Obtain image data to be classified; Utilize the approximate submodular function and continuous learning joint optimization strategy to optimize the prompt words corresponding to the image; The image data and the optimized prompt words are input into the multimodal classification model for image-text matching, and the image classification result is output; The process of implementing a joint optimization strategy of approximate submodular functions and continuous learning for prompt words includes: constructing a set of candidate prompt words, designing a combined objective function based on the properties of the approximate submodular function, solving the combined objective function using a greedy selection algorithm that combines random perturbations, multiple rounds of iteration, and a task adaptation mechanism to achieve optimal selection of the candidate prompt word set, gradually selecting the prompt word with the largest gain from the candidate prompt word set and adding it to the optimized subset. Each time a new prompt word is selected, a local optimization is performed. After all target prompt words have been selected, all prompt words are jointly optimized using an alternating optimization strategy in a continuous space, and iterate continuously until the optimized prompt word is obtained. Specifically, constructing a candidate word set includes: extracting candidate prompt words corresponding to the image, encoding and filtering the candidate prompt words using the BPE word segmenter, retaining the prompt words encoded as a single token, and the retained prompt words forming a candidate prompt word set, each prompt word is a semantically valid, moderately frequent atomic term that can be directly processed by CLIP after multiple screenings; Designing a combination objective function based on the properties of approximate submodular functions, including: combining task loss and semantic diversity to design a combination objective function based on the properties of approximate submodular functions. The combination objective function based on the properties of approximate submodular functions consists of a classification performance term and a semantic diversity regularization term, and the combination loss exhibits the properties of approximate submodular functions, including a marginally decreasing improvement in the combination loss with the expansion of the candidate cue word set, and a higher gain for a certain cue word for a smaller set than for a larger set. The combination objective function is solved by a greedy selection algorithm that combines random perturbations, multiple rounds of iterations, and a task-adaptive mechanism. The goal of solving the combination objective function by using a greedy selection algorithm that combines random perturbations, multiple rounds of iterations, and a task-adaptive mechanism is to select an optimized subset from the set of candidate prompt words without exceeding a set length, so that the combined loss of the combination objective function is minimized. The greedy selection algorithm process includes: initializing the optimized subset of prompt words to an empty set, and in each round of iteration, traversing all candidate prompt words and calculating the loss reduction that will be brought about by adding them to the optimized subset. The candidate prompt word that brings the largest loss reduction is selected and added to the current optimized subset. Iterations are repeated until the maximum number of words is reached or the loss reduction is less than a preset threshold. The iteration is stopped, and the optimized subset under the current greedy iteration is obtained. To prevent the greedy algorithm from falling into local optimality and improve the coverage of the search space, two enhancement mechanisms are designed: adding random perturbations, starting the greedy process in parallel from different starting points, and randomly shuffling the candidate prompt words before each iteration to introduce perturbations to improve the diversity of the search path. In the marginal gain calculation, the prompt words are weighted by integrating category distribution information and adaptive weight adjustment is performed. After all target cues are selected, an alternating optimization strategy is used to jointly optimize all cues in the continuous space, and the optimization is repeated until the optimized cues are obtained. The method includes: in the initial stage, a greedy algorithm is used to gradually select the cues with the largest gain from the candidate cues set; each time a new cues is selected, the existing cues are frozen, and only the new cues are locally fine-tuned; after all target cues are selected, the joint optimization stage is started, all cues parameters are released, and joint fine-tuning is performed in the continuous embedding space; the joint fine-tuning process uses the same classification loss as the approximate submodular function evaluation process, namely the comparative cross-entropy loss, which is used to measure the similarity between image features and the true category cues. The comparative cross-entropy loss is used as the only optimization target, and by gradually strengthening the optimization target, the cues are finally semantically highly consistent with the image category.

2. A prompt word optimization system based on approximate submodular functions and continuous learning, specifically implementing the prompt word optimization method based on approximate submodular functions and continuous learning as described in claim 1, characterized by comprising: A data acquisition module, used to acquire image data to be classified; The image-text matching classification module is used to optimize the prompt words corresponding to the image using an approximate submodular function and a continuous learning joint optimization strategy; the image data and the optimized prompt words are input into the multimodal classification model for image-text matching, and the image classification result is output; Among them, the process of implementing the joint optimization strategy of approximate submodular function and continuous learning for prompt words includes: constructing a set of candidate prompt words, designing a combined objective function based on the properties of approximate submodular function, and solving the combined objective function using a greedy selection algorithm that combines random perturbation, multiple rounds of iteration and task adaptation mechanism to achieve optimal selection of the candidate prompt word set, gradually selecting the prompt word with the largest gain from the candidate prompt word set and adding it to the optimized subset. Each time a new prompt word is selected, a local optimization is performed. When all target prompt words are selected, all prompt words are jointly optimized in the continuous space using an alternating optimization strategy, and iterate continuously until the optimized prompt word is obtained.

3. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the prompt word optimization method based on approximate submodular function and continuous learning according to any one of claims 1 is implemented.

4. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the prompt word optimization method based on approximate submodular function and continuous learning according to any one of claims 1 is implemented.

5. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the prompt word optimization method based on approximate submodule function and continuous learning as described in any one of claim 1.

Images