Vision-language model cue word evolution generation method based on genetic algorithm

Abstract

The invention relates to a visual-language model cue word evolution generation method based on a genetic algorithm, which comprises the following steps of: randomly generating different cue words under a target task to construct an initial cue word set, preprocessing the randomly generated cue words, and reserving N groups of cue words as an initial population; and designing a multi-dimensional fitness evaluation function to evaluate the performance quality of each group of cue words in a vision-language task, and selecting high-quality cue word individuals based on an elitism strategy. Performing crossover and mutation operation on the selected high-quality cue word individuals by referring to a genetic algorithm to generate a new-generation cue word population; and finally, carrying out iterative optimization for multiple times until a preset termination condition is met, and outputting an optimal cue word set. By the adoption of the visual-language model cue word evolution generation method based on the genetic algorithm, high-quality and diversified cue words can be automatically generated, the performance of a visual-language model is remarkably improved, and meanwhile the visual-language model cue word evolution generation method has good interpretability and adaptability.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of artificial intelligence and computational linguistics, and in particular to a method for evolving prompt words for a visual-language model based on a genetic algorithm. The method is particularly suitable for optimizing prompt engineering for visual-language models (such as CLIP, BLIP, and Flamingo), and can be widely used in tasks such as image generation, visual question answering, image description generation, and cross-modal retrieval. Background Art

[0002] Vision-language models have been a significant breakthrough in the field of artificial intelligence in recent years, capable of understanding and generating natural language descriptions related to visual content. These models are typically based on large-scale multimodal pre-training and possess powerful cross-modal understanding and generation capabilities. However, the performance of vision-language models depends largely on the quality of the input prompts, and different prompts can lead to significantly different model performance. Currently, prompt design relies primarily on manual experience and trial-and-error, resulting in low manual design efficiency, a lack of systematic optimization methods, poor adaptability, and difficulty in quantifying and evaluating prompt quality and diversity. Prior art research on prompt optimization has primarily focused on manual template design, gradient-based optimization, reinforcement learning, and retrieval-based methods. Genetic algorithms, as a swarm intelligence optimization method, offer unique advantages in solving complex optimization problems. By simulating natural selection and genetic mechanisms, they can efficiently search for high-quality solutions in high-dimensional spaces. However, applying genetic algorithms to prompt generation in vision-language models faces the following challenges: designing an effective fitness function to evaluate prompt quality, balancing multiple objectives in prompt optimization, designing appropriate genetic operations to maintain semantic plausibility, and improving the computational efficiency of the optimization process.

[0003] To address the above issues, the present invention proposes a method for evolving prompt words for a visual-language model based on a genetic algorithm. This method automatically generates high-quality and diverse prompt words through a multi-dimensional optimization framework, significantly improving the performance of the visual-language model. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a highly efficient and adaptable method for the evolutionary generation of visual-language model prompt words based on genetic algorithms. This method can automatically generate high-quality and diverse prompt words, significantly improving the performance of the visual-language model.

[0005] In order to achieve the above-mentioned purpose, the vision-language model prompt word evolution generation method based on genetic algorithm of the present invention is as follows:

[0006] The main feature of the genetic algorithm-based visual-language model prompt word evolution generation method is that the method comprises the following steps:

[0007] (1) Construct an initial prompt word set and randomly generate different prompt words for the target task;

[0008] (2) Preprocess the randomly generated prompt words and retain N groups of prompt words as the initial population;

[0009] (3) Design a multi-dimensional fitness evaluation function to evaluate the performance quality of each group of prompt words in the visual-language task;

[0010] (4) Select high-quality prompt word individuals based on the elite retention strategy;

[0011] (5) Perform crossover and mutation operations on the selected high-quality prompt word individuals to generate a new generation of prompt word populations;

[0012] (6) Repeat steps (3) to (5) for iterative optimization until the preset termination condition is met;

[0013] (7) Output the final optimal prompt word set.

[0014] Preferably, the step (1) of constructing the initial prompt word set includes the following operations:

[0015] (1.1) Clarify the target tasks of the vision-language model, including human target recognition, vehicle target recognition, and specific event recognition, and construct multi-attribute prompt words based on the target task type;

[0016] (1.2) Determine the population size, maximum number of iterations, crossover probability, mutation probability, and length range of each prompt word [L min ,L max ], where L min is the minimum length of the prompt word, L max is the maximum length of the prompt word;

[0017] (1.3) Randomly select prompt words from the predefined target task vocabulary to generate N groups of length [L min ,L max ] to complete the construction of the initial prompt word set.

[0018] Preferably, the specific method of pretreatment in step (2) is:

[0019] (2.1) Perform a preliminary grammatical check on the generated prompt words to ensure basic readability. This includes: first, using natural language processing tools (such as LanguageTool) to perform part-of-speech tagging and dependency syntax analysis on the prompt words to detect obvious grammatical errors; then, using statistical methods (such as n-gram language models) to evaluate the rationality of the vocabulary collocation and execution results; finally, setting a threshold to filter out low-confidence results;

[0020] (2.2) After the initial grammatical verification, the prompt words are further subjected to preliminary semantic verification to ensure consistency with the target task. This includes: first, mapping the prompt words and the description text of the target task into a vector space, and quantifying the semantic matching degree by cosine similarity; then, determining the similarity threshold (e.g., cosine similarity ≥ 0.6) based on the preset initial population size N, and combining it with manual rules (e.g., task types must match), screening out the final N groups of prompt words as the initial population.

[0021] Preferably, the multi-dimensional fitness evaluation function in step (3) includes:

[0022] (3.1) Task performance metrics: This refers to the task completion accuracy of the prompt word on the vision-language model, which is quantitatively evaluated by comparing the matching degree between the generated results and the expected target.

[0023] (3.2) Diversity index: the semantic difference between the prompt word and other prompt words in the population, calculated based on the distribution distance of semantic embedding.

[0024] Preferably, the specific calculation method of the task performance index is:

[0025] The prompt word is input into the visual-language model, and the accuracy, recall or F1 value output by the evaluation model is used as the performance score, wherein the accuracy is the ratio of the number of samples correctly predicted by the statistical model to the total number of samples, the recall is the ratio of the correctly predicted positive examples to all true positive examples, and the F1 value is the harmonic mean of the comprehensive accuracy and recall.

[0026] Preferably, the diversity index is calculated as follows:

[0027] Use a pre-trained language model (such as BERT) to extract the semantic embedding vector of the prompt word; calculate the average cosine distance with the vectors of other prompt words in the population, take the arithmetic average of all similarity values, and then subtract the average from 1, so that exactly the same prompt words have a score of 0 and completely unrelated prompt words are close to 1; the calculation result is used as the diversity score to measure semantic similarity.

[0028] Preferably, the step (4) comprises the following steps:

[0029] (4.1) Sort the prompt words in the population according to the fitness score. Specifically, first calculate the comprehensive fitness score of each prompt word in the population according to the multi-dimensional fitness evaluation function, and then use the quick sort or heap sort algorithm to sort all individuals in descending order according to the score;

[0030] (4.2) The top K best prompt word individuals are retained as elites and directly enter the next generation. Specifically, the top K prompt words are directly selected from the sorted population, and their original texts and corresponding parameters are retained without crossover or mutation operations.

[0031] (4.3) The remaining individuals are selected for subsequent crossover and mutation operations according to the fitness ratio. Specifically, the fitness of all remaining individuals is calculated first, and the fitness of each individual is divided by the total to obtain its probability of being selected. Then, individuals are randomly selected as parents according to the probability through random number sampling method to ensure that individuals with high fitness have a greater chance of being selected. Finally, the selected individuals are used for subsequent crossover and mutation operations.

[0032] Preferably, the crossover operation in step (5) is specifically as follows:

[0033] (5.1) Randomly selecting two parent cue words and determining the intersection position, including: first randomly selecting two cue words from the parent candidate pool selected by fitness, and then randomly generating a crossover position;

[0034] (5.2) Swapping the parts of the two prompt words before and after the intersection to generate two new offspring prompt words, including: dividing the two parent prompt words into two parts at the intersection, and swapping the latter parts to form a new individual;

[0035] (5.3) Adjust the length of the offspring prompt words and perform semantic verification, including: checking whether the offspring length exceeds the preset range. If it is too long, it will be truncated; if it is too short, it will be supplemented with a random fragment from the parent. Then, a pre-trained semantic verification model (such as the NLI model) is called to perform reasoning analysis on the generated content, calculate its logical consistency and factual rationality score, and filter out semantically coherent candidate content by setting a threshold. Finally, the optimization result that meets the semantic constraints is output.

[0036] Preferably, the mutation operation in step (5) specifically includes at least one of the following:

[0037] (5.4) Word replacement: Randomly select one or more words from the prompt words, use a context-aware embedding model (such as BERT) to calculate the similarity of word vectors, and find semantically similar alternative words to replace them;

[0038] (5.5) Word insertion: insert context-related words at random positions of the prompt word. The context-related words are selected and constructed through a topic model (such as LDA);

[0039] (5.6) Word Deletion: First, perform part-of-speech tagging (e.g., marking modifiers such as adjectives and adverbs) and dependency syntax analysis (identifying core components such as subject, predicate, and object) on the prompt words. Count the word frequencies to filter out redundant words with low semantic contribution (e.g., repeated expressions or unnecessary modifiers), and randomly delete these redundant words.

[0040] (5.7) Word order adjustment: Determine the semantic units that can be adjusted through grammatical tree analysis and randomly swap their positions.

[0041] Preferably, the termination condition in step (6) includes at least one of the following:

[0042] (6.1) Reaching the preset maximum number of iterations: When the current number of iterations is equal to the preset maximum, the evolution process is terminated immediately, and the prompt word with the highest fitness in the current population is output as the final result;

[0043] (6.2) The fitness improvement for several consecutive generations is less than the threshold: Dynamically monitor the fitness change of the best individual in each generation. If the fitness improvement for N consecutive generations is less than the threshold, the optimization process is considered to be over.

[0044] (6.3) Obtaining the ideal hint word that satisfies all constraints: Check whether there are individuals that simultaneously satisfy the hard constraints of performance, diversity, and interpretability during each generation evaluation.

[0045] Preferably, the step (7) is specifically as follows:

[0046] (7.1) Output the top M prompt words with the highest fitness in the final population;

[0047] (7.2) Through manual evaluation and in combination with specific application requirements, the most suitable prompt word or prompt word combination is selected from the optimal set.

[0048] This paper adopts the genetic algorithm-based visual-language model prompt word evolutionary generation method. It automatically optimizes the prompt word quality through a multi-dimensional fitness evaluation function, improving the performance of the visual-language model in tasks such as image generation and visual question answering. Through an elite retention strategy and diversity maintenance mechanism, it solves the problem of low efficiency and difficulty in balancing multiple objectives in manually designed prompt words. Through semantically reasonable cross-mutation operations, the generated prompt words have both natural language interpretability and task adaptability. This method innovatively introduces genetic algorithms into the field of prompt engineering, playing a key role in automatically generating high-quality and diverse prompt words, significantly reducing the cost of manual tuning and improving the model's generalization ability. This method not only provides an efficient prompt engineering automation solution for visual-language models such as CLIP and BLIP, but its modular design can also be extended to application scenarios such as image generation and cross-modal retrieval. It has important practical value in fields such as visual-language model evaluation, multimodal model robustness enhancement, and intelligent system interaction optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the genetic algorithm-based visual-language model prompt word evolution generation method of the present invention. DETAILED DESCRIPTION

[0050] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0051] The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm of the present invention comprises the following steps:

[0052] (1) Construct an initial prompt word set and randomly generate different prompt words for the target task;

[0053] (2) Preprocess the randomly generated prompt words and retain N groups of prompt words as the initial population;

[0054] (3) Design a multi-dimensional fitness evaluation function to evaluate the performance quality of each group of prompt words in the visual-language task;

[0055] (4) Select high-quality prompt word individuals based on the elite retention strategy;

[0056] (5) Perform crossover and mutation operations on the selected high-quality prompt word individuals to generate a new generation of prompt word populations;

[0057] (6) Repeat steps (3) to (5) for iterative optimization until the preset termination condition is met;

[0058] (7) Output the final optimal prompt word set.

[0059] As a preferred embodiment of the present invention, the construction of the initial prompt word set in step (1) includes the following operations:

[0060] (1.1) Clarify the target tasks of the vision-language model, such as person target recognition, vehicle target recognition, specific event recognition, etc.

[0061] (1.2) Determine the length range of the prompt word [L min ,L max ];

[0062] (1.3) Randomly select words from the predefined target task vocabulary to generate N groups of length [L min ,L max ] prompt words within the range.

[0063] As a preferred embodiment of the present invention, the specific method of pretreatment in step (2) is:

[0064] (2.1) Perform a preliminary grammatical check on the generated prompt words to ensure basic readability. This includes: first, performing a quick match based on a grammatical rule library (e.g., punctuation specifications, part-of-speech collocations) to detect obvious errors (e.g., missing periods, subject-verb inconsistency); then, using language tools or grammar checking models for fine-grained verification, calculating the probability of grammatical errors, and filtering out low-quality results;

[0065] (2.2) Perform a preliminary semantic verification on the generated prompt words to ensure consistency with the target task. This includes: first, mapping the prompt word and the target task description text into a vector space and quantifying the semantic match using cosine similarity; simultaneously, training a binary classification model (with the input being the prompt word + task type) and outputting a consistency score;

[0066] (2.3) The adjacency matrix is ​​sparsified to preserve dependencies, including: first calculating the weight distribution of all edges in the adjacency matrix, retaining the top-K (e.g., K = 5) strong connections or edges exceeding a dynamic threshold; then checking dependency integrity through transitive closure to ensure that the critical path is not truncated.

[0067] As a preferred embodiment of the present invention, the multi-dimensional fitness evaluation function in step (3) includes the following evaluation dimensions:

[0068] (3.1) Task performance metrics: This refers to the task completion accuracy of the prompt word on the vision-language model, which is quantitatively evaluated by comparing the matching degree between the generated results and the expected target.

[0069] (3.2) Diversity index: the semantic difference between the prompt word and other prompt words in the population, calculated based on the distribution distance of semantic embedding.

[0070] (3.3) Interpretability index: language fluency and logical coherence score of the prompt words;

[0071] (3.4) Robustness index: the performance stability of the prompt word under different parameter settings.

[0072] As a preferred embodiment of the present invention, the specific calculation method of the task performance index is:

[0073] The prompt word is input into the visual-language model, and the accuracy, recall or F1 value output by the evaluation model is used as the performance score, wherein the accuracy is the ratio of the number of samples correctly predicted by the statistical model to the total number of samples, the recall is the ratio of the correctly predicted positive examples to all true positive examples, and the F1 value is the harmonic mean of the comprehensive accuracy and recall.

[0074] As a preferred embodiment of the present invention, the specific calculation method of the diversity index is:

[0075] Use a pre-trained language model (such as BERT) to extract the semantic embedding vector of the prompt word; calculate the average cosine distance with the vectors of other prompt words in the population, take the arithmetic average of all similarity values, and then subtract the average from 1, so that exactly the same prompt words have a score of 0 and completely unrelated prompt words are close to 1; the calculation result is used as the diversity score to measure semantic similarity.

[0076] As a preferred embodiment of the present invention, the step (4) is specifically as follows:

[0077] (4.1) Sort the prompt words in the population according to the fitness score. Specifically, first calculate the comprehensive fitness score of each prompt word in the population according to the multi-dimensional fitness evaluation function, and then use the quick sort or heap sort algorithm to sort all individuals in descending order according to the score;

[0078] (4.2) The top K best prompt word individuals are retained as elites and directly enter the next generation. Specifically, the top K prompt words are directly selected from the sorted population, and their original texts and corresponding parameters are retained without crossover or mutation operations.

[0079] (4.3) The remaining individuals are selected for subsequent crossover and mutation operations according to the fitness ratio. Specifically, the fitness of all remaining individuals is calculated first, and the fitness of each individual is divided by the total to obtain its probability of being selected. Then, individuals are randomly selected as parents according to the probability through random number sampling method to ensure that individuals with high fitness have a greater chance of being selected. Finally, the selected individuals are used for subsequent crossover and mutation operations.

[0080] As a preferred embodiment of the present invention, the crossover operation in step (5) is specifically as follows:

[0081] (5.1) Randomly selecting two parent cue words and determining the intersection position, including: first randomly selecting two cue words from the parent candidate pool selected by fitness, and then randomly generating a crossover position;

[0082] (5.2) Swapping the parts of the two prompt words before and after the intersection to generate two new offspring prompt words, including: dividing the two parent prompt words into two parts at the intersection, and swapping the latter parts to form a new individual;

[0083] (5.3) Adjust the length of the offspring prompt words and perform semantic verification, including: checking whether the offspring length exceeds the preset range. If it is too long, it will be truncated; if it is too short, it will be supplemented with a random fragment from the parent. Then, a pre-trained semantic verification model (such as the NLI model) is called to perform reasoning analysis on the generated content, calculate its logical consistency and factual rationality score, and filter out semantically coherent candidate content by setting a threshold. Finally, the optimization result that meets the semantic constraints is output.

[0084] As a preferred embodiment of the present invention, the mutation operation in step (5) specifically includes at least one of the following:

[0085] (5.4) Word replacement: Randomly select one or more words from the prompt words, use a context-aware embedding model (such as BERT) to calculate the similarity of word vectors, and find semantically similar alternative words to replace them;

[0086] (5.5) Word insertion: insert context-related words at random positions of the prompt word. The context-related words are selected and constructed through a topic model (such as LDA);

[0087] (5.6) Word Deletion: First, perform part-of-speech tagging (e.g., marking modifiers such as adjectives and adverbs) and dependency syntax analysis (identifying core components such as subject, predicate, and object) on the prompt words. Count the word frequencies to filter out redundant words with low semantic contribution (e.g., repeated expressions or unnecessary modifiers), and randomly delete these redundant words.

[0088] (5.7) Word order adjustment: Determine the semantic units that can be adjusted through grammatical tree analysis and randomly swap their positions.

[0089] As a preferred embodiment of the present invention, the termination condition in step (6) includes at least one of the following:

[0090] (6.1) Reaching the preset maximum number of iterations: When the current number of iterations is equal to the preset maximum, the evolution process is terminated immediately, and the prompt word with the highest fitness in the current population is output as the final result;

[0091] (6.2) The fitness improvement for several consecutive generations is less than the threshold: Dynamically monitor the fitness change of the best individual in each generation. If the fitness improvement for N consecutive generations is less than the threshold, the optimization process is considered to be over.

[0092] (6.3) Obtaining the ideal hint word that satisfies all constraints: Check whether there are individuals that simultaneously satisfy the hard constraints of performance, diversity, and interpretability during each generation evaluation.

[0093] As a preferred embodiment of the present invention, the step (7) is specifically as follows:

[0094] (7.1) Output the top M prompt words with the highest fitness in the final population;

[0095] (7.2) Through manual evaluation and in combination with specific application requirements, the most suitable prompt word or prompt word combination is selected from the optimal set.

[0096] In specific embodiments of the present invention, by automatically optimizing prompt word quality through a multi-dimensional fitness evaluation function, the performance of vision-language models in tasks such as image generation and visual question answering is improved. An elite retention strategy and diversity preservation mechanism address the inefficiency and multi-objective balancing issues of manually designed prompt words. Through semantically sound crossover and mutation operations, the generated prompt words achieve both natural language interpretability and task adaptability. This invention introduces genetic algorithms into the field of prompt engineering, playing a key role in automatically generating high-quality and diverse prompt words. This technology currently lacks similar technology in this field.

[0097] The flowchart of the genetic algorithm-based visual-language model prompt word evolution generation method of the present invention is as follows: Figure 1 As shown, in order to more clearly describe the technical content of the present invention, the model includes CLIP, BLIP, Flamingo, etc. When the visual language model used is CLIP, a specific embodiment is further described below:

[0098] (1) The target task is determined to be person recognition. 200 adjectives and nouns related to person image attributes are constructed, including the following categories:

[0099] Gender: male, female;

[0100] Age: children, youth, middle-aged, elderly;

[0101] Attributes: clothing color, hair length, whether wearing glasses, whether wearing a hat, etc.;

[0102] Behavior: running, standing, lying, riding a bicycle, etc.;

[0103] Shooting angles: front, side, top view, top view, close-up.

[0104] Clearly define the population size N = 50, the maximum number of iterations G = 20, and the crossover probability P c =0.8, mutation probability P m =0.1, prompt word length range [L min ,L max ]=[3,6] words.

[0105] (2) Randomly select 3 to 6 words from the vocabulary to form prompt words, for example: "a photo of [behavior][gender][age]". Use a pre-trained language model (such as GPT-2) to calculate the perplexity of the generated prompt words and filter out grammatically incoherent combinations. Calculate the semantic similarity of the prompt words in the initial population using BERT embedding, ensuring that the cosine distance is greater than 0.5 to avoid repeated semantics. Finally, retain 100 groups of prompt words as the initial population.

[0106] (3) Combine the prompt word P with the ImageNet category label (such as "a photo of a standing male child"), input it into the CLIP model, and calculate the classification accuracy on the ImageNet validation set, F1 = Accuracy(P); use BERT to extract the semantic embedding vector E(P) of the prompt word, and calculate the average cosine distance F2 between the current prompt word and other prompt words in the population. The larger the value, the better the diversity; calculate the perplexity S through GPT-2 Perplexity , the lower the value, the better, the artificial design rule score S logic (e.g. reasonableness of the collocation of adjectives and nouns), with a full score of 10 points, confirm the comprehensive score: F3 = 0.7 × (1 / S Perplexity )+0.3×S logic ; Test the prompt word performance under different parameter configurations of CLIP (such as temperature coefficient τ = 0.01 / 0.05 / 0.1) and calculate the accuracy variance: F4 = 1 / Variance(Accuracy);

[0107] The final calculated fitness value is: F = 0.5×F1+0.2×F2+0.2×F3+0.1×F4.

[0108] (4) The top 10% of individuals (10) in fitness are retained in each generation and directly enter the next generation. The remaining individuals select their parents according to their fitness ratio to ensure that high-quality prompt words have a higher probability of reproduction.

[0109] (5) According to the crossover probability P c = 0.8, perform crossover operation, randomly select two parent generation prompt words and crossover point (such as after the second word), and generate offspring; according to the mutation probability P m= 0.1, perform mutation operations, including randomly replacing a word, inserting new words at random positions, and swapping the order of words.

[0110] (6) Repeat steps (3) to (5) for iterative optimization. The iteration is terminated when the maximum number of iterations G = 20 is reached or the fitness of the best individual for five consecutive generations is improved by < 0.5%.

[0111] (7) After the iteration is completed, the optimal prompt word set is output, and the prompt words can be further evaluated and tested.

[0112] The specific implementation scheme of this embodiment can be found in the relevant descriptions in the above embodiments and will not be repeated here.

[0113] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0114] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is at least two.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0116] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0118] Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0119] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0120] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0121] This paper adopts the genetic algorithm-based visual-language model prompt word evolutionary generation method. It automatically optimizes the prompt word quality through a multi-dimensional fitness evaluation function, improving the performance of the visual-language model in tasks such as image generation and visual question answering. Through an elite retention strategy and diversity maintenance mechanism, it solves the problem of low efficiency and difficulty in balancing multiple objectives in manually designed prompt words. Through semantically reasonable cross-mutation operations, the generated prompt words have both natural language interpretability and task adaptability. This method innovatively introduces genetic algorithms into the field of prompt engineering, playing a key role in automatically generating high-quality and diverse prompt words, significantly reducing the cost of manual tuning and improving the model's generalization ability. This method not only provides an efficient prompt engineering automation solution for visual-language models such as CLIP and BLIP, but its modular design can also be extended to application scenarios such as image generation and cross-modal retrieval. It has important practical value in fields such as visual-language model evaluation, multimodal model robustness enhancement, and intelligent system interaction optimization.

[0122] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method for evolving prompt words in a visual-language model based on a genetic algorithm, characterized in that: The method comprises the following steps: (1) Construct an initial prompt word set and randomly generate different prompt words for the target task; (2) Preprocess the randomly generated prompt words and retain N groups of prompt words as the initial population; (3) Design a multi-dimensional fitness evaluation function to evaluate the performance quality of each group of prompt words in the visual-language task; (4) Select high-quality prompt word individuals based on the elite retention strategy; (5) Perform crossover and mutation operations on the selected high-quality prompt word individuals to generate a new generation of prompt word populations; (6) Repeat steps (3) to (5) for iterative optimization until the preset termination condition is met; (7) Output the final optimal prompt word set.

2. The method for evolving and generating visual-language model prompt words based on genetic algorithm according to claim 1, characterized in that: The construction of the initial prompt word set in step (1) includes the following operations: (1.1) Clarify the target tasks of the vision-language model, including human target recognition, vehicle target recognition, and specific event recognition, and construct multi-attribute prompt words based on the target task type; (1.2) Determine the population size, maximum number of iterations, crossover probability, mutation probability, and length range of each prompt word [L min ,L max ], where L min is the minimum length of the prompt word, L max is the maximum length of the prompt word; (1.3) Randomly select prompt words from the predefined target task vocabulary to generate N groups of length [L min ,L max ] to complete the construction of the initial prompt word set.

3. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 1, characterized in that: The specific method of pretreatment in step (2) is: (2.1) Perform a preliminary grammatical check on the generated prompt words to ensure basic readability. This includes: first, using natural language processing tools to perform part-of-speech tagging and dependency syntax analysis on the prompt words to detect obvious grammatical errors; then, using statistical methods to evaluate the rationality of the vocabulary collocation and execution results; and finally, setting a threshold to filter out low-confidence results. (2.2) After the initial grammatical verification, the prompt words are further subjected to preliminary semantic verification to ensure consistency with the target task. This includes: first, mapping the prompt words and the description text of the target task into a vector space, and quantifying the semantic matching degree through cosine similarity; then, determining the similarity threshold based on the preset initial population size N, and combining manual rules to screen out the final N groups of prompt words as the initial population.

4. The method for evolving and generating visual-language model prompt words based on genetic algorithm according to claim 1, characterized in that: The multi-dimensional fitness evaluation function in step (3) includes: (3.1) Task performance metrics: This refers to the task completion accuracy of the prompt word on the vision-language model, which is quantitatively evaluated by comparing the matching degree between the generated results and the expected target. (3.2) Diversity index: the semantic difference between the prompt word and other prompt words in the population, calculated based on the distribution distance of semantic embedding.

5. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 4, characterized in that: The specific calculation method of the task performance index is: The prompt word is input into the visual-language model, and the accuracy, recall or F1 value output by the evaluation model is used as the performance score, wherein the accuracy is the ratio of the number of samples correctly predicted by the statistical model to the total number of samples, the recall is the ratio of the correctly predicted positive examples to all true positive examples, and the F1 value is the harmonic mean of the comprehensive accuracy and recall.

6. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 4, characterized in that: The specific calculation method of the diversity index is: Use a pre-trained language model to extract the semantic embedding vector of the prompt word; calculate the average cosine distance with the vectors of other prompt words in the population, take the arithmetic average of all similarity values, and then subtract the average from 1, so that exactly the same prompt words have a score of 0 and completely unrelated prompt words are close to 1; the calculation result is used as the diversity score to measure semantic similarity.

7. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 1, wherein: The step (4) comprises the following steps: (4.1) Sort the prompt words in the population according to the fitness score. Specifically, first calculate the comprehensive fitness score of each prompt word in the population according to the multi-dimensional fitness evaluation function, and then use the quick sort or heap sort algorithm to sort all individuals in descending order according to the score; (4.2) The top K best prompt word individuals are retained as elites and directly enter the next generation. Specifically, the top K prompt words are directly selected from the sorted population, and their original texts and corresponding parameters are retained without crossover or mutation operations. (4.3) The remaining individuals are selected for subsequent crossover and mutation operations according to the fitness ratio. Specifically, the fitness of all remaining individuals is calculated first, and the fitness of each individual is divided by the total to obtain its probability of being selected. Then, individuals are randomly selected as parents according to the probability through random number sampling method to ensure that individuals with high fitness have a greater chance of being selected. Finally, the selected individuals are used for subsequent crossover and mutation operations.

8. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 7, characterized in that: The crossover operation in step (5) is specifically as follows: (5.1) Randomly selecting two parent cue words and determining the intersection position, including: first randomly selecting two cue words from the parent candidate pool selected by fitness, and then randomly generating a crossover position; (5.2) Swapping the parts of the two prompt words before and after the intersection to generate two new offspring prompt words, including: dividing the two parent prompt words into two parts at the intersection, and swapping the latter parts to form a new individual; (5.3) Adjust the length of the offspring prompt words and perform semantic verification, including: checking whether the offspring length exceeds the preset range. If it is too long, it will be truncated; if it is too short, it will be supplemented with a random fragment from the parent. Then, the pre-trained semantic verification model is called to perform reasoning analysis on the generated content, calculate its logical consistency and factual rationality score, and filter out semantically coherent candidate content by setting a threshold. Finally, the optimization result that meets the semantic constraints is output.

9. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 8, characterized in that: The mutation operation in step (5) specifically includes at least one of the following: (5.4) Word replacement: Randomly select one or more words from the prompt words, use the context-aware embedding model to calculate the word vector similarity, and find semantically similar alternative words to replace them; (5.5) Word insertion: insert context-related words at random positions of the prompt word. The context-related words are selected and constructed through the topic model; (5.6) Word Deletion: First, perform part-of-speech tagging and dependency syntax analysis on the prompt words. Filter out redundant words with low semantic contribution by counting word frequencies, and randomly delete these redundant words. (5.7) Word order adjustment: Determine the semantic units that can be adjusted through grammatical tree analysis and randomly swap their positions.

10. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to claim 9, characterized in that: The termination condition in step (6) includes at least one of the following: (6.1) Reaching the preset maximum number of iterations: When the current number of iterations is equal to the preset maximum, the evolution process is terminated immediately, and the prompt word with the highest fitness in the current population is output as the final result; (6.2) The fitness improvement for several consecutive generations is less than the threshold: Dynamically monitor the fitness change of the best individual in each generation. If the fitness improvement for N consecutive generations is less than the threshold, the optimization process is considered to be over. (6.3) Obtaining the ideal hint word that satisfies all constraints: Check whether there are individuals that simultaneously satisfy the hard constraints of performance, diversity, and interpretability during each generation evaluation.

11. The method for evolving and generating prompt words of a visual-language model based on a genetic algorithm according to any one of claims 1 to 10, characterized in that: The step (7) is specifically as follows: (7.1) Output the top M prompt words with the highest fitness in the final population; (7.2) Through manual evaluation and in combination with specific application requirements, the most suitable prompt word or prompt word combination is selected from the optimal set.

Images