Multi-modal large model-based cue word automatic generation model training method and system
By using a closed-loop training framework and a composite loss function, combined with simple and complex multimodal models, high-quality multimodal prompt words are automatically generated. This solves the problems of reliance on manual writing and unstable quality in existing technologies, and achieves efficient and logical prompt word generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-03-03
AI Technical Summary
The generation of prompts for existing multimodal large models relies on manual writing, which is costly and inefficient. Existing automated methods cannot effectively generate high-quality multimodal prompts and ignore the model analysis process, resulting in unstable quality of the generated prompts.
By constructing a closed-loop training framework, utilizing cross-attention mechanism and composite loss function, and combining simple and complex multimodal models, prompt words are automatically generated and optimized. "Analysis process" supervision is introduced, and a critical learning mechanism is adopted to generate high-quality prompt words.
It achieves automated generation of multimodal prompts, significantly improving efficiency and quality, reducing costs, ensuring the logic and reliability of generated prompts, and shortening the training cycle.
Smart Images
Figure CN121600511A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for training a prompt word automatic generation model based on a multimodal large model. Background Technology
[0002] With the development of artificial intelligence technology, multimodal large models have shown great potential in assisting human life. For example, in terminal devices such as smart glasses that provide services to visually impaired users (blind people), multimodal large models are needed to help users understand their surroundings and answer questions in order to achieve normal life. Multimodal large models used in such scenarios not only need to have high accuracy, but also extremely fast analysis speed to provide real-time environmental feedback.
[0003] To balance accuracy and speed, a common approach is to select a multimodal model with a relatively simple structure and fast inference speed, and guide it to perform accurate and efficient analysis using high-quality prompts. However, currently, the generation of high-quality prompts mainly relies on prompt engineers to manually write and debug them. This approach has several inherent drawbacks: First, manual writing depends on the engineer's personal understanding of the task, which may lead to misunderstandings; second, manually written prompts are often one-sided, failing to encompass the full range of task requirements and exhibiting poor generalization ability; third, the manual debugging process requires significant manpower and time, resulting in high costs and low efficiency.
[0004] To address this issue, the industry has proposed several algorithms based on large models to automatically optimize suggestion words. However, existing solutions still suffer from the following problems: 1. Existing automatic prompt word optimization algorithms mainly focus on single-modal text large model fields, and their application in multimodal large model fields has not yet been realized, making it impossible to effectively handle complex tasks that combine images and text.
[0005] 2. Most existing text prompt optimization methods are based on the analysis and optimization functions of a ready-made, powerful large model to refine and improve existing prompts. They cannot generate prompts from scratch and are also limited by the upper limit of the capabilities of the large model itself.
[0006] 3. Existing technologies often focus only on whether the final output "answer" is correct during model optimization, while ignoring the "analysis process" or "reasoning path" by which the model arrives at the answer. This may lead to the optimization process getting stuck in local optima and the quality of the generated prompts being unstable.
[0007] Therefore, how to automatically and efficiently generate high-quality cue words that can adapt to multimodal tasks in order to improve the analytical capabilities of simple multimodal models is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0008] This invention primarily addresses the technical problems of existing technologies, such as reliance on manual writing of multimodal prompt words, high costs, bias, and the lack of or poor effectiveness of automated optimization methods. It provides a method and system for training an automatic prompt word generation model based on a large multimodal model.
[0009] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a method for training an automatic prompt word generation model based on a multimodal large model, comprising the following steps: S1: Receives image data and question text, and generates prompt words to be evaluated based on guidance information and prompt words generation model (e.g., qwen2.5 vl 7b); the initial value of the guidance information is manually set before model training; S2: Input the image data, question text, and prompt words to be evaluated into the first multimodal model (e.g., qwen vl 7B) and the second multimodal model (e.g., chat gpt 4o) respectively to obtain the first output result and the second output result containing the answer and the analysis process respectively; the model capability of the second multimodal model is higher than that of the first multimodal model; S3: Calculate the composite loss value based on the first output result, the second output result, and the preset labeled data; S4: Based on the composite loss value, update the model parameters of the prompt word automatic generation model through the backpropagation algorithm; S5: Based on the comparative analysis of the analysis process in the first output result and the analysis process in the second output result or the analysis process in the labeled data, generate new guidance information for the next training iteration.
[0010] The entire process is a closed loop: old guidance information - generate prompts - evaluate performance (calculate loss) - update the model - analyze process differences - generate new guidance information - use the new guidance information in the next round... This cycle continues until the model's loss converges or the preset number of training rounds is reached. Each round not only updates the guidance information but also typically uses new training data.
[0011] The core idea of this invention is to construct a self-evolving closed-loop training framework of "generation-evaluation-analysis-feedback". Unlike existing technologies, this invention introduces three core model roles: an automatic prompt generation model to be trained, a first multimodal model (simple model) acting as the guided student, and a second multimodal model (complex model) acting as the reference teacher. The entire process does not simply use the results of the complex model to supervise the simple model. Instead, a crucial "metacognitive" step is introduced in step S5: by comparing the analysis processes of the simple and complex models (or manually labeled models), the reasoning flaws of the simple model are identified, and these flaws are transformed into structured guidance information, which is fed back to the prompt generation model in the next iteration. This mechanism ensures that prompt optimization is no longer blindly fitting answers, but rather learns how to better guide the model's thinking, thereby generating more logical and guiding prompts. This achieves a shift from simply providing answers to teaching how to fish, bringing unexpected technical benefits.
[0012] Preferably, step S1, which involves generating the prompt words to be evaluated based on the guidance information, specifically includes: S11: Input the guidance information into the first text encoder for encoding to obtain guidance features; S12: Input the image data and question text into the encoding layer of the prompt word automatic generation model to obtain the original features; S13: The guidance features are fused with the original features through a cross-attention mechanism to generate target features carrying the guidance information, and the prompt words to be evaluated are generated based on the target features.
[0013] The formula is as follows: feature mix =Cross_Attention(feature 提示 ,Reshap(BGE(context 指导信息 )); Reshap represents dimension transformation, Cross_Attention represents the cross-attention mechanism, and feature 提示 This refers to the feature map of the first stage of the prompt word automatic generation model encoding layer. BGE is the first text encoder, such as a clip text encoder or BGE embedding, and context. 指导信息 It provides guidance information, directing the model generation process.
[0014] In existing technologies, guidance information is typically presented as a text concatenation prefix to the model input. This approach is relatively superficial and may not be fully understood and utilized by the model. This invention introduces a cross-attention mechanism to deeply integrate the text-encoded guidance features with the original features of the image and question within the model's encoding layer. This means that guidance information is no longer a simple external instruction, but rather a weight signal that directly adjusts and guides the model's focus and understanding of the original input at the feature level. For example, if the guidance information is "pay attention to and recognize the text in the image," this mechanism enables the model to automatically assign higher attention weights to the text regions in the image when processing the original features, thereby generating prompts that are more effective in guiding OCR (Optical Character Recognition) functions. This approach makes the utilization of guidance information more efficient and targeted, significantly accelerating model convergence.
[0015] Preferably, the calculation process for the composite loss value in step S3 is as follows: S31: Simple to calculate - loss from manual answer A_简_人 : loss A_简_人 =1-cos(BGE(result A_简 ),BGE(result A_人 )); Among them, result A_简 The answer in the first output result, result A_人 For the answers in the labeled data, BGE is the text encoder, and cos represents the calculation of cosine similarity; S32: Simple to calculate - loss due to manual processes P_简_人 : loss P_简_人 =1-cos(BGE(result P_简 ),BGE(result P_人 )); Among them, result P_简 For the analysis process in the first output result, result P_人 This refers to the analysis process within the labeled data; S33: Computational complexity - loss due to human answers A_复_人 : loss A_复_人 =1-cos(BGE(result A_复 ),BGE(result A_人 )); Among them, result A_复 This is the answer in the second output result; S34: If complex - human answer lossA_复_人 If the loss exceeds the threshold for complex answers, then the simple-complex process loss is applied. P_简_复 If the value is 0, then the simple-complex process loss is calculated using the following formula. P_简_复 : loss P_简_复 =1-cos(BGE(result P_简 ),BGE(result P_复 )); Among them, result P_复 This refers to the analysis process in the second output result; S35: Calculate the composite loss value: loss sum_简 =loss A_简_人 +loss P_简_人 +(1-loss A_复_人 )×loss P_简_复 .
[0016] This invention designs an ingenious composite loss function with a "critical learning" mechanism, which is one of its core technical contributions. The purpose of this loss function is to ensure that the prompt word generation model learns high-quality, correct reasoning processes, rather than erroneous noise information. A_简_人 and loss P_简_人 These constitute the basic supervision terms, ensuring that the generated prompts guide the simple model to approximate the manually labeled standard answer in both answer and process. The most crucial element is the loss function. A_复_人 and loss P_简_复构成 The conditional weighting terms. The 1-loss in steps S34 and S35. A_复_人 The weighting factor embodies a profound technical idea: only when the teacher model (the second multimodal model) itself provides a correct answer to the question (i.e., the loss) can the weighting factor be applied correctly. A_复_人 Only when the loss is very small (close to 0) does the solution process it provides have reference value. At this point, 1-loss A_复_人 When the loss is close to 1, P_简_复 Giving them higher weights encourages simpler models to learn the superior reasoning paths of complex models. Conversely, if the teacher model itself answers incorrectly (loss...), it hinders the learning process. A_复_人 If the value is greater than the threshold, then its solution process is likely misleading, and in this case, 1-loss A_复_人 The loss is close to 0 or negative. P_简_复The weights are significantly reduced or even ignored (directly set to 0 via step S34). This "critical" or "selective" learning mechanism effectively avoids the model being misled by incorrect teacher knowledge during training, significantly improving the stability of training and the reliability of the final generated prompts, which is not available in existing knowledge distillation or model optimization methods.
[0017] Preferably, step S5, the step of generating new guidance information, specifically includes: When complex - human answer loss A_复_人 When the loss threshold for complex answers is less than the threshold, the analysis process of the first output result is compared with the analysis process of the second output result by an analysis model (e.g., DeepseekR1), the defects in the analysis process of the first output result are identified, and the defects are used as new guidance information. Otherwise, by comparing the analysis process of the first output result with the analysis process of the labeled data through the analysis model, defects in the analysis process of the first output result are identified, and the defects are used as new guiding information.
[0018] This is how the feedback loop is implemented. To achieve a deep understanding and comparison of the unstructured text of the "analysis process," this invention introduces an independent and more powerful analysis model (e.g., based on DeepseekR1). This model is specifically responsible for performing the "metacognitive" task, that is, comparing the reasoning paths of the two models to find out "why the simple model performs poorly." For example, the analysis process of the simple model might be "there is a car in the picture," while the analysis process of the complex model is "first locate the moving object, identify it as a bus, and then use OCR to identify the bus number as route 88." After comparing the analysis models, a defect description can be generated, namely "a key OCR recognition step was missed." This text then becomes the guiding information for the next round, guiding the prompt word generation model to generate prompt words that are more likely to trigger the OCR function.
[0019] Preferably, the model parameters of the first multimodal model and the second multimodal model are frozen during the execution of the training method. This is done to clarify the optimization objectives during training and ensure the stability and efficiency of the training process.
[0020] In this invention, the first and second multimodal models are fixed as student and teacher models, respectively, existing as the evaluation and guidance environment, and their parameters should not be changed. The sole object of optimization is the "automatic prompt generation model." By freezing the parameters of the student and teacher models, all gradient information can be used to update the parameters of the prompt generation model, making the optimization direction very clear and avoiding the non-convergence problem that may arise from training multiple models simultaneously.
[0021] Preferably, the image data originates from the image sensor of a terminal device serving visually impaired users; The calculation of the composite loss value further includes: applying a preset penalty weight to the identification error of the predefined critical security object in the first output result, so as to increase the composite loss value including the error.
[0022] This solution can be tailored to specific application scenarios (such as assisting visually impaired users). While general methods aim to improve overall accuracy, specific scenarios are more sensitive to certain errors. For example, when assisting visually impaired users, failing to identify critical safety objects such as "steps" or "puddles" is far more serious than failing to identify the color of a "billboard." Therefore, this solution introduces a penalty weight term into the composite loss function, imposing a greater penalty on errors in identifying these critical safety objects. This allows the model to prioritize learning how to correctly identify these safety-critical objects during training, resulting in generated prompts that better guide the model to focus on hazards in the environment, thus improving the practicality and safety of the solution in specific scenarios.
[0023] A training system for an automatic prompt word generation model based on a multimodal large model, running the automatic prompt word generation model training method as described above, includes: The model providing unit is used to provide an automatic prompt word generation model, a first multimodal model, and a second multimodal model, wherein the model capability of the second multimodal model is higher than that of the first multimodal model; The prompt word generation unit is used to receive image data and question text, and generate prompt words to be evaluated based on guidance information through the prompt word automatic generation model; The model reasoning unit is used to input the image data, question text, and prompt words to be evaluated into the first multimodal model and the second multimodal model, respectively, so as to obtain a first output result and a second output result containing the answer and the analysis process, respectively. The loss calculation unit is used to calculate the composite loss value based on the first output result, the second output result, and the preset labeled data; The parameter update unit is used to update the model parameters of the prompt word automatic generation model according to the composite loss value; The guidance information update unit is used to generate new guidance information for the next training iteration based on a comparative analysis of the analysis process in the first output result and the analysis process in the second output result or the analysis process in the labeled data.
[0024] This system can be a dedicated hardware device or a software system running on a general-purpose computer. Its internal functional units or modules correspond one-to-one with the method steps, such as including a model providing unit, a prompt word generation unit, a model inference unit, a loss calculation unit, a parameter update unit, and a guidance information update unit.
[0025] It should be noted that the training object of this invention—the automatic prompt word generation model—is not directly aimed at solving specific problems for end users (for example, it does not directly answer a blind person's question, "What's in front of me?"). Instead, it exists as a prompt word factory or a meta-tool for model training. The final application of the prompt words generated by the automatic prompt word generation model is a multimodal model (i.e., the first multimodal model in this invention) that needs to be deployed on terminal devices, has a relatively simple model structure, and fast inference speed. The core value of this invention lies in obtaining a high-performance automatic prompt word generation model through an offline, complex training process. Then, in the actual application deployment phase, this trained model can be used to automatically and in batches generate optimal prompt words for various specific tasks or scenarios.
[0026] An example illustrating its application process: 1. Training phase (offline): Following the method described in this invention, a large amount of computing resources are invested to train a powerful automatic prompt word generation model.
[0027] 2. Deployment Phase (Online / Offline): Scenario 1 (General Prompt Word Generation): For a general task, such as "object recognition," the task description is fed into a pre-trained prompt word generation model to generate a high-performance general prompt word. This prompt word will be embedded into a simple multimodal model deployed on terminal devices such as glasses for the blind, serving as its default system-level prompt word.
[0028] Scenario 2 (Customized Prompt Word Generation): For a specific, new task, such as "identifying drug instructions", developers only need to input the description of this new task into the prompt word automatic generation model to quickly obtain a prompt word highly optimized for the task, without having to do tedious manual writing and debugging.
[0029] Scenario 3 (Dynamic Prompt Generation): In some more complex applications, the automatic prompt generation model itself can also be integrated into the cloud. When a simple model on the terminal device encounters difficulties, the current scenario (image, question) can be uploaded to the cloud. The prompt generation model in the cloud will then dynamically generate a one-time refined prompt for that specific scenario and send it to the terminal model for execution.
[0030] In summary, the direct output of this invention is an instruction generator capable of generating instructions. It provides high-quality prompt words for simple and efficient models in various downstream practical applications, thereby improving the overall service quality and capability ceiling of artificial intelligence applications without increasing the computational burden on terminal devices.
[0031] The substantial effects of this invention are: 1. This invention automates the generation of multimodal prompt words, significantly improving efficiency and reducing costs. It constructs a complete closed-loop training framework capable of automatically generating and optimizing high-quality multimodal prompt words from scratch. This eliminates the reliance on manual writing and debugging by prompt word engineers, shortening the manual debugging process, which originally took weeks or even months, to machine training time, greatly improving R&D efficiency and reducing labor costs.
[0032] 2. By introducing "analysis process" supervision, the quality of generated prompts and the "interpretability" of the model are improved. Unlike existing technologies that only focus on the correctness of the final answer, this invention innovatively incorporates the model's "analysis process" or "reasoning path" into the supervision and feedback process. This makes the optimization process no longer a black-box fitting, but rather guides the prompt generation model to learn how to construct more logically rigorous and clearly structured reasoning. The final generated prompts not only enable simple models to "answer correctly," but also enable them to "answer correctly with sound reasoning," fundamentally improving the model's analytical capabilities and the reliability of its output.
[0033] 3. The stability and reliability of the training process are enhanced through a composite loss function based on "critical learning." The conditionally weighted composite loss function designed in this invention can dynamically evaluate the reliability of the "teacher model's" output and determine whether to adopt its analysis process as a learning sample. This "critical" learning mechanism effectively avoids the model being misled by erroneous or low-quality teacher knowledge during training, solves the "teacher noise" problem that may occur in existing knowledge distillation methods, significantly improves the stability of the training process, and ensures the lower limit of the quality of the final generated prompt words.
[0034] 4. By deeply integrating dynamic guidance information with the model, model convergence is accelerated. This invention employs a cross-attention mechanism to deeply integrate iteratively generated guidance information into the model's encoding layer, enabling direct interaction between the guidance signal and the original features at the feature level. Compared to simple text concatenation, this approach allows the guidance information to more directly and efficiently guide the model's attention allocation and feature extraction. Each iteration of optimization is more targeted, thereby significantly accelerating the model's convergence speed and shortening the training cycle. Attached Figure Description
[0035] Figure 1This is a schematic diagram of the training system structure of an automatic prompt word generation model based on a multimodal large model according to the present invention; Figure 2 This is a flowchart of a training method for an automatic prompt word generation model based on a multimodal large model according to the present invention; In the diagram: 101 - Automatic prompt word generation model; 102 - First multimodal model; 103 - Second multimodal model; 104 - Analysis model. Detailed Implementation
[0036] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.
[0037] Example: Refer to Figure 1 This embodiment discloses a training system for an automatic prompt word generation model based on a multimodal large model. The system can be deployed on one or more servers and mainly includes the following model entities: The prompt word automatic generation model 101 is the core training object of this invention. It can be built on a modified base model, for example, based on the qwen2.5-7b model, with a cross-attention module added to its encoder structure for receiving guidance information.
[0038] First multimodal model 102 (simple model): This serves as a student model guided by prompts to perform tasks, requiring a small model size and fast inference speed. For example, the qwen-vl-7b model can be used.
[0039] The second type of multimodal model, 103 (complex model), serves as a teacher model providing high-quality references and requires strong model capabilities, as well as superior analytical and reasoning abilities. For example, industry-leading multimodal models such as GPT-4O can be selected.
[0040] Analysis Model 104: Used for comparative analysis processes and generating defect reports (i.e., new guidance information). This model requires strong text understanding and logical analysis capabilities; for example, the DeepSeekR1 model could be used.
[0041] Below, refer to Figure 1 Taking a smart glasses application scenario serving visually impaired users as an example, the training method and process of the present invention are explained in detail.
[0042] Step S1: Generate prompts to be evaluated. Before training begins, the system initializes a general guidance message, such as "Please describe the image content in detail and answer the questions." In one iteration of training, the system retrieves a batch of data from the training dataset. For example, one sample might be: Image data: A picture of a bus stop taken by the smart glasses' camera.
[0043] Question text: "Which bus number is in front of me?"
[0044] Data annotation: {Answer: “Route 88”, Analysis process: “First, a large vehicle was located in the center of the image and identified as a bus. Then, OCR optical character recognition was performed on the electronic display screen area at the front of the bus, and the number '88' was identified.”}
[0045] The prompt word automatic generation model 101 receives image data and question text, and integrates the current guidance information into its encoding process through a cross-attention mechanism. Finally, it decodes and generates a prompt word to be evaluated, such as: "Please identify the bus route number in the picture."
[0046] Step S2: Model Inference. The image data, the question text, and the prompt words to be evaluated generated in the previous step are taken as a complete input and fed into the first multimodal model 102 and the second multimodal model 103, respectively.
[0047] The first output of the first multimodal model 102 (simple model) may be: {Answer: "Route 88", Analysis process: "Route 88 bus identified."}.
[0048] The second output of the second multimodal model 103 (complex model) may be: {Answer: "Route 88", Analysis process: "There is a bus pulling into the station in the image, and the front of the bus shows Route 88."}.
[0049] Step S3: Calculate the composite loss value. The system calculates the composite loss value according to the steps defined in claim 3.
[0050] 31: Calculate the loss A_简_人 By comparing the answer "88" from the simple model with the answer "88" from the labeled data, the cosine similarity loss is calculated, and the value is very small.
[0051] S32: Calculate the loss P_简_人 Comparing the analysis process of the simple model, "identifying bus route 88," with the analysis process of labeled data, "first...OCR...", the loss value will be larger due to the significant difference in the level of detail in the process description.
[0052] S33: Calculate the loss A_复_人 The loss value is very small when comparing the answer "88" from the complex model with the answer "88" from the labeled data.
[0053] S34: Determine the loss A_复_人 Is it greater than a threshold (e.g., 0.1)? Since this value is very small, it is determined as no. Therefore, the loss is calculated further.P_简_复 This loss value also exists when comparing the process of a simple model with that of a complex model.
[0054] S35: According to the formula loss sum_简 =loss A_简_人 +loss P_简_人 +(1-loss A_复_人 )×loss P_简_复 The final composite loss value is calculated.
[0055] Step S4: Update model parameters. The system updates the model parameters based on the calculated loss. sum_简 The gradient is calculated using the backpropagation algorithm, and the model parameters of the prompt word automatic generation model 101 are updated. During this process, the parameters of the first multimodal model 102 and the second multimodal model 103 remain frozen.
[0056] Step S5: Generate new guidance information. Due to loss A_复_人 If the result is less than the threshold, it indicates that the complex model is reliable. The system inputs the analysis process of the first output result and the analysis process of the second output result into the analysis model 104.
[0057] After comparing the analysis model 104, it was found that the process description of the first output result was too simple and lacked key steps. Therefore, a defect description was generated, namely the new guidance information: "The analysis process should include steps such as location, identification and key information extraction (such as OCR).
[0058] Iterative Training: In the next training iteration, the system will use this new guidance information to process the next batch of training data. Under this new guidance, the prompt word generation model 101 is more likely to generate more specific and guiding high-quality prompt words, such as "Please locate the bus in the image, perform OCR recognition on its front, and report the recognized numbers." The entire training process involves continuous iterative iteration until the composite loss value converges to a stable, low level, at which point training ends. The trained prompt word generation model can then be deployed to real-world applications.
[0059] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
[0060] Although this document uses various terms extensively, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would contradict the spirit of the invention.
Claims
1. A method for training an automatic prompt word generation model based on a multimodal large model, characterized in that, Includes the following steps: S1: Receive image data and question text, and generate prompt words to be evaluated based on guidance information and prompt word generation model; S2: Input the image data, question text, and the prompt words to be evaluated into the first multimodal model and the second multimodal model respectively to obtain the first output result and the second output result containing the answer and the analysis process respectively; The second multimodal model has higher modeling capabilities than the first multimodal model; S3: Calculate the composite loss value based on the first output result, the second output result, and the preset labeled data; S4: Based on the composite loss value, update the model parameters of the prompt word automatic generation model through the backpropagation algorithm; S5: Based on the comparative analysis of the analysis process in the first output result and the analysis process in the second output result or the analysis process in the labeled data, generate new guidance information for the next training iteration.
2. The method for training an automatic prompt word generation model based on a multimodal large model according to claim 1, characterized in that, In step S1, the step of generating the prompt words to be evaluated based on the guidance information specifically includes: S11: Input the guidance information into the first text encoder for encoding to obtain guidance features; S12: Input the image data and question text into the encoding layer of the prompt word automatic generation model to obtain the original features; S13: The guidance features are fused with the original features through a cross-attention mechanism to generate target features carrying the guidance information, and the prompt words to be evaluated are generated based on the target features.
3. The method for training an automatic prompt word generation model based on a multimodal large model according to claim 1, characterized in that, In step S3, the calculation process for the composite loss value is as follows: S31: Simple to calculate - loss from manual answer A_简_人 : loss A_简_人 =1-cos(BGE(result A_简 ),BGE(result A_人 )); Among them, result A_简 The answer in the first output result, result A_人 For the answers in the labeled data, BGE is the text encoder, and cos represents the calculation of cosine similarity; S32: Simple to calculate - loss due to manual processes P_简_人 : loss P_简_人 =1-cos(BGE(result P_简 ),BGE(result P_人 )); Among them, result P_简 For the analysis process in the first output result, result P_人 This refers to the analysis process within the labeled data; S33: Computational complexity - loss due to human answers A_复_人 : loss A_复_人 =1-cos(BGE(result A_复 ),BGE(result A_人 )); Among them, result A_复 This is the answer in the second output result; S34: If complex - human answer loss A_复_人 If the loss exceeds the threshold for complex answers, then the simple-complex process loss is applied. P_简_复 If the value is 0, then the simple-complex process loss is calculated using the following formula. P_简_复 : loss P_简_复 =1-cos(BGE(result P_简 ),BGE(result P_复 )); Among them, result P_复 This refers to the analysis process in the second output result; S35: Calculate the composite loss value: loss sum_简 =loss A_简_人 +loss P_简_人 +(1-loss A_复_人 )×loss P_简_复 。 4. The method for training an automatic prompt word generation model based on a multimodal large model according to claim 1, characterized in that, In step S5, the step of generating the new guidance information specifically includes: When complex - human answer loss A_复_人 When the loss threshold for complex answers is less than the threshold, the analysis model compares the analysis process of the first output result with that of the second output result to identify the defects in the analysis process of the first output result and uses the defects as new guidance information. Otherwise, by comparing the analysis process of the first output result with the analysis process of the labeled data through the analysis model, defects in the analysis process of the first output result are identified, and the defects are used as new guiding information.
5. A method for training an automatic prompt word generation model based on a multimodal large model according to any one of claims 1-4, characterized in that, During the execution of the training method, the model parameters of the first multimodal model and the second multimodal model are frozen.
6. The method for training an automatic prompt word generation model based on a multimodal large model according to any one of claims 5, characterized in that, The image data originates from the image sensor of a terminal device that serves visually impaired users. The calculation of the composite loss value further includes: applying a preset penalty weight to the identification error of the predefined critical security object in the first output result, so as to increase the composite loss value including the error.
7. A training system for an automatic prompt word generation model based on a multimodal large model, running the automatic prompt word generation model training method as described in claim 1, characterized in that, Include: The model providing unit is used to provide an automatic prompt word generation model, a first multimodal model, and a second multimodal model, wherein the model capability of the second multimodal model is higher than that of the first multimodal model; The prompt word generation unit is used to receive image data and question text, and generate prompt words to be evaluated based on guidance information through the prompt word automatic generation model; The model reasoning unit is used to input the image data, question text, and prompt words to be evaluated into the first multimodal model and the second multimodal model, respectively, so as to obtain a first output result and a second output result containing the answer and the analysis process, respectively. The loss calculation unit is used to calculate the composite loss value based on the first output result, the second output result, and the preset labeled data; The parameter update unit is used to update the model parameters of the prompt word automatic generation model according to the composite loss value; The guidance information update unit is used to generate new guidance information for the next training iteration based on a comparative analysis of the analysis process in the first output result and the analysis process in the second output result or the analysis process in the labeled data.