A prompt word optimization training method and system based on multimodal large model supervision

Abstract

The present invention discloses a method and system for optimizing prompt words based on multimodal large model supervision. The method comprises: screening and optimizing multiple initial prompt words using a simple multimodal large model, a complex multimodal large model, and a first text large model to obtain simple optimal prompt words and complex optimal prompt words; inputting the simple optimal prompt words, a training set, and a manually calibrated problem into the simple multimodal large model for training to obtain a simple total loss value; determining whether the simple total loss value is within a preset threshold range, and if so, terminating the training; otherwise, repeating the optimization process until the simple total loss value is within the preset threshold range, terminating the training, and using the simple optimal prompt word obtained from the last update as the target optimal prompt word. The method reduces manual intervention through the collaborative training of the complex large model and the simple large model; combining the manually calibrated data with the reasoning process of the complex large model to improve the accuracy of the prompt words through dual supervision.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and multimodal large models, and in particular to a method and system for optimizing and training prompt words based on multimodal large model supervision. Background Art

[0002] Existing port navigation monitoring mainly relies on traditional target detection models for target recognition, which has problems such as category limitations and insufficient information fusion capabilities. Multimodal large models have the ability to integrate vision and semantics, can perform open set recognition, and adapt to the complex environment of ports. However, since the reasoning performance of multimodal large models is highly dependent on the quality of prompt words, the existing prompt word optimization usually relies on manual design, which has problems such as understanding bias and low efficiency. Although attempts to automatically optimize prompt words have appeared in recent years, they are single-modal optimizations based on reinforcement learning. In complex multimodal scenarios, especially in port monitoring applications, there is no mature method to achieve automatic and efficient optimization of prompt words. Therefore, there is an urgent need for an automated, high-precision multimodal prompt word optimization method. Summary of the Invention

[0003] The embodiments of the present invention provide a prompt word optimization training method and system based on multimodal large model supervision to solve the problems of relying on manual design of prompt words in the prior art, which have misunderstanding bias and low efficiency.

[0004] To achieve the above objectives, the present invention provides, on the one hand, a method for optimizing prompt words and training them based on supervision of a multimodal large model, the method comprising: S1, constructing a training set including manually calibrated questions, a manual analysis process, and manual answers; S2, using the complex multimodal large model to generate multiple initial prompt words from the training set; S3, screening the multiple initial prompt words using a simple multimodal large model to obtain simple initial prompt words; and S4, screening the multiple initial prompt words using the complex multimodal large model to obtain complex initial prompt words. S4. Optimize the simple initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt word; optimize the complex initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the complex optimal prompt word; S5. Input the simple optimal prompt word, the training set, and the manual calibration problem into the simple multimodal large model for training to obtain the simple total loss value, the simple answer for each picture, and the simple analysis process for each picture; determine whether the simple total loss value is within the preset threshold range. If so, end the training; otherwise, use the simple optimal prompt word as the simple initial prompt word and the complex optimal prompt word as the complex initial prompt word, repeat S4~S5 until the simple total loss value is within the preset threshold range, stop training, and use the simple optimal prompt word obtained from the last update as the target optimal prompt word.

[0005] Optionally, after S5, the following steps are included: inputting each picture corresponding to the simple answer error obtained in the last training into the visual feature extraction model encoding layer for feature extraction to obtain a visual feature vector of each picture; inputting the manual analysis process of each picture corresponding to the simple answer error obtained in the last training into the embedding encoder for feature extraction to obtain a text feature vector of each picture; fusing the visual feature vector and the text feature vector of the current picture to obtain a fused feature vector of the current picture; clustering the fused feature vectors of all pictures to obtain multiple clusters; and adding the simple analysis process of the picture corresponding to the cluster center of each cluster to the target optimal prompt word.

[0006] Optionally, S2 includes: inputting the training set and the manual calibration problem of each picture in the training set into the complex multimodal large model to obtain the keywords of the problem; obtaining a reference prompt word template based on the keyword network search; inputting the reference prompt word template and the manual calibration problem of all pictures into the complex multimodal large model to obtain multiple initial prompt words.

[0007] Optionally, the method of screening multiple initial prompt words using a simple multimodal large model to obtain simple initial prompt words includes: taking each initial prompt word as the current initial prompt word in turn; inputting the current initial prompt word, training set, and manual calibration problem into the simple multimodal large model for training to obtain a simple reasoning answer and a simple reasoning analysis process for each picture; calculating a simple analysis process similarity loss value for each picture based on the manual analysis process and the simple reasoning analysis process for each picture; calculating a score corresponding to the current initial prompt word based on the simple analysis process similarity loss value for each picture, the simple reasoning answer for each picture, and the manual answer for each picture; and taking the initial prompt word corresponding to the highest score as the simple initial prompt word.

[0008] Optionally, the simple initial prompt words are optimized through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt words, including: inputting the simple initial prompt words, the training set, and the manual calibration problem into the simple multimodal large model for model training to obtain the simple reasoning answer, the simple analysis process, and the simple analysis process similarity loss value of each picture; inputting the complex initial prompt words, the training set, and the manual calibration problem into the complex multimodal large model for model training to obtain the complex reasoning answer, the complex analysis process, and the complex analysis process similarity loss value of each picture; according to the simple reasoning answer, the simple analysis process, and the simple analysis process similarity loss value of each picture, the complex reasoning answer, the complex analysis process, and the complex analysis process similarity loss value of each picture, the simple initial prompt words are optimized. The prompt words are trained through the first text large model to obtain the first simple correct prompt word, the second simple correct prompt word, the first simple wrong prompt word, and the second simple wrong prompt word; the first simple correct prompt word, the second simple correct prompt word, the first simple wrong prompt word, and the second simple wrong prompt word are input into the first text large model for training to obtain simple common prompt words; the first simple correct prompt word, the second simple correct prompt word, the first simple wrong prompt word, the second simple wrong prompt word, the simple common prompt word, and the training set are input into the simple multimodal large model to obtain the answer obtained by reasoning each prompt word for each picture, as well as the analysis process; based on the answer obtained by reasoning each prompt word for each picture, as well as the analysis process, the simple common prompt word is optimized using the first text large model to obtain the simple optimal prompt word.

[0009] Optionally, the simple initial prompt word is trained through the first text large model according to the simple reasoning answer, simple analysis process and similarity loss value of the simple analysis process of each picture, the complex reasoning answer, complex analysis process and similarity loss value of the complex analysis process of each picture to obtain the first simple correct prompt word, the second simple correct prompt word, the first simple wrong prompt word and the second simple wrong prompt word, including: the simple reasoning analysis process and the manual analysis process of the picture whose simple reasoning answer is correct and the simple analysis process similarity loss value is greater than or equal to the first threshold are input into the first text large model for training to obtain the first comprehensive analysis process; the simple initial prompt word and the first comprehensive analysis process are input into the first text large model for training to obtain the first simple correct prompt word; the complex reasoning analysis process of the picture whose simple reasoning answer is correct and the simple analysis process similarity loss value is less than the first threshold, the complex reasoning analysis process of the picture whose complex reasoning answer is correct and the complex analysis process similarity loss value is greater than or equal to the second threshold, and the simple The reasoning analysis process and the manual analysis process are input into the first text model for training to obtain the second comprehensive analysis process; the simple initial prompt words and the second comprehensive analysis process are input into the first text model for training to obtain the second simple correct prompt words; the simple reasoning analysis process of the picture whose simple reasoning answer is wrong and the complex reasoning answer is correct is subjected to difference analysis with the manual analysis process and the complex reasoning analysis process respectively; the difference analysis is input into the first text model for training to obtain the third comprehensive analysis process; the simple initial prompt words and the third comprehensive analysis process are input into the first text model for training to obtain the first simple wrong prompt words; the manual analysis process of the picture whose simple reasoning answer is wrong and the complex reasoning answer is correct is subjected to difference analysis with the simple reasoning analysis process and the complex reasoning analysis process respectively; the difference analysis is input into the first text model for training to obtain the fourth comprehensive analysis process; the simple initial prompt words and the fourth comprehensive analysis process are input into the first text model for training to obtain the second simple wrong prompt words.

[0010] Optionally, the similarity loss value of the simple analysis process of each image is calculated according to the following formula:

[0011]

[0012] in, is the similarity loss value of the simple analysis process of the current picture, This is a simple reasoning and analysis process for the current picture. is the manual analysis process of the current picture, is the embedded encoder.

[0013] Optionally, the score corresponding to the current initial prompt word is calculated according to the following formula:

[0014]

[0015] in, is the score corresponding to the current initial prompt word, is the number of images in the training set, is the similarity loss value of the simple analysis process of the i-th picture, Whether the simple reasoning answer of the i-th picture is the same as the manual answer, 1 if they are the same, and 0 if they are different.

[0016] On the other hand, the present invention provides a prompt word optimization training system based on multimodal large model supervision, which includes: a manual calibration unit for constructing a training set including manual calibration questions, manual analysis process and manual answers; a generation unit for generating multiple initial prompt words from the training set using a complex multimodal large model; a screening unit for screening multiple initial prompt words using a simple multimodal large model to obtain simple initial prompt words; screening multiple initial prompt words using a complex multimodal large model to obtain complex initial prompt words; an optimization unit for optimizing the simple initial prompt words through a simple multimodal large model, a complex multimodal large model and a first text large model to obtain simple optimal prompt words; and The word is optimized through a simple multimodal large model, a complex multimodal large model, and a first text large model to obtain a complex optimal prompt word; a judgment unit is used to input the simple optimal prompt word, a training set, and a manual calibration problem into the simple multimodal large model for training to obtain a simple total loss value, a simple answer to each picture, and a simple analysis process for each picture; it is judged whether the simple total loss value is within a preset threshold range, and if so, the training is terminated; otherwise, the simple optimal prompt word is used as the simple initial prompt word, and the complex optimal prompt word is used as the complex initial prompt word, and S4~S5 are repeated until the simple total loss value is within the preset threshold range, and the training is stopped, and the simple optimal prompt word obtained by the last update is used as the target optimal prompt word.

[0017] Optionally, it also includes: an adding unit, which is used to: input each picture corresponding to the simple answer error obtained in the last training into the visual feature extraction model encoding layer for feature extraction to obtain the visual feature vector of each picture; input the manual analysis process of each picture corresponding to the simple answer error obtained in the last training into the embedding encoder for feature extraction to obtain the text feature vector of each picture; fuse the visual feature vector and text feature vector of the current picture to obtain the fused feature vector of the current picture; cluster the fused feature vectors of all pictures to obtain multiple clusters; and add the simple analysis process of the picture corresponding to the cluster center of each cluster to the target optimal prompt word.

[0018] Beneficial effects of the present invention:

[0019] The present invention provides a prompt word optimization training method and system based on multimodal large model supervision. The method reduces manual intervention and reduces the cost of prompt word optimization through the collaborative training of complex large models and simple large models. The accuracy and generalization ability of prompt words are improved through dual supervision by combining manual calibration data with the reasoning process of the complex large model. The dual supervision of question answer loss and reasoning process loss is introduced to balance accuracy and reasoning rationality. The prompt words are simultaneously optimized in the two dimensions of logical analysis and visual recognition to solve the adaptation problem of complex scenarios. Error cases are analyzed by fusing visual and text features, typical difficult examples are discovered based on clustering, and prompt words are optimized in reverse. The method of the present invention greatly reduces the degree of manual participation and improves the automation level of prompt word generation and optimization. It significantly improves the comprehensive reasoning ability of the simple multimodal large model in port navigation monitoring tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of a prompt word optimization training method based on multimodal large model supervision provided by an embodiment of the present invention;

[0021] Figure 2 This is a structural diagram of a prompt word optimization training system based on multimodal large model supervision provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0023] Figure 1 This is a flowchart of a prompt word optimization training method based on multimodal large model supervision provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0024] S1. Construct a training set including manually calibrated questions, manual analysis process and manual answers;

[0025] In an optional embodiment, the training set is a ship image training set; assuming that the ship image training set includes 10 images, each image is manually calibrated with a question, and the analysis process and the answer to the question are manually judged.

[0026] Each image is manually labeled in the form of a multiple-choice question, with answers manually given based on the question. For example: How many cargo ships are docked at the port in the video? A: 5, B: 6, C: 7.

[0027] Manual analysis process: There are 11 ships in the picture, 5 of which are on the sea and 6 are parked at the dock.

[0028] Human answer: B.

[0029] The manual calibration problems for each image in the ship image training set are the same or similar.

[0030] S2. Generate multiple initial prompts using a complex multimodal model for the training set.

[0031] In an optional embodiment, the S2 includes:

[0032] S21, inputting the training set and the manual calibration problem of each image in the training set into the complex multimodal large model to obtain keywords of the problem;

[0033] Input all the manual calibration questions for each image in the training set into a complex multimodal large model (such as the Qwen2.5 VL 72B model). Instruct the large model to extract keywords for all questions. For example, keywords include: [pier], [docking], [cargo ship], [quantity statistics].

[0034] S22, obtaining a reference prompt word template based on a keyword online search;

[0035] The keywords extracted in the previous step are searched online (for example, by accessing a search engine or an industry knowledge base); and existing prompt words (prompt words) related to the keywords are searched.

[0036] S23: Input the reference prompt word template and the manual calibration problems of all images into the complex multimodal large model to obtain multiple initial prompt words.

[0037] Specifically, the retrieved reference prompt word template and the manual calibration problems of the above 10 images are input into the complex multimodal large model together; the complex large model is required to rewrite and expand the prompt words based on the reference prompt word template for all manual calibration problems, and generate at least multiple versions of the initial prompt words (for example, 3 to 5).

[0038] S3. Screening the multiple initial prompt words using a simple multimodal large model to obtain simple initial prompt words; screening the multiple initial prompt words using a complex multimodal large model to obtain complex initial prompt words;

[0039] In an optional embodiment, the screening of the multiple initial prompt words using a simple multimodal large model to obtain the simple initial prompt words includes:

[0040] S311, taking each initial prompt word as the current initial prompt word in turn;

[0041] S312: Input the current initial prompt word, training set, and manual calibration questions into a simple multimodal large model for training, and obtain a simple reasoning answer for each image, as well as a simple reasoning analysis process;

[0042] In this application, the simple multimodal large model is the qwen2.5 vl 7B model.

[0043] S313, calculating the simple analysis process similarity loss value of each image based on the manual analysis process and the simple reasoning analysis process of each image;

[0044] The similarity loss value of the simple analysis process of each image is calculated according to the following formula:

[0045]

[0046] in, is the similarity loss value of the simple analysis process of the current picture, This is a simple reasoning and analysis process for the current picture. is the manual analysis process of the current picture, is the embedded encoder.

[0047] S314, calculating the score corresponding to the current initial prompt word based on the similarity loss value of the simple analysis process of each image, the simple reasoning answer of each image, and the manual answer of each image;

[0048] The score corresponding to the current initial prompt word is calculated according to the following formula:

[0049]

[0050] in, is the score corresponding to the current initial prompt word, is the number of pictures in the training set (in the above embodiment, the number of pictures is 10), is the similarity loss value of the simple analysis process of the i-th picture, The simple reasoning answer for the i-th picture is the same as the manual answer. If they are the same, it is 1 (that is, the answer is correct); if they are different, it is 0 (that is, the answer is wrong).

[0051] S315: Use the initial prompt word corresponding to the highest score as a simple initial prompt word.

[0052] The complex initial prompt words are screened using a complex multimodal large model to obtain complex initial prompt words including:

[0053] S321, taking each initial prompt word as the current initial prompt word in turn;

[0054] S322: Input the current initial prompt word, training set, and manual calibration question into the complex multimodal large model for training, and obtain the complex reasoning answer and complex reasoning analysis process for each image;

[0055] S323, calculating a complex analysis process similarity loss value of each image based on the manual analysis process and the complex reasoning analysis process of each image;

[0056] The similarity loss value of the complex analysis process of each image is calculated according to the following formula:

[0057]

[0058] in, is the similarity loss value of the complex analysis process of the current picture, is the complex reasoning and analysis process of the current picture, is the manual analysis process of the current picture, is the embedded encoder.

[0059] S324, calculating a score corresponding to the current initial prompt word based on the similarity loss value of the complex analysis process of each image, the complex reasoning answer of each image, and the manual answer of each image;

[0060] The score corresponding to the current initial prompt word is calculated according to the following formula:

[0061]

[0062] in, is the score corresponding to the current initial prompt word, is the number of pictures in the training set (in the above embodiment, the number of pictures is 10), is the similarity loss value of the simple analysis process of the i-th picture, The simple reasoning answer for the i-th picture is the same as the manual answer. If they are the same, it is 1 (that is, the answer is correct); if they are different, it is 0 (that is, the answer is wrong).

[0063] S325: The initial prompt word corresponding to the highest score is used as the complex initial prompt word.

[0064] S4. Optimize the simple initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt word; optimize the complex initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the complex optimal prompt word;

[0065] In the present application, the first large text model may be a deepseek R1 large text model.

[0066] In an optional embodiment, optimizing the simple initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt word includes:

[0067] S411, inputting the simple initial prompt words, training set, and manual calibration questions into the simple multimodal large model for model training, obtaining a simple reasoning answer, a simple analysis process, and a similarity loss value of the simple analysis process for each image; inputting the complex initial prompt words, training set, and manual calibration questions into the complex multimodal large model for model training, obtaining a complex reasoning answer, a complex analysis process, and a similarity loss value of the complex analysis process for each image;

[0068] Compare the simple reasoning answer and the manual answer for each picture, with correct (i.e., the same) being 1 and incorrect (i.e., different) being 0;

[0069] Compare the complex reasoning answer and the manual answer for each picture, with correct (i.e., the same) valued as 1 and incorrect (i.e., different) valued as 0;

[0070] The calculation method of the similarity loss value of the simple analysis process and the similarity loss value of the complex analysis process is the same as that in S3 above.

[0071] S412: Based on the simple reasoning answer, simple analysis process, and similarity loss value of the simple analysis process for each image, and the complex reasoning answer, complex analysis process, and similarity loss value of the complex analysis process for each image, the simple initial prompt word is trained using the first large text model to obtain a first simple correct prompt word, a second simple correct prompt word, a first simple incorrect prompt word, and a second simple incorrect prompt word;

[0072] In an optional implementation, the S412 includes:

[0073] Input the simple reasoning analysis process and the manual analysis process of the image whose simple reasoning answer is correct and whose simple analysis process similarity loss value is greater than or equal to the first threshold into the first text large model for training to obtain a first comprehensive analysis process; input the simple initial prompt word and the first comprehensive analysis process into the first text large model for training to obtain a first simple correct prompt word;

[0074] For example: Among 10 pictures, there are 5 pictures with correct simple reasoning answers (that is, the simple reasoning answers are the same as the manual answers). Among these 5 pictures, the similarity loss values ​​of the simple analysis process of 2 pictures are greater than or equal to the first threshold (set to 0.6 in the present invention). Then the simple reasoning analysis process and the manual analysis process of these 2 pictures are input into the first text model for training to obtain the first comprehensive analysis process (that is, the advantages and disadvantages of all analysis processes from the visual and logical aspects are combined); the simple initial prompt word combines the advantages and disadvantages of vision and logic to regenerate the first simple correct prompt word.

[0075] The complex reasoning analysis process of the pictures whose simple reasoning answers are correct and whose simple analysis process similarity loss values ​​are less than a first threshold, the complex reasoning analysis process of the pictures whose complex reasoning answers are correct and whose complex analysis process similarity loss values ​​are greater than or equal to a second threshold, and the simple reasoning analysis process and manual analysis process of the remaining pictures are input into the first text large model for training to obtain a second comprehensive analysis process; the simple initial prompt word and the second comprehensive analysis process are input into the first text large model for training to obtain a second simple correct prompt word;

[0076] For example: among 10 pictures, there are 5 pictures with correct simple reasoning answers (that is, the simple reasoning answers are the same as the manual answers), and the similarity loss values ​​of the simple analysis process of 3 of the 5 pictures are less than the first threshold (set to 0.6 in the present invention), and the complex reasoning answers of 2 of the 3 pictures are correct (that is, the complex reasoning answers are the same as the manual answers) and the similarity loss values ​​of the complex analysis process are greater than or equal to the second threshold, then the complex reasoning analysis processes of the 2 pictures, and the simple reasoning analysis process and the manual analysis process of 1 picture (that is, the picture other than the 2 pictures with correct complex reasoning answers among the 3 pictures) are input into the first text model for training to obtain the second comprehensive analysis process (that is, the shortcomings of all analysis processes from both visual and logical aspects); the simple initial prompt word combines the visual and logical shortcomings to regenerate the second simple correct prompt word.

[0077] The simple reasoning analysis process of the picture with an incorrect simple reasoning answer and a correct complex reasoning answer is subjected to difference analysis with the manual analysis process and the complex reasoning analysis process respectively; the difference analysis is input into the first text large model for training to obtain a third comprehensive analysis process; the simple initial prompt word and the third comprehensive analysis process are input into the first text large model for training to obtain a first simple error prompt word;

[0078] For example, among 10 pictures, there are 5 pictures with incorrect simple reasoning answers (i.e., the simple reasoning answers are different from the manual answers), and 2 of the 5 pictures have correct complex reasoning answers (i.e., the complex reasoning answers are the same as the manual answers). Then, a difference analysis is performed between the simple reasoning analysis process and the manual analysis process of the 2 pictures, and a difference analysis is performed between the simple reasoning analysis process and the complex reasoning analysis process of the 2 pictures. All difference analyses are input into the first large text model for training to obtain a third comprehensive analysis process (i.e., the similarities and differences of all analysis processes from both visual and logical aspects are comprehensively analyzed). The simple initial prompt word is combined with the visual and logical analysis reasons to regenerate the first simple error prompt word.

[0079] The manual analysis process of the pictures corresponding to the simple reasoning answers being wrong and the complex reasoning errors is subjected to difference analysis with the simple reasoning analysis process and the complex reasoning analysis process respectively; the difference analysis is input into the first text large model for training to obtain the fourth comprehensive analysis process; the simple initial prompt words and the fourth comprehensive analysis process are input into the first text large model for training to obtain the second simple error prompt words.

[0080] For example, among 10 pictures, there are 5 pictures with incorrect simple reasoning answers (i.e., the simple reasoning answers are different from the manual answers), and 3 of the 5 pictures have incorrect complex reasoning answers (i.e., the complex reasoning answers are different from the manual answers). Then, a difference analysis is performed between the manual analysis process and the simple reasoning analysis process of the 3 pictures, and a difference analysis is performed between the manual analysis process and the complex reasoning analysis process of the 3 pictures. All difference analyses are input into the first large text model for training to obtain the fourth comprehensive analysis process (i.e., the similarities and differences of all analysis processes from both visual and logical aspects are combined); the simple initial prompt word is combined with the visual and logical analysis reasons to regenerate the second simple error prompt word.

[0081] S413: Input the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, and the second simple incorrect prompt word into the first large text model for training to obtain a simple common prompt word;

[0082] That is, the first text large model analyzes the commonalities of the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, and the second simple incorrect prompt word to obtain a simple common prompt word.

[0083] S414. Input the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, the second simple incorrect prompt word, the simple common prompt word, and the training set into the simple multimodal large model to obtain the answer obtained by reasoning each prompt word for each picture, as well as the analysis process; based on the answer obtained by reasoning each prompt word for each picture, as well as the analysis process, the simple common prompt word is optimized using the first text large model to obtain the simple optimal prompt word.

[0084] Based on the answers obtained by reasoning each prompt word for each picture and the analysis process, the simple common prompt words are optimized using the first large text model to obtain the simple optimal prompt words including:

[0085] Take each picture as the current picture in turn. If the answer obtained by simple common prompt word reasoning for the current picture is wrong, but the answer obtained by the first simple correct prompt word, or the second simple correct prompt word, or the first simple wrong prompt word, or the second simple wrong prompt word is correct, then the analysis process obtained by simple common prompt word reasoning for the current picture and the analysis process obtained by reasoning for the prompt word corresponding to the correct answer of the current picture are input into the first text model for training to obtain the fifth comprehensive analysis process; the simple common prompt word and the fifth comprehensive analysis process are input into the first text model for training to obtain the simple optimal prompt word.

[0086] For example, among 10 pictures, the answers obtained by inferring simple common prompt words for 5 pictures are incorrect. Among these 5 pictures, the answer obtained by inferring the first simple correct prompt word for the first picture is correct, the answer obtained by inferring the second simple correct prompt word for the second picture is correct, the answer obtained by inferring the first simple incorrect prompt word for the third picture is correct, the answer obtained by inferring the second simple incorrect prompt word for the fourth picture is correct, and the answers obtained by inferring the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, and the second simple incorrect prompt word for the fifth picture are all incorrect.

[0087] The analysis process of the first picture obtained by simple common prompt word reasoning, and the analysis process of the first picture obtained by the first simple correct prompt word reasoning; the analysis process of the second picture obtained by simple common prompt word reasoning, and the analysis process of the second picture obtained by the second simple correct prompt word reasoning; the analysis process of the third picture obtained by simple common prompt word reasoning, and the analysis process of the third picture obtained by the first simple wrong prompt word reasoning; the analysis process of the fourth picture obtained by simple common prompt word reasoning, and the analysis process of the fourth picture obtained by the second simple wrong prompt word reasoning are input into the first text model for training to obtain the fifth comprehensive analysis process (that is, comprehensive analysis of all analysis processes from both visual and logical aspects); the simple common prompt words combine the visual and logical defects to regenerate the simple optimal prompt words.

[0088] The complex initial prompt word is optimized through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the complex optimal prompt word, which includes:

[0089] S421. Inputting the simple initial prompt words, training set, and manual calibration questions into a simple multimodal large model for model training, obtaining a simple reasoning answer, a simple analysis process, and a similarity loss value of the simple analysis process for each image; inputting the complex initial prompt words, training set, and manual calibration questions into a complex multimodal large model for model training, obtaining a complex reasoning answer, a complex analysis process, and a similarity loss value of the complex analysis process for each image;

[0090] S422: Based on the complex reasoning answer, the complex analysis process, and the similarity loss value of the complex analysis process for each image, and the simple reasoning answer, the simple analysis process, and the similarity loss value of the simple analysis process for each image, the complex initial prompt word is trained using the first large text model to obtain a first complex correct prompt word, a second complex correct prompt word, a first complex incorrect prompt word, and a second complex incorrect prompt word;

[0091] The S422 includes:

[0092] Input the complex reasoning analysis process and the manual analysis process of the picture whose complex reasoning answer is correct and whose complex analysis process similarity loss value is greater than or equal to the first threshold into the first text large model for training to obtain a sixth comprehensive analysis process; input the complex initial prompt word and the sixth comprehensive analysis process into the first text large model for training to obtain a first complex correct prompt word;

[0093] Input the simple reasoning analysis process of the pictures whose complex reasoning answers are correct and whose complex analysis process similarity loss values ​​are less than the first threshold, the simple reasoning analysis process of the pictures whose simple reasoning answers are correct and whose simple analysis process similarity loss values ​​are greater than or equal to the second threshold, and the complex reasoning analysis process and manual analysis process of the remaining pictures into the first text large model for training to obtain the seventh comprehensive analysis process; input the complex initial prompt word and the seventh comprehensive analysis process into the first text large model for training to obtain the second complex correct prompt word;

[0094] Perform difference analysis on the complex reasoning analysis process of the picture with an incorrect complex reasoning answer and a correct simple reasoning answer, the manual analysis process, and the simple reasoning analysis process respectively; input the difference analysis into the first text large model for training to obtain the eighth comprehensive analysis process; input the complex initial prompt word and the eighth comprehensive analysis process into the first text large model for training to obtain the first complex error prompt word;

[0095] The manual analysis process of the pictures corresponding to the complex reasoning answers being wrong and the simple reasoning errors is subjected to difference analysis with the complex reasoning analysis process and the simple reasoning analysis process respectively; the difference analysis is input into the first text large model for training to obtain the ninth comprehensive analysis process; the simple initial prompt words and the ninth comprehensive analysis process are input into the first text large model for training to obtain the second complex error prompt words.

[0096] S423: Input the first complex correct prompt word, the second complex correct prompt word, the first complex incorrect prompt word, and the second complex incorrect prompt word into the first large text model for training to obtain complex common prompt words;

[0097] S424. Input the first complex correct prompt word, the second complex correct prompt word, the first complex incorrect prompt word, the second complex incorrect prompt word, the complex common prompt word, and the training set into the complex multimodal large model to obtain the answer obtained by reasoning each prompt word for each picture, as well as the analysis process; based on the answer obtained by reasoning each prompt word for each picture, as well as the analysis process, the complex common prompt word is optimized using the first text large model to obtain the complex optimal prompt word.

[0098] S5. Input the simple optimal prompt word, training set, and manual calibration problem into the simple multimodal large model for training to obtain the simple total loss value, the simple answer for each picture, and the simple analysis process for each picture; determine whether the simple total loss value is within the preset threshold range. If so, end the training; otherwise, use the simple optimal prompt word as the simple initial prompt word and the complex optimal prompt word as the complex initial prompt word, repeat S4~S5 until the simple total loss value is within the preset threshold range, stop training, and use the simple optimal prompt word obtained from the last update as the target optimal prompt word.

[0099] In an optional embodiment, the simple total loss value is calculated according to the following formula:

[0100]

[0101] in, is the simple total loss value, is the number of images in the training set, is the similarity loss value of the simple analysis process of the i-th image (calculated based on the simple analysis process and manual analysis process of the i-th image, the same as the calculation method in S3 above), Whether the simple answer of the i-th picture is the same as the manual answer, 1 if they are the same, and 0 if they are different.

[0102] Determine whether the simple total loss value is within the preset threshold range. If so, end the training; otherwise, use the simple optimal prompt word as the simple initial prompt word and the complex optimal prompt word as the complex initial prompt word, repeat S4~S5 until the simple total loss value is within the preset threshold range, stop training, and use the simple optimal prompt word obtained by the last update as the target optimal prompt word.

[0103] After said S5, the following steps are included:

[0104] Each image (i.e., all error cases) corresponding to the simple answer error obtained in the last training (i.e., the simple answer is different from the manual answer) is input into the encoding layer of the visual feature extraction model for feature extraction to obtain the visual features of each image. The visual features of each image are converted into 1*2048-dimensional feature values ​​to obtain the visual feature vector of each image.

[0105] The manual analysis process of each image corresponding to the simple answer error obtained in the last training (i.e., the simple answer is different from the manual answer) is input into the embedding encoder (i.e., beg_embedding encoder) for feature extraction to obtain the text features of each image. The text features of each image are converted into 1*2048-dimensional feature values ​​to obtain the text feature vector of each image;

[0106] Fuse the visual feature vector and text feature vector of the current image to obtain the fused feature vector of the current image;

[0107] The fusion feature vector of the current image is calculated according to the following formula:

[0108]

[0109] in, is the fusion feature vector of the current picture, is the visual feature vector of the current picture, is the text feature vector of the current image.

[0110] The fused feature vectors of all images are clustered to obtain multiple clusters; a simple analysis process of the image corresponding to the cluster center of each cluster is added to the target optimal prompt word.

[0111] In an optional embodiment, the fused feature vectors of all images (i.e., the fused feature vectors based on all error cases) are divided into 5 categories using a k-mean clustering operation to obtain 5 clusters; the cluster centers of the 5 clusters are taken as typical error cases and added to the target optimal prompt word.

[0112] Adding typical error cases to the target optimal prompt words can help us reversely enhance the prompt words, allowing the prompt words to "consider" such situations in advance, thereby guiding the model to avoid errors.

[0113] After obtaining the optimal target cue, the ship image to be analyzed, the manual calibration problem for the ship image to be analyzed (this problem is similar to the training set problem mentioned above), and the optimal target cue are input into a simple multimodal large model for training. This results in an inference answer for the ship image to be analyzed, and the inference analysis process. Guided by the optimal target cue, the accuracy of the inference answer for the ship image to be analyzed can be greatly improved.

[0114] Figure 2 is a structural diagram of a prompt word optimization training system based on multimodal large model supervision provided by an embodiment of the present invention; Figure 2 As shown, the system includes:

[0115] A manual calibration unit 201 is used to construct a training set including manual calibration questions, manual analysis processes and manual answers;

[0116] A generating unit 202 is configured to generate a plurality of initial prompt words from the training set using a complex multimodal large model;

[0117] The screening unit 203 is configured to screen the multiple initial prompt words using a simple multimodal large model to obtain simple initial prompt words; and screen the multiple initial prompt words using a complex multimodal large model to obtain complex initial prompt words;

[0118] The optimization unit 204 is configured to optimize the simple initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain a simple optimal prompt word; and optimize the complex initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain a complex optimal prompt word;

[0119] The judgment unit 205 is used to input the simple optimal prompt word, the training set, and the manual calibration problem into the simple multimodal large model for training, and obtain a simple total loss value, a simple answer for each image, and a simple analysis process for each image; determine whether the simple total loss value is within a preset threshold range, and if so, terminate the training; otherwise, use the simple optimal prompt word as the simple initial prompt word and the complex optimal prompt word as the complex initial prompt word, repeat S4-S5 until the simple total loss value is within the preset threshold range, stop training, and use the simple optimal prompt word obtained from the last update as the target optimal prompt word.

[0120] In an optional embodiment, the system further comprises: an adding unit, configured to:

[0121] Each image corresponding to the simple answer error obtained in the last training is input into the encoding layer of the visual feature extraction model for feature extraction to obtain the visual feature vector of each image;

[0122] The manual analysis process of each image corresponding to the simple answer error obtained in the last training is input into the embedding encoder for feature extraction to obtain the text feature vector of each image;

[0123] Fuse the visual feature vector and text feature vector of the current image to obtain the fused feature vector of the current image;

[0124] The fused feature vectors of all images are clustered to obtain multiple clusters; a simple analysis process of the image corresponding to the cluster center of each cluster is added to the target optimal prompt word.

[0125] The system of the present application corresponds to the above-mentioned method, and the specific implementation of the system will not be repeated here.

[0126] The method of the present invention reduces manual intervention and lowers the cost of prompt word optimization through the collaborative training of complex large models and simple large models; combines manual calibration data with the reasoning process of the complex large model to improve the accuracy and generalization ability of prompt words through dual supervision; introduces dual supervision of question answer loss and reasoning process loss to balance accuracy and reasoning rationality; synchronously optimizes prompt words in the two dimensions of logical analysis and visual recognition to solve the adaptation problem of complex scenarios; analyzes error cases through the fusion of visual and text features, discovers typical difficult examples based on clustering, and reversely optimizes prompt words; the method of the present invention greatly reduces the degree of manual participation and improves the automation level of prompt word generation and optimization; and significantly improves the comprehensive reasoning ability of simple multimodal large models in port navigation monitoring tasks.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A prompt word optimization training method based on multimodal large model supervision, characterized in that: include: S1. Construct an image training set including manual calibration questions, manual analysis process and manual answers; S2, using a complex multimodal large model to generate multiple initial prompt words from the image training set; S3, screening multiple initial prompt words using a simple multimodal large model to obtain a simple initial prompt word; Multiple initial prompt words are screened using a complex multimodal large model to obtain complex initial prompt words; S4, optimizing the simple initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt word; Optimize the complex initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the complex optimal prompt word; S5. Input the simple optimal prompt word and the image training set into the simple multimodal large model for training, obtain a simple total loss value, a simple answer for each image, and a simple analysis process for each image; determine whether the simple total loss value is within a preset threshold range, and if so, terminate the training; Otherwise, the simple optimal prompt word is used as the simple initial prompt word, and the complex optimal prompt word is used as the complex initial prompt word. S4 to S5 are repeated until the simple total loss value is within the preset threshold range. The training is stopped and the simple optimal prompt word obtained by the last update is used as the target optimal prompt word. The S2 includes: Inputting the image training set and the manual calibration problem of each image in the image training set into the complex multimodal large model to obtain keywords of the problem; Obtain reference word templates based on keyword online search; Inputting the reference prompt word template and the manual calibration problems of all images into the complex multimodal large model to obtain multiple initial prompt words; The simple multimodal large model is used to screen multiple initial prompt words to obtain simple initial prompt words including: Each initial prompt word is used as the current initial prompt word in turn; Input the current initial prompt words and picture training set into a simple multimodal large model for training, and obtain a simple reasoning answer for each picture, as well as a simple reasoning analysis process; The similarity loss value of the simple analysis process of each image is calculated based on the manual analysis process and the simple reasoning analysis process of each image; The score corresponding to the current initial prompt word is calculated based on the similarity loss value of the simple analysis process of each image, the simple reasoning answer of each image, and the manual answer of each image; The initial prompt word corresponding to the highest score is used as the simple initial prompt word; The complex initial prompt words are screened using a complex multimodal large model to obtain complex initial prompt words including: Each initial prompt word is used as the current initial prompt word in turn; Input the current initial prompt words and picture training set into the complex multimodal large model for training, and obtain the complex reasoning answer and complex reasoning analysis process for each picture; The similarity loss value of the complex analysis process of each image is calculated based on the manual analysis process and the complex reasoning analysis process of each image; The score corresponding to the current initial prompt word is calculated based on the similarity loss value of the complex analysis process of each image, the complex reasoning answer of each image, and the manual answer of each image; The initial prompt word corresponding to the highest score is used as the complex initial prompt word; The simple initial prompt word is optimized through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt word, which includes: Simple initial prompt words, image training sets, and manual calibration questions are input into a simple multimodal large model for model training, and a simple reasoning answer, a simple analysis process, and a similarity loss value of the simple analysis process are obtained for each image. Complex initial prompt words, image training sets, and manual calibration questions are input into a complex multimodal large model for model training, and a complex reasoning answer, a complex analysis process, and a similarity loss value of the complex analysis process are obtained for each image. Based on the simple reasoning answer, simple analysis process, and similarity loss value of the simple analysis process for each image, and the complex reasoning answer, complex analysis process, and similarity loss value of the complex analysis process for each image, the simple initial prompt word is trained through the first large text model to obtain a first simple correct prompt word, a second simple correct prompt word, a first simple incorrect prompt word, and a second simple incorrect prompt word; Inputting the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, and the second simple incorrect prompt word into the first large text model for training to obtain a simple common prompt word; Input the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, the second simple incorrect prompt word, the simple common prompt word, and the image training set into the simple multimodal large model to obtain the answer obtained by reasoning each prompt word for each image and the analysis process; based on the answer obtained by reasoning each prompt word for each image and the analysis process, the simple common prompt word is optimized using the first text large model to obtain the simple optimal prompt word; The simple total loss value is calculated according to the following formula: Among them, loss is the simple total loss value, N 训练集 is the number of pictures in the picture training set, is the similarity loss value of the simple analysis process of the i-th picture, Whether the simple answer of the i-th picture is the same as the manual answer, 1 if they are the same, and 0 if they are different.

2. The method according to claim 1, characterized in that After said S5, the following steps are included: Each image corresponding to the simple answer error obtained in the last training is input into the encoding layer of the visual feature extraction model for feature extraction to obtain the visual feature vector of each image; The manual analysis process of each image corresponding to the simple answer error obtained in the last training is input into the embedding encoder for feature extraction to obtain the text feature vector of each image; Fuse the visual feature vector and text feature vector of the current image to obtain the fused feature vector of the current image; The fused feature vectors of all images are clustered to obtain multiple clusters; a simple analysis process of the image corresponding to the cluster center of each cluster is added to the target optimal prompt word.

3. The method according to claim 1, characterized in that The simple initial prompt word is trained through the first large text model based on the simple reasoning answer, simple analysis process and similarity loss value of the simple analysis process of each picture, the complex reasoning answer, complex analysis process and similarity loss value of the complex analysis process of each picture, and the first simple correct prompt word, the second simple correct prompt word, the first simple wrong prompt word and the second simple wrong prompt word are obtained. Input the simple reasoning analysis process and the manual analysis process of the image whose simple reasoning answer is correct and whose simple analysis process similarity loss value is greater than or equal to the first threshold into the first text large model for training to obtain a first comprehensive analysis process; input the simple initial prompt word and the first comprehensive analysis process into the first text large model for training to obtain a first simple correct prompt word; The complex reasoning analysis process of the pictures whose simple reasoning answers are correct and whose simple analysis process similarity loss values ​​are less than a first threshold, the complex reasoning analysis process of the pictures whose complex reasoning answers are correct and whose complex analysis process similarity loss values ​​are greater than or equal to a second threshold, and the simple reasoning analysis process and manual analysis process of the remaining pictures are input into the first text large model for training to obtain a second comprehensive analysis process; the simple initial prompt word and the second comprehensive analysis process are input into the first text large model for training to obtain a second simple correct prompt word; The simple reasoning analysis process of the picture with an incorrect simple reasoning answer and a correct complex reasoning answer is subjected to difference analysis with the manual analysis process and the complex reasoning analysis process respectively; the difference analysis is input into the first text large model for training to obtain a third comprehensive analysis process; the simple initial prompt word and the third comprehensive analysis process are input into the first text large model for training to obtain a first simple error prompt word; The manual analysis process of the pictures corresponding to the simple reasoning answers being wrong and the complex reasoning errors is subjected to difference analysis with the simple reasoning analysis process and the complex reasoning analysis process respectively; the difference analysis is input into the first text large model for training to obtain the fourth comprehensive analysis process; the simple initial prompt words and the fourth comprehensive analysis process are input into the first text large model for training to obtain the second simple error prompt words.

4. The method according to claim 1, wherein The similarity loss value of the simple analysis process of each image is calculated according to the following formula: smi_value 简单-人 =beg_embeding(analyse 简单 )·beg_embeding(analyse 人 ) Among them, smi_value 简单-人 is the similarity loss value of the simple analysis process of the current picture, analyze 简单 It is a simple reasoning and analysis process for the current picture. 人 It is the manual analysis process of the current picture, and beg_embeding is the embedding encoder.

5. The method according to claim 1, wherein The score corresponding to the current initial prompt word is calculated according to the following formula: Among them, score is the score corresponding to the current initial prompt word, N 训练集 is the number of pictures in the picture training set, is the similarity loss value of the simple analysis process of the i-th picture, Whether the simple reasoning answer of the i-th picture is the same as the manual answer, 1 if they are the same, and 0 if they are different.

6. A prompt word optimization training system based on multimodal large model supervision, characterized in that: include: Manual calibration unit, used to construct an image training set including manual calibration questions, manual analysis process and manual answers; A generation unit, used to generate multiple initial prompt words from the image training set using a complex multimodal large model; A screening unit, configured to screen multiple initial prompt words using a simple multimodal large model to obtain a simple initial prompt word; Multiple initial prompt words are screened using a complex multimodal large model to obtain complex initial prompt words; An optimization unit, configured to optimize the simple initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain a simple optimal prompt word; Optimize the complex initial prompt word through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the complex optimal prompt word; The judgment unit is used to input the simple optimal prompt word and the image training set into the simple multimodal large model for training, obtain a simple total loss value, a simple answer for each image, and a simple analysis process for each image; determine whether the simple total loss value is within a preset threshold range, and if so, terminate the training; On the contrary, the simple optimal prompt word is used as the simple initial prompt word, and the complex optimal prompt word is used as the complex initial prompt word. The optimization unit and the judgment unit are repeated until the simple total loss value is within the preset threshold range. The training is stopped and the simple optimal prompt word obtained by the last update is used as the target optimal prompt word. The generating unit includes: Inputting the image training set and the manual calibration problem of each image in the image training set into the complex multimodal large model to obtain keywords of the problem; Obtain reference word templates based on keyword online search; Inputting the reference prompt word template and the manual calibration problems of all images into the complex multimodal large model to obtain multiple initial prompt words; The simple multimodal large model is used to screen multiple initial prompt words to obtain simple initial prompt words including: Each initial prompt word is used as the current initial prompt word in turn; Input the current initial prompt words and picture training set into a simple multimodal large model for training, and obtain a simple reasoning answer for each picture, as well as a simple reasoning analysis process; The similarity loss value of the simple analysis process of each image is calculated based on the manual analysis process and the simple reasoning analysis process of each image; The score corresponding to the current initial prompt word is calculated based on the similarity loss value of the simple analysis process of each image, the simple reasoning answer of each image, and the manual answer of each image; The initial prompt word corresponding to the highest score is used as the simple initial prompt word; The complex initial prompt words are screened using a complex multimodal large model to obtain complex initial prompt words including: Each initial prompt word is used as the current initial prompt word in turn; Input the current initial prompt words and picture training set into the complex multimodal large model for training, and obtain the complex reasoning answer and complex reasoning analysis process for each picture; The similarity loss value of the complex analysis process of each image is calculated based on the manual analysis process and the complex reasoning analysis process of each image; The score corresponding to the current initial prompt word is calculated based on the similarity loss value of the complex analysis process of each image, the complex reasoning answer of each image, and the manual answer of each image; The initial prompt word corresponding to the highest score is used as the complex initial prompt word; The simple initial prompt word is optimized through the simple multimodal large model, the complex multimodal large model, and the first text large model to obtain the simple optimal prompt word, which includes: Simple initial prompt words, image training sets, and manual calibration questions are input into a simple multimodal large model for model training, and a simple reasoning answer, a simple analysis process, and a similarity loss value of the simple analysis process are obtained for each image. Complex initial prompt words, image training sets, and manual calibration questions are input into a complex multimodal large model for model training, and a complex reasoning answer, a complex analysis process, and a similarity loss value of the complex analysis process are obtained for each image. Based on the simple reasoning answer, simple analysis process, and similarity loss value of the simple analysis process for each image, and the complex reasoning answer, complex analysis process, and similarity loss value of the complex analysis process for each image, the simple initial prompt word is trained through the first large text model to obtain a first simple correct prompt word, a second simple correct prompt word, a first simple incorrect prompt word, and a second simple incorrect prompt word; Inputting the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, and the second simple incorrect prompt word into the first large text model for training to obtain a simple common prompt word; Input the first simple correct prompt word, the second simple correct prompt word, the first simple incorrect prompt word, the second simple incorrect prompt word, the simple common prompt word, and the image training set into the simple multimodal large model to obtain the answer obtained by reasoning each prompt word for each image and the analysis process; based on the answer obtained by reasoning each prompt word for each image and the analysis process, the simple common prompt word is optimized using the first text large model to obtain the simple optimal prompt word; The simple total loss value is calculated according to the following formula: Among them, loss is the simple total loss value, N 训练集 is the number of pictures in the picture training set, is the similarity loss value of the simple analysis process of the i-th picture, Whether the simple answer of the i-th picture is the same as the manual answer, 1 if they are the same, and 0 if they are different.

7. The system according to claim 6, characterized in that Also includes: Add units for: Each image corresponding to the simple answer error obtained in the last training is input into the encoding layer of the visual feature extraction model for feature extraction to obtain the visual feature vector of each image; The manual analysis process of each image corresponding to the simple answer error obtained in the last training is input into the embedding encoder for feature extraction to obtain the text feature vector of each image; Fuse the visual feature vector and text feature vector of the current image to obtain the fused feature vector of the current image; The fused feature vectors of all images are clustered to obtain multiple clusters; a simple analysis process of the image corresponding to the cluster center of each cluster is added to the target optimal prompt word.

Images