A Multimodal Data Construction Method Based on VLM-LLM Collaborative Verification and Instruction-Driven Approach

Abstract

This application relates to a multimodal data construction method based on VLM-LLM collaborative verification and instruction-driven approach. Through the collaborative verification design mechanism of VLM and LLM, the entire process is automated, reducing manual annotation steps and improving the efficiency of data construction for complex scenarios. Furthermore, through the multiple cross-verification mechanism of LLM, self-correction capability is established, significantly reducing the error annotation rate and greatly reducing the cost of subsequent corrections. By automatically scheduling expert identities in LLM, excellent domain transfer capability is achieved, effectively solving the problem of dependence on human experts. It can automatically adapt to new image domains without human intervention, improving the flexibility and adaptability of multimodal data construction. Compared with traditional manual annotation schemes and semi-automated annotation schemes, this invention overcomes the technical disadvantages of insufficient semantic understanding, modal fragmentation, and poor dynamic adaptability, as well as the disadvantages of high cost, large quality fluctuations, and low iteration efficiency, providing a solution for the efficient production of high-quality multimodal datasets.

Description

Technical Field

[0001] This application relates to the field of large model application technology, and in particular to a method, apparatus, system, computer device and computer-readable storage medium for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven methods. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models are being used more and more widely in various fields. However, the construction of high-quality multimodal datasets has always been a key bottleneck restricting the improvement of model performance. At present, the construction of multimodal datasets mainly relies on three methods: manual annotation, semi-automatic annotation, and automatic generation of large models.

[0003] In traditional manual annotation schemes, annotators need to understand the image content and generate corresponding text-image descriptions and question-and-answer pairs. However, this method still requires a lot of manual intervention, which is not only costly, but also the annotation quality is greatly affected by the professional level of the annotators, resulting in problems such as insufficient semantic understanding and modal fragmentation.

[0004] Semi-automated annotation schemes improve efficiency to some extent by introducing automated tools to assist manual annotation; however, these methods still have technical disadvantages such as rough generated content and lack of multimodal verification, and the cost of later correction is high and the domain transfer is difficult.

[0005] In recent years, with the development of large-scale model technology, automated data construction methods based on large-scale models have gradually emerged. These methods use large-scale artificial intelligence models to perform deep learning and semantic understanding of expert knowledge text features, generating question-and-answer pairs. However, existing large-scale model solutions still face a series of technical problems: First, the separation of description and reasoning leads to a lack of deep correlation between the generated textual descriptions and the question-and-answer content; second, insufficient data diversity makes it difficult to meet the needs of different scenarios and users; third, the lack of effective verification mechanisms easily leads to the risk of error propagation, with the model illusion problem being particularly prominent.

[0006] Furthermore, existing technologies generally struggle to achieve fine-grained control over the data generation process, resulting in a lack of diversity in style, depth, and structure among the generated multimodal datasets. Therefore, how to efficiently and accurately construct multimodal datasets for training and optimizing models has become a pressing technical problem to be solved. Summary of the Invention

[0007] This application provides a method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven methods, the method comprising:

[0008] The VLM (Visual-Language Model) model is used to obtain multiple sets of image and text descriptions of the image to be processed.

[0009] The LLM (Large Language Model) model is used to perform logical judgments on the text and image descriptions to obtain target text and image descriptions that conform to logic and common sense.

[0010] Using an instruction-driven mechanism, based on the target graphic description and external domain knowledge base, controllable QA generation is performed to obtain multiple sets of detailed question-and-answer pairs with different styles, depths, and structures.

[0011] Based on the graphic description, the external domain knowledge base, and multiple sets of detailed question-and-answer pairs, cross-validation is performed to obtain target detailed question-and-answer pairs that meet preset requirements.

[0012] Based on the target detailed question-answer pairs, the target image and text description, and the image to be processed, a multimodal dataset is constructed for model training and optimization.

[0013] In some embodiments, the LLM model is used to perform logical judgments on the graphic descriptions to obtain target graphic descriptions that conform to logic and common sense, including:

[0014] The input consists of multiple sets of image and text descriptions. Based on the second Prompt instruction, the LLM model determines whether the semantic similarity between any set of image and text descriptions is greater than a preset similarity threshold. If so, the set of image and text descriptions is established as the target image and text description; otherwise, the set of image and text descriptions is discarded.

[0015] In some embodiments, the second Prompt is configured to utilize the language analysis expert capabilities of the LLM model to construct content similarity analysis logic, and obtain abnormal text and image descriptions caused by model illusions through similarity analysis and threshold judgment.

[0016] In some embodiments, an instruction-driven mechanism is used to generate controllable QA based on the target graphic description and an external domain knowledge base, resulting in multiple sets of detailed question-and-answer pairs with different styles, depths, and structures, including:

[0017] The LLM model is instructed by the third prompt to perform domain identification on the target image and text description using MOE expert capabilities and match the domain expert role corresponding to the target image and text description.

[0018] Leveraging the capabilities of the domain expert role and the external domain knowledge base, and based on the Prompt instruction and the target textual description, multiple sets of detailed question-and-answer pairs are constructed, wherein...

[0019] During the construction of the detailed question-answer pairs, the third Prompt instructs the LLM model to dynamically adjust the question-answer style, question-answer depth, and question-answer format.

[0020] In some embodiments, the method further includes:

[0021] The detailed question-and-answer pairs, target graphic descriptions, and external domain knowledge base data are input into the LLM model;

[0022] The LLM model is instructed by the fourth Prompt to perform secondary cross-validation and fact verification on the input data in the role of a logical reasoning expert. If the verification passes, the detailed question-and-answer pair is output.

[0023] In some embodiments, constructing the multimodal dataset based on the target detail question-answer pair, the target image-text description, and the image to be processed includes:

[0024] A candidate pool of image and text descriptions is constructed based on the target image and text descriptions, and a candidate pool of question and answer is constructed based on the target detail question and answer pairs;

[0025] A set of image and text descriptions is randomly selected from the image and text description candidate pool through a random selection mechanism, and a set of detailed question and answer pairs is randomly selected from the question and answer candidate pool.

[0026] The randomly selected image and text descriptions are combined with detailed question-and-answer pairs, and the multimodal dataset is constructed based on multiple combination results.

[0027] In some embodiments, obtaining multiple sets of image and text descriptions of the image to be processed through a VLM model includes:

[0028] The image to be processed is preprocessed, and the VLM model is instructed by the first Prompt to process the image in parallel to obtain multiple sets of image and text descriptions.

[0029] Specifically, global semantic extraction is performed using the first VLM model, and local detail semantic extraction is performed using the second VLM model, resulting in two sets of differentiated initial image and text descriptions.

[0030] The initial graphic description is combined to obtain the graphic description.

[0031] Secondly, embodiments of this application provide a multimodal data construction system based on VLM-LLM collaborative verification and instruction-driven architecture. The system includes an acquisition module, a construction module, and a combination module.

[0032] The acquisition module is used to acquire multiple sets of image and text descriptions of the image to be processed through the VLM model;

[0033] The construction module is used to perform logical judgment on the graphic description through the LLM model to obtain the target graphic description that conforms to logic and common sense.

[0034] Furthermore, by utilizing an instruction-driven mechanism, based on the target graphic description and external domain knowledge base, controllable QA generation is performed to obtain multiple sets of detailed question-and-answer pairs with different styles, depths, and structures.

[0035] Furthermore, based on the graphic description, the external domain knowledge base, and multiple sets of detailed question-and-answer pairs, a cross-validation is performed to obtain target detailed question-and-answer pairs that meet preset requirements;

[0036] The combined module is used to construct a multimodal dataset for model training and optimization based on the target detailed question-answer pair, the target image and text description, and the image to be processed.

[0037] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0039] Compared to related technologies, the multimodal data construction method based on VLM-LLM collaborative verification and instruction-driven approach provided in this application reduces manual annotation steps and improves the efficiency of complex scene data construction through the collaborative verification design mechanism of VLM and LLM. Furthermore, by establishing a self-correcting capability through the multiple cross-verification mechanism of LLM, the error annotation rate is significantly reduced, greatly decreasing the cost of subsequent corrections. By automatically scheduling expert identities in LLM, excellent domain transfer capability is achieved, effectively solving the problem of dependence on human experts. It can automatically adapt to new image domains without human intervention, improving the flexibility and adaptability of multimodal dataset construction. Compared with traditional manual annotation schemes and semi-automated annotation schemes, this invention overcomes the technical disadvantages of insufficient semantic understanding, modal fragmentation, and poor dynamic adaptability, as well as the disadvantages of high cost, large quality fluctuations, and low iteration efficiency, providing a solution for the efficient production of high-quality multimodal datasets. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 This is a flowchart of a method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven methods according to an embodiment of this application;

[0042] Figure 2 This is a flowchart of another method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven implementation according to an embodiment of this application;

[0043] Figure 3 This is a structural block diagram of a multimodal data construction system based on VLM-LLM collaborative verification and instruction-driven implementation, according to an embodiment of this application.

[0044] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0046] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0047] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] This application provides a method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven approach. Figure 1 This is a flowchart of a method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven implementation according to an embodiment of this application, such as... Figure 1 As shown, this method constructs a high-quality multimodal dataset through the collaborative work of a Visual Language Model (VLM) and a Large Language Model (LLM), combined with an instruction-driven mechanism. Specific implementation steps include:

[0050] S101: Obtain multiple sets of image and text descriptions of the image to be processed through the VLM model.

[0051] The images to be processed undergo preprocessing, including image resizing, color balance adjustment, and noise removal. The resolution of the preprocessed images is uniformly adjusted to 1024×1024 pixels to ensure consistency in subsequent processing. Using the first Prompt instruction to the VLM model, multiple sets of image and text descriptions are obtained through parallel processing of the preprocessed images.

[0052] Optionally, an example of the content of the first Prompt is: "Please describe the content of this image in detail, including the main object, scene environment, color features, spatial relationships and possible activities."

[0053] Two different VLM models are used for parallel processing: the first VLM model (such as Gemma3) is used for global semantic extraction to generate descriptions that focus on the overall scene and main objects; the second VLM model (such as InternVL2.5) is used for local detail semantic extraction to generate descriptions that focus on detailed features and attributes; the two models respectively produce two sets of differentiated initial image and text descriptions.

[0054] In one exemplary embodiment:

[0055] Example of global semantic description generated by the first VLM model: "This is a photo of an outdoor park. In the center of the picture, a young woman is sitting on a bench reading a book. There are lush green plants and open lawns around her. The sky is clear and the sunlight shines through the leaves, creating dappled light and shadow."

[0056] Example of local detail description generated by the second VLM model: "The woman in the picture is wearing a light blue dress, with her long brown hair tied in a ponytail, and holding a book with a red cover. The bench is made of wood and is dark brown. Three tall oak trees can be seen in the background, and there is a small fountain in the distance on the right. The ground is paved with gray stone slabs."

[0057] Furthermore, in this application, the two initial sets of graphic descriptions are combined to obtain a complete graphic description. Optionally, the combination method adopts the principle of semantic complementarity to ensure that both global information and local details are preserved.

[0058] The final example of the image description is: "This is a photo of an outdoor park. In the center of the picture, a young woman is sitting on a bench reading a book. She is wearing a light blue dress, her long brown hair is tied in a ponytail, and she is holding a book with a red cover. The bench is made of wood and is dark brown. There is lush green vegetation and an open lawn area around her. Three tall oak trees can be seen in the background, and there is a small fountain in the distance on the right. The ground is paved with gray flagstones. The sky is clear, and sunlight filters through the leaves, creating dappled light and shadow."

[0059] In the specific implementation, the above process is repeated 3-5 times for each image to be processed, using slightly different variations of the prompt words each time to obtain multiple sets of differentiated image and text descriptions. These descriptions constitute the basic dataset for subsequent processing.

[0060] S102 uses the LLM model to perform logical judgments on the graphic descriptions to obtain target graphic descriptions that conform to logic and common sense.

[0061] The input consists of multiple sets of image and text descriptions. Based on the second Prompt instruction, the LLM model determines whether the semantic similarity between any set of image and text descriptions is greater than a preset similarity threshold. If so, the set of image and text descriptions is established as the target image and text description; otherwise, the set of image and text descriptions is discarded.

[0062] The second Prompt is set to utilize the language analysis expert capabilities of the LLM model to construct content similarity analysis logic, and obtain abnormal text and image descriptions caused by model illusions through similarity analysis and threshold judgment.

[0063] The specific example of the second Prompt is: "As a language analysis expert, please analyze the following multiple sets of descriptions about the same image. Calculate the semantic similarity between them and identify inconsistent descriptions that may be caused by model hallucinations. If the semantic similarity between any pair of descriptions is less than 0.75 (out of 1), please mark the descriptions that may be hallucinating and explain why."

[0064] It should be noted that the preset similarity threshold is set to 0.75. This value was determined experimentally and effectively balances accuracy and diversity. LLM models (such as GPT-4) receive multiple sets of image and text descriptions and calculate the semantic similarity between them. The calculation method is based on a comprehensive score of three dimensions: key entity recognition, attribute matching, and scene consistency.

[0065] Furthermore, for each set of text and image descriptions, the LLM model extracts core entities (such as "young woman," "bench," "park"), attributes (such as "light blue dress," "red-covered book"), and scene elements (such as "oak tree," "fountain," "cobblestone path"), and cross-references them with other descriptions. If a set of descriptions contains key elements that are inconsistent with other descriptions (such as mentioning "man" instead of "woman," or "café" instead of "park"), it is marked as a description that may contain hallucinations.

[0066] Finally, the LLM model outputs semantic similarity analysis results and assigns a credibility score to each group of descriptions. Descriptions with a similarity score greater than a preset threshold are retained as target text / image descriptions, while descriptions that do not meet the requirements are discarded. This step effectively filters out potentially misleading content generated by the VLM model, improving the data quality for subsequent processing.

[0067] S103 utilizes an instruction-driven mechanism to generate controllable QA based on target graphic descriptions and external domain knowledge bases, resulting in multiple sets of detailed question-and-answer pairs with different styles, depths, and structures.

[0068] Specifically, the LLM model is instructed by the third prompt, and the MOE expert capabilities are used to perform domain identification on the target graphic description and match the domain expert role corresponding to the target graphic description.

[0069] The third prompt is exemplified as follows: "Please analyze the following image descriptions, identify the main areas they involve (such as natural scenery, urban architecture, human activities, works of art, etc.), and, as an expert in that area, generate 10 sets of in-depth question-and-answer pairs based on the description content. The questions and answers should cover different cognitive levels (from basic descriptions to in-depth analysis) and include various question-and-answer formats (such as open-ended, closed-ended, comparative, etc.)."

[0070] In the implementation phase, the LLM model first analyzes the theme and content of the target graphic description to identify its domain. For example, the description of the aforementioned park scene identifies two main domains: "urban public space" and "leisure activities." Based on this identification, the most suitable expert role is selected from a pre-set expert role database, such as "urban planning expert" and "leisure sociology expert." External domain knowledge bases are then accessed, including relevant knowledge on urban park design principles, public space usage patterns, and the psychology of leisure activities.

[0071] Furthermore, leveraging the capabilities of domain experts and external domain knowledge bases, multiple sets of detailed question-answer pairs are constructed based on Prompt prompts and target textual descriptions. During the construction of these detailed question-answer pairs, a third Prompt instructs the LLM model to dynamically adjust its question-answering style, depth, and format.

[0072] Optional: The question-and-answer style includes four basic styles: academic, educational, conversational, and exploratory.

[0073] Academic Q&A focuses on professional terminology and theoretical frameworks; educational Q&A emphasizes knowledge transfer and concept explanation; conversational Q&A simulates natural communication; and exploratory Q&A encourages creative thinking and hypothesis analysis.

[0074] The depth of question-and-answer sessions is divided into four levels: foundational, comprehension, application, and analysis. The foundational level focuses on directly observable facts; the comprehension level explores the principles behind phenomena; the application level discusses the practical application of knowledge; and the analysis level involves critical thinking and multi-faceted evaluation.

[0075] Optionally, the question-and-answer format includes five types: open-ended, closed-ended, comparative, hypothetical, and evaluative. Open-ended questions have no fixed answers; closed-ended questions have explicit answers; comparative questions require comparative analysis; hypothetical questions explore possibilities; and evaluative questions require value judgments.

[0076] Specifically, an example of the generated question-and-answer pairs is as follows:

[0077] 1. Basic level, closed-campus, educational:

[0078] Question: What is the bench the woman in the picture is sitting on made of?

[0079] A: "The bench in the picture is made of wood and is dark brown. This material is common in park environments because wood has a natural beauty, harmonizes with the surrounding environment, and provides a comfortable seating experience."

[0080] 2. Understanding levels, open-ended, academic style:

[0081] Q: "From an urban planning perspective, what design considerations are there for the placement of these park benches?"

[0082] A: "From an urban planning perspective, the placement of park benches usually takes into account a number of factors: first, the visual landscape, with the bench located in a tree-lined area, providing a good view; second, functional zoning, situated in a transitional zone between activity areas and quiet areas; third, microclimate considerations, with the shade of trees providing natural shading; and finally, social distancing design, providing privacy without complete isolation. This arrangement reflects the 'human-scale' design philosophy, emphasizing user experience."

[0083] 3. Application level, comparative, dialogic:

[0084] Q: "In the digital age, what are the differences in experience between reading a physical book in a park, as shown in the picture, and reading on an electronic device?"

[0085] A: "Reading a physical book in a park is indeed a very different experience compared to using an electronic device. Reading a physical book allows you to be completely immersed in the natural environment, without screen glare or worries about running out of battery. The tactile experience and the sound of turning pages enhance the sensory enjoyment, making it easier to blend into the surroundings. While electronic devices are portable and can carry a lot of books, they are prone to distractions like social media, and the screen may be difficult to read in direct sunlight. Which reading method do you prefer?"

[0086] 4. Analytical levels, evaluative, and exploratory approaches:

[0087] Q: "How would you evaluate the potential impact of the public space design shown in the diagram on promoting the mental health of urban residents?"

[0088] A: "The impact of the public space design shown in the image on the mental health of urban residents can be assessed from multiple dimensions: First, the presence of green vegetation provides a 'natural recovery' environment, which studies have shown can reduce stress hormone levels; second, the mixed layout of open and semi-private spaces meets different social needs, supporting both solitude and interaction; third, multi-sensory experiences (visual aesthetics, natural sounds, fresh air) promote psychological relaxation; and finally, facilities supporting static activities such as reading encourage deep thinking and mindfulness experiences. This design embodies the concept of a 'healing landscape' and may be an effective buffer against the stresses of urban life. It is worth considering how to make these spaces more inclusive, serving people of different ages and abilities."

[0089] Understandably, in this scheme, 20-30 sets of question-answer pairs with different styles, depths and forms are generated for each image, forming a rich pool of detailed question-answer candidates.

[0090] S104, based on the text and image description, external domain knowledge base and multiple sets of detailed question-answer pairs, performs cross-validation to obtain target detailed question-answer pairs that meet the preset requirements.

[0091] Specifically, detailed question-and-answer pairs, target graphic descriptions, and external domain knowledge base data are input into the LLM model. The fourth Prompt instructs the LLM model to perform secondary cross-validation and factual verification on the input data, acting as a logical reasoning expert. If the verification passes, detailed question-and-answer pairs are output.

[0092] An example of the fourth Prompt is: "As a logic reasoning expert, please conduct a rigorous factual and logical review of the following question-and-answer pairs. Check whether the content of the questions and answers is consistent with the image description, conforms to common sense and domain knowledge, and whether there are any logical contradictions. For each question-and-answer pair, please give a 'pass' or 'fail' assessment and explain the reasons. For the failed questions and answers, please provide suggestions for correction."

[0093] The LLM model performs the following verification process:

[0094] 1. Fact Consistency Check: Verify whether the factual statements in the question-and-answer pair are consistent with the target image / text description. For example, if the question-and-answer pair mentions "the woman is wearing a red dress," while the image / text description says "a light blue dress," then it is marked as inconsistent.

[0095] 2. Logical coherence check: Verify whether there are logical contradictions within the question-and-answer pair. For example, if the question asks "Why did this lady choose to read indoors?" while the image description clearly shows an outdoor scene, it is marked as a logical error.

[0096] 3. Knowledge Accuracy Check: Verifies whether the professional knowledge statements in the question-and-answer pairs are consistent with an external domain knowledge base. For example, if an answer mentions "this park design style belongs to the Baroque style," the system will check the knowledge base for information on park design styles to verify this.

[0097] 4. Reasoning Reasoning Check: Verify whether the reasoning process in the question-and-answer pair is reasonable. For example, if the answer directly infers "she is a literature major" from "the woman is reading," it is marked as over-reasoning.

[0098] For each question-answer pair, the LLM model provides an evaluation result and reasoning. Question-answer pairs that pass the validation are marked as target detail question-answer pairs; those that fail are discarded or re-validated after adjustments based on correction suggestions.

[0099] Verification example:

[0100] The correct answer is: "The fountain in the picture is in the Baroque style, which suggests that the park may have been built in the 17th or 18th century."

[0101] Verification result: Failed. Reason:

[0102] 1) The description of the picture and text only mentions a small fountain in the distance on the right, without specifying the style;

[0103] 2) Inferring the entire age of the park from the style of a single fountain is a logical leap;

[0104] 3) Lack of sufficient visual detail to support the Baroque style judgment. Suggested correction: Explicitly label speculative content as assumptions, or simply describe the visible fountain features without making a style judgment.

[0105] It is understandable that this secondary cross-validation process is used to select high-quality target detail question-and-answer pairs to ensure their factual accuracy, logical coherence, and knowledge reliability.

[0106] S105 is a multimodal dataset for model training and optimization, based on target detail question-answer pairs, target image and text descriptions, and the image to be processed.

[0107] A candidate pool of target image and text descriptions is constructed based on the target image and text descriptions, and a candidate pool of target detail question-and-answer pairs is constructed based on the target detail question-and-answer pairs. The image and text description candidate pool contains all target image and text descriptions that pass the logical judgment, and the question-and-answer candidate pool contains all target detail question-and-answer pairs that pass the cross-validation.

[0108] Through a random selection mechanism, a set of image and text descriptions is randomly selected from the image and text description candidate pool, and a set of detailed question and answer pairs is randomly selected from the question and answer candidate pool.

[0109] Randomly selected image and text descriptions are combined with detailed question-and-answer pairs, and a multimodal dataset is constructed based on multiple combination results. Specifically, in this embodiment, the combination methods include, but are not limited to:

[0110] Basic combination: Directly associate images, text descriptions, and question-and-answer pairs to form triplet data.

[0111] Hierarchical combination: Using images as the root node, text and image descriptions as the first-level child nodes, and question-answer pairs as the second-level child nodes, a tree-like data structure is formed.

[0112] Interactive Combination: Integrating text and image descriptions into question-and-answer pairs to form context-enhanced question-and-answer data.

[0113] For each image to be processed, this scheme generates 5-10 different combinations to increase data diversity. Optionally, the final multimodal dataset includes the following fields: Image ID: unique identifier, Image Path: original image storage location, Image Feature Vector: pre-computed image feature representation, Image Description: selected target image description, Question-Answer Pair Set: selected target detailed question-answer pair set, and Metadata: including data generation time, model version used, validation status, etc.

[0114] The completed multimodal dataset can be used for model training and optimization for various downstream tasks, including but not limited to: multimodal dialogue system training, visual question answering model optimization, image description generation model training, cross-modal retrieval system development, multimodal understanding ability assessment. The dataset also includes quality scores and difficulty level annotations, which facilitates the selection of appropriate training samples for different training stages.

[0115] also, Figure 2 This is a flowchart of another method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven methods according to an embodiment of this application.

[0116] Through steps S101 to S105 above, the multimodal data construction method based on VLM-LLM collaborative verification and instruction-driven approach provided in this application reduces manual annotation steps and improves the efficiency of complex scene data construction by utilizing the collaborative verification design mechanism of VLM and LLM. Furthermore, through the multiple cross-verification mechanism of LLM, self-correction capability is established, significantly reducing the error annotation rate and greatly reducing subsequent correction costs. By automatically scheduling expert identities in LLM, excellent domain transfer capability is achieved, effectively solving the problem of dependence on human experts. It can automatically adapt to new image domains without human intervention, improving the flexibility and adaptability of multimodal dataset construction. Compared with traditional manual annotation schemes and semi-automatic annotation schemes, this invention overcomes the technical disadvantages of insufficient semantic understanding, modal fragmentation, and poor dynamic adaptability, as well as the disadvantages of high cost, large quality fluctuations, and low iteration efficiency, providing a solution for the efficient production of high-quality multimodal datasets.

[0117] This application also provides a multimodal data construction system based on VLM-LLM collaborative verification and instruction-driven operation. Figure 3 This is a structural block diagram of a multimodal data construction system based on VLM-LLM collaborative verification and instruction-driven implementation, according to an embodiment of this application. Figure 3 As shown, the system includes: an acquisition module 30, a construction module 31, and a combination module 32, wherein,

[0118] The acquisition module 30 is used to acquire multiple sets of image and text descriptions of the image to be processed through the VLM model;

[0119] The process involves preprocessing the images to be processed, including image resizing, color balance adjustment, and noise removal. The resolution of the preprocessed images is uniformly adjusted to 1024×1024 pixels to ensure consistency in subsequent processing. Using a first Prompt to instruct the VLM model, multiple sets of image-text descriptions are generated based on the preprocessed images and processed in parallel. The first Prompt reads: "Please describe the content of this image in detail, including the main object, scene environment, color features, spatial relationships, and possible activities."

[0120] In this process, the system employs two different VLM models for parallel processing: a first VLM model (e.g., CLIP-ViT-L-14) performs global semantic extraction to generate descriptions focusing on the overall scene and main objects; a second VLM model (e.g., BLIP-2) performs local detail semantic extraction to generate descriptions focusing on detailed features and attributes. These two models yield two sets of differentiated initial image and text descriptions.

[0121] The construction module 31 is used to: logically determine the text and image descriptions through the LLM model to obtain target text and image descriptions that conform to logic and common sense; and, using an instruction-driven mechanism, generate controllable QA based on the target text and image descriptions and external domain knowledge bases to obtain multiple sets of detailed question-and-answer pairs with different styles, depths and structures; and, perform cross-validation based on the text and image descriptions, external domain knowledge bases and multiple sets of detailed question-and-answer pairs to obtain target detailed question-and-answer pairs that meet preset requirements.

[0122] The input consists of multiple sets of image and text descriptions. Based on the second Prompt instruction, the LLM model determines whether the semantic similarity between any set of image and text descriptions is greater than a preset similarity threshold. If so, the set of image and text descriptions is established as the target image and text description; otherwise, the set of image and text descriptions is discarded.

[0123] This module uses a third prompt to instruct the LLM model to perform domain identification on the target image / text description, matching it with the corresponding domain expert role. It also integrates an external domain knowledge base, utilizing the capabilities of the domain expert roles and the external knowledge base, and constructs multiple sets of detailed question-and-answer pairs based on the prompt instructions and the target image / text description. During the construction of these detailed question-and-answer pairs, the third prompt instructs the LLM model to dynamically adjust the question-and-answer style, depth, and format.

[0124] The detailed question-and-answer pairs, target graphic descriptions, and external domain knowledge base data are then input into the LLM model. The fourth Prompt instructs the LLM model to perform secondary cross-validation and factual verification on the input data, acting as a logical reasoning expert. If the verification passes, detailed question-and-answer pairs are output.

[0125] The combination module 32 is used to construct a multimodal dataset for model training and optimization based on target detail question-answer pairs, target image and text descriptions, and the image to be processed.

[0126] A candidate pool of target image and text descriptions is constructed based on the target image and text descriptions, and a candidate pool of target detail question-and-answer pairs is constructed based on the target detail question-and-answer pairs. The image and text description candidate pool contains all target image and text descriptions that pass the logical judgment, and the question-and-answer candidate pool contains all target detail question-and-answer pairs that pass the cross-validation.

[0127] A random selection mechanism is used to randomly select a set of image and text descriptions from the image and text description candidate pool, and a set of detailed question-and-answer pairs from the question-and-answer candidate pool. The randomly selected image and text descriptions and detailed question-and-answer pairs are then combined, and a multimodal dataset is constructed based on multiple combination results. The random selection mechanism employs a weighted random sampling algorithm, with weights based on the completeness score of the description and the quality score of the question-and-answer pairs.

[0128] This system achieves full-process automation through a collaborative verification design mechanism of VLM and LLM, reducing manual annotation steps and improving the efficiency of constructing complex scene data. Furthermore, the LLM's multiple cross-checking mechanism establishes self-correction capabilities, significantly reducing the error annotation rate and greatly decreasing post-correction costs. By automatically scheduling expert identities in the LLM, excellent domain transfer capabilities are achieved, effectively solving the problem of reliance on human experts. It can automatically adapt to new image domains without human intervention, improving the flexibility and adaptability of multimodal dataset construction. Compared with traditional manual and semi-automated annotation schemes, this invention overcomes the technical disadvantages of insufficient semantic understanding, modal fragmentation, and poor dynamic adaptability, as well as the disadvantages of high cost, large quality fluctuations, and low iteration efficiency, providing a solution for the efficient production of high-quality multimodal datasets.

[0129] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, this electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides the environment for the operating system, the computer programs are executed by the processor to implement a VLM-LLM-based co-verification and instruction-driven multimodal data construction method, and the database stores data.

[0130] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. Specifically, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the VLM-LLM co-verification and instruction-driven multimodal data construction methods described in the above embodiments.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for constructing multimodal data based on VLM-LLM collaborative verification and instruction-driven approach, characterized in that, The method includes: The VLM model is used to obtain multiple sets of image and text descriptions of the image to be processed. By using the LLM model, the graphic description is logically judged to obtain the target graphic description that conforms to logic and common sense. Using an instruction-driven mechanism, based on the target image and text description and an external domain knowledge base, controllable QA generation is performed to obtain multiple sets of detailed question-and-answer pairs with different styles, depths, and structures, specifically including: The LLM model is instructed by the third prompt to perform domain identification on the target image and text description using MOE expert capabilities and match the domain expert role corresponding to the target image and text description. Leveraging the capabilities of the domain expert role and the external domain knowledge base, and based on the Prompt instruction and the target textual description, multiple sets of detailed question-and-answer pairs are constructed, wherein... During the construction of the detailed question-answer pairs, the third Prompt instructs the LLM model to dynamically adjust the question-answer style, question-answer depth, and question-answer format. Based on the graphic description, the external domain knowledge base, and multiple sets of detailed question-and-answer pairs, cross-validation is performed to obtain target detailed question-and-answer pairs that meet preset requirements. Based on the target detailed question-answer pairs, the target image and text description, and the image to be processed, a multimodal dataset is constructed for model training and optimization.

2. The method according to claim 1, characterized in that, By using the LLM model, logical judgment is performed on the graphic description to obtain target graphic descriptions that conform to logic and common sense, including: The input consists of multiple sets of image and text descriptions. Based on the second Prompt instruction, the LLM model determines whether the semantic similarity between any set of image and text descriptions is greater than a preset similarity threshold. If so, the set of image and text descriptions is established as the target image and text description; otherwise, the set of image and text descriptions is discarded.

3. The method according to claim 2, characterized in that, The second Prompt is configured to utilize the language analysis expert capabilities of the LLM model to construct content similarity analysis logic, and obtain abnormal text and image descriptions caused by model illusions through similarity analysis and threshold judgment.

4. The method according to claim 1, characterized in that, The method further includes: The detailed question-and-answer pairs, target graphic descriptions, and external domain knowledge base data are input into the LLM model; The LLM model is instructed by the fourth Prompt to perform secondary cross-validation and fact verification on the input data in the role of a logical reasoning expert. If the verification passes, the detailed question-and-answer pair is output.

5. The method according to claim 1, characterized in that, Based on the target detailed question-and-answer pairs, the target image-text description, and the image to be processed, the construction of the multimodal dataset includes: A candidate pool of image and text descriptions is constructed based on the target image and text descriptions, and a candidate pool of question and answer is constructed based on the target detail question and answer pairs; A set of image and text descriptions is randomly selected from the image and text description candidate pool through a random selection mechanism, and a set of detailed question and answer pairs is randomly selected from the question and answer candidate pool. The randomly selected image and text descriptions are combined with detailed question-and-answer pairs, and the multimodal dataset is constructed based on multiple combination results.

6. The method according to claim 1, characterized in that, Using the VLM model, multiple sets of image and text descriptions of the image to be processed are obtained, including: The image to be processed is preprocessed, and the VLM model is instructed by the first Prompt to process the image in parallel to obtain multiple sets of image and text descriptions. Specifically, global semantic extraction is performed using the first VLM model, and local detail semantic extraction is performed using the second VLM model, resulting in two sets of differentiated initial image and text descriptions. The initial graphic description is combined to obtain the graphic description.

7. A multimodal data construction system based on VLM-LLM collaborative verification and instruction-driven operation, characterized in that, The system includes an acquisition module, a construction module, and a combination module, wherein, The acquisition module is used to acquire multiple sets of image and text descriptions of the image to be processed through the VLM model; The construction module is used to perform logical judgment on the graphic description through the LLM model to obtain the target graphic description that conforms to logic and common sense. Furthermore, using an instruction-driven mechanism, based on the target graphic description and an external domain knowledge base, controllable QA generation is performed to obtain multiple sets of detailed question-and-answer pairs with different styles, depths, and structures, specifically including: The LLM model is instructed by the third prompt to perform domain identification on the target image and text description using MOE expert capabilities and match the domain expert role corresponding to the target image and text description. Leveraging the capabilities of the domain expert role and the external domain knowledge base, and based on the Prompt instruction and the target textual description, multiple sets of detailed question-and-answer pairs are constructed, wherein... During the construction of the detailed question-answer pairs, the third Prompt instructs the LLM model to dynamically adjust the question-answer style, question-answer depth, and question-answer format. Furthermore, based on the graphic description, the external domain knowledge base, and multiple sets of detailed question-and-answer pairs, a cross-validation is performed to obtain target detailed question-and-answer pairs that meet preset requirements; The combined module is used to construct a multimodal dataset for model training and optimization based on the target detailed question-answer pair, the target image and text description, and the image to be processed.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images