A method, device and equipment for constructing multimodal cross-domain question-answering data

Abstract

The present invention discloses a method, device, and apparatus for constructing multimodal, cross-domain question-and-answer data, comprising: performing syntactic analysis on question text in acquired general-domain image-text question-and-answer data to generate question templates, and constructing a general-domain question-and-answer template library based on the question templates; performing feature extraction on target-domain image data to be processed to obtain image description information; generating target-domain question text by combining the general-domain question-and-answer template library and the image description information; inputting the target-domain question text and the target-domain image data into a multimodal question-and-answer model to generate answer text, and using the answer text and the target-domain question text as target-domain image question-and-answer pair data. This application can achieve efficient, flexible, and accurate generation of question-and-answer data for different fields.

Description

Technical Field

[0001] The present invention relates to the field of visual question answering technology, and in particular to a method, device, and equipment for constructing multimodal cross-domain question answering data. Background Art

[0002] With the development of artificial intelligence, especially multimodal learning, visual question answering (VQA) systems have achieved remarkable results in general image understanding and natural language processing tasks. However, when applied to specific fields such as military reconnaissance, industrial inspection, and cultural heritage protection, some limitations exist:

[0003] General-purpose models perform poorly in specific areas. For example, in the military, tasks such as accurately identifying combat equipment types in satellite imagery, assessing troop movements, or analyzing battlefield situation maps require deep military expertise. Due to a lack of domain-specific labeled examples in training data, general-purpose VQA models struggle to accurately capture specialized terminology, concepts, and key information in complex scenarios. This limits their depth of understanding and accuracy of responses, and they also lack the ability to integrate information in multimodal scenarios.

[0004] Constructing domain-specific question-answering datasets relies on extensive manual annotation. This not only requires the in-depth involvement of domain experts but also consumes considerable time to ensure data accuracy. This is particularly true in areas like military reconnaissance, where obtaining a sufficient number of diverse image-question pairs is challenging. This significantly increases the cost and difficulty of data construction, limiting the scalability and application of traditional multimodal data construction methods.

[0005] Rule-based templates generate a limited number of question types. While rule-based approaches can ensure the consistency and grammatical correctness of generated questions, they strictly adhere to preset templates and struggle to adapt to complex semantic variations and diverse question formats. This results in a limited variety of generated questions, an inability to fully explore the potential of the target domain, and a compromised model's generalization capabilities and practical application effectiveness. Furthermore, they struggle to meet the requirements for covering rich interactive relationships in multimodal environments. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to propose a multimodal and cross-domain question-answering data construction method, device and equipment, aiming to solve the problems of existing visual question-answering systems in specific fields, such as poor domain adaptability, high data construction cost, and insufficient diversity of generated templates.

[0007] To achieve the above objectives, the present invention provides a method for constructing multimodal and cross-domain question-answering data, the method comprising:

[0008] Performing syntactic analysis on the question text in the acquired general domain image-text question-answering data to generate question templates, and building a general question-answering template library based on the question templates;

[0009] Perform feature extraction on the target domain image data to be processed to obtain image description information;

[0010] Generate a target domain question text by combining the general question and answer template library and the image description information;

[0011] The target domain question text and the target domain image data are input into a multimodal question-answering model to generate an answer text, and the answer text and the target domain question text are used as target domain image question-answering pair data.

[0012] Preferably, the step of performing syntactic analysis on the question text in the acquired general-domain graphic question-and-answer data to generate a question template, and constructing a general-purpose question-and-answer template library based on the question template includes:

[0013] Using dependency parsing tools to identify the sentence structure of the question text to obtain core semantic information, including question words, predicates, entities, and attributes;

[0014] Replacing the entities and attributes identified in the question text with placeholders to obtain the question template;

[0015] Calculating the similarity of the question templates using a sentence vector model and merging the question templates with similar semantics;

[0016] The question templates are classified according to question types to obtain the general question and answer template library, where the question types include object query, attribute query, and action description.

[0017] Preferably, the feature extraction of the target domain image data to be processed to obtain image description information includes:

[0018] Extracting a global visual feature vector of the target domain image data using a pre-trained ResNet model, and inputting the target domain image data into a visual language model to generate a global description;

[0019] Identify target areas in the target area image data using a target detection model, and obtain bounding box coordinates and local visual feature vectors corresponding to each target area;

[0020] Cropping the target area image data to obtain a plurality of cropped area images, inputting the plurality of cropped area images into a visual language model, and generating a plurality of candidate description words corresponding to each of the cropped area images;

[0021] Determining a target entity name corresponding to the cropped area image according to the plurality of candidate description words, and generating a regional structured description corresponding to each of the cropped area images based on the global description and the target entity name;

[0022] A visual relationship triplet is generated based on the bounding box coordinates and the region structured description, and the global visual feature vector, the global description, the local visual feature vector, and the visual relationship triplet are used as the image description information.

[0023] Preferably, determining the target entity name corresponding to the cropped area image according to the plurality of candidate descriptors includes:

[0024] Counting the frequency of occurrence of each candidate description word in the image corresponding to the cropped area to obtain a word frequency weight;

[0025] Calculating the feature similarity between each candidate description word and the corresponding cropped area image to obtain a visual feature similarity weight;

[0026] A saliency value is obtained by calculating according to the word frequency weight and the visual feature similarity weight, and the candidate description word with the highest saliency value is selected as the target entity name corresponding to the cropped area image.

[0027] Preferably, generating a regional structured description corresponding to each of the cropped region images based on the global description and the target entity name includes:

[0028] Analyzing the global description using a dependency parsing tool to obtain a plurality of description phrases;

[0029] Entity matching is performed on the plurality of description phrases with the target entity name to obtain a target description phrase, and attribute extraction is performed on the target description phrase to obtain the regional structured description, wherein the regional structured description includes the object name, state attributes and spatial location information.

[0030] Preferably, generating a visual relationship triple based on the bounding box coordinates and the region structured description includes:

[0031] Performing spatial relationship detection on the bounding box coordinates to generate a spatial relationship triplet;

[0032] Performing semantic analysis on the region structured description to generate a semantic relationship triple, and using the spatial relationship triple and the semantic relationship triple as the semantic relationship triple;

[0033] A large language model is used to perform scene consistency verification on the global description and the visual relationship triples, and the visual relationship triples that pass the verification are retained.

[0034] Preferably, generating the target domain question text by combining the general question and answer template library and the image description information includes:

[0035] Extracting entities from the image description information, and filling the extracted entities into corresponding slots in the question template according to the question type to obtain the target domain question text;

[0036] The multimodal pre-training model is used to calculate the similarity between the target domain question text and the target domain image data. When the similarity is greater than a preset value, the target domain question text is retained; otherwise, the target domain question text is deleted.

[0037] To achieve the above objectives, the present invention further provides a device for constructing multimodal and cross-domain question-answer data, the device comprising:

[0038] A template library construction unit is used to perform syntactic analysis on the question text in the acquired general domain image-text question and answer data, generate question templates, and construct a general question and answer template library based on the question templates;

[0039] A feature extraction unit is used to extract features from the target domain image data to be processed to obtain image description information;

[0040] A question generation unit, configured to generate a target domain question text by combining the general question and answer template library and the image description information;

[0041] The question-answer pair construction unit is used to input the target domain question text and the target domain image data into the multimodal question-answering model, generate an answer text, and use the answer text and the target domain question text as the target domain image question-answer pair data.

[0042] In order to achieve the above-mentioned objectives, the present invention also proposes a multimodal and cross-domain question-and-answer data construction device, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a multimodal and cross-domain question-and-answer data construction method as described in the above-mentioned embodiment.

[0043] In order to achieve the above-mentioned objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of a multimodal and cross-domain question-and-answer data construction method as described in the above-mentioned embodiment.

[0044] In order to achieve the above-mentioned objectives, the present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a multimodal and cross-domain question-and-answer data construction method as described in the above-mentioned embodiment.

[0045] Beneficial effects:

[0046] The above solution constructs a template library by extracting question templates from general-domain question-answering data, combining feature extraction and description information from target-domain images to generate target-domain question text, and then using a multimodal question-answering model to generate answers, thus achieving cross-domain question-answering data construction. This method effectively reduces the reliance on extensive manual annotation, improves the efficiency and flexibility of question-answering data construction, and addresses the issues of general-purpose models performing poorly in specific domains and the limited number of question types generated by rule-based templates. It can automatically generate question-answering data adapted to specific domains, thereby improving the performance and application scope of VQA systems in these specific domains.

[0047] Using dependency parsing tools to identify the sentence structure of question texts, including key components such as question words, predicates, entities, and attributes, we can accurately extract the core semantic information of the question and replace entities with placeholders to form templates, thereby improving the versatility and adaptability of the templates. Using sentence vector models to calculate template similarity and merge semantically similar templates reduces template redundancy and optimizes the structure of the template library. Furthermore, by categorizing templates according to question type, the template library becomes more systematic and organized, facilitating the subsequent selection of appropriate templates for different question-and-answer needs. Overall, this improves the quality and practicality of the template library, laying the foundation for the subsequent generation of high-quality question text in the target domain.

[0048] By using the ResNet model to extract global visual feature vectors from the target domain image data, the overall scene features of the image can be quickly and accurately represented. At the same time, the image data is input into a visual language model to generate a global description, further enriching the semantic information of the image. The target region in the image is identified through the target detection model, and bounding box coordinates and local visual feature vectors are obtained, enabling the capture of local image details. Candidate descriptors are generated for the cropped region image, and the target entity name is determined. A structured regional description is also generated, which can describe the content and attributes of each region in the image in detail. Visual relationship triplets are generated based on the bounding box coordinates and the structured regional description, further exploring the spatial and semantic relationships between entities in the image. The resulting comprehensive and rich image description information provides strong support for the subsequent generation of accurate and diverse target domain question text.

[0049] By counting the frequency of occurrence of candidate descriptive words to obtain word frequency weights, and calculating the feature similarity between the candidate descriptive words and the cropped area image to obtain visual feature similarity weights, the importance of the vocabulary in the text description and the degree of matching with the image visual features are comprehensively considered; the saliency value is calculated based on these two weights, and the candidate descriptive word with the highest saliency value is selected as the target entity name. This can more accurately determine the core entity of the image area, improve the accuracy and reliability of the image description, and thus provide more accurate entity information for the subsequent generation of high-quality target domain question text, further improving the quality of question and answer data construction.

[0050] Using dependency parsing tools, the global description is analyzed to obtain multiple descriptive phrases, which can parse key information from the global description at a semantic level. The descriptive phrases are then matched with the target entity names to obtain the target description phrases. Attribute extraction from the target description phrases yields a structured regional description containing the object name, state attributes, and spatial location information. This allows for a more detailed and accurate description of the entities and attributes of each region in the image. This structured description not only enriches the semantic information of the image but also provides more comprehensive entity and attribute information for the subsequent generation of specific and accurate question text, improving the accuracy and sophistication of the question-answering data.

[0051] By performing spatial relationship detection on bounding box coordinates to generate spatial relationship triplets, and semantically parsing the region's structured description to generate semantic relationship triplets, the system comprehensively captures the various relationships between entities in the image. Using a large language model to perform scene consistency verification on the global description and visual relationship triplets, and selecting triplets that pass verification, the generated visual relationship triplets are ensured to be consistent with the image scene, with high accuracy and rationality. The resulting visual relationship triplets more realistically reflect the interactive relationships between entities in the image, providing reliable semantic relationship support for the subsequent generation of logically sound, realistic target domain question text.

[0052] By extracting entities from image descriptions and filling them into corresponding question template slots based on question type, the system can quickly generate preliminary target domain question text. A multimodal pre-trained model is used to calculate the similarity between the generated question text and the target domain image. Highly relevant question text is then filtered based on a similarity threshold, effectively removing questions that are inconsistent with or have low relevance to the image content. This ensures that the generated target domain question text is highly relevant and accurate to the image content. This similarity-based screening mechanism improves the quality and reliability of question-answering data construction, ensuring that the resulting question-answer pairs are more aligned with practical application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A flowchart of a method for constructing multimodal and cross-domain question-and-answer data provided by one embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the overall process of a question-and-answer data construction method provided in another embodiment of the present invention.

[0056] Figure 3 A schematic diagram of a process for extracting features from a target domain image according to an embodiment of the present invention.

[0057] Figure 4 A schematic diagram of the structure of a multimodal and cross-domain question-and-answer data construction device provided by one embodiment of the present invention.

[0058] The realization of the objectives of the invention, the functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] The present invention is described in detail below with reference to the embodiments.

[0061] Reference Figure 1 The figure is a flow chart of a method for constructing multimodal and cross-domain question-and-answer data according to an embodiment of the present invention.

[0062] In this embodiment, the method includes:

[0063] S11, performing syntactic analysis on the question text in the acquired general domain image-text question and answer data, generating a question template, and building a general question and answer template library based on the question template.

[0064] Furthermore, in step S11, the question text in the acquired general field graphic question and answer data is subjected to syntactic analysis to generate a question template, and a general question and answer template library is constructed based on the question template, including:

[0065] S11-1, using a dependency parsing tool to identify the sentence structure of the question text and obtain core semantic information, wherein the core semantic information includes question words, predicates, entities, and attributes;

[0066] S11-2, replacing the entities and attributes identified in the question text with placeholders to obtain the question template;

[0067] S11-3, using a sentence vector model to calculate the similarity of the question templates, and merging the question templates with similar semantics;

[0068] S11-4, classifying the question templates according to question types to obtain the general question and answer template library, wherein the question types include object query, attribute query and action description.

[0069] Reference Figure 2 The overall process diagram of question-answer data construction is shown. In this embodiment, taking the VCR dataset as an example, the original question text of the VCR dataset is analyzed by using a dependency syntax analysis tool (such as the Spacy tool) to identify the core structure of the original question text. For example, for the original question "Why is the person smiling?", the core semantic information is extracted, including: question words (Why), predicates (is smiling), entities (person), and attributes (smiling state); functional vocabulary (such as prepositions, question words, etc.) in the question text is retained, and entities and attributes in the question text are replaced with placeholders. For example, the original question text "Why is the person smiling?" can be abstracted to "Why is <person> <action>?” question template, the original question “What is the man holding?” can be abstracted to “What is <person>holding?" template; use sentence vector models (such as BERT-wwm model) to calculate the similarity of question templates, merge semantically similar question templates, classify question templates according to question types (such as object / cause query, attribute query, action description, etc.) to mark their uses, and establish a question and answer template library for subsequent cross-domain question and answer generation matching, such as "What is <person>doing?" template and "What is <person>The "holding?" template has a high similarity and can be classified as an "action query" type. The uses of annotation templates can include: reason query (Why), action description (What...doing), object attribute (Whatcolor...), etc.

[0070] S12, performing feature extraction on the target domain image data to be processed to obtain image description information.

[0071] Furthermore, in step S12, feature extraction is performed on the target area image data to be processed to obtain image description information, including:

[0072] S12-1, extracting a global visual feature vector of the target domain image data using a pre-trained ResNet model, and inputting the target domain image data into a visual language model to generate a global description;

[0073] S12-2, identifying target areas in the target area image data using a target detection model, and obtaining bounding box coordinates and local visual feature vectors corresponding to each target area;

[0074] S12-3, cropping the target area image data to obtain a plurality of cropped area images, inputting the plurality of cropped area images into a visual language model, and generating a plurality of candidate descriptive words corresponding to each of the cropped area images;

[0075] S12-4, determining a target entity name corresponding to the cropped area image according to the plurality of candidate description words, and generating a regional structured description corresponding to each of the cropped area images based on the global description and the target entity name;

[0076] S12-5, generating a visual relationship triplet based on the bounding box coordinates and the region structured description, and using the global visual feature vector, the global description, the local visual feature vector and the visual relationship triplet as the image description information.

[0077] Wherein, in step S12-4, determining the target entity name corresponding to the cropped area image according to the plurality of candidate descriptors includes:

[0078] Counting the frequency of occurrence of each candidate description word in the image corresponding to the cropped area to obtain a word frequency weight;

[0079] Calculating the feature similarity between each candidate description word and the corresponding cropped area image to obtain a visual feature similarity weight;

[0080] A saliency value is obtained by calculating according to the word frequency weight and the visual feature similarity weight, and the candidate description word with the highest saliency value is selected as the target entity name corresponding to the cropped area image.

[0081] Wherein, in step S12-4, generating a regional structured description corresponding to each of the cropped region images based on the global description and the target entity name includes:

[0082] Analyzing the global description using a dependency parsing tool to obtain a plurality of description phrases;

[0083] Entity matching is performed on the plurality of description phrases with the target entity name to obtain a target description phrase, and attribute extraction is performed on the target description phrase to obtain the regional structured description, wherein the regional structured description includes the object name, state attributes and spatial location information.

[0084] Furthermore, in step S12-5, generating a visual relationship triple based on the bounding box coordinates and the region structured description includes:

[0085] S12-5-1, performing spatial relationship detection on the bounding box coordinates to generate a spatial relationship triplet;

[0086] S12-5-2, performing semantic analysis on the region structured description to generate a semantic relationship triple, and using the spatial relationship triple and the semantic relationship triple as the semantic relationship triple;

[0087] S12-5-3, using a large language model to perform scene consistency verification on the global description and the visual relationship triples, and retaining the visual relationship triples that pass the verification.

[0088] Reference Figure 3 As shown, in this embodiment, a pre-trained deep convolutional neural network is used for feature extraction, that is, to extract the global visual feature vector of the target domain image data to represent the overall scene of the image, where the target domain can be the military field, the medical field, etc. For example, the ResNet-152 model is used to extract the global visual feature vector of the target domain image to represent the overall scene, such as "kitchen scene" and "multi-person conversation"; as shown below: global_feature = [0.12, -0.05,..., 0.78];

[0089] global_scene = "A kitchen scene with multiple people interacting".

[0090] Use a target detection model (such as the Faster R-CNN model) to detect salient areas of the target domain image based on the target detection object (such as a person or object) and filter low-confidence results (such as filtering results with a confidence level < 0.7) to obtain the bounding box coordinates and local visual feature vector of the corresponding target area; as shown below:

[0091] regions = [

[0092] {"bbox": [120, 80, 250, 300], "feature": [0.34, ...,-0.21], "class":"person"},

[0093] {"bbox": [300, 150, 400, 250], "feature": [0.02, ..., 0.45], "class":"cup"};].

[0094] The target domain image is cropped to obtain multiple cropped regions. The visual language model BLIP is used to generate candidate description words for each cropped region, such as:

[0095] Area 1: ["man", "smiling", "man", "person", "man", "blue shirt"];

[0096] Region 2: ["cup", "white mug", "coffee cup"];

[0097] The frequency of occurrence of candidate descriptors in the corresponding regions is counted to obtain word frequency weights. The multimodal pre-trained model is used to calculate the feature similarity between each candidate descriptor and the corresponding cropped region (obtained by calculating the dot product using cosine similarity), resulting in a visual feature similarity weight. The candidate descriptor with the highest overall weight is selected as the most prominent target entity name in each region through a weighted calculation of the word frequency weights and the visual feature similarity weights. The steps for calculating the feature similarity between each candidate descriptor and the corresponding cropped region include: encoding the cropped region image through CLIP's image encoder to obtain a normalized visual feature vector; inputting each candidate descriptor into CLIP's text encoder for encoding to obtain a normalized text feature vector; and performing a dot product operation on the two feature vectors (multiplying the corresponding elements and then summing them) to obtain feature similarity. For example, the word frequency weight of "man" in region 1 is 3 / 6 = 0.5. Using the multimodal pre-trained model CLIP to calculate the feature similarity between "man" and region 1, we obtain a visual feature similarity weight of 0.85. Setting the weight factor to 0.4, we calculate the saliency value of "man" using the preset formula: weight factor * word frequency weight + (1 - weight factor) * visual similarity weight. This gives us a saliency value of 0.75 for each candidate descriptor. Similarly, after calculating the saliency value for each candidate descriptor, we output the most salient entity in each region, as follows:

[0098] { "region_1": {"entity": "man", "attributes": ["smiling", "blueshirt"]},

[0099] "region_2": {"entity": "cup", "attributes": ["white", "coffee"]}}.

[0100] Use a visual language model (such as the BLIP model) to generate a global description of the overall content of the target domain image, such as "A man in a blue shirt is smiling while holding a white coffee cup in a kitchen." Use the Spacy tool to analyze the global description and obtain the following description phrase: ["man in a blueshirt", "smiling", "holding a cup"]. Perform entity matching on this description phrase with the most salient entity (such as "man") obtained above to obtain the target description phrase. Use the Spacy tool to extract attributes from the target description phrase "man in ablue shirt" to obtain a regional structured description (such as including object name, state attributes, and spatial location information), as shown below:

[0101] attributes = { "appearance": "blue shirt",

[0102] "action": "smiling",

[0103] "holding": "cup"}.

[0104] Then, by the bounding box coordinates obtained above (representing the spatial relationship) such as:

[0105] {"id": "person-1", "bbox": [120, 80, 250, 300], "class": "man"},

[0106] {"id": "cup-1", "bbox": [300, 150, 400, 250], "class": "cup"},

[0107] {"id": "table-1", "bbox": [280, 200, 500, 350], "class": "table"}, performs spatial relationship detection including containment, adjacency, and direction (up and down, left and right), and generates spatial relationship triples. For example, if it is detected that the bounding box of person (person-1) is adjacent to the bounding box of table (table-1) and the cup (cup-1) is located on the top area of ​​table (table-1), the following spatial relationship triples are generated:

[0108] {"subject": "person-1", "relation": "adjacent_to", "object": "table-1", "confidence": 0.89},

[0109] {"subject": "cup-1", "relation": "on_top_of", "object": "table-1", "confidence": 0.92}.

[0110] And, a structural description of the region (representing semantic relationships) such as:

[0111] { "person-1": {

[0112] "entity": "man",

[0113] "attributes": {

[0114] "action": "smiling",

[0115] "interaction": {"holding": "cup-1"}}},

[0116] "cup-1": { "entity": "cup",

[0117] "attributes": {"color": "white", "content": "coffee"}}}, performs semantic analysis including action extraction and attribute extraction to generate semantic relationship triples, for example:

[0118] Action extraction: generated from the interaction field<person-1, holding, cup-1> ;

[0119] Attribute extraction: generated from the color field<cup-1, color, white> ; Generate semantic relationship triples as follows: {"subject": "person-1", "relation": "holding", "object": "cup-1", "source": "interaction"},

[0120] {"subject": "cup-1", "relation": "color", "object": "white", "source": "attribute"}.

[0121] By constructing prompt words, a large language model (such as the Qwen2.5-72B-Instruct model) is used to verify the scene consistency of the obtained global description and semantic relationship triples. The prompt words are as follows:

[0122] Please strictly judge whether the following relationship is reasonable based on the scenario description and answer only True or False:

[0123] Scene: A man in a blue shirt is smiling while holding a white coffeecup in a kitchen.

[0124] Relation 1:<person-1, holding, cup-1>

[0125] Relation 2:<cup-1, on_top_of, table-1>

[0126] Relation 3:<cup-1, color, white>

[0127] Relationship 4:<person-1, standing_on, beach>

[0128] Answer: "

[0129] By parsing the output of the large language model, the visual relationship triplets with the answer of True (reasonable relationship) are retained; by fusing the processing results of the above steps, including global visual feature vectors, local visual feature vectors, global description, etc. as image description information, a description file in JSON format is output.

[0130] S13, generating a target domain question text by combining the general question and answer template library and the image description information.

[0131] Furthermore, in step S13, generating a target domain question text by combining the general question and answer template library and the image description information includes:

[0132] S13-1, extracting entities from the image description information, and filling the extracted entities into corresponding slots in the question template according to the question type to obtain the target domain question text;

[0133] S13-2, using a multimodal pre-training model to calculate the similarity between the target domain question text and the target domain image data, when the similarity is greater than a preset value, retaining the target domain question text, otherwise, deleting the target domain question text.

[0134] S14, inputting the target domain question text and the target domain image data into a multimodal question-answering model to generate an answer text, and using the answer text and the target domain question text as target domain image question-answering pair data.

[0135] In this embodiment, the entity names corresponding to the entities (corresponding to the most prominent entities in each area) are extracted from the image description information, including object entity lists, phenomenon entity lists, fault entity lists, etc. The extracted entities are filled into the slots of the corresponding template according to the question type to generate the target domain question text. Specifically:

[0136] Populate the structured entity output above:

[0137] For example, the "Why is <person> <action>?”:

[0138] Input entity "person-1: smiling" and generate the question "Why is the person smiling?";

[0139] If the entity is "person-1: laughing", generate "Why is the person laughing?";

[0140] For example, the "Action Query" template "What is <person>holding?"

[0141] Fill in the entity "person-1: man" and generate the question "What is the man holding?";

[0142] If the entity is "person-1: woman", generate "What is the woman holding?".

[0143] By using a multimodal pre-trained model (such as CLIP), the similarity between the generated question text and the target domain image is calculated. A threshold is set to verify the consistency between the generated question text and the target domain image, and those that pass the threshold are retained. The generated question text "Why is the person smiling?" and the corresponding target domain image are then fed into a multimodal question-answering model (Qwen2.5-VL-72B-Instruct model) as visual context, generating the answer text "Because he is holding a cup of coffee." The question text "What is the man holding?" and the corresponding target domain image are then fed into the question-answering model, generating the answer text "A white cup."

[0144] Through this embodiment, the dependence on the domain knowledge base can be effectively reduced, and the quality of cross-domain question and answer can be ensured through semantic matching verification. Compared with the traditional manual construction method, it significantly reduces the domain adaptation cost and solves the problem of strong dependence on professional knowledge and low migration efficiency in the construction of cross-domain question and answer data. The use of visual-language joint modeling, combined with structured description generation and dynamic slot filling, overcomes the problem of separation of visual and text features in the construction of traditional single-modal question and answer data, improves the degree of automation and scalability of question and answer data construction, and ensures that high-quality cross-modal question and answer pairs can still be generated in different fields and complex scenarios.

[0145] Reference Figure 4 FIG2 is a schematic diagram showing the structure of a multimodal and cross-domain question-answer data construction device provided by an embodiment of the present invention.

[0146] In this embodiment, the device 20 includes:

[0147] A template library construction unit 21 is used to perform syntactic analysis on the question text in the acquired general domain image-text question and answer data, generate question templates, and construct a general question and answer template library based on the question templates;

[0148] A feature extraction unit 22 is used to extract features from the target area image data to be processed to obtain image description information;

[0149] A question generating unit 23 is configured to generate a target domain question text by combining the general question and answer template library and the image description information;

[0150] The question-answer pair construction unit 24 is used to input the target domain question text and the target domain image data into the multimodal question-answering model, generate an answer text, and use the answer text and the target domain question text as target domain image question-answer pair data.

[0151] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0152] The embodiment of the present invention further provides a multimodal cross-domain question-answer data construction device, which includes the multimodal cross-domain question-answer data construction device as described above, wherein the multimodal cross-domain question-answer data construction device can adopt Figure 4 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments and will not be repeated here.

[0153] The device includes: a mobile phone, digital camera, tablet computer, or other device with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0154] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0155] The input unit can be used to receive input digital, character, or image information, and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment can include not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0156] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Furthermore, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0157] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The computer-readable storage medium may be a read-only memory, a disk, or an optical disk.

[0158] The embodiment of the present invention further provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A multimodal and cross-domain question-answering data construction method is shown.

[0159] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus embodiments, device embodiments, and storage medium embodiments, since they are generally similar to the method embodiments, their descriptions are relatively simple. For relevant parts, reference can be made to the descriptions of the method embodiments.

[0160] Furthermore, as used herein, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0161] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.< / person> < / action> < / person> < / person> < / person> < / person> < / action> < / person>

Claims

1. A method for constructing multimodal and cross-domain question-answering data, characterized in that: The method comprises: Performing syntactic analysis on the question text in the acquired general domain image-text question-answering data to generate question templates, and building a general question-answering template library based on the question templates; Perform feature extraction on the target domain image data to be processed to obtain image description information; The feature extraction of the target domain image data to be processed to obtain image description information includes: Extracting a global visual feature vector of the target domain image data using a pre-trained ResNet model, and inputting the target domain image data into a visual language model to generate a global description; Identify target areas in the target area image data using a target detection model, and obtain bounding box coordinates and local visual feature vectors corresponding to each target area; Cropping the target area image data to obtain a plurality of cropped area images, inputting the plurality of cropped area images into a visual language model, and generating a plurality of candidate description words corresponding to each of the cropped area images; Determining a target entity name corresponding to the cropped area image according to the plurality of candidate description words, and generating a regional structured description corresponding to each of the cropped area images based on the global description and the target entity name; generating a visual relationship triplet based on the bounding box coordinates and the region structured description, and using the global visual feature vector, the global description, the local visual feature vector, and the visual relationship triplet as the image description information; Generating a target domain question text by combining the general question and answer template library and the image description information; wherein generating the target domain question text by combining the general question and answer template library and the image description information includes: Extracting entities from the image description information, and filling the extracted entities into corresponding slots in the question template according to the question type to obtain the target domain question text; Calculating the similarity between the target domain question text and the target domain image data using a multimodal pre-training model; retaining the target domain question text when the similarity is greater than a preset value; otherwise, deleting the target domain question text; The target domain question text and the target domain image data are input into a multimodal question-answering model to generate an answer text, and the answer text and the target domain question text are used as target domain image question-answering pair data.

2. A multimodal cross-domain question-answering data construction method according to claim 1, characterized in that: The step of performing syntactic analysis on the question text in the acquired general domain graphic question and answer data to generate a question template, and constructing a general question and answer template library based on the question template includes: Using dependency parsing tools to identify the sentence structure of the question text to obtain core semantic information, including question words, predicates, entities, and attributes; Replacing the entities and attributes identified in the question text with placeholders to obtain the question template; Calculating the similarity of the question templates using a sentence vector model and merging the question templates with similar semantics; The question templates are classified according to question types to obtain the general question and answer template library, where the question types include object query, attribute query, and action description.

3. The method for constructing multimodal cross-domain question-answering data according to claim 1, characterized in that: The determining of the target entity name corresponding to the cropped area image according to the plurality of candidate descriptors includes: Counting the frequency of occurrence of each candidate description word in the image corresponding to the cropped area to obtain a word frequency weight; Calculating the feature similarity between each candidate description word and the corresponding cropped area image to obtain a visual feature similarity weight; A saliency value is obtained by calculating according to the word frequency weight and the visual feature similarity weight, and the candidate description word with the highest saliency value is selected as the target entity name corresponding to the cropped area image.

4. The method for constructing multimodal cross-domain question-answering data according to claim 1, characterized in that: Generating a regional structured description corresponding to each of the cropped region images based on the global description and the target entity name includes: Analyzing the global description using a dependency parsing tool to obtain a plurality of description phrases; Entity matching is performed on the plurality of description phrases with the target entity name to obtain a target description phrase, and attribute extraction is performed on the target description phrase to obtain the regional structured description, wherein the regional structured description includes the object name, state attributes and spatial location information.

5. The method for constructing multimodal cross-domain question-answering data according to claim 1, characterized in that: Generating a visual relationship triple based on the bounding box coordinates and the region structured description includes: Performing spatial relationship detection on the bounding box coordinates to generate a spatial relationship triplet; Performing semantic analysis on the region structured description to generate a semantic relationship triple, and using the spatial relationship triple and the semantic relationship triple as the semantic relationship triple; A large language model is used to perform scene consistency verification on the global description and the visual relationship triples, and the visual relationship triples that pass the verification are retained.

6. A multimodal and cross-domain question-answering data construction device, characterized in that: The device comprises: A template library construction unit is used to perform syntactic analysis on the question text in the acquired general domain image-text question and answer data, generate question templates, and construct a general question and answer template library based on the question templates; A feature extraction unit is used to extract features from the target domain image data to be processed to obtain image description information; Wherein, the feature extraction unit is further used for: Extracting a global visual feature vector of the target domain image data using a pre-trained ResNet model, and inputting the target domain image data into a visual language model to generate a global description; Identify target areas in the target area image data using a target detection model, and obtain bounding box coordinates and local visual feature vectors corresponding to each target area; Cropping the target area image data to obtain a plurality of cropped area images, inputting the plurality of cropped area images into a visual language model, and generating a plurality of candidate description words corresponding to each of the cropped area images; Determining a target entity name corresponding to the cropped area image according to the plurality of candidate description words, and generating a regional structured description corresponding to each of the cropped area images based on the global description and the target entity name; generating a visual relationship triplet based on the bounding box coordinates and the region structured description, and using the global visual feature vector, the global description, the local visual feature vector, and the visual relationship triplet as the image description information; A question generation unit, configured to generate a target domain question text by combining the general question and answer template library and the image description information; Wherein, the question generating unit is further used for: Extracting entities from the image description information, and filling the extracted entities into corresponding slots in the question template according to the question type to obtain the target domain question text; Calculating the similarity between the target domain question text and the target domain image data using a multimodal pre-training model; retaining the target domain question text when the similarity is greater than a preset value; otherwise, deleting the target domain question text; The question-answer pair construction unit is used to input the target domain question text and the target domain image data into the multimodal question-answering model, generate an answer text, and use the answer text and the target domain question text as the target domain image question-answer pair data.

7. A multimodal and cross-domain question-answering data construction device, characterized in that: The system comprises a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a multimodal and cross-domain question-answering data construction method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of a multimodal and cross-domain question-answering data construction method as described in any one of claims 1 to 5.

Images