Visual question and answer automatic labeling method and device based on multi-modal large model

Abstract

The invention relates to the technical field of multi-modal artificial intelligence and intelligent data processing, in particular to a visual question and answer automatic labeling method and device based on a multi-modal large model, and the method comprises the steps: collecting basic image data used for constructing different scenes of visual question and answer; preprocessing the basic image data, constructing a picture content instance list, and associating related entries; based on related entries, screening and correcting the picture content instance list, generating visual question and answer pairs, and constructing a visual question and answer pair data set; and judging whether the visual question and answer pair data set meets a preset quality condition, generating an interference sample according to the visual question and answer pair data set, and completing visual question and answer automatic labeling of the multi-modal large model. Therefore, the problems that in related technologies, semantic deviation, context splitting or knowledge noise is prone to occurring in a single-mode model labeling result, and due to factors such as model illusion and data deviation of a multi-mode large model, the actual labeling quality is remarkably affected, and efficiency and accuracy are reduced are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal artificial intelligence and intelligent data processing, in particular to a visual question answering automatic labeling method and device based on a multi-modal large model. BACKGROUND

[0002] At present, with the breakthrough of multi-modal large model technology, the cross-modal understanding ability of vision and language has been significantly improved, which promotes the VQA (Visual Question Answering) task to evolve from relying on artificial labeling to automation. Such models can analyze image semantics and generate natural language responses simultaneously through the pre-training framework of fusing visual encoders and language decoders, which provides a new technical path for automatic labeling.

[0003] In related technologies, the automatic method based on single-modal models can partially replace artificial labeling work, and multi-modal large models can realize higher-level semantic association and more accurate understanding of the complex relationship between visual and language information through end-to-end joint learning mechanisms.

[0004] However, in related technologies, single-modal models have limitations in cross-modal semantic alignment, fine-grained relationship reasoning, and multi-modal knowledge retrieval, which leads to problems such as semantic deviation, context fragmentation, or knowledge noise in labeling results. Due to factors such as model hallucination and data bias, the actual labeling quality of multi-modal large models is significantly affected, which needs to be improved. SUMMARY

[0005] The present application provides a visual question answering automatic labeling method and device based on a multi-modal large model to solve the problems in related technologies, such as the significant limitations of single-modal models in cross-modal semantic alignment, fine-grained relationship reasoning, and multi-modal knowledge retrieval, which leads to problems such as semantic deviation, context fragmentation, or knowledge noise in labeling results. Due to factors such as model hallucination and data bias, the actual labeling quality of multi-modal large models is significantly affected, which leads to problems such as a significant decrease in labeling efficiency and accuracy.

[0006] The first aspect embodiment of the application provides a visual question answering automatic labeling method based on a multi-modal large model, including the following steps: collecting basic image data for constructing a visual question answering, wherein the basic image data at least includes image data of different scenes; preprocessing the basic image data to construct a picture content instance list and associate the picture content instances in the picture content instance list with relevant terms; based on the terms associated with the picture content instances in the picture content instance list, screening and correcting the picture content instance list to generate a visual question answering pair and construct a visual question answering pair dataset; determining whether the visual question answering pair dataset meets a preset quality condition, wherein if the preset quality condition is met, generating interference samples according to the visual question answering pair dataset to complete the visual question answering automatic labeling of the multi-modal large model.

[0007] The embodiment of the application can collect multi-scene images, construct a diverse dataset, cover multi-dimensional visual scenes, avoid model deviation caused by single data, and improve the semantic understanding ability of the multi-modal large model for complex scenes; after manually screening low-quality images, the visual feature extraction model and the clustering algorithm are used to screen the maximum difference images and associate the relevant terms, reduce data redundancy, convert visual information into structured semantic data, provide accurate semantic support for question and answer pair generation, and improve the accuracy and efficiency of subsequent labeling; introduce multi-modal large model verification instance labeling, select instances based on term semantic similarity, generate question and answer pairs, cover multi-dimensional attributes of instances, automatically identify and label errors, generate diversified question and answer pairs, comprehensively train the cross-modal reasoning ability of the model, and improve the coverage of the dataset for complex semantic scenes; calculate the accuracy of the question and answer pair, generate pseudo descriptions and interference categories based on the multi-modal large model, construct negative samples, quantify and evaluate the quality of the dataset, enhance the noise resistance of the model through interference samples, simulate real scene semantic variation, and improve the labeling reliability and model robustness.

[0008] Optionally, in an embodiment of the application, the preprocessing of the basic image data includes: selecting, from the basic image data, maximum difference images that have the largest similarity difference gap among a preset number of pictures.

[0009] The embodiment of the application can extract high-dimensional features of images through the DINO model, cluster images into K clusters using K-means clustering, select the center images of each cluster as maximum difference samples, compress the data size while retaining 90% information entropy, reduce the data processing amount, focus on core visual features, and improve the recognition ability of the model for long-tail scenes.

[0010] Optionally, in an embodiment of the present application, the constructing the picture content instance list comprises: counting the number of category instances contained in the image, and performing numbering processing on the missing categories to label the instances in the image one by one, and giving each instance a unique index, and supplementing the corresponding description information for all labeled instances.

[0011] The embodiment of the present application can count the number of instance categories, number the newly added categories according to the M+C rule, label each instance with a unique index, a position box and semantic description, form a structured list, realize fine management of instances, avoid labeling confusion, and provide accurate position and semantic basis for question and answer pair generation.

[0012] Optionally, in an embodiment of the present application, the generating the visual question and answer pair comprises: generating a count type question and answer pair based on the number of instances of the same category; generating a reference type question and answer pair based on different description instances; generating an existence type question and answer pair based on whether the instance exists; and generating a region type question and answer pair based on the instance labeling position.

[0013] The embodiment of the present application can generate question and answer pairs according to a predefined template, such as a count type based on the number of instances, a reference type based on attribute description, an existence type based on the existence of instances, and a region type based on coordinate position, combine a T5 model to rewrite problem variants, cover multi-dimensional semantic requirements, and strengthen the reasoning ability of the model in the dimensions of quantity calculation and spatial positioning.

[0014] Optionally, in an embodiment of the present application, it further comprises: calculating the accuracy of the visual question and answer pair; and obtaining the correct response ability of the multi-modal large model to a specific question and answer pair according to the accuracy of the visual question and answer pair, to obtain a visual question and answer quality result.

[0015] The embodiment of the present application can use mainstream multi-modal models to predict the answers of question and answer pairs, calculate the accuracy, analyze the error distribution of different types of question and answer pairs, generate a quality evaluation report, provide quantitative basis for data set optimization, locate model weaknesses, and guide subsequent training strategy adjustment.

[0016] The second aspect embodiment of the application provides a visual question answering automatic labeling device based on a multi-modal large model, comprising: an acquisition module configured to acquire basic image data used to construct a visual question answering, wherein the basic image data at least comprises image data of different scenes; an association module configured to preprocess the basic image data to construct a picture content instance list and associate a term related to a picture content instance in the picture content instance list; a correction module configured to filter and correct the picture content instance list based on the term related to the picture content instance in the picture content instance list to generate a visual question answering pair and construct a visual question answering pair data set; and a labeling module configured to determine whether the visual question answering pair data set meets a preset quality condition, wherein if the preset quality condition is met, an interference sample is generated according to the visual question answering pair data set, and visual question answering automatic labeling of the multi-modal large model is completed.

[0017] Optionally, in an embodiment of the application, the association module comprises a selection unit configured to select a maximum difference image with the largest similarity difference gap among pictures under a preset number from the basic image data.

[0018] Optionally, in an embodiment of the application, the association module comprises a supplement unit configured to count the number of category instances contained in the image, perform numbering processing on the missing categories, label the instances in the image one by one, assign a unique index to each instance, and supplement the corresponding description information for all labeled instances.

[0019] Optionally, in an embodiment of the application, the generation of the visual question answering pair comprises: generating a count type question and answer pair based on the number of the same category instances; generating a reference type question and answer pair based on different description instances; generating an existence question and answer pair based on whether the instance exists; and generating a region type question and answer pair based on the location of the instance.

[0020] Optionally, in an embodiment of the application, further comprising: a calculation module configured to calculate the accuracy of the visual question and answer pair; and an acquisition module configured to obtain the correct response ability of the multi-modal large model to a specific question and answer pair according to the accuracy of the visual question and answer pair to obtain a visual question and answer quality result.

[0021] The third aspect embodiment of the application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the visual question answering automatic labeling method based on the multi-modal large model as described in the above embodiments.

[0022] The fourth aspect of the application provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for automatically labeling visual question answering based on a multi-modal large model is realized.

[0023] The fifth aspect of the application provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the method for automatically labeling visual question answering based on a multi-modal large model is realized.

[0024] The embodiments of the application can collect multi-scene basic image data and pre-process, construct a picture content instance list associated with a term, screen and correct to generate a visual question and answer pair and a data set, generate interference samples to complete automatic labeling after meeting the quality conditions, so as to make the labeling result accurate, and further enhance the robustness of the multi-modal large model to data bias, and significantly improve the quality and reliability of visual question and answer labeling. Therefore, the problems that in the related art, the single-modal model has performance limitations in cross-modal semantic alignment, fine-grained relationship reasoning, multi-modal knowledge retrieval, etc., leading to semantic bias, context fragmentation or knowledge noise in the labeling result, and the multi-modal large model is significantly affected by the actual labeling quality due to model illusion, data bias and other factors are solved.

[0025] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and / or additional aspects and advantages of the application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a method for automatically labeling visual question answering based on a multi-modal large model according to an embodiment of the application is provided.

[0028] Figure 2 A flowchart of constructing a picture content instance list according to an embodiment of the application is provided.

[0029] Figure 3 A picture instance information constitution diagram according to an embodiment of the application is provided.

[0030] Figure 4 A flowchart of automatically generating a question and answer pair and evaluation according to an embodiment of the application is provided.

[0031] Figure 5 A structural diagram of a method and device for automatically labeling visual question answering based on a multi-modal large model according to an embodiment of the application is provided.

[0032] Figure 6 A structural schematic diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0033] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which like or similar elements or elements having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0034] A multi-modal large model-based visual question answering automatic labeling method and device of an embodiment of the present application are described below with reference to the accompanying drawings. In view of the problems in the related art mentioned above, the single-modal model has limitations in cross-modal semantic alignment, fine-grained relationship reasoning, and multi-modal knowledge retrieval, which leads to semantic deviation, context fragmentation, or knowledge noise in the labeling result. The multi-modal large model is affected by model hallucination and data bias, which significantly affects the actual labeling quality. The present application provides a multi-modal large model-based visual question answering automatic labeling method. In the method, multi-scene basic image data can be collected and preprocessed, picture content instance list associated terms can be constructed, visual question and answer pairs and data sets can be screened and corrected, and interference samples can be generated to complete automatic labeling after meeting the quality conditions, so that the labeling result is accurate, and the robustness of the multi-modal large model to data bias is enhanced, and the quality and reliability of the visual question answering labeling are significantly improved. Thus, the problems in the related art, such as the limitations of the single-modal model in cross-modal semantic alignment, fine-grained relationship reasoning, and multi-modal knowledge retrieval, which lead to semantic deviation, context fragmentation, or knowledge noise in the labeling result, and the multi-modal large model is affected by model hallucination and data bias, which significantly affects the actual labeling quality, are solved.

[0035] Specifically, Figure 1 A flowchart of a multi-modal large model-based visual question answering automatic labeling method provided by an embodiment of the present application is shown in FIG. 4.

[0036] As Figure 1 shown, the multi-modal large model-based visual question answering automatic labeling method includes the following steps:

[0037] In step S101, basic image data for constructing visual questions and answers is collected, wherein the basic image data at least includes image data of different scenes.

[0038] It can be understood that the visual question and answer data set in the embodiment of the application is composed of three types of data, i.e., image, question and answer pair, and knowledge entry, wherein the relationship between the image data and the question and answer pair is one-to-many, i.e., one image can correspond to multiple question and answer pairs. The relationship between the knowledge entry and the instance is one-to-one, but since one image usually contains multiple instances, the relationship between the image data and the knowledge entry is also one-to-many.

[0039] In actual execution, the embodiment of the application can construct a visual question and answer data set, and the required data includes image data covering different scenes. When constructing the visual question and answer data set, image data needs to be collected first, then relevant knowledge entries are obtained, and then the annotation of the question and answer pair can be performed. Image acquisition is based on real scenes, covering various typical spatial scenes such as schools, airports, and restaurants. In order to ensure the normativity and integrity of the data set construction, it is necessary to first confirm that the number of collected scene categories is S i , wherein i∈{1,…,n}; each large category contains a number of small categories, denoted as S ij , wherein j∈{1,…,n}, and an independent coding index is assigned to each category of scene.

[0040] Specifically, in the data collection process, a specified number of images should be collected for each category of scene to ensure the diversity and representativeness of the data set under multiple scene conditions. For each collected image, a unique index {S ij ,N m ,T n} is constructed, wherein N represents the unique number of images in the scene category, and T n represents the time stamp of image collection.

[0041] The embodiment of the application can collect basic image data for constructing visual question and answer, provide rich visual semantic basis for subsequent visual question and answer pair generation, and thus improve the understanding ability of the multi-modal large model for complex scenes.

[0042] In step S102, the basic image data is preprocessed to construct a picture content instance list, and the entries related to the picture content instances in the picture content instance list are associated.

[0043] It can be understood that the picture content instance list in the embodiment of the application is a structured record of specific targets in the image, containing information such as instance category, position, and attribute.

[0044] In actual execution, the embodiment of the application can filter out the most representative images with the greatest difference from all collected images through image preprocessing. According to the requirements of the constructed data set, low-quality images are first manually filtered out, and the filtered images are not used as basic data for subsequent question and answer pair construction. On this basis, the image content instance list is manually constructed.

[0045] The embodiment of the present application can preprocess the basic image data, construct a picture content instance list, greatly improve the data quality and diversity, and provide high-quality and diversified data basis for subsequent question and answer pair generation.

[0046] Optionally, in an embodiment of the present application, preprocessing the basic image data comprises: selecting, from the basic image data, a maximum difference image that has the largest similarity difference gap among pictures under a preset number.

[0047] In actual execution process, the embodiment of the present application can filter out representative images with the largest difference from all collected images. Let the total number of collected images be NXS, where N represents the number of images collected under each scene category, and S represents the total number of scene categories. K images with the largest difference are selected from these images, where K<NXS.

[0048] Specifically, a visual feature extraction model such as DINO or CLIP is used to extract semantic features of all images to obtain high-dimensional feature representation of each image. Then, a common clustering algorithm such as K-means is used to cluster the extracted image features K times to finally obtain K cluster centers. The images corresponding to each cluster center are representative feature images in the cluster, and thus can be regarded as the K representative images with the largest difference in the entire image set.

[0049] The embodiment of the present application can extract high-dimensional features of images through the DINO model, cluster the images into K clusters using K-means clustering, select the center images of each cluster as the maximum difference samples, compress the data size while retaining 90% information entropy, reduce the data processing amount, focus on core visual features, and accurately select representative images in a quantitative manner, thereby improving the recognition ability of multi-modal large models in the scene.

[0050] Optionally, in an embodiment of the present application, constructing the picture content instance list comprises: counting the number of category instances contained in the image, and performing numbering processing on the missing categories to label the instances in the image one by one, give each instance a unique index, and supplement the corresponding description information for all labeled instances.

[0051] As shown in FIG. 1, in actual execution process, the process of manually constructing the image content instance list can include the following steps: Figure 2

[0052] Step S201: Count the instance categories and their quantities contained in the image, and perform numbering management on the missing categories.

[0053] ​Specifically, the number of instances contained in an image varies, and the categories are diverse. If a fixed number of instance categories is pre-set, it may not cover all the instance categories that actually appear. In this embodiment of the application, the number of existing categories can be set to M, and the number of newly added instances in each image can be set to C. These instances are then assigned an index range of c∈{1,2,…,M+C}. In this way, instances of different categories are uniquely indexed, enabling effective differentiation and management.

[0054] Step S202: manually label each instance in the image one by one and assign a unique index to it.

[0055] like Figure 3 As shown in the figure, for each instance in each annotated image, its information structure includes the following: S301 is the unique index of the instance in the image, used to distinguish different instances in the image; S302 is the category index of the instance; S303 is the semantic feature description of the instance; S304 is the location information box of the instance, represented by [x, y, h, w], where x and y are the coordinates of the upper left corner of the annotation box in the image, and h and w are the height and width of the annotation box; S305 represents the target image of the annotation. Each instance in the image is manually annotated one by one and assigned a unique index.

[0056] Step S203: Detailed description information may be optionally added to all labeled instances to enhance the completeness and semantic differentiation capability of data annotation.

[0057] For example, the detailed description information in the embodiments of this application is located at Figure 3 For example, the position shown in S303 in the image is “the person wearing red clothes in the lower left corner of the image”. This type of semantic expression helps to distinguish instances of the same category. Its semantic information is unique in the image, ensuring clear identification between instances.

[0058] The embodiment of the present application can count the number of instance categories, number the newly added categories according to the M+C rule, label each instance with a unique index, location box and semantic description, form a structured list, realize refined instance management, avoid labeling confusion, and provide accurate location and semantic basis for question and answer pair generation, thereby improving the reliability and semantic understanding accuracy of the visual question answering system.

[0059] In step S103, based on the terms related to the image content instances in the image content instance list, the image content instance list is screened and modified to generate visual question-answer pairs and construct a visual question-answer pair dataset.

[0060] In actual execution, the embodiment of the application can introduce a multi-modal visual large model to verify and correct the annotation results, automatically identify and point out potential category annotation errors, and thus generate high-quality instance images after cleaning. After completing image cleaning, corresponding visual question and answer pairs are automatically generated based on the images, the constructed instance list is screened and corrected, and finally visual question and answer pairs are automatically generated to construct a visual question and answer pair dataset.

[0061] The embodiment of the application can introduce a multi-modal visual large model to verify and correct the annotation results, realize efficient and accurate generation of question and answer pairs, and thus comprehensively cover multi-modal semantic understanding scenarios and effectively improve the generalization ability of multi-modal large models in complex visual question and answer tasks.

[0062] Optionally, in an embodiment of the application, generating a visual question and answer pair includes: generating a count type question and answer pair based on the number of instances of the same category; generating a reference type question and answer pair based on different description instances; generating an existence question and answer pair based on whether the instance exists; and generating a region type question and answer pair based on the instance annotation position.

[0063] For example, after completing image cleaning, the embodiment of the application automatically generates corresponding visual question and answer pairs based on the images.

[0064] As shown in Figure 4 The generation of question and answer pairs depends on the following predefined question and answer template rules:

[0065] S401: Use a large model to preliminarily clean up examples with incorrect annotations in the picture.

[0066] S402: Automatically generate question and answer pairs.

[0067] S403 Count: Generate a count type question and answer pair based on the number of instances of the same category: count the number of instances of each category in the image, and generate a question and answer pair according to the following template:

[0068] Q: How many [categories] are there in the image? Please output the number.

[0069] A: [The number of categories]

[0070] S404 Reference: Generate a reference type question and answer pair based on instance description information: for instances containing detailed description information in the image, ask for their position information in the image, and generate a question and answer pair in the following format:

[0071] Q: Where is [description] located in the image? Please output the coordinates.

[0072] A: [Coordinates of the instance]

[0073] S405 Existence: Existence class question-answer pairs are generated based on instance existence: whether the labeled instance in the image exists is determined, and the following template question-answer pair is generated:

[0074] Q: Does [description] exist in the image? Please output "exists" or "does not exist".

[0075] A: exists

[0076] S406 Region: Region class question-answer pairs are generated based on instance location: the class of the instance with known coordinate position is determined, and the following type of question-answer pair is generated:

[0077] Q: What is [instance coordinate]? Please select from [correct class], [interference class].

[0078] A: [correct class]

[0079] S407: Integrate the data set.

[0080] S408: Use model evaluation.

[0081] Based on the above question-answer templates, for each image with completed instance labeling, the corresponding visual question-answer pair is automatically generated. The automatic generation process is performed by a program in a computer-readable medium, and the construction of the question-answer content is completed according to the pre-defined template rules.

[0082] The embodiments of the present application can generate question-answer pairs according to pre-defined templates, such as counting class based on instance quantity, reference class based on attribute description, existence class based on instance existence, and region class based on coordinate position. Combined with T5 model rewriting problem variants, multiple semantic requirements are covered, the reasoning ability of the model in quantity calculation and spatial positioning dimensions is strengthened, and the image instance information is systematically mined, thereby significantly improving the comprehensive performance of the model in complex visual question-answer tasks.

[0083] In step S104, it is determined whether the visual question-answer pair data set meets the pre-set quality condition, wherein if the pre-set quality condition is met, interference samples are generated according to the visual question-answer pair data set, and the visual question-answer automatic labeling of the multi-modal large model is completed.

[0084] It can be understood that the pre-set quality condition in the embodiments of the present application can be the accuracy rate of the model's answers to all questions; the accuracy of the visual question-answer pair is defined as the proportion of the number of question-answer pairs correctly answered by the model to the total number of question-answer pairs.

[0085] In actual execution, the embodiment of the application can perform quality evaluation on the generated visual question and answer pair dataset, generate interference samples for the generated visual question and answer dataset, and further construct interference samples on the basis of the generated question and answer pairs to enhance the robustness and diversity of the visual question and answer dataset. Different types of question and answer pairs have different construction methods of interference samples.

[0086] Specifically, in an image with completed instance labeling, there are description information of several instances. The description information is input into a multi-modal large model to generate "pseudo description information" that is similar to the original description semantics but does not exist in the image. The existence question and answer template constructed by using the pseudo description information generates interference question and answer pairs as negative samples in the dataset. The question and answer template is as follows:

[0087] Q: Does [new description] exist in the image? Please output existence or nonexistence.

[0088] A: Nonexistence

[0089] Further, for the region class question and answer pair, the model is required to select among multiple classes in the question design. One is the correct class, and the others are interference classes. In order to enhance the confusion of the question and answer, a group of interference classes with similar semantics are automatically generated by inputting the correct class into the multi-modal large model. For example, if the correct class is "red small car", the generated interference classes include "red bus", "red truck", etc.

[0090] The embodiment of the application can integrate the constructed interference question and answer pairs with the original question and answer pairs to form an image question and answer dataset with perfect structure and containing positive and negative samples, which is used to improve the generalization ability and anti-interference ability of the subsequent multi-modal question and answer model.

[0091] Optionally, in an embodiment of the application, the method further includes: calculating the accuracy of the visual question and answer pair; and obtaining the correct response ability of the multi-modal large model to the specific question and answer pair according to the accuracy of the visual question and answer pair to obtain a visual question and answer quality result.

[0092] In actual execution, the embodiment of the application can use all generated visual question and answer pairs to constitute a complete image question and answer dataset. In order to effectively evaluate the quality of the dataset, the current mainstream multi-modal large model for visual question and answer is used for testing, and the accuracy of the model in answering all questions and answers is calculated.

[0093] The embodiment of the application can use the mainstream multi-modal model to predict the answers of the question and answer pairs, calculate the accuracy, analyze the error distribution of different types of question and answer pairs, generate a quality evaluation report, provide a quantitative basis for dataset optimization, locate the weaknesses of the model, and guide the adjustment of subsequent training strategies.

[0094] According to the method for automatically labeling visual question answering based on a multimodal large model provided in the embodiments of the present application, multi-scene basic image data can be collected and preprocessed, picture content instance list-related terms are constructed, visual question answering pairs and data sets are screened and corrected, interference samples are generated to complete automatic labeling after meeting the quality conditions, so that the labeling result is accurate, and the robustness of the multimodal large model to data bias is enhanced, and the quality and reliability of visual question answering labeling are significantly improved. Thus, the problems in the related art that the single-modal model has limitations in cross-modal semantic alignment, fine-grained relationship reasoning, and multimodal knowledge retrieval, leading to semantic bias, context fragmentation, or knowledge noise in the labeling result, and that the multimodal large model is significantly affected by model hallucination and data bias, etc., resulting in a significant impact on the actual labeling quality, are solved.

[0095] Secondly, the device for automatically labeling visual question answering based on a multimodal large model according to the embodiments of the present application is described with reference to the accompanying drawings.

[0096] Figure 5 is a block schematic diagram of the device for automatically labeling visual question answering based on a multimodal large model according to the embodiments of the present application.

[0097] As shown in Figure 5 , the device for automatically labeling visual question answering based on a multimodal large model 10 comprises an acquisition module 100, an association module 200, a correction module 300, and a labeling module 400.

[0098] The acquisition module 100 is configured to acquire basic image data for constructing visual question answering, wherein the basic image data at least includes image data of different scenes.

[0099] The association module 200 is configured to pre-process the basic image data to construct a picture content instance list and associate terms related to picture content instances in the picture content instance list.

[0100] The correction module 300 is configured to screen and correct the picture content instance list based on the terms related to the picture content instances in the picture content instance list to generate visual question answering pairs and construct a visual question answering pair data set.

[0101] The labeling module 400 is configured to determine whether the visual question answering pair data set meets a preset quality condition, wherein if the preset quality condition is met, interference samples are generated according to the visual question answering pair data set to complete the automatic labeling of visual question answering of the multimodal large model.

[0102] Optionally, in an embodiment of the present application, the association module 200 comprises a selection unit.

[0103] The selecting unit is configured to select, from the basic image data, a maximum difference image with the largest similarity difference gap among a preset number of images.

[0104] Optionally, in an embodiment of the present application, the association module 200 comprises a supplement unit.

[0105] The supplement unit is configured to count the number of category instances contained in the image, and perform numbering processing on the missing categories to label the instances in the image one by one, and assign a unique index to each instance, and supplement the corresponding description information for all labeled instances.

[0106] Optionally, in an embodiment of the present application, generating the visual question and answer pair comprises: generating a counting type question and answer pair based on the number of the same category instances; generating a reference type question and answer pair based on different description instances; generating an existence type question and answer pair based on whether the instance exists; and generating a region type question and answer pair based on the instance labeling position.

[0107] Optionally, in an embodiment of the present application, the method further comprises a calculation module and an acquisition module.

[0108] The calculation module is configured to calculate the accuracy of the visual question and answer pair.

[0109] The acquisition module is configured to obtain, according to the accuracy of the visual question and answer pair, the correct response ability of the multi-modal large model to a specific question and answer pair, to obtain a visual question and answer quality result.

[0110] It should be noted that the foregoing explanation and description of the embodiment of the method for automatically labeling visual question and answer based on a multi-modal large model also applies to the embodiment of the device for automatically labeling visual question and answer based on a multi-modal large model, which will not be described here.

[0111] The device for automatically labeling visual question and answer based on a multi-modal large model according to the embodiment of the present application can collect multi-scene basic image data and pre-process, construct a picture content instance list associated with a term, screen and correct to generate a visual question and answer pair and a data set, generate an interference sample to complete automatic labeling after meeting the quality condition, so as to make the labeling result accurate, and further enhance the robustness of the multi-modal large model to data bias, and significantly improve the quality and reliability of visual question and answer labeling. Thus, the problems of semantic bias, context fragmentation or knowledge noise in the labeling result caused by the limitations of single-modal models in cross-modal semantic alignment, fine-grained relationship reasoning, multi-modal knowledge retrieval, etc. in related technologies are solved, and the actual labeling quality is significantly affected by the model illusion and data bias of the multi-modal large model.

[0112] Figure 6 The structure schematic diagram of the electronic device provided in the embodiment of the present application. The electronic device can comprise:

[0113] The memory 601, the processor 602 and the computer program stored on the memory 601 and executable on the processor 602.

[0114] The processor 602 implements the multi-modal large model-based visual question answering automatic labeling method provided in the above embodiments when executing the program.

[0115] Further, the electronic device further comprises:

[0116] The communication interface 603 is used for communication between the memory 601 and the processor 602.

[0117] The memory 601 is used to store the computer program executable on the processor 602.

[0118] The memory 601 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0119] If the memory 601, the processor 602 and the communication interface 603 are independently implemented, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0120] Optionally, in specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0121] The processor 602 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0122] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the visual question answering automatic labeling method based on a multi-modal large model.

[0123] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the visual question answering automatic labeling method based on a multi-modal large model.

[0124] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0125] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0126] Any process or method descriptions in flow charts or otherwise described herein represent embodiments that can be understood as a module, segment, or portion of code that includes one or N executable instructions for implementing the specified logical function or process, and the scope of the preferred embodiments of the present application includes additional implementation in which the functions described are performed in a different order, including substantially simultaneously, or in reverse order, as will be understood by those skilled in the art of the embodiments to which the present application pertains.

[0127] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0128] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0129] Those of skill in the art would understand that the steps of the methods carried out above can be carried out wholly or partly by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0130] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0131] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for automatic annotation of visual question answering based on a multimodal large model, characterized by: The following steps are involved: Collecting basic image data for constructing visual question answering, wherein the basic image data at least includes image data of different scenes; Preprocessing the basic image data to construct a picture content instance list, and associating terms related to the picture content instances in the picture content instance list; Based on the terms related to the image content instances in the image content instance list, screening and revising the image content instance list to generate visual question-answer pairs and construct a visual question-answer pair dataset; Determine whether the visual question-answering pair dataset meets a preset quality condition, wherein if the preset quality condition is met, generate interference samples based on the visual question-answering pair dataset to complete the automatic labeling of the visual question-answering of the multimodal large model.

2. The method according to claim 1, characterized in that The preprocessing of the basic image data includes: From the basic image data, a maximum difference image with the largest similarity difference among a preset number of images is selected.

3. The method according to claim 1, characterized in that The step of constructing a list of picture content instances includes: Count the number of category instances contained in the image and number the missing categories to mark the instances in the image one by one, assign a unique index to each instance, and supplement the corresponding descriptive information for all marked instances.

4. The method according to claim 1, wherein Generating a visual question-answer pair includes: Generate count-based question-answer pairs based on the number of instances of the same category; Generate reference question-answer pairs based on different description instances; Generate existence question-answer pairs based on whether the instance exists; Generate region-class question-answer pairs based on instance annotation locations.

5. The method according to claim 1, wherein: Also includes: Calculating the accuracy of the visual question-answer pair; According to the accuracy of the visual question-answer pair, the correct response ability of the multimodal large model to a specific question-answer pair is obtained to obtain the visual question-answer quality result.

6. A visual question answering automatic annotation device based on a multimodal large model, characterized by: include: An acquisition module, configured to acquire basic image data for constructing visual question answering, wherein the basic image data at least includes image data of different scenes; an association module, configured to pre-process the basic image data to construct a picture content instance list and associate terms related to the picture content instances in the picture content instance list; a correction module, configured to filter and correct the image content instance list based on the terms related to the image content instance in the image content instance list to generate visual question-answer pairs and construct a visual question-answer pair dataset; A labeling module is used to determine whether the visual question-answering dataset meets preset quality conditions. If the preset quality conditions are met, interference samples are generated based on the visual question-answering dataset to complete the automatic labeling of visual question-answering for the multimodal large model.

7. The device according to claim 6, characterized in that The association module includes: The selection unit is configured to select, from the basic image data, a maximum difference image with the largest similarity difference among a preset number of images.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the automatic labeling method for visual question answering based on a multimodal large model as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the automatic annotation method for visual question answering based on a multimodal large model as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that The computer program is executed to implement a visual question answering automatic annotation method based on a multimodal large model as described in any one of claims 1 to 5.

Images