Method and device for generating VQA data set, electronic equipment and readable medium

Abstract

The embodiment of the invention provides a method and device for generating a VQA data set, electronic equipment and a readable medium. The method comprises the steps that a collected image data set is acquired; a visual language model is adopted to determine a counting problem proposed for each image; extracting an entity word from each counting question to obtain an entity word corresponding to each counting question; detecting the position and the number of each entity word in the corresponding image by using an open set target detection model to obtain detection information corresponding to each entity word; wherein the detection information comprises position information and quantity information; fusing each counting question and the corresponding detection information into a target statement of each counting question, and taking the target statement of each counting question as an answer of each counting question; the VQA data set is generated by using each image, the counting question proposed for each image and the answer to each counting question, and compared with an existing method, the method not only can improve the generation efficiency, but also can generate a high-quality VQA data set.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and in particular to a method for generating a VQA dataset, a device for generating a VQA dataset, an electronic device and a readable medium. Background Art

[0002] Counting and localization is an important subtask in the VQA task. It requires the model to answer questions based on the number and location of objects in the image, such as "How many people are there in this image?", "How many apples are there in this image?", etc. In addition, the counting and localization task requires the model to have not only counting capabilities, but also localization capabilities, that is, the ability to generate the coordinate frame of the object. The counting and localization task places higher requirements on the localization and counting capabilities of the multimodal model.

[0003] Currently, existing methods use large language models (such as GPT-4) to generate instructions to fine-tune VQA data. These methods usually require providing image-related annotations and quantities, and then use large language models to generate corresponding question-answer pairs.

[0004] However, since it requires manual annotation of information such as object category, location, description, etc. in the image, there is at least a problem of low efficiency in generating VQA datasets. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and readable medium for generating a VQA dataset to solve the problem of low efficiency in generating a VQA dataset in the prior art.

[0006] The embodiment of the present application discloses a method for generating a VQA dataset, comprising: acquiring a collected image dataset; wherein the image dataset comprises multiple images of different scenes and different objects; using a visual language model to determine a counting question posed for each image; extracting entity words from each counting question to obtain an entity word corresponding to each counting question; using an open set target detection model to detect the position and quantity of each entity word in the corresponding image to obtain detection information corresponding to each entity word; wherein the detection information comprises position information and quantity information; fusing each counting question and its corresponding detection information into a target sentence for each counting question, and using the target sentence for each counting question as an answer to each counting question; and generating a VQA dataset using each image, the counting question posed for each image, and the answer to each counting question.

[0007] In a possible embodiment, the visual language model is any one of the Qwen2-VL model, the InternVL2 model and the CogVLM model.

[0008] In a possible embodiment, entity words are extracted from each counting question to obtain entity words corresponding to each counting question, including: using a large language model to extract entity words from each counting question respectively to obtain entity words corresponding to each counting question.

[0009] In a possible embodiment, the large language model is any one of the Qwen2.5 model, the TeleChat2 model and the Baichuan2 model.

[0010] In a possible embodiment, each counting question and its corresponding detection information are fused into a target sentence for each counting question, including: using a large language model to fuse each counting question and its corresponding detection information into a target sentence for each counting question.

[0011] In a possible embodiment, the open set object detection model is a GroundingDINO model.

[0012] In one possible embodiment, a VQA dataset is generated using each image, a counting question posed for each image, and an answer to each counting question, including: evaluating the answer to each counting question to obtain an evaluation result of each counting question; if the evaluation result of each counting question meets expectations, a VQA dataset is generated using each image, a counting question posed for each image, and an answer to each counting question.

[0013] An embodiment of the present application also provides a device for generating a VQA dataset, including: an acquisition module, used to acquire a collected image dataset; wherein the image dataset includes multiple images of different scenes and different objects; a determination module, used to use a visual language model to determine the counting questions raised for each image; an extraction module, used to extract entity words from each counting question to obtain the entity words corresponding to each counting question; a detection module, used to use an open set target detection model to detect the position and quantity of each entity word in the corresponding image to obtain detection information corresponding to each entity word; wherein the detection information includes position information and quantity information; a fusion module, used to fuse each counting question and its corresponding detection information into a target sentence for each counting question, and use the target sentence of each counting question as the answer to each counting question; a generation module, used to generate a VQA dataset using each image, the counting question raised for each image, and the answer to each counting question.

[0014] The present application also provides a storage medium in an embodiment, on which a computer program is stored. When the computer program is executed by a processor, the method described in the embodiment of the present application is executed.

[0015] The present application also provides an electronic device in an embodiment, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the method described in the embodiment of the present application is performed.

[0016] The present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the method described in the embodiments of the present application.

[0017] The embodiments of the present application include the following advantages:

[0018] The embodiments of the present application provide a method, apparatus, electronic device and readable medium for generating a VQA dataset, by acquiring a collected image dataset, wherein the image dataset includes multiple images of different scenes and different objects, and using a visual language model to determine a counting question posed for each image, and extracting entity words from each counting question to obtain the entity words corresponding to each counting question, and using an open set target detection model to detect the position and quantity of each entity word in the corresponding image to obtain detection information corresponding to each entity word; wherein the detection information includes position information and quantity information, and each counting question and its corresponding detection information are fused into a target sentence for each counting question, and the target sentence of each counting question is used as the answer to each counting question, and each image, the counting question posed for each image and the answer to each counting question are used to generate a VQA dataset. Compared with the existing methods, it can not only improve the generation efficiency, but also generate a high-quality VQA dataset.

[0019] In order to make the above-mentioned objectives, features and advantages to be achieved by the embodiments of the present application more obvious and understandable, the following specifically cites preferred embodiments and describes them in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a method for generating a VQA dataset provided by an embodiment of the present application is shown;

[0022] Figure 2A A schematic diagram of an original image provided in an embodiment of the present application is shown;

[0023] Figure 2B A schematic diagram of the result of detecting the entity word "ball" in the GroundingDINO model image 2A provided in an embodiment of the present application is shown;

[0024] Figure 3 A structural block diagram of a device for generating a VQA dataset provided in an embodiment of the present application is shown;

[0025] Figure 4 A block diagram of an electronic device provided by an embodiment of the present application is shown;

[0026] Figure 5 It is a schematic diagram of a readable medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] A multimodal model is a model that can handle different types of data, such as images, text, audio, etc. In addition, multimodal models are widely used in many tasks, such as visual question answering (VQA), image description, image retrieval, etc. VQA is a task that requires a model to generate a natural language answer given an image and a natural language question. The VQA task can evaluate the understanding and reasoning ability of multimodal models, and also raises some challenges, such as diversity, uncertainty, multimodal fusion, etc.

[0029] In addition, counting and localization is an important subtask in the VQA task, which requires the model to answer questions based on the number and location of objects in the image, such as "How many people are there in this image?", "How many apples are there in this image?", etc. The counting and localization task requires the model to have not only counting capabilities, but also localization capabilities, that is, the ability to generate the coordinate frame of the object. The counting and localization task places higher requirements on the localization and counting capabilities of the multimodal model.

[0030] At present, there are some datasets used for VQA task research, such as VQA2.0, CLEVR, GQA, etc. However, there are fewer counting and localization problems in these datasets, and they are often related to specific fields or scenarios. For example, the problems in the CLEVR dataset are all about geometric shapes. The limitations of these datasets make it difficult to fully train and evaluate the localization and counting capabilities of multimodal models, and are not conducive to improving the generalization ability of multimodal models.

[0031] To solve this problem, some existing methods try to use large language models (such as GPT-4) to generate instructions to fine-tune VQA data. In addition, these methods usually require providing image-related annotations and quantities, and then use large language models to generate corresponding question-answer pairs. However, these methods have the following disadvantages:

[0032] High labeling cost: Since GPT-4 cannot see real images, it is necessary to manually label the object category, location, description and other information in the image. This labeling work is time-consuming and labor-intensive, and it is difficult to cover all the information in the image;

[0033] Low answer quality: Since GPT-4 can only rely on limited annotation information to generate questions and answers, it is easy to ignore the details in the image or generate answers that are inconsistent with the image. For example, when there are two people and a table in the image, GPT-4 may generate "this image has three people and a table", or "this image has two people and a chair".

[0034] Therefore, there is an urgent need for a method that can generate high-quality question-answer pairs and a counting and localization VQA dataset with low annotation cost.

[0035] Based on this, the embodiments of the present application provide a method, device, electronic device and readable medium for generating a VQA dataset, by acquiring a collected image dataset, wherein the image dataset includes multiple images of different scenes and different objects, and using a visual language model to determine the counting questions raised for each image, and extracting entity words from each counting question to obtain the entity words corresponding to each counting question, and using an open set target detection model to detect the position and quantity of each entity word in the corresponding image to obtain detection information corresponding to each entity word; wherein the detection information includes position information and quantity information, and each counting question and its corresponding detection information are fused into a target sentence for each counting question, and the target sentence of each counting question is used as the answer to each counting question, and using each image, the counting question raised for each image and the answer to each counting question to generate a VQA dataset. Compared with the existing method, it can not only improve the generation efficiency, but also generate a high-quality VQA dataset.

[0036] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0037] In order to facilitate understanding of the embodiments of the present application, the terms involved in the present application are explained as follows:

[0038] "Instruction fine-tuning": It is an NLP technology that aims to further train a pre-trained language model to better understand and follow instructions or commands given by humans. This process is achieved by building a dataset containing instructions and expected outputs and using the dataset to fine-tune the model. In addition, the core of instruction fine-tuning is to improve the performance of the model on specific tasks so that it can generate more accurate results that meet the requirements of instructions.

[0039] Visual Question Answering (VQA) dataset: It is a dataset for visual question answering tasks, which aims to generate answers from images and questions. It consists of images, questions, and corresponding answers, and each sample contains an image, a question, and an answer. The image can be a photo of a real scene or a synthetic image, the question is a natural language question about the image content, and the answer is the answer to the question.

[0040] "FSC-147 Dataset": It is a dataset consisting of 147 object categories, which can include more than 6,000 images, and this dataset is designed to solve the problem that existing visual counting research mainly focuses on a single specific category, such as people, animals, and cells, and cannot count objects of any category.

[0041] "Qwen2-VL model": It is a multimodal pre-trained model designed to understand and generate content that combines visual and language information. This model combines the techniques of natural language processing (NLP) and computer vision (CV) and can perform well in a variety of tasks.

[0042] "Qwen2.5 model": It is a language model, and the Qwen2.5 model contains a series of models with various B parameters, and in practical applications, you can choose any model from this series to use.

[0043] "GroundingDINO model": It is a Transformer-based object detection model that combines the advantages of CLIP (Contrastive Language-Image Pre-training) and DINO (Deformable DETR) to achieve open-set object detection. The main feature of GroundingDINO is that it can detect any object based on text description without retraining the model, which makes it perform well in zero-shot object detection.

[0044] "GPT-4 model": It is a multimodal model that can process inputs in multiple modes, such as text and images. It can generate response text, classify, analyze and extract implicit semantics from visual elements, and demonstrate excellent response capabilities. This multimodal processing capability makes GPT-4 more advantageous in handling complex tasks.

[0045] "InternVL2 model": It is a multimodal pre-trained model specifically designed to handle joint vision and language tasks. InternVL2 performs well in multiple benchmarks, especially in tasks such as image description generation, visual question answering (VQA), and text-to-image generation.

[0046] "CogVLM model": It is a visual language model that aims to improve the processing capabilities of multimodal tasks by deeply integrating visual and language features. In addition, CogVLM introduces a trainable visual expert module to deeply integrate language encoding and visual encoding in the attention layer and feedforward network layer, achieving deep fusion of visual language features without sacrificing the performance of any natural language processing (NLP) tasks.

[0047] "TeleChat2 Model": It is a language model.

[0048] "Baichuan2 model": It is an open source large language model, mainly including two versions, Baichuan2-7B and Baichuan2-13B. These models are trained based on 2.6 trillion tokens of high-quality multilingual data, and have achieved the best results of the same size on multiple authoritative Chinese, English and multilingual general domain benchmarks.

[0049] See also Figure 1 , Figure 1 A flowchart of a method for generating a VQA dataset provided by an embodiment of the present application is shown. It should be understood that the method can be executed by a device for generating a VQA dataset, and the specific device of the device can be set according to actual needs, and the embodiment of the present application is not limited thereto. For example, the device can be a computer, or a server, etc. Specifically, the steps of the method include:

[0050] Step S110, obtaining a collected image data set, wherein the image data set includes multiple images of different scenes and different objects, so that it is possible to consider collecting images of different scenes and objects to cover a wider range of counting problems.

[0051] It should be understood that the images in the image dataset are images used to generate count question-answer pairs.

[0052] It should also be understood that the specific data set of the image data set and the images contained therein can also be set according to actual needs, and the embodiments of the present application are not limited to this.

[0053] For example, the image dataset may be a FSC-147 dataset, or other non-counting dedicated datasets.

[0054] For another example, the images in the image dataset may involve multiple scenes, and the objects in the images corresponding to each of the multiple scenes are also different.

[0055] Step S120: using a visual language model to determine a counting question for each image.

[0056] It should be understood that the specific model of the visual language model can be set according to actual needs, as long as it can determine the counting problem of the image, and the embodiments of the present application are not limited to this.

[0057] For example, the visual language model is any one of the Qwen2-VL model, the InternVL2 model, and the CogVLM model. Also, the effects of different models in generating count question-answer pairs can be compared, so as to select the optimal visual language model.

[0058] To facilitate understanding of step S120, a specific embodiment is used to describe it below.

[0059] Specifically, all images in the image dataset can be preprocessed to obtain a preprocessed dataset. And, each image in the preprocessed dataset is input into the visual language model to obtain the counting questions for the image output by the visual language model, such as "how many people are there in the image", "how many apples are there in the image", "how many cats are there in the image", etc., so as to obtain the image and corresponding question dataset VQ.

[0060] It should be understood that the specific process of preprocessing can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0061] For example, the pre-processing may include resizing and the like.

[0062] Step S130, extracting entity words from each counting question to obtain entity words corresponding to each counting question.

[0063] Specifically, a large language model is used to extract entity words from each counting question to obtain the entity words corresponding to each counting question.

[0064] For example, for each counting question, you can design prompt words to extract entity words, use a large language model to extract entity words in the question, and get a dataset VQE about images, questions, and entity words.

[0065] It should be understood that the specific model of the large language model can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0066] For example, the large language model is any one of the Qwen2.5 model, the TeleChat2 model, and the Baichuan2 model. Also, the effects of different models in generating count question-answer pairs can be compared, so as to select the optimal large language model.

[0067] Step S140, using an open set object detection model to detect the position and quantity of each entity word in the corresponding image, and obtain detection information corresponding to each entity word. The detection information includes position information and quantity information, and the position information can be coordinates.

[0068] It should be understood that the specific model of the open set target detection model can be set according to actual needs, as long as it can detect the detection information corresponding to the entity words in the image. The embodiments of the present application are not limited to this.

[0069] For example, the open set target detection model is a GroundingDINO model.

[0070] To facilitate understanding of step S140, a specific embodiment is used to describe it below.

[0071] Specifically, for each image and corresponding entity word data, the GroundingDINO model is used for detection to obtain the coordinate information and quantity of each entity word. GroundingDINO's target detection capability is stronger than that of the multimodal model, so the detection results of GroundingDINO are taken as the credible results of the problem.

[0072] Step S150, each counting question and its corresponding detection information are integrated into a target sentence of each counting question, and the target sentence of each counting question is used as the answer to each counting question.

[0073] Specifically, a large language model is used to fuse each counting question and its corresponding detection information into a target sentence for each counting question, and the target sentence for each counting question is used as the answer to each counting question.

[0074] For example, a large language model is used to combine the entity word detection information (quantity, coordinates) and the corresponding question into a complete and coherent sentence, and the sentence is used as the answer to the question.

[0075] It should be noted here that the large language model used in step S150 and the large language model used in step S130 may be the same model or different models.

[0076] Step S160 , generating a VQA dataset using each image, the counting question posed for each image, and the answer to each counting question.

[0077] Specifically, the answer evaluation model can be used to evaluate the answer to each counting question to obtain the evaluation result of each counting question. If the evaluation result does not meet expectations, the model can be adjusted iteratively to improve the quality of the answer until the evaluation result of each counting question meets expectations. Then, each image, the counting question raised for each image, and the answer to each counting question can be integrated into the required format to generate a VQA dataset, thereby ensuring that the format and structure of the VQA dataset meet the requirements of the specified task.

[0078] It should be understood that the specific model of the answer evaluation model can be set according to actual needs, as long as the answer evaluation model can realize the evaluation of answers to counting questions. The embodiments of the present application are not limited to this.

[0079] For example, the answer evaluation model may be a GPT4 model or a Qwen2.5-72B-Instruct model, etc.

[0080] Therefore, with the help of the above technical solution, this application solves the problem of generating VQA data sets for multimodal model instruction fine-tuning counting tasks. Also, if the multimodal model is used directly to directly generate the answer in the question-answer pair, it may be inaccurate, and the counting task has very high requirements for the accuracy of the answer. Therefore, a large language model is introduced to extract entity words from the question, and then the open set target detection model is used to detect the entity words. This can provide more accurate quantity and coordinate information, thereby improving the accuracy of the answer. Finally, by using the large language model to merge the obtained quantity, coordinate information and question into a complete and fluent sentence, a more natural and fluent answer can be generated.

[0081] In addition, the technical solution of this application can generate a more accurate, natural and fluent counting question and answer pair dataset, providing high-quality fine-tuning data for counting VQA tasks, which will help improve the performance of multimodal models in counting question and answering.

[0082] Moreover, compared with traditional object detection models, GroundingDINO in this application provides higher accuracy and detailed detection capabilities, especially when dealing with complex scenes and diverse entity words. Moreover, it is an open-set object detection model with strong zero-shot capability, avoiding the need to train a closed-set object detection model for specific targets.

[0083] Furthermore, this application generates more accurate and natural answers by combining specific quantity information and the spatial location of objects with the help of a large language model. Compared with manually annotating answers to questions, the efficiency of generating answers is higher and the labor cost is lower.

[0084] Also, large language models are used to extract entity words, so that large language models can more accurately understand and process complex natural language queries, especially in extracting and understanding entity words, which improves the accuracy of question processing and lays a solid foundation for subsequent target detection and answer generation.

[0085] In addition, using visual language models (such as Qwen2-VL) to generate counting questions, and multimodal models such as Qwen2-VL can intuitively understand the image content and generate relevant counting questions based on this. This method is more natural and closer to actual application scenarios than traditional rule-based question generation, and improves the relevance and diversity of the questions.

[0086] In addition, by collecting diverse image data sets, especially non-counting-specific data sets, the present application can cover a wider range of counting problems, enhance the versatility and applicability of the model, and is not limited to data sets for specific scenarios such as crowd counting. The method of generating VQA data can be implemented for any image, thereby improving the adaptability of the model in different scenarios.

[0087] In order to facilitate understanding of the embodiments of the present application, a description is given below using specific embodiments.

[0088] Specifically, the FSC-147 dataset is obtained, and the images in the FSC-147 dataset are preprocessed to obtain the preprocessed dataset. Also, the preprocessed dataset is input into the Qwen2-VL model, and prompt words are designed to ask counting questions for the image. For example, the prompt words that guide the model to ask questions: "You are a counting master who is good at seeing the number of objects in the picture. Please ask a question based on the number of objects in the picture. For example, how many cats are there in the picture? How many apples are there in the picture?" The Qwen2-VL model will generate questions about counting based on the prompt words.

[0089] And, input the generated questions into the Qwen2.5 model, and design prompt words to extract entity words. For example, "You are a language assistant who can help extract information from a given sentence. Given a sentence, extract the entities in the sentence for me. Extract common objects and generalize them into general categories without duplication, and merge objects that are similar in nature. Avoid extracting abstract or non-specific entities. Extract entities in singular form. Output all extracted item types in a row, and separate each object type with a period. If there is nothing to output, output a "none". Example: Sentence: How many apples and bananas are there in this photo? Output: apple.banana. Sentence: How many cats are there in this picture. Output: cat" and so on. The Qwen2.5 model will extract entity words in the question based on the prompt words.

[0090] And, the extracted entity words are input into the GroundingDINO model, and objects are detected to obtain coordinates and numbers. Figure 2A and Figure 2B As shown, the coordinate box information of the entity words can be saved and the number can be calculated.

[0091] Also, the prompt words are designed so that the Qwen2.5 model can combine the question and the quantity and coordinate information of the entity words into a complete and fluent sentence. For example, "You are a master of language. According to the question, organize the entity words, quantity, and coordinate positions into a complete and fluent sentence as the answer to the question." The Qwen2.5 model will generate the answer to the question based on the prompt words. For example, "There are 2 apples in the picture, and they are distributed in the areas of [0.245, 0.571, 0.258, 0.393], [0.357, 0.318, 0.148, 0.228]."

[0092] In addition, the generated answers can be verified and corrected. For example, other models can be used to evaluate the answers, or manual spot checks can be performed to ensure the accuracy and rationality of the answers. If the answers do not meet expectations, iterative adjustments to the model can be considered to improve the quality of the answers.

[0093] And, organize images, questions and their answers into the data format required for model training to generate a VQA dataset.

[0094] It should be understood that the above method for generating a VQA dataset is only exemplary, and those skilled in the art may perform various modifications based on the above method, and the modified schemes also fall within the protection scope of the present application.

[0095] See also Figure 3 , Figure 3The structural block diagram of a device 300 for generating a VQA dataset provided by an embodiment of the present application is shown. It should be understood that the device 300 can perform each step in the above method embodiment. The specific functions of the device 300 can be referred to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device 300 includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device 300. Specifically, the device 300 includes:

[0096] An acquisition module 310 is used to acquire a collected image data set; wherein the image data set includes multiple images of different scenes and different objects;

[0097] A determination module 320 for determining a counting question for each image using a visual language model;

[0098] An extraction module 330 is used to extract entity words from each counting question to obtain entity words corresponding to each counting question;

[0099] A detection module 340 is used to detect the position and quantity of each entity word in the corresponding image using an open set object detection model to obtain detection information corresponding to each entity word; wherein the detection information includes position information and quantity information;

[0100] A fusion module 350, for fusing each counting question and its corresponding detection information into a target sentence for each counting question, and using the target sentence for each counting question as an answer to each counting question;

[0101] The generating module 360 ​​is used to generate a VQA dataset using each image, the counting question asked for each image, and the answer to each counting question.

[0102] In a possible embodiment, the visual language model is any one of the Qwen2-VL model, the InternVL2 model and the CogVLM model.

[0103] In a possible embodiment, the extraction module 330 is specifically configured to extract entity words from each counting question respectively using a large language model to obtain entity words corresponding to each counting question.

[0104] In a possible embodiment, the large language model is any one of the Qwen2.5 model, the TeleChat2 model and the Baichuan2 model.

[0105] In a possible embodiment, the fusion module 350 is specifically configured to use a large language model to fuse each counting question and its corresponding detection information into a target sentence for each counting question.

[0106] In a possible embodiment, the open set object detection model is a GroundingDINO model.

[0107] In one possible embodiment, the generation module 360 ​​is specifically used to: evaluate the answer to each counting question to obtain an evaluation result of each counting question; if the evaluation result of each counting question meets expectations, generate a VQA dataset using each image, the counting question raised for each image, and the answer to each counting question.

[0108] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0109] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0110] Memory 403, used for storing computer programs;

[0111] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:

[0112] Get the current frame video image in the campus surveillance video;

[0113] Acquire a collected image data set; wherein the image data set includes multiple images of different scenes and different objects;

[0114] Use visual language models to determine the counting questions to ask for each image;

[0115] Extract entity words from each counting question to obtain the entity words corresponding to each counting question;

[0116] Using an open set object detection model to detect the position and quantity of each entity word in the corresponding image, and obtaining detection information corresponding to each entity word; wherein the detection information includes position information and quantity information;

[0117] Each counting question and its corresponding detection information are fused into a target sentence for each counting question, and the target sentence for each counting question is used as the answer to each counting question;

[0118] A VQA dataset is generated using each image, the counting questions asked for each image, and the answers to each counting question.

[0119] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0120] The communication interface is used for communication between the above terminal and other devices.

[0121] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0122] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0123] like Figure 5 As shown, in another embodiment provided by the present invention, a readable medium 501 is also provided, in which instructions are stored, and when the instructions are run on a computer, the computer executes the abnormal behavior identification method on campus described in the above embodiment. The readable medium refers to a computer-readable storage medium.

[0124] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the campus abnormal behavior identification method described in the above embodiment.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method, and will not be elaborated here.

[0126] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0127] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0130] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0131] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0132] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating a VQA dataset, characterized in that: include: Acquire a collected image data set; wherein the image data set includes multiple images of different scenes and different objects; Determining a counting question to be asked for each of the images using a visual language model; Extracting entity words from each of the counting questions to obtain entity words corresponding to each of the counting questions; Using an open set object detection model to detect the position and quantity of each entity word in the corresponding image, and obtain detection information corresponding to each entity word; wherein the detection information includes position information and quantity information; Merging each of the counting questions and the corresponding detection information into a target sentence for each of the counting questions, and using the target sentence for each of the counting questions as an answer to each of the counting questions; The VQA dataset is generated using each image, a counting question posed for each image, and an answer to each counting question.

2. The method according to claim 1, characterized in that The visual language model is any one of the Qwen2-VL model, the InternVL2 model and the CogVLM model.

3. The method according to claim 1, characterized in that The extracting entity words from each of the counting questions to obtain the entity words corresponding to each of the counting questions includes: A large language model is used to extract entity words from each of the counting questions to obtain entity words corresponding to each of the counting questions.

4. The method according to claim 3, characterized in that: The large language model is any one of the Qwen2.5 model, the TeleChat2 model and the Baichuan2 model.

5. The method according to claim 3, characterized in that: The step of fusing each of the counting problems and the corresponding detection information into a target sentence for each of the counting problems includes: The large language model is used to fuse each of the counting questions and its corresponding detection information into a target sentence for each of the counting questions.

6. The method according to claim 1, characterized in that The open set target detection model is the GroundingDINO model.

7. The method according to claim 1, characterized in that The generating the VQA dataset by using each image, the counting question raised for each image and the answer to each counting question comprises: Evaluate the answer to each of the counting questions to obtain an evaluation result of each of the counting questions; If the evaluation result of each of the counting questions meets expectations, the VQA dataset is generated using each of the images, the counting questions posed for each of the images, and the answers to each of the counting questions.

8. A device for generating a VQA dataset, characterized in that: include: An acquisition module, used to acquire a collected image data set; wherein the image data set includes multiple images of different scenes and different objects; A determination module, configured to determine a counting question posed for each of the images using a visual language model; An extraction module, used to extract entity words from each of the counting questions to obtain entity words corresponding to each of the counting questions; A detection module, used to detect the position and quantity of each entity word in the corresponding image using an open set object detection model, and obtain detection information corresponding to each entity word; wherein the detection information includes position information and quantity information; A fusion module, used for fusing each of the counting questions and the corresponding detection information into a target sentence of each of the counting questions, and using the target sentence of each of the counting questions as an answer to each of the counting questions; A generating module is used to generate the VQA dataset by using each image, the counting questions raised for each image and the answers to each counting questions.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the campus abnormal behavior identification method as described in any one of claims 1-7 when executing the program stored in the memory.

10. A readable medium having instructions stored thereon, which, when executed by one or more processors, causes the processors to execute the campus abnormal behavior identification method according to any one of claims 1 to 7.

Images