A visual question answering method

Abstract

The present invention provides a visual question answering processing method, which relates to the field of artificial intelligence technology. The method comprises: obtaining a question text and question image data associated with the question text; performing intent recognition processing on the question text to obtain the intent type corresponding to the question text; for each question image, using the intent type and the question text, extracting target information from the question image; performing text recognition processing on each question image to obtain the image text corresponding to the question image; fusing the target information and the image text to generate a reply text corresponding to the question text. The image text extracted in the present invention can provide more details for understanding the question text, and the extracted target information can focus on the information related to the question text in subsequent reasoning, avoiding the interference of irrelevant information, thereby reducing model hallucinations and improving the accuracy of visual question answering.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a visual question answering processing method. Background Art

[0002] The Visual Language Model (VLM), also known as a multimodal large model, can understand visual images and text content, integrate visual information and language information, and complete image understanding tasks such as image description generation and visual question answering.

[0003] However, in visual question answering tasks, VLM has obvious visual defects, such as the inability to count the number of objects in the image and inaccurate text recognition. At the same time, the visual language model also has a serious hallucination problem and is prone to generating some erroneous content, resulting in low accuracy in visual question answering. Summary of the invention

[0004] In view of the above problems, the object of the present invention is to provide a visual question answering processing method to improve the accuracy of visual question answering.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides a visual question answering processing method, the method comprising:

[0007] Acquire a question text and question image data associated with the question text, wherein the question image data includes at least one question image;

[0008] Performing intent recognition processing on the question text to obtain the intent type corresponding to the question text;

[0009] For each of the question images, extract target information from the question image using the intention type and the question text;

[0010] Performing text recognition processing on each of the problem images to obtain image text corresponding to the problem image;

[0011] The target information and the image text are integrated to generate a reply text corresponding to the question text.

[0012] Optionally, the obtaining of the question text and the question image data associated with the question text includes:

[0013] Acquire a question text and original image data associated with the question text, wherein the original image data includes at least one original image;

[0014] If the original image data only includes one original image, determining the original image data as problematic image data;

[0015] If the original image data includes multiple original images, screening out the problematic image from the multiple original images;

[0016] The problem image data is generated using all the problem images.

[0017] Optionally, screening out problematic images from the multiple original images includes:

[0018] Performing sampling processing on the multiple original images to obtain images to be processed;

[0019] For each of the images to be processed, taking the image before the image to be processed as the designated image of the image to be processed;

[0020] Calculate the importance score of the image to be processed by fusing the image to be processed and the designated image;

[0021] Based on the importance score of each of the to-be-processed images, problematic images are screened out from the multiple original images.

[0022] Optionally, the fusing the image to be processed and the designated image to calculate the importance score of the image to be processed includes:

[0023] Calculating a parameter of a degree of change between the image to be processed and the designated image;

[0024] Calculating image feature parameters of the image to be processed by using the image features of the image to be processed;

[0025] Calculating motion feature parameters of the image to be processed by using the motion feature of the image to be processed;

[0026] A weighted calculation is performed on the change degree parameter, the image feature parameter and the motion feature parameter of the image to be processed to obtain an importance parameter of the image to be processed.

[0027] Optionally, performing intent recognition processing on the question text to obtain the intent type corresponding to the question text includes:

[0028] Obtaining a first prompt word template, wherein the first prompt word template includes a first question slot, multiple intent recognition examples, and intent recognition rules corresponding to the intent type;

[0029] Fill the question text into the first question slot to generate an intention prompt word;

[0030] The intention prompt word is input into the intention recognition model so that the intention recognition model understands the intention recognition rule based on the multiple intention recognition examples, and uses the intention recognition rule to infer the intention type corresponding to the question text.

[0031] Optionally, the intention type includes a detail type and an overall type, and for each of the question images, extracting target information from the question image using the intention type and the question text includes:

[0032] For each of the question images, if the intent type is a detail type, entity extraction is performed on the question text to obtain an entity extraction result;

[0033] If the entity extraction result does not contain the problem entity, the problem image is used as target information;

[0034] If the entity extraction result includes a problem entity, detecting an entity object corresponding to the problem entity in the problem image to obtain a detection result;

[0035] Extracting target information from the problem image using the detection result;

[0036] If the intention type is an overall type, the problem image is used as target information.

[0037] Optionally, performing entity extraction processing on the question text to obtain an entity extraction result includes:

[0038] Acquire a second prompt word template, wherein the second prompt word template includes a second question slot, an extraction rule, an extraction example, and an output format;

[0039] Filling the question text into the second question slot to obtain an extraction prompt word;

[0040] Inputting the extraction prompt words into an entity extraction model so that the entity extraction model understands the extraction rules based on the extraction examples and extracts question entities from the question text based on the extraction rules;

[0041] The problem entities are organized according to the output format to obtain entity extraction results.

[0042] Optionally, the extracting target information from the problem image using the detection result includes:

[0043] If the detection result is empty, the problem image and the specified sentence are used as target information;

[0044] If the detection result is not empty, filtering the detection result based on a preset confidence and the confidence of each entity object in the detection result to obtain a target detection result;

[0045] If the target detection result is empty, the problem image and the specified sentence are used as target information;

[0046] If the target detection result is not empty, target information is extracted from the problem image based on the position information and category information of each entity object in the target detection result.

[0047] Optionally, extracting target information from the problem image based on the position information and category information of each entity object in the target detection result includes:

[0048] Using the position information of each entity object, calculate the position information of the clipping area;

[0049] Using the position information of the cropping area, cropping the cropping area from the problem image;

[0050] According to the category information, the number of the entity objects is counted to obtain object statistical information;

[0051] The object statistics information and the clipping area are used as the target information.

[0052] Optionally, the fusing the target information and the image text to generate a reply text corresponding to the question text includes:

[0053] Acquire a visual prompt word template, wherein the visual prompt word template includes a third question slot, a target slot, and a text slot;

[0054] Fill the target information into the target slot, fill the question text into the third question slot, and fill the image text into the text slot to obtain a visual prompt word;

[0055] The visual prompt words are input into a visual language model so that the visual language model generates a reply text corresponding to the question text based on the target information and the image text.

[0056] On the other hand, an embodiment of the present invention further provides a visual question answering processing device, the device comprising:

[0057] An acquisition module, used for acquiring a question text and question image data associated with the question text, wherein the question image data includes at least one question image;

[0058] An intention module is used to perform intention recognition processing on the question text to obtain the intention type corresponding to the question text;

[0059] An extraction module, for extracting target information from each of the problem images by using the intention type and the problem text;

[0060] A recognition module, used to perform text recognition processing on each of the problem images to obtain image text corresponding to the problem image;

[0061] The reply module is used to fuse the target information and the image text to generate a reply text corresponding to the question text.

[0062] On the other hand, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any one of the visual question answering processing methods provided by the embodiments of the present invention.

[0063] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the visual question answering processing methods provided by the embodiments of the present invention.

[0064] On the other hand, an embodiment of the present invention further provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any one of the visual question answering processing methods provided by the embodiments of the present invention.

[0065] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0066] The embodiment of the present invention can obtain question text and question image data associated with the question text, perform intent recognition processing on the question text to obtain the intent type corresponding to the question text; for each question image, use the intent type and question text to extract target information from the question image; perform text recognition processing on each question image to obtain the image text corresponding to the question image; fuse the target information and the question text to generate the answer text corresponding to the question text. The extracted image text can provide more details for understanding the question text, and the extracted target information can focus on the information related to the question text in subsequent reasoning, avoiding the interference of irrelevant information, thereby reducing model hallucinations and improving the accuracy of visual question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 is a schematic diagram of an application scenario of the visual question answering processing method provided by an embodiment of the present invention;

[0069] Figure 2 is a flowchart of a visual question answering processing method provided by an embodiment of the present invention;

[0070] Figure 3 is a schematic diagram of a process for extracting target information provided by an embodiment of the present invention;

[0071] Figure 4 is a schematic diagram of calculating position information of a clipping area provided by an embodiment of the present invention;

[0072] Figure 5 Schematic diagram of the overall architecture of the visual question answering method provided by an embodiment of the present invention;

[0073] Figure 6 is a schematic diagram of the structure of a visual question answering processing device provided by an embodiment of the present invention;

[0074] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0076] It is understandable that in the specific implementation of the present invention, data related to user information, etc., needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0077] The present invention provides a visual question answering method, which can be found in Figure 1, showing a schematic diagram of an application scenario of a visual question-answering processing method. The application scenario may include a terminal 101 and a server 102, and data may be exchanged between the terminal 101 and the server 102 via a network. An application program related to question-answering may be installed on the terminal 101, and the application program may be referred to as an intelligent agent. The terminal 101 may be a mobile phone, a tablet computer, a smart Bluetooth device, a computer, a large screen device, a robot, etc.; the server 102 may be a single server or a server cluster consisting of multiple servers.

[0078] The user can send the question text and question image data to the server 102 through the terminal 101, so that the server 102 can obtain the question text and question image data; then the server 102 can perform intent recognition processing on the question text to obtain the intent type corresponding to the question text; for each question image, the intent type and question text can be used to extract the target information from the question image; perform text recognition processing on each question image to obtain the image text corresponding to the question image; fuse the target information and the image text to generate the reply text corresponding to the question text. Then the server 102 can send the reply text to the terminal 101 so that the terminal 101 can display the reply text to the user.

[0079] In this embodiment, a visual question answering method is provided. Figure 2 As shown, the specific process of the visual question answering processing method can be as follows:

[0080] S110, obtaining question text and question image data associated with the question text.

[0081] The question text is the textual data of the question that needs to be answered, and the question image data is the image data needed to answer the question. The question image data may contain at least one question image, and the question image refers to the image that will be used to answer the question later. The question text can be input in the form of voice or text. If the question text is input in the form of voice, it is necessary to convert the voice into text through speech recognition technology (Automatic Speech Recognition, ASR) to obtain the question text. The question image data can be in the form of image or video, etc., which is not specifically limited here.

[0082] The question text and question image data can be input by the user, wherein the user and the intelligent agent can have multiple rounds of dialogues. The question text and question image data can be input in the same round of dialogue or in different rounds of dialogues, and can be set according to actual needs.

[0083] As an implementation method, when obtaining question text and question image data, the question text and original image data associated with the question text may be obtained, wherein the original image data includes at least one original image; if the original image data only includes one original image, the original image data is determined as question image data; if the original image data includes multiple original images, the question image is filtered out from the original image data; and the question image data is generated using all the question images.

[0084] The image data input by the user into the agent can be called original image data. If the original image data contains only one original image, the original image data can be directly used as the problem image data. When the original image data contains multiple original images, such as video data, in order to avoid data redundancy and increase the amount of additional data processing, some images can be screened out from the original image data as problem image data.

[0085] Among them, the multiple original images may be video data or original images input successively. In either case, the multiple original images have a certain order in the time dimension and can be regarded as video data. The following is a detailed description using video data as an example. Optionally, when screening problem images from multiple original images, the multiple original images may be sampled to obtain images to be processed; for each of the images to be processed, the previous image of the image to be processed is used as the designated image of the image to be processed; the image to be processed and the designated image are fused to calculate the importance score of the image to be processed; based on the importance score of each of the images to be processed, the problem images are screened out from the multiple original images.

[0086] Sampling multiple original images can reduce the number of original images. Taking video data as an example, sampling can be performed at a fixed frame interval, for example, at an interval of 2 or 4 frames, to obtain an image to be processed. For each image to be processed, the importance score of the image to be processed can be calculated using the image to be processed and the designated image. The designated image is the previous image of the image to be processed. For example, there are 10 frames of original images, and after sampling, the 1st, 3rd, 5th, 7th, and 9th frames of original images are left. At this time, the designated image of the 3rd frame of the image to be processed is the 1st frame; the designated image of the 5th frame of the image to be processed is the 3rd frame.

[0087] Among them, in order to comprehensively and accurately estimate the importance of the image to be processed, it can be determined based on the comprehensive parameters of the image to be processed in multiple dimensions. Optionally, it can be to calculate the change degree parameter between the image to be processed and the specified image; use the image features of the image to be processed to calculate the image feature parameters of the image to be processed; use the motion features of the image to be processed to calculate the motion feature parameters of the image to be processed; perform weighted calculation on the change degree parameter, image feature parameters and motion feature parameters of the image to be processed to obtain the importance parameter of the image to be processed.

[0088] The specified image is the previous image of the image to be processed. In the video, the scene between two frames may change. Compare the difference between the two images. If the difference is significant, it can be considered that the scene has changed. Optionally, the absdiff function in OpenCV can be used to calculate the difference between the two images to obtain a difference image; traverse each pixel of the difference image, add up its values, and get the total; divide the total by the total number of pixels in the difference image to get the change degree parameter. The change degree parameter reflects the degree of overall difference between the two images. The larger the value of the change degree parameter, the greater the difference.

[0089] For each image to be processed in the video data, a pre-trained image classification model can be used to extract the feature vector of each image to be processed, and the feature vectors are subjected to absolute value operation and then summed to obtain a scalar value, which is the image feature parameter. The larger the value of the image feature parameter, the richer the information contained in the image.

[0090] The optical flow algorithm can be used to analyze the motion patterns of pixels or feature points in an image sequence over time to estimate the motion field, thereby calculating the motion characteristics of the image to be processed and calculating the motion score to quantify the degree of motion of each image.

[0091] First, each image to be processed can be converted into a grayscale image to reduce the interference caused by color. Then, the calcOpticalFlowFarneback algorithm in OpenCV can be used to calculate the dense optical flow matrix between the current image to be processed and its corresponding specified image, and then the amplitude of the optical flow vector is calculated and averaged to obtain the motion feature parameters of the image to be processed. Among them, the larger the motion feature parameter, the more significant position changes have occurred in the pixels or feature points between the two images, and more dynamic information is contained.

[0092] The corresponding weight can be pre-set for each dimension, combined with the change degree parameter, image feature parameter and motion feature parameter, and then weighted summed to obtain the importance parameter of the image to be processed. For example, the change degree parameter is x, and its corresponding weight is a; the image feature parameter is y, and its corresponding weight is b; the motion feature parameter is z, and its corresponding weight is c, then the importance parameter can be expressed as ax+by+cz, and the larger the value of the importance parameter, the more important the image to be processed.

[0093] Then, the importance parameters are sorted in descending order, and the top N importance parameters are taken, and the to-be-processed images corresponding to the top N importance parameters are used as the problem images to obtain the problem image data. Wherein, N is a positive integer, which can be set according to actual needs and is not specifically limited here, and in the problem image data, the problem images are still arranged in chronological order.

[0094] S120: Perform intent recognition processing on the question text to obtain the intent type corresponding to the question text.

[0095] Intent recognition can be used to determine the intent or purpose expressed by the question text. The intent recognition process can be regarded as a classification task. At least two intent types can be predefined, and the question text can be divided into corresponding intent types to identify the specific intent of the question text.

[0096] As an implementation method, an intent recognition model may be pre-trained and the intent type may be pre-defined, the question text may be directly input into the intent recognition model, and the intent type output by the intent recognition model may be used as the intent of the question text.

[0097] As another implementation, a large language model may be used for intent recognition processing. For example, a first prompt word template may be obtained, which includes a first question slot, multiple intent recognition examples, and intent recognition rules corresponding to the intent type; the question text is filled into the first question slot to generate an intent prompt word; the intent prompt word is input into the intent recognition model so that the intent recognition model understands the intent recognition rule based on the multiple intent recognition examples, and uses the intent recognition rule to infer the intent type corresponding to the question text.

[0098] The first prompt word template is a preset template of prompt words to be used in intent recognition. The first prompt word template may include a first question slot, multiple intent recognition examples, and intent recognition rules corresponding to the intent type.

[0099] Among them, the first question slot can be used to fill in the text that needs to be recognized for intent. The intent recognition example is a sample of intent recognition provided to the large language model so that the large language model can learn to perform more complex intent recognition tasks by observing these samples. The intent recognition example can include questions and corresponding intent types. The intent type can be pre-set, and different intent types can correspond to different intent recognition rules. The intent recognition rules allow the large language model to understand the specific recognition method, so as to perform intent recognition on the text in the question slot and determine the intent of the text.

[0100] The first prompt word template is as follows:

[0101] “You are an intelligent assistant that is specifically designed to determine the user’s question intent in a visual question answering scenario. Your task is to analyze the user’s question and determine whether it is necessary to use object detection tools to crop out a sub-image of a specific object region.

[0102] Analyze the user's questions carefully and categorize them into one of the following two intents:

[0103] 1. Object detection is required: If the question involves details of a specific object, person, or area, the image needs to be cropped to obtain a more precise answer.

[0104] 2. No object detection required: If the problem is about the overall scene, a general problem, or does not require special attention to a specific part of the image.

[0105] When answering, please give your judgment directly ("target detection is required" or "target detection is not required")

[0106] <example>

[0107] User question: What breed is the dog in the picture?

[0108] Answer: Object detection is required.

[0109] User question: Is the person in the picture next to the red car?

[0110] Answer: Object detection is required.

[0111] User question: Is this scene indoors or outdoors?

[0112] Answer: No object detection is required.

[0113] User question: How to output the text in the picture?

[0114] Answer: No object detection is required.

[0115] < / example>

[0116] User question: {{ query}}

[0117] answer:

[0118] ”

[0119] In order to determine the intention of the question text, the question text can be filled into the first question slot to generate the intention prompt word. That is, the question text is filled into "{{query}}" in the aforementioned first prompt word template to obtain the intention prompt word.

[0120] Then, the intent prompt words can be input into the intent recognition model to determine the intent type corresponding to the question text. The intent recognition model is a large language model. After the intent prompt words are input into the intent recognition model, the intent recognition model can understand the intent recognition rules based on the intent recognition examples in the intent recognition prompt words, and use the intent recognition rules to infer the recognition results corresponding to the question text.

[0121] In the aforementioned first prompt word template, the recognition result obtained by the intention recognition model is that target detection is required or not required. If the recognition result is that target detection is required, the intention type of the question text can be determined to be a detail type; if the recognition result is that target detection is not required, the intention type of the question text can be determined to be an overall type. Through intention recognition, the intention type corresponding to the question text can be accurately determined, so that different processing can be performed for different intention types in the future, and different question texts can be adapted to each other.

[0122] S130: For each of the question images, extract target information from the question image using the intention type and the question text.

[0123] The target information refers to the key information in the question image that is used to answer the question text. In order to improve processing efficiency and accuracy of question answering, the target information can be extracted from the question image using the intent type and question text. The extracted target information can help focus on the key information when understanding the question text, avoid interference from irrelevant information, and thus improve the accuracy of reasoning. Figure 3 , showing a schematic diagram of the process of extracting target information, the following will be combined Figure 3 Provide detailed explanation.

[0124] In some embodiments, when extracting target information from question image data, for each of the question images, if the intent type is a detail type, entity extraction is performed on the question text to obtain an entity extraction result; if the entity extraction result does not contain a question entity, the question image is used as target information; if the entity extraction result contains a question entity, an entity object corresponding to the question entity is detected in the question image to obtain a detection result; using the detection result, target information is extracted from the question image; if the intent type is an overall type, the question image is used as target information.

[0125] If the intent type is a detail type, it indicates that when answering the question text, attention should be paid to the entities in the question text. When performing entity extraction processing on the question text, a second prompt word template may be obtained, wherein the second prompt word template includes a second question slot, extraction rules, extraction examples, and an output format; the question text is filled into the second question slot to obtain extraction prompt words; the extraction prompt words are input into the entity extraction model so that the entity extraction model understands the extraction rules based on the extraction examples, and extracts question entities from the question text based on the extraction rules; the question entities are organized according to the output format to obtain entity extraction results.

[0126] The second prompt word template is a preset prompt word template used in entity extraction. The second prompt word template may include a second question slot, extraction rules, extraction examples, and output format.

[0127] Among them, the second question slot can be used to fill in the text that needs to be extracted. The extraction example is a sample provided to the large language model so that the large language model can learn to perform more complex entity extraction tasks by observing these samples. The extraction example can include the text to be extracted and the corresponding entity extraction results. The extraction rules refer to the rules to be followed when performing entity extraction, for example, only extracting nominal entities, etc. The output format is a pre-set format for organizing the extraction results. The unified output format can simplify the subsequent processing of the extraction results and improve processing efficiency.

[0128] The second prompt word template is as follows:

[0129] “You are a professional information extraction assistant. Your task is to extract key entities or objects from the user’s question that may be related to the content in the image. Please analyze the user’s question carefully and output the result in JSON format.

[0130] Output format requirements:

[0131] ```json

[0132] {

[0133] "entities": ["entities1", " entities2", ...]

[0134] }

[0135] Note:

[0136] 1. Only extract nominal entities or objects, excluding adjectives, verbs, etc.

[0137] 2. If the question involves multiple related entities or objects, please list them all.

[0138] 3. If there is no clear entity or object in the question, please output "no clear entity or object".

[0139] 4. Keep the extracted entities or objects in their original form and output them in English without interpretation or expansion.

[0140] 5. Make sure the output JSON format is correct and can be parsed directly.

[0141] <example>

[0142] User question: What breed is the dog in the picture?

[0143] Output:

[0144] ```json

[0145] {

[0146] "entities": ["dog"]

[0147] }

[0148] User question: Are there several books and an apple on the table?

[0149] Output:

[0150] ```json

[0151] {

[0152] "entities": ["table", "book", "apple"]

[0153] }

[0154] User question: Describe this photo

[0155] Output:

[0156] ```json

[0157] {

[0158] "entities": []

[0159] }

[0160] User question: {{ query}}

[0161] Output:

[0162] ”

[0163] In order to extract the question entity from the question text, the question text can be filled into the second question template to generate the extraction prompt word. That is, the question text is filled into {{ query}} in the second prompt word template to obtain the extraction prompt word.

[0164] Then, the extraction prompt words can be input into the entity extraction model to extract the question entity from the question text. The entity extraction model is a large language model. After the extraction prompt words are input into the entity extraction model, the entity extraction model can understand the extraction rules based on the extraction examples in the extraction prompt words, and use the extraction rules to extract the question entity from the question text.

[0165] Then organize the extracted problem entities according to the output format to obtain the entity extraction results. Based on the extraction rules in the aforementioned second prompt word template, it can be known that when there is no clear entity or object in the question text, the entity extraction result is a preset sentence. After obtaining the entity extraction result, it can be determined whether the entity extraction result contains the problem entity by detecting the preset sentence in the entity extraction result. For example, in the aforementioned second prompt word, the preset sentence is "no clear entity or object", and the preset sentence "no clear entity or object" can be detected in the entity extraction result; if the preset sentence is detected in the entity extraction result, it is determined that the entity extraction result does not contain the problem entity; if the preset sentence is not detected in the entity extraction result, it is determined that the entity extraction result does not contain the problem entity.

[0166] If the entity extraction result does not contain the problem entity, the problem image can be directly used as the target information. If the entity extraction result contains the problem entity, the entity object corresponding to the problem entity can be detected in the problem image to obtain the detection result.

[0167] As an implementation method, when detecting the entity object corresponding to the problem entity in the problem image and obtaining the detection result, the entity extraction result can be used as the detection prompt word; for each problem image data, the problem image data and the detection prompt word are input into the detection model together to obtain the detection result. The detection result can include the detection box sequence, confidence sequence and category sequence of the entity object.

[0168] The detection box sequence refers to the sequence obtained by arranging the position information of each entity object in the order of entity extraction results. A position information can contain 4 values, which can be recorded as [x1, y1, x2, y2]. (x1, y1) represents the upper left corner coordinate of the detection box in the problem image data, and (x2, y2) represents the lower right corner coordinate of the detection box in the problem image data.

[0169] The confidence sequence refers to the sequence composed of the credibility or accuracy of the corresponding entity objects; the category sequence refers to the category corresponding to the detected entity objects, such as book, apple, etc.

[0170] For example, if the entity extraction result is ["table", "book", "apple"], the detection result can be the detection box sequence boxes = [[0.8521, 0.5847, 0.1285, 0.1697], [0.5988, 0.4731, 0.9261,0.4761], [0.3398, 0.6043, 0.1304, 0.1807], [0.7347, 0.6629, 0.0943, 0.0458]]. The confidence sequence Confidences = [0.8231, 0.8778, 0.6979, 0.5847]. The category sequence Labels = ["table", "book", "apple", "apple"].

[0171] It should be noted that if an entity object is detected in the problem image data, the detection model can output the detection result in the above example; if no entity object is detected in the problem image, the detection result output by the detection model is empty.

[0172] For each problem image data, the detection result output by the detection model can be used to extract target information from the problem image data. For example, if the detection result is empty, the problem image and the specified sentence are used as target information; if the detection result is not empty, the detection result is filtered based on the preset confidence and the confidence of each entity object in the detection result to obtain the target detection result; if the target detection result is empty, the problem image and the specified sentence are used as target information; if the target detection result is not empty, the target information is extracted from the problem image based on the location information and category information of each entity object in the target detection result.

[0173] If the detection result is empty, the problem image and the specified sentence can be used as the target information, wherein the specified sentence is a pre-set sentence, which can be "the image does not contain the target detection result".

[0174] If the detection result is not empty, the detection result can be filtered to obtain the target detection result. Based on the above description, it can be known that the detection result may include the location information, confidence and category information of the detected entity object. In order to ensure the accuracy of the detection result, in some embodiments, the detection result can be filtered based on the preset confidence. Among them, the preset confidence can be set according to actual needs. In an embodiment of the present invention, it can be set to a higher value, such as 0.8, 0.6, etc. Delete the data of the entity object with less than the preset confidence in the detection result to obtain the target detection result.

[0175] For example, in the detection results of the above examples, if the preset confidence is 0.8, the target detection results are the detection box sequence boxes = [[0.8521, 0.5847, 0.1285, 0.1697], [0.5988, 0.4731, 0.9261,0.4761]]. The confidence sequence Confidences = [0.8231, 0.8778]. The category sequence Labels = ["table", "book"].

[0176] After filtering the detection results, the target detection result may be empty. If the target detection result is empty, the entire question image and the specified sentence can be directly used as the target information. If the target detection result is not empty, the location information and category information of each entity object in the target detection result can be used to extract the target information from the question image.

[0177] As an implementation method, when extracting target information from a problem image using the location information and category information of the entity objects, the location information of a cropping area may be used to calculate the location information of each entity object; the location information of the cropping area may be used to crop the cropping area from the problem image; the number of the entity objects may be counted according to the category information to obtain object statistical information; and the object statistical information and the cropping area may be used as the target information.

[0178] The cropping area is the minimum area of ​​the detection box containing all entity objects in the problem image. The position information of each entity object includes the coordinates of the upper left corner and the lower right corner of the detection box, and each coordinate includes the value on the first coordinate axis and the second coordinate axis in the specified coordinate system. When calculating the position information of the cropping area, the minimum and maximum values ​​of the position information of all entity objects on the first coordinate axis can be obtained as the first minimum value and the first maximum value; the minimum and maximum values ​​of the position information of all entity objects on the second coordinate axis can be obtained as the second minimum value and the second maximum value; the first minimum value and the second maximum value are used as the first coordinates, and the first maximum value and the second minimum value are used as the second coordinates to obtain the position information of the cropping area.

[0179] The first coordinate axis may be an x-axis, and the second coordinate axis may be a y-axis. For example, see Figure 4 , shows a schematic diagram of calculating the position information of the cropping area. The gray area is the detection box of each entity object, and the area of ​​the thick solid line box is the entire problem image. In the x-axis direction, the minimum value is x1 and the maximum value is x4; in the y-axis direction, the minimum value is y6 and the maximum value is y3. The first coordinate of the cropping area is (x1, y3) and the second coordinate is (x4, y6). The first coordinate is the coordinate of the upper left corner of the cropping area, and the second coordinate is the coordinate of the lower right corner of the cropping area. The area of ​​the dotted line box is the cropping area.

[0180] Based on the position information of the cropped area, the question image can be cropped to obtain the cropped area. The cropped area is obtained by cropping the question image so that the content of the cropped area is related to the question text. When performing answer reasoning later, only the content in the cropped area can be included to avoid the influence of other image areas. This is particularly useful when understanding high-resolution images. According to the category information of each entity object, the number of entity objects can be counted to obtain object statistics. For example, if the category sequence is Labels = ["table", "book", "apple", "apple"], the object statistics are 2apples, 1table, 1book. This object statistics can improve the accuracy of answers in counting statistics questions. Then the object statistics and the cropped area can be used as target information.

[0181] S140: Perform text recognition processing on each of the problem images to obtain image text corresponding to the problem image.

[0182] The problem image data may generally include multiple problem images, and the problem image may include at least one of text, person, object, and scene. For each problem image in the problem image data, text recognition processing may be performed on the problem image to extract image text from the problem image.

[0183] As an implementation method, a text recognition model can be used to recognize each question image to extract the text in each question image as the image text corresponding to the question image, wherein the text recognition model can be an optical character recognition (OCR) model. For tasks that require understanding the text in the image, such as reading the content on a menu, report, or bulletin board, the text extracted by OCR can be directly analyzed by the visual language model to reduce model hallucinations, and subsequent analysis combines the image and the extracted text to better understand the scene in the image and improve the accuracy of visual question answering.

[0184] S150, integrating the target information and the image text to generate a reply text corresponding to the question text.

[0185] For each question image, the corresponding target information and image text will be obtained. By integrating all the target information and image text, the answer text corresponding to the question text can be generated.

[0186] Optionally, a visual cue word template may be obtained, wherein the visual cue word template includes a third question slot, a target slot and a text slot; the target information is filled into the target slot, the question text is filled into the third question slot, and the image text is filled into the text slot to obtain a visual cue word; and the visual cue word is input into a visual language model so that the visual language model generates a reply text corresponding to the question text based on the target information and the image text.

[0187] The visual prompt word template is a preset prompt word template used when generating the answer text corresponding to the question text. The third question slot can be used to fill in the question text, the target slot can be used to fill in the extracted target information, and the text slot can be used to fill in the image text.

[0188] According to the above content, the aforementioned problem image data may contain one problem image or multiple problem images, and the target information extracted from them may also be different. Multiple visual prompt word templates can be set according to different situations, or only one general template can be set to explain special situations. The visual prompt word template will be described in detail below.

[0189] As mentioned above, the aforementioned problem image data includes at least one problem image, and the problem image may be multiple or a single one. Moreover, the target information includes different contents in different situations. The following will be described in detail in combination with the prompt word template and the specific situation.

[0190] Case 1: When the question image data contains only one question image and the intent type is the overall type, the target visual cue word template can be as follows:

[0191] prompt_vlm = """

[0192]

[0193] The text information contained in the picture is as follows:

[0194] {ocr_text}

[0195] question:

[0196] {query}

[0197] """

[0198] in, {ocr_text} is the target slot, into which the problem image in the target information can be filled; {ocr_text} is the text slot, into which the image text can be filled. If the image text is empty, then the text filled here is also empty.

[0199] If the problem image data contains only one problem image, and the target information corresponding to the problem image is the problem image and the specified sentence. The problem image in the target information can be filled in , and add the specified sentence to the visual cue word template.

[0200] Case 2: When the question image data contains only one question image and the intent type is detail type, the visual cue word template can be as follows:

[0201] prompt_vlm = """

[0202]

[0203] The image contains some of the following text:

[0204] {ocr_text}

[0205] Additional object detection information:

[0206] {labels_info}

[0207] question:

[0208] {query}

[0209] """

[0210] in, The cropped area in the target information can be filled in {ocr_text}; the image text can be filled in {labels_info}; the object statistics in the target information can be filled in {query}; the question text can be filled in.

[0211] Case 3: When the question image data contains multiple question images and the intent type is the overall type, the visual prompt word template can be as follows:

[0212] prompt_vlm = """

[0213] <image1>

[0214] <image2>

[0215] …

[0216] <imagen>

[0217] The text information contained in the 1st to 3rd frame images is as follows:

[0218] {ocr_text1}

[0219] The text information contained in the Nth frame image is as follows:

[0220] {ocr_textN}

[0221] question:

[0222] {query}

[0223] """

[0224] in, <image1>to <imagen>Multiple question images can be filled in sequentially; {ocr_text1} and {ocr_textN} can be filled in with the image text extracted from the corresponding question images; {query} can be filled in with the question text.

[0225] Case 4: When the question image data contains multiple question images and the intent type is detail type, the visual prompt word template can be as follows:

[0226] prompt_vlm = """

[0227] <sub_image1>

[0228] < sub_image2>

[0229] …

[0230] < sub_imageN>

[0231] Some text information contained in the 1st to 3rd frame images is as follows:

[0232] {ocr_text1}

[0233] The text information contained in the Nth frame is as follows:

[0234] {ocr_textN}

[0235] <Additional object detection information>

[0236] In the first frame:

[0237] {labels_info1}

[0238] In the Nth frame image:

[0239] {labels_infoN}

[0240] question:

[0241] {query}

[0242] """

[0243] in,<sub_image1> to<sub_imageN> The cropped area of ​​the corresponding problem image can be filled in in sequence; the image text extracted from the corresponding problem image can be filled in {ocr_text1} and {ocr_textN}; the object statistical information of the corresponding problem image can be filled in {labels_info1} and {labels_infoN}; the question text can be filled in {query}.

[0244] According to different situations, the corresponding content can be filled into the visual prompt word template to generate the corresponding visual prompt word. The generated visual prompt word can be input into the visual language model so that the visual language model can generate the answer text corresponding to the question text based on the target information and the image text and other information.

[0245] For a more detailed description of the visual question answering approach, see Figure 5 , showing a schematic diagram of the overall architecture of the visual question answering processing method. Among them, the visual question answering processing method can be run in an intelligent system, and the intelligent system can include an intent recognition model, an entity extraction model, a text recognition model, and a visual language model. After the user inputs the question text and the question image data, the question text is processed by the intent recognition model to determine the intent type of the question text. When the intent type is a detail type, the question entity can be extracted from the question text by the entity extraction model to obtain the entity extraction result. The question image is detected by using the detection model to detect the question entity in the entity extraction result in the question image, and then the target detection result can be obtained. Based on the content in the target detection result, the question image is cropped and the category information is statistically obtained to obtain the target information. The text recognition model is used to perform text recognition on the question image, and the image text is extracted therefrom. The target information and the image text are input into the visual language model together to obtain the reply text.

[0246] The visual question answering processing method provided by the embodiment of the present invention can be applied to image-text question answering scenarios, such as high-resolution image understanding, document and chart visual understanding, visual counting statistics, general visual question answering, and other scenarios. The method can use a variety of models to build an intelligent system, and through the collaboration between multiple models, the image understanding ability of the model can be enhanced. Extracting image text through the OCR model can provide more detailed information for generating reply text, and cropping the question image helps the visual language model focus on key areas in image understanding, avoiding interference from irrelevant information, thereby improving the accuracy of visual question answering.

[0247] In order to better implement the above method, the embodiment of the present invention further provides a visual question answering processing device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0248] For example, in this embodiment, the method of the embodiment of the present invention is described in detail by taking the visual question answering processing device being specifically integrated in a server as an example.

[0249] For example, Figure 6 As shown, the visual question answering processing device 200 may include an acquisition module 210 , an intention module 220 , an extraction module 230 , a recognition module 240 and a reply module 250 .

[0250] An acquisition module 210 is used to acquire a question text and question image data associated with the question text, wherein the question image data includes at least one question image;

[0251] The intention module 220 is used to perform intention recognition processing on the question text to obtain the intention type corresponding to the question text;

[0252] An extraction module 230, for extracting target information from each of the problem images by using the intention type and the problem text;

[0253] The recognition module 240 is used to perform text recognition processing on each of the problem images to obtain image text corresponding to the problem image;

[0254] The reply module 250 is used to fuse the target information and the image text to generate a reply text corresponding to the question text.

[0255] In some embodiments, the acquisition module 210 is specifically used to:

[0256] Acquire a question text and original image data associated with the question text, wherein the original image data includes at least one original image;

[0257] If the original image data only includes one original image, determining the original image data as problematic image data;

[0258] If the original image data includes multiple original images, screening out the problematic image from the multiple original images;

[0259] The problem image data is generated using all the problem images.

[0260] In some embodiments, the acquisition module 210 is specifically used to:

[0261] Performing sampling processing on the multiple original images to obtain images to be processed;

[0262] For each of the images to be processed, taking the image before the image to be processed as the designated image of the image to be processed;

[0263] Calculate the importance score of the image to be processed by fusing the image to be processed and the designated image;

[0264] Based on the importance score of each of the to-be-processed images, problematic images are screened out from the multiple original images.

[0265] In some embodiments, the acquisition module 210 is specifically used to:

[0266] Calculating a parameter of a degree of change between the image to be processed and the designated image;

[0267] Calculating image feature parameters of the image to be processed by using the image features of the image to be processed;

[0268] Calculating motion feature parameters of the image to be processed by using the motion feature of the image to be processed;

[0269] A weighted calculation is performed on the change degree parameter, the image feature parameter and the motion feature parameter of the image to be processed to obtain an importance parameter of the image to be processed.

[0270] In some embodiments, the intention module 220 is specifically used to:

[0271] Obtaining a first prompt word template, wherein the first prompt word template includes a first question slot, multiple intent recognition examples, and intent recognition rules corresponding to the intent type;

[0272] Fill the question text into the first question slot to generate an intention prompt word;

[0273] The intention prompt word is input into the intention recognition model so that the intention recognition model understands the intention recognition rule based on the multiple intention recognition examples, and uses the intention recognition rule to infer the intention type corresponding to the question text.

[0274] In some embodiments, the intent type includes a detail type and an overall type, and the extraction module 230 is specifically used to:

[0275] For each of the question images, if the intent type is a detail type, entity extraction is performed on the question text to obtain an entity extraction result;

[0276] If the entity extraction result does not contain the problem entity, the problem image is used as target information;

[0277] If the entity extraction result includes a problem entity, detecting an entity object corresponding to the problem entity in the problem image to obtain a detection result;

[0278] Extracting target information from the problem image using the detection result;

[0279] If the intention type is an overall type, the problem image is used as target information.

[0280] In some embodiments, the extraction module 230 is specifically used to:

[0281] Acquire a second prompt word template, wherein the second prompt word template includes a second question slot, an extraction rule, an extraction example, and an output format;

[0282] Filling the question text into the second question slot to obtain an extraction prompt word;

[0283] Inputting the extraction prompt words into an entity extraction model so that the entity extraction model understands the extraction rules based on the extraction examples and extracts question entities from the question text based on the extraction rules;

[0284] The problem entities are organized according to the output format to obtain entity extraction results.

[0285] In some embodiments, the extraction module 230 is specifically used to:

[0286] If the detection result is empty, the problem image and the specified sentence are used as target information;

[0287] If the detection result is not empty, filtering the detection result based on a preset confidence and the confidence of each entity object in the detection result to obtain a target detection result;

[0288] If the target detection result is empty, the problem image and the specified sentence are used as target information;

[0289] If the target detection result is not empty, target information is extracted from the problem image based on the position information and category information of each entity object in the target detection result.

[0290] In some embodiments, the extraction module 230 is specifically used to:

[0291] Using the position information of each entity object, calculate the position information of the clipping area;

[0292] Using the position information of the cropping area, cropping the cropping area from the problem image;

[0293] According to the category information, the number of the entity objects is counted to obtain object statistical information;

[0294] The object statistics information and the clipping area are used as the target information

[0295] In some embodiments, the reply module 250 is specifically configured to:

[0296] Acquire a visual prompt word template, wherein the visual prompt word template includes a third question slot, a target slot, and a text slot;

[0297] Fill the target information into the target slot, fill the question text into the third question slot, and fill the image text into the text slot to obtain a visual prompt word;

[0298] The visual prompt words are input into a visual language model so that the visual language model generates a reply text corresponding to the question text based on the target information and the image text.

[0299] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0300] As can be seen from the above, the visual question answering processing device of this embodiment can obtain the question text and the question image data associated with the question text, perform intent recognition processing on the question text to obtain the intent type corresponding to the question text; for each question image, use the intent type and question text to extract the target information from the question image; perform text recognition processing on each question image to obtain the image text corresponding to the question image; fuse the target information and the question text to generate the answer text corresponding to the question text. The extracted image text can provide more details for understanding the question text, and the extracted target information can focus on the information related to the question text in subsequent reasoning, avoiding the interference of irrelevant information, thereby reducing the model hallucination and improving the accuracy of visual question answering.

[0301] The embodiment of the present invention further provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like; the server may be a single server or a server cluster composed of multiple servers, or the like.

[0302] In some embodiments, the visual question and answer processing device can also be integrated into multiple electronic devices. For example, the visual question and answer processing device can be integrated into multiple servers, and the visual question and answer processing method of the present invention can be implemented by multiple servers.

[0303] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present invention, specifically:

[0304] The electronic device may include components such as a processor 310 with one or more processing cores, a processor 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0305] The processor 310 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the processor 320, and calling data stored in the processor 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 310.

[0306] The processor 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the processor 320. The processor 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the processor 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the processor 320 may also include a memory controller to provide the processor 310 with access to the processor 320.

[0307] The electronic device also includes a power supply 330 for supplying power to various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 330 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0308] The electronic device may further include an input module 340, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0309] The electronic device may further include a communication module 350. In some embodiments, the communication module 350 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 350, thereby providing the user with wireless broadband Internet access. For example, the communication module 350 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0310] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 310 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the processor 320 according to the following instructions, and the processor 310 will run the application programs stored in the processor 320, thereby implementing the steps in the methods of various embodiments of the present invention.

[0311] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0312] As can be seen from the above, the electronic device provided by the embodiment of the present invention can obtain the question text and the question image data associated with the question text, perform intent recognition processing on the question text to obtain the intent type corresponding to the question text; for each question image, use the intent type and question text to extract the target information from the question image; perform text recognition processing on each question image to obtain the image text corresponding to the question image; fuse the target information and the question text to generate the answer text corresponding to the question text. The extracted image text can provide more details for understanding the question text, and the extracted target information can focus on the information related to the question text in subsequent reasoning, avoiding the interference of irrelevant information, thereby reducing the model hallucination and improving the accuracy of visual question answering.

[0313] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0314] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any visual question answering processing method provided by the embodiment of the present invention.

[0315] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0316] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program comprising a computer program / instruction, the computer program / instruction being stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the electronic device executes the method provided in various optional implementations of the visual question answering processing provided in the above embodiments.

[0317] Since the instructions stored in the storage medium can execute the steps in any visual question answering processing method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any visual question answering processing method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0318] The above is a detailed introduction to a visual question answering processing method provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.< / imagen> < / imagen> < / image2> < / image1> < / example>

Claims

1. A visual question answering method, characterized in that: The method comprises: Acquire a question text and question image data associated with the question text, wherein the question image data includes at least one question image; Performing intent recognition processing on the question text to obtain the intent type corresponding to the question text includes: obtaining a first prompt word template, wherein the first prompt word template includes a first question slot, multiple intent recognition examples, and an intent recognition rule corresponding to the intent type; filling the question text into the first question slot to generate an intent prompt word; inputting the intent prompt word into an intent recognition model so that the intent recognition model understands the intent recognition rule based on the multiple intent recognition examples, and uses the intent recognition rule to infer the intent type corresponding to the question text, wherein the intent type includes a detail type and an overall type; For each of the problem images, the target information is extracted from the problem image using the intention type and the problem text, including: for each of the problem images, if the intention type is a detail type, entity extraction is performed on the problem text to obtain an entity extraction result; if the entity extraction result does not contain a problem entity, the problem image is used as the target information; if the entity extraction result contains a problem entity, an entity object corresponding to the problem entity is detected in the problem image to obtain a detection result; using the detection result, the target information is extracted from the problem image; if the intention type is an overall type, the problem image is used as the target information; Performing text recognition processing on each of the problem images to obtain image text corresponding to the problem image; The target information and the image text are integrated to generate a reply text corresponding to the question text.

2. The method according to claim 1, characterized in that The step of obtaining the question text and the question image data associated with the question text includes: Acquire a question text and original image data associated with the question text, wherein the original image data includes at least one original image; If the original image data only includes one original image, determining the original image data as problematic image data; If the original image data includes multiple original images, screening out the problematic image from the multiple original images; The problem image data is generated using all the problem images.

3. The method according to claim 2, characterized in that The step of screening out problematic images from the plurality of original images comprises: Performing sampling processing on the multiple original images to obtain images to be processed; For each of the images to be processed, taking the image before the image to be processed as the designated image of the image to be processed; Calculate the importance score of the image to be processed by fusing the image to be processed and the designated image; Based on the importance score of each of the to-be-processed images, problematic images are screened out from the multiple original images.

4. The method according to claim 3, characterized in that The step of fusing the image to be processed and the designated image to calculate the importance score of the image to be processed includes: Calculating a parameter of a degree of change between the image to be processed and the designated image; Calculating image feature parameters of the image to be processed by using the image features of the image to be processed; Calculating motion feature parameters of the image to be processed by using the motion feature of the image to be processed; A weighted calculation is performed on the change degree parameter, the image feature parameter and the motion feature parameter of the image to be processed to obtain an importance parameter of the image to be processed.

5. The method according to claim 1, characterized in that The entity extraction process is performed on the question text to obtain an entity extraction result, including: Acquire a second prompt word template, wherein the second prompt word template includes a second question slot, an extraction rule, an extraction example, and an output format; Filling the question text into the second question slot to obtain an extraction prompt word; Inputting the extraction prompt words into an entity extraction model so that the entity extraction model understands the extraction rules based on the extraction examples and extracts question entities from the question text based on the extraction rules; The problem entities are organized according to the output format to obtain entity extraction results.

6. The method according to claim 1, characterized in that The step of extracting target information from the problem image using the detection result includes: If the detection result is empty, the problem image and the specified sentence are used as target information; If the detection result is not empty, filtering the detection result based on a preset confidence and the confidence of each entity object in the detection result to obtain a target detection result; If the target detection result is empty, the problem image and the specified sentence are used as target information; If the target detection result is not empty, target information is extracted from the problem image based on the position information and category information of each entity object in the target detection result.

7. The method according to claim 6, characterized in that The step of extracting target information from the problem image based on the position information and category information of each entity object in the target detection result includes: Using the position information of each entity object, calculate the position information of the clipping area; Using the position information of the cropping area, cropping the cropping area from the problem image; According to the category information, the number of the entity objects is counted to obtain object statistical information; The object statistics information and the clipping area are used as the target information.

8. The method according to claim 1, characterized in that: The step of fusing the target information and the image text to generate a reply text corresponding to the question text includes: Acquire a visual prompt word template, wherein the visual prompt word template includes a third question slot, a target slot, and a text slot; Fill the target information into the target slot, fill the question text into the third question slot, and fill the image text into the text slot to obtain a visual prompt word; The visual prompt words are input into a visual language model so that the visual language model generates a reply text corresponding to the question text based on the target information and the image text.

Images