Multi-modal image-text question and answer method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a multi-mode image-text question answering method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving a to-be-processed picture and a question text; performing task identification based on the question text, and determining a task type; according to the task type, performing feature extraction of different dimensions on the to-be-processed picture to obtain visual word lists of different types; fusing the visual word lists of different types to obtain a target visual word list; and according to the target visual word list and the task type, generating a question and answer result. The question and answer result can be accurately generated by adopting the method.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal image-text question-answering method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, deep learning technology has emerged. Through the training of multi-layer neural networks, deep learning can automatically extract features from data and is widely used in tasks such as image recognition and natural language processing. The emergence of deep learning has greatly promoted the application of artificial intelligence in various fields, especially in visual processing, speech recognition and autonomous driving, bringing significant breakthroughs.

[0003] However, existing deep learning-based image and text processing technologies still have some shortcomings in practical applications, especially in scenarios where images and texts are combined. Although deep learning can extract rich features from images and texts, existing technologies still face certain challenges in terms of the accuracy of the answer results. Specifically, existing methods often cannot effectively combine the visual information of the image with the semantic information in the text, resulting in the final answer results deviating from the actual situation, or in complex scenarios, failing to accurately understand the question and generate a reasonable answer. Summary of the invention

[0004] Based on this, it is necessary to provide a multimodal image-text question-and-answer method, apparatus, computer equipment, computer-readable storage medium and computer program product that can effectively integrate multimodal diseases to generate question-and-answer results in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a multimodal picture-text question-answering method, comprising:

[0006] Receive the image to be processed and the question text;

[0007] Performing task identification based on the question text to determine the task type;

[0008] According to the task type, extracting features of different dimensions from the image to be processed to obtain different types of visual word lists;

[0009] fusing the different types of visual word lists to obtain a target visual word list;

[0010] Generate a question-answering result according to the target visual vocabulary and the task type.

[0011] In one embodiment, according to the task type, the feature extraction of different dimensions is performed on the image to be processed to obtain different types of visual word lists.

[0012] Through different attention mechanisms and the task types, features of different dimensions are extracted from the image to be processed to obtain a global visual vocabulary and a local visual vocabulary; the task type is used to guide the attention mechanism to perform feature extraction.

[0013] In one of the embodiments, the attention mechanism includes a first attention mechanism and a second attention mechanism, wherein the first attention mechanism is used to extract the global visual vocabulary; and the second attention mechanism is used to extract the local visual vocabulary.

[0014] In one embodiment, the first attention mechanism includes multiple attention layers; the extraction process of the global visual vocabulary includes:

[0015] Dividing the image to be processed into image blocks to obtain an initial image block sequence;

[0016] According to the task type, adjusting the attention weights of the preset number of attention layers to obtain a first attention weight, and adjusting the attention weights of other attention layers to obtain a second attention weight; the first attention weight is used to extract global semantic features; the second attention weight is used to extract local detail features;

[0017] Performing feature extraction on the initial image block sequence according to the first attention weight to obtain a first initial vocabulary;

[0018] Performing feature extraction on the initial image block sequence according to the second attention weight to obtain a second initial vocabulary;

[0019] According to the task type, weighted fusion is performed on the first initial vocabulary and the second initial vocabulary to obtain the global visual vocabulary.

[0020] In one embodiment, generating a question-answering result according to the target visual vocabulary and the task type includes:

[0021] Determining a semantic template corresponding to the task type;

[0022] extracting associated attributes from the target visual vocabulary according to the semantic template;

[0023] A visual semantic description is constructed according to the associated attributes to obtain the question-answering result.

[0024] In one embodiment, before extracting features of different dimensions from the image to be processed according to the task type to obtain different types of visual word lists, the method further includes:

[0025] Preprocessing the to-be-processed object;

[0026] The preprocessing includes: one or more of image normalization, image size adjustment and image enhancement.

[0027] In a second aspect, the present application also provides a multimodal graphic question-and-answer device, comprising:

[0028] A receiving module, used for receiving the image to be processed and the question text;

[0029] A type identification module, used to identify the task based on the question text and determine the task type;

[0030] A feature extraction module, used to extract features of different dimensions on the image to be processed according to the task type, to obtain different types of visual word lists;

[0031] A feature fusion module, used for fusing different types of visual vocabulary to obtain a target visual vocabulary;

[0032] The answer module is used to generate a question-answering result according to the target visual vocabulary and the task type.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0035] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0036] The above-mentioned multimodal image-text question-answering method, apparatus, computer device, computer-readable storage medium and computer program product, after receiving the image to be processed and the question text, determine the task type according to the question text, and based on the task type, ensure that the most relevant visual vocabulary is extracted for different tasks, and then fuse the visual vocabulary of different dimensions to generate the target visual vocabulary, and finally, accurately generate the question-answering result according to the target visual vocabulary. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of a process flow of a multi-model graphic question-and-text question answering method in one embodiment;

[0039] Figure 2 A flowchart of the steps of a multimodal graphic question-and-answer method in another embodiment;

[0040] Figure 3 is a structural block diagram of a multi-model graphic question-and-answer device in one embodiment;

[0041] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In one embodiment, Figure 1 As shown, a multimodal graphic question-and-answer method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0044] Step 102, receiving the image to be processed and the question text.

[0045] The image to be processed refers to an input image, including visual information that needs to be semantically analyzed and processed; the question text refers to the question raised by the user, which describes the needs and goals related to the image to be processed, such as "how many dogs are there in this picture" or "what color clothes are the people in the picture wearing"

[0046] Optionally, the user uploads the image to be processed and inputs the question text at the same time. For example, the user takes a picture with a mobile device and inputs “What kind of plant is this?”

[0047] Step 104: Perform task identification based on the question text to determine the task type.

[0048] In this embodiment, the user needs are analyzed and the type of task is determined by parsing the question text. The task type defines the category of the method to solve the problem. For example, "what is this" can be determined as a classification task, and "how many dogs are there in the picture" can be determined as a counting task.

[0049] Optionally, the identification of task types can be done by a pre-wired classification model, which inputs an embedded representation of the question text and outputs a task type and label.

[0050] Optionally, you can first identify the subject and verb in the text, and then quickly identify the task type by matching specific keywords in the question, such as "how many", "color", etc.

[0051] Step 106: According to the task type, feature extraction of different dimensions is performed on the image to be processed to obtain different types of visual vocabulary.

[0052] Among them, the visual vocabulary is a semantic representation of image features, which is used to represent the visual features in image or video data. The visual vocabulary is a set of visual feature vectors extracted from image or video datasets. These feature vectors can be obtained through various image processing and feature extraction techniques, such as local feature descriptors (such as SIFT, SURF, ORB), deep learning convolutional neural network (CNN) features, etc.

[0053] For example, for a picture containing a dog, the visual vocabulary may include information such as "category: dog", "attribute: brown", "action: running", etc. This information can help the model understand the key elements in the image and then answer questions related to the image.

[0054] The dimensions and focus of feature extraction may vary for different task types. The requirements of different task types determine the angle and method of extracting image information. For example, for a classification task, the focus is usually on identifying the target category in the image. For counting tasks, more attention is paid to the number and spatial distribution of different targets in the image, which will strengthen the distribution characteristics of the target. For description tasks, it is necessary to comprehensively consider the appearance, movement and relationship of the target with the background to generate richer semantic information.

[0055] The task type can guide the extraction of features in different dimensions, better understand the information in the image, and optimize the extracted features according to the specific task requirements. For example, when dealing with classification tasks, you may rely more on local features, such as the shape, color and other information in the image block to identify the target category; while in counting tasks, you may pay attention to the density and position distribution of targets in different areas, and complete the counting task by focusing on the number of occurrences and arrangement of targets in the image. In description tasks, it is necessary not only to understand the basic features of the target, such as color, size, etc., but also to model the target's actions, states, and relationship with the environment to generate a more comprehensive and accurate semantic description.

[0056] Guided by the task type, different dimensions of the image can receive different degrees of attention, and then the feature dimensions most relevant to the task can be extracted. Due to the diversity of task types and image content, it is crucial to extract features in different dimensions. Only by fully mining the information of the image in each dimension can we ensure accurate processing of various tasks. Feature extraction in different dimensions helps to enhance the specificity of the task, enabling optimization for specific needs, and ultimately providing more accurate answers in the picture-text question-answering task.

[0057] Step 108: different types of visual vocabulary are fused to obtain a target visual vocabulary.

[0058] Different types of visual vocabularies usually represent different feature dimensions of an image. For example, a local feature vocabularies may emphasize the details of a specific area in an image, such as texture, shape, and color, while a global feature vocabularies usually focus on the overall semantics of an image, such as the main object, background, and overall layout in the image. Since these vocabularies represent different aspects of an image, a more comprehensive target visual vocabularies that is highly relevant to the current task type can be generated by fusing these vocabularies.

[0059] The purpose of generating a target visual vocabulary is to provide a semantic representation that can effectively answer user questions by integrating various feature information according to specific task requirements. For example, in a counting task, the target visual vocabulary may strengthen the number and distribution information of the target; while in a description task, the target visual vocabulary may integrate various information such as the target's category, color, action, and relationship with the environment. This fusion process ensures that the target visual vocabulary can contain the key information required for the task, thereby generating accurate question-answering results.

[0060] Optionally, different types of visual vocabulary can be weighted fused, and the weighted fusion method assigns different weights to different types of visual vocabulary according to their importance to the task. For example, a classification task may rely more on global features, while a counting task may rely more on local features. In this case, by assigning different weights to each visual vocabulary, the contribution of different features in the final target visual vocabulary can be effectively adjusted.

[0061] Optionally, different types of visual vocabularies can be fused by concatenation, in which all visual vocabularies are directly concatenated together to form a long vector representing all types of visual features. The advantage of concatenation is that it can retain all feature information in each vocabulary at the same time without losing any details.

[0062] Step 110, generating a question-answering result according to the target visual vocabulary and the task type.

[0063] In the process of generating question-answering results, the target visual vocabulary is first matched with the task type to generate a latent semantic representation. The latent semantic representation is a structured intermediate representation that can effectively connect visual information and language requirements. This representation includes associated attributes extracted from the target visual vocabulary, and by matching with the task type, a visual semantic description that meets the task requirements is constructed. In this way, the model can connect the visual information in the image with the requirements of the question text, and then generate accurate answers for the task type.

[0064] For example, when the target visual vocabulary is: [category: dog, quantity: 2], and the task type is a counting task, the latent semantic representation can be: [target: dog, answer: 2]. This representation clearly reflects the core information of the task: the category of the target object and its quantity. Next, through the language generation model, the latent semantic representation is converted into natural language form to obtain the final question-answering result.

[0065] Optionally, the output results will be verified to verify whether they are consistent with the target visual vocabulary to ensure the accuracy of the answer.

[0066] For example, the generated natural language answer is compared with the information in the target visual vocabulary. For example, if the target visual vocabulary clearly states that the number of "dogs" is 2, then the generated answer should contain the same number information.

[0067] For example, text matching and semantic similarity calculation can be used to perform semantic verification on the natural language output to ensure that it is consistent with the information in the target visual vocabulary.

[0068] In the above multimodal image-text answering method, after receiving the image to be processed and the question text, the task type is determined according to the question text. Based on the task type, the most relevant visual vocabulary can be extracted for different tasks, and then the visual vocabulary of different dimensions is fused to generate the target visual vocabulary. Finally, the question-answering result is accurately generated according to the target visual vocabulary.

[0069] Furthermore, in one embodiment, according to the task type, feature extraction of different dimensions is performed on the processed image to obtain different types of visual lexicons, and feature extraction of different dimensions is performed on the processed image through different attention mechanisms and task types to obtain a global visual lexicon and a local visual lexicon; the task type is used to guide the attention mechanism for feature extraction.

[0070] In this embodiment, feature extraction of different dimensions of the processed image is achieved through different attention mechanisms to obtain a global visual vocabulary and a local visual vocabulary, wherein the task type is used to guide the attention mechanism to perform feature extraction.

[0071] Optionally, the task type guides the attention mechanism to extract features by directing it to focus on features in the image that are relevant to the task type. By adjusting the focus of the attention mechanism according to the task type, the feature extraction process can be optimized to ensure that the extracted visual information is directly related to the task requirements, thereby improving the accuracy of the subsequent question-answering results.

[0072] Furthermore, in one embodiment, the above-mentioned attention mechanism includes a first attention mechanism and a second attention mechanism, the first attention mechanism is used to extract a global visual vocabulary; the second attention mechanism is used to extract a local visual vocabulary.

[0073] Optionally, features of different dimensions can be extracted from the processed images through the attention mechanism in VIT (Visual Transformer) and CNN.

[0074] The global visual vocabulary is realized through the attention mechanism in VIT, which can model the entire image through the self-attention mechanism to capture the relationship and contextual information between different regions in the image. The advantage of the VIT model is that it can extract the global information of the image and understand the overall semantic structure of the image. In this way, VIT can effectively extract high-level global features in the image and generate a global visual vocabulary.

[0075] Among them, the local visual vocabulary is realized through the convolution and attention mechanism in CNN. CNN extracts local features in the image through local receptive field and convolution operations, and after combining with the attention mechanism, the model can focus on specific areas in the image more accurately. In this way, CNN can extract rich local features from the details of the image and generate a local visual vocabulary.

[0076] Furthermore, the above-mentioned first attention mechanism includes multiple attention layers; the extraction process of the global visual vocabulary includes: dividing the image to be processed into image blocks to obtain an initial image block sequence; according to the task type, adjusting the attention weights of a preset number of attention layers to obtain a first attention weight, and adjusting the attention weights of other attention layers to obtain a second attention weight; the first attention weight is used to extract global semantic features; the second attention weight is used to extract local detail features; according to the first attention weight, feature extraction is performed on the initial image block sequence to obtain a first initial vocabulary; according to the second attention weight, feature extraction is performed on the initial image block sequence to obtain a second initial vocabulary; according to the task type, weighted fusion of the first initial vocabulary and the second initial vocabulary is performed to obtain a global visual vocabulary.

[0077] First, the image to be processed is divided into image blocks to obtain an initial image block sequence. Image blocks refer to the division of an image into several small areas, each of which contains local information of the image. Through this division, the complexity of the image can be reduced, so that the model can process the image block by block and extract local features. The division process is to divide the image into multiple small blocks of fixed size through a sliding window or other strategies. In this way, it is easy to gradually extract local information and capture details in the image. Among them, the initial image block sequence refers to a sequence composed of multiple image blocks after division.

[0078] Next, different attention layers are assigned different weights according to the task type to guide the focus of feature extraction. The task type determines which parts of the image are more important to focus on. Specifically, the first preset number of attention layers, such as the first few layers of Transformer, will be used to extract global semantic features that are closely related to the task type. These layers pay more attention to the overall structure and contextual information of the image. Other attention layers focus on extracting local features, such as details in the image, local objects, or the saliency of specific areas.

[0079] Among them, the first attention weight is used to focus on the global semantic features of the image, such as the general target category, overall layout, relationship between targets in the image, etc. The second attention weight is used to focus on the local detail features of the image, such as the specific shape, texture, color and other details of the target.

[0080] Optionally, according to the task type, determine the part that the attention mechanism needs to pay attention to, and then adjust the attention weight according to the proportion of global semantic features and local detail features. Exemplarily, the task type is counting, which requires accurate local positioning and quantity recognition of the target in the image. In the counting task, the global semantic features include what type of objects are contained in the image, while the local detail features need to identify the specific location and quantity of each object. In the counting task, the type and approximate location of the objects in the image are key, so more attention needs to be paid to the global semantic features. At this time, the sensitivity of the model to the object category can be enhanced by adjusting the first attention weight, and the attention to irrelevant background or details can be weakened, thereby improving the ability to recognize the type of objects in the image. For local detail features, relatively speaking, although it is also important to accurately locate the position of each object, it is not as critical as the recognition of the object category. Therefore, the weight of the second attention layer can be appropriately reduced so that the attention mechanism pays more attention to global features rather than local features. In this way, by adjusting the weight of the attention layer, the model can extract key information in the image more efficiently for the counting task.

[0081] Optionally, according to the first attention weight, feature extraction is performed on the initial image block sequence to obtain a first initial vocabulary; similarly, according to the second attention weight, feature extraction is performed on the initial image block sequence to obtain a second initial vocabulary.

[0082] Finally, according to the task type, the first initial vocabulary and the second initial vocabulary are weighted and fused to obtain the global visual vocabulary. This weighted fusion process takes into account the specific needs of the task and balances the contribution of global semantic features and local detail features through different weight ratios. For example, if the task requires high details, the weight of local features may be larger; if the task requires overall semantic understanding, it may focus more on global features.

[0083] For example, assuming the task type is "object recognition", the global semantic features may be mainly used to identify the major categories of objects, such as animals, plants, etc., while the local features help identify the specific features of objects, such as the fur color and shape of a dog. Finally, through weighted fusion, the global visual vocabulary obtained will contain both global category information and local detail information.

[0084] In the above embodiment, by weighted fusion of global semantic features and local detail features, the overall and local information of the image can be fully utilized, thereby improving the model's ability to understand the image.

[0085] In other embodiments, the local visual vocabulary is extracted through the attention mechanism in CNN. Specifically, through multiple convolutional layers and attention mechanisms in CNN, the model can capture local area features in the image. In this process, the attention mechanism weights the convolution feature map so that the model can automatically focus on important local areas in the image, such as object edges, details, etc., and suppress irrelevant background information. In this way, the local visual vocabulary extracted by the attention mechanism in CNN can accurately represent the local feature information in the image, and provide detailed visual semantic support for subsequent image analysis tasks.

[0086] In one embodiment, the above-mentioned generation of question and answer results based on the target visual vocabulary and the task type includes: determining a semantic template corresponding to the task type; extracting associated attributes from the target visual vocabulary according to the semantic template; constructing a visual semantic description according to the associated attributes to obtain a question and answer result.

[0087] Among them, a semantic template is a predefined structure that describes the form and structure of the answer required according to the task type. Semantic templates usually contain a fixed grammatical framework and provide a standardized output format for different types of tasks. The task type, such as counting, classification, description, etc., determines which semantic template is needed. For example, a counting task may require a template containing "target category" and "quantity", while a description task may require a template containing "target category" and "action" or "attribute". The purpose of matching semantic templates is to ensure that the format of the generated answer is highly consistent with the task requirements and avoid generating answers that do not match the task. Different task types require different grammatical structures and contents. Semantic templates can help the system organize and express information accurately according to requirements, thereby improving the accuracy and relevance of answers.

[0088] Among them, the associated attributes refer to the visual information or semantic information related to the task type, which can be extracted from the target visual vocabulary and is the key to generating question-answering results. The associated attributes include target category, quantity, position, action, etc., and the specific attributes depend on the task type. For example, in a counting task, the associated attributes may be "object category" and "quantity"; in a description task, the associated attributes may be "object category" and "attribute", such as color, shape, etc. The reason for extracting the associated attributes is to obtain key information related to the task from the target visual vocabulary. By extracting this key information from the visual vocabulary, it can be ensured that the generated answers accurately reflect the actual content in the image and meet the task requirements. Without these attributes, the generated answers may lack pertinence or accuracy.

[0089] Optionally, each task type has its own predefined template structure. For example, the template for counting tasks focuses on the category and quantity of objects, the classification task only focuses on the category of objects, and the description task covers the category, attributes, and actions of objects. The system automatically selects the appropriate template through the mapping relationship between task types and semantic templates to ensure that the format and content of the generated answer meet the task requirements, thereby achieving accurate answer generation.

[0090] Optionally, the process of extracting associated attributes is mainly to filter the relevant attributes based on the information in the target visual vocabulary and the requirements of the task type. The target visual vocabulary contains information such as the category, position, color, quantity, and action of the objects in the image, while the task type determines which attributes are most critical for generating answers. For example, in counting tasks, associated attributes usually include the type and quantity of objects; in description tasks, associated attributes may be the category, color, or action of objects. Depending on the task type, the system will automatically extract these relevant attributes from the visual vocabulary and only focus on the attributes that are closely related to the task type. In this way, the information in the visual vocabulary can be effectively mapped to the task requirements, thereby providing key information for constructing semantic descriptions.

[0091] Optionally, the process of constructing a visual semantic description based on the associated attributes is actually to combine the associated attributes required by the task type with the semantic template to generate a structured description that meets the task requirements. In the counting task, the constructed visual semantic description may be similar to: "target: dog, number: 2", and in the description task, a semantic description similar to "target: dog, attribute: brown, action: running" may be constructed.

[0092] In one embodiment, before extracting features of different dimensions of the image to be processed according to the task type to obtain different types of visual vocabulary, the method further includes: preprocessing the image to be processed; the preprocessing includes: one or more of image normalization, image resizing, and image enhancement.

[0093] In this embodiment, the preprocessing of the image to be processed is to improve the effect and accuracy of subsequent feature extraction.

[0094] Image normalization is to unify the image pixel values ​​into a standard range (such as [0, 1] or [-1, 1]). Images taken by different devices may have different brightness, contrast, and color distribution. Normalization can reduce the impact of these differences on image recognition.

[0095] Since different image sizes will affect the input layer size of the subsequent neural network model, the image needs to be resized to a uniform size. Especially for deep learning models, such as CN, the model usually requires the input image size to be consistent. Optionally, the image size can be adjusted by interpolation or cropping according to the preset target size so that the image can adapt to the input requirements of the model.

[0096] Image enhancement is used to generate diverse training samples to improve the robustness of the model. Through image enhancement, the model can learn from more transformed data, avoid overfitting, and improve the generalization ability of the model on unknown data. Optionally, image enhancement can be performed by random rotation, random flipping, random cropping, and color change.

[0097] In an exemplary embodiment, in combination Figure 2 As shown, Figure 2 The present invention is a flowchart of the steps of a multimodal text-image question-answering method in one embodiment.

[0098] In this embodiment, the attention mechanism in VIT and CNN is used to extract features from the image to be processed, and a global visual vocabulary and a local visual vocabulary are obtained respectively. The question-answering result is generated by the LLM model.

[0099] First, the task type corresponding to the image to be processed is identified, and then the image to be processed is input into VIT and CNN respectively to obtain the global visual vocabulary and the local visual vocabulary respectively. The features of the global visual vocabulary and the local visual vocabulary are fused through a linear network to obtain the target visual vocabulary.

[0100] Finally, the target visual vocabulary is input into the LLM model to obtain the question-answering result.

[0101] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiment of the present application also provides a multimodal graphic question-and-answer device for implementing the multimodal graphic question-and-answer method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multimodal graphic question-and-answer device embodiments provided below can refer to the limitations of the multimodal graphic question-and-answer method above, and will not be repeated here.

[0103] In an exemplary embodiment, Figure 3 As shown, a multimodal image-text question-answering device is provided, comprising: a receiving module, a type recognition module, a feature extraction module, a feature fusion module and an answering module, wherein:

[0104] The receiving module is used to receive the image to be processed and the question text.

[0105] The type identification module is used to identify tasks based on question text and determine the task type.

[0106] The feature extraction module is used to extract features of different dimensions from the images to be processed according to the task type, and obtain different types of visual vocabulary.

[0107] The feature fusion module is used to fuse different types of visual vocabulary to obtain the target visual vocabulary.

[0108] The answer module is used to generate question-answering results based on the target visual vocabulary and task type.

[0109] In one embodiment, the feature extraction module includes:

[0110] The attention mechanism extraction unit is used to extract features of different dimensions from the processed images through different attention mechanisms and task types to obtain a global visual vocabulary and a local visual vocabulary; the task type is used to guide the attention mechanism to extract features.

[0111] In one embodiment, the attention mechanism extraction unit includes a first attention mechanism and a second attention mechanism, wherein the first attention mechanism is used to extract a global visual vocabulary; and the second attention mechanism is used to extract a local visual vocabulary.

[0112] In one embodiment, the attention mechanism extraction unit includes a first extraction subunit, wherein the first extraction subunit includes:

[0113] The image division subunit is used to divide the image to be processed into image blocks to obtain an initial image block sequence.

[0114] The weight adjustment subunit is used to adjust the attention weights of a preset number of attention layers according to the task type to obtain a first attention weight, and to adjust the attention weights of other attention layers to obtain a second attention weight; the first attention weight is used to extract global semantic features; the second attention weight is used to extract local detail features.

[0115] The first vocabulary extraction subunit is used to extract features from the initial image block sequence according to the first attention weight to obtain a first initial vocabulary.

[0116] The second vocabulary extraction subunit is used to extract features from the initial image block sequence according to the second attention weight to obtain a second initial vocabulary.

[0117] The weighted subunit is used to perform weighted fusion on the first initial vocabulary and the second initial vocabulary according to the task type to obtain a global visual vocabulary.

[0118] In one embodiment, the feature fusion module includes:

[0119] The template determination unit is used to determine the semantic template corresponding to the task type.

[0120] The attribute extraction unit is used to extract relevant attributes from the target visual vocabulary according to the semantic template.

[0121] The semantic description unit is used to construct a visual semantic description based on the associated attributes to obtain the question-answering result.

[0122] In one embodiment, the above device further comprises:

[0123] The preprocessing module is used to preprocess the image to be processed; wherein the preprocessing includes one or more of image normalization, image size adjustment and image enhancement.

[0124] Each module in the above multimodal graphic question-and-answer device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal graphic question and answer method is implemented.

[0126] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0130] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multimodal picture-text question answering method, characterized in that: The method comprises: Receive the image to be processed and the question text; Performing task identification based on the question text to determine the task type; According to the task type, extracting features of different dimensions from the image to be processed to obtain different types of visual word lists; fusing the different types of visual word lists to obtain a target visual word list; Generate a question-answering result according to the target visual vocabulary and the task type.

2. The method according to claim 1, characterized in that According to the task type, the features of the image to be processed are extracted in different dimensions to obtain different types of visual word lists. Extracting features of different dimensions from the image to be processed by using different attention mechanisms and the task types to obtain a global visual vocabulary and a local visual vocabulary; The task type is used to guide the attention mechanism to perform feature extraction.

3. The method according to claim 2, characterized in that The attention mechanism includes a first attention mechanism and a second attention mechanism, wherein the first attention mechanism is used to extract the global visual vocabulary; and the second attention mechanism is used to extract the local visual vocabulary.

4. The method according to claim 3, characterized in that The first attention mechanism includes multiple attention layers; The extraction process of the global visual vocabulary includes: Dividing the image to be processed into image blocks to obtain an initial image block sequence; According to the task type, adjusting the attention weights of the preset number of attention layers to obtain a first attention weight, and adjusting the attention weights of other attention layers to obtain a second attention weight; the first attention weight is used to extract global semantic features; the second attention weight is used to extract local detail features; Performing feature extraction on the initial image block sequence according to the first attention weight to obtain a first initial vocabulary; Performing feature extraction on the initial image block sequence according to the second attention weight to obtain a second initial vocabulary; According to the task type, weighted fusion is performed on the first initial vocabulary and the second initial vocabulary to obtain the global visual vocabulary.

5. The method according to claim 1, characterized in that Generating a question-answering result according to the target visual vocabulary and the task type includes: Determining a semantic template corresponding to the task type; extracting associated attributes from the target visual vocabulary according to the semantic template; A visual semantic description is constructed according to the associated attributes to obtain the question-answering result.

6. The method according to claim 1, characterized in that Before extracting features of different dimensions from the image to be processed to obtain different types of visual word lists according to the task type, the method further includes: Preprocessing the to-be-processed object; The preprocessing includes: one or more of image normalization, image size adjustment and image enhancement.

7. A multi-modal graphic question-and-answer device, characterized in that: The device comprises: A receiving module, used for receiving the image to be processed and the question text; A type identification module, used to identify the task based on the question text and determine the task type; A feature extraction module, used to extract features of different dimensions on the image to be processed according to the task type, to obtain different types of visual word lists; A feature fusion module, used for fusing different types of visual vocabulary to obtain a target visual vocabulary; The answer module is used to generate a question-answering result according to the target visual vocabulary and the task type.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images