Multi-modal question and answer method and device based on multi-angle image, and electronic equipment

Abstract

The invention provides a multi-modal question answering method and device based on multi-angle images and electronic equipment, and the method comprises the steps: obtaining an original image and a question text input by a user, and generating a plurality of multi-angle images based on the original image, the multi-angle images being used for representing different angles of the same entity in the original image; preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; obtaining a trainable soft prompt corresponding to each embedded vector, and carrying out multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; and performing prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text. By making full use of the visual information contained in different images, the visual information obtained by the model when the answer is generated is richer and more comprehensive, so that the illusion problem existing in the multi-modal question and answer process is relieved, and the answer generation precision of the model is higher.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-modal question-answering method, device, and electronic device based on multi-angle images. Background Art

[0002] Large Language Models (LLMs) are natural language processing (NLP) models based on deep learning technology. They can generate grammatically correct and contextual language output by pre-training on massive text data. In recent years, with the advancement of model architecture (such as Transformer) and training methods, LLMs have made remarkable achievements in language understanding and generation. LLMs have demonstrated excellent performance in many fields such as text generation, machine translation, and automatic question answering, promoting the development of the NLP field.

[0003] However, LLMs also face some challenges, including the problem of hallucination: LLMs sometimes generate fictitious content that has nothing to do with actual semantics, a phenomenon known as "hallucination." For example, when answering questions, LLMs may generate non-existent facts or unfounded inferences.

[0004] This shows that the visual input information of the multimodal large model in the related technology is too single and has the defect of easily causing the "hallucination problem". Summary of the invention

[0005] The present invention provides a multimodal question-answering method, device, and electronic device based on multi-angle images, which are used to solve the defect in the prior art that the visual input information of the multimodal large model is too single, thus easily causing the "hallucination problem". By making full use of the visual information contained in different images, the model obtains richer and more comprehensive visual information when generating answers, thereby alleviating the hallucination problem existing in the multimodal question-answering process and making the model generate answers with higher accuracy.

[0006] The present invention provides a multimodal question-answering method based on multi-angle images, comprising the following steps: obtaining an original image and a question text input by a user, and generating a plurality of multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; obtaining trainable soft prompts corresponding to each of the embedded vectors respectively, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; performing prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0007] According to a multimodal question-answering method based on multi-angle images provided by the present invention, multi-view feature perception processing is performed based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation, including: injecting the visual information contained in each of the embedded vectors into the corresponding trainable soft prompts based on a cross-attention mechanism to obtain soft prompts containing visual information; calculating the attention weights of the embedded vectors based on a multi-head attention mechanism to obtain weight values ​​corresponding to each of the embedded vectors; and performing weighted summation processing on the soft prompts containing visual information based on the weight values ​​to obtain the multi-angle visual feature representation.

[0008] According to a multimodal question-answering method based on multi-angle images provided by the present invention, the visual information contained in each of the embedded vectors is injected into the corresponding trainable soft prompt based on the cross-attention mechanism to obtain a soft prompt containing visual information, including: for each of the embedded vectors, the corresponding trainable soft prompt is used as a query variable, the embedded vector is used as a key and a value, and cross-attention processing is performed based on the query variable, the key and the value to obtain the soft prompt containing visual information.

[0009] According to a multimodal question-answering method based on multi-angle images provided by the present invention, the method also includes: for each of the embedded vectors, mapping the embedded vectors into multiple groups of vector representations of the same dimension; calculating the attention weights of the embedded vectors based on the multi-head attention mechanism to obtain the weight values ​​corresponding to each of the embedded vectors, including: for each group of the vector representations, using the corresponding trainable soft prompt as a query variable, using the vector representation as a key, and performing multi-head attention processing based on the query variable and the key to obtain the weight score corresponding to each attention head; calculating the average value of the weight scores corresponding to each attention head, and using the average value as the weight value corresponding to each of the embedded vectors.

[0010] According to a multimodal question-answering method based on multi-angle images provided by the present invention, the multi-angle images are pre-processed respectively to obtain embedded vectors corresponding to the multi-angle images respectively, including: performing image encoding processing on the multi-angle images respectively to obtain image embedding vectors corresponding to the multi-angle images respectively; and mapping processing on the image embedding vectors to obtain embedded vectors corresponding to the multi-angle images respectively.

[0011] According to a multimodal question-answering method based on multi-angle images provided by the present invention, the prediction processing is performed based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text, including: performing feature extraction processing on the question text to obtain text features; splicing the multi-angle visual feature representation and the text features to obtain spliced ​​features; and inputting the spliced ​​features into a pre-trained language model for prediction processing to obtain an answer corresponding to the question text.

[0012] The present invention also provides a multimodal question-answering device based on multi-angle images, comprising the following modules: an acquisition module, used to acquire an original image and a question text input by a user, and generate a plurality of multi-angle images based on the original image, wherein the multi-angle image is used to represent different angles of the same entity in the original image; a preprocessing module, used to preprocess the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; a feature perception module, used to acquire trainable soft prompts corresponding to each of the embedded vectors respectively, and perform multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; a prediction module, used to perform prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multimodal question-answering method based on multi-angle images as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal question-answering methods based on multi-angle images as described above.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multimodal question-answering method based on multi-angle images as described above is implemented.

[0016] The multimodal question-answering method based on multi-angle images provided by the present invention obtains the original image and question text input by the user, and generates a number of multi-angle images of different angles for representing the same entity in the original image based on the original image, thereby making full use of the visual information contained in different images; then, multi-view feature perception processing is performed according to the embedded vectors corresponding to the multi-angle images and the trainable soft prompts, and a multi-angle visual feature representation is obtained; finally, prediction processing is performed based on the multi-angle visual feature representation and the question text to obtain the answer corresponding to the question text. By making full use of the visual information contained in different images, the visual information obtained by the model when generating the final text output is richer and more comprehensive, and the image information from different angles can further enhance the model's perception of image details, thereby alleviating the hallucination problem that is widely present in the answer generation process, so that the model produces a more reasonable and reliable answer output. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the multimodal question-answering method based on multi-angle images provided by the present invention.

[0019] Figure 2 This is the second flow chart of the multimodal question-answering method based on multi-angle images provided by the present invention.

[0020] Figure 3 This is the third flow chart of the multimodal question-answering method based on multi-angle images provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the structure of the multi-view feature sensor provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the multi-head sampler provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of a multi-modal question-answering device based on multi-angle images provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Traditional multimodal models mostly rely on visual input from a single angle, such as processing only image information from a single camera or a single perspective. However, this design has obvious limitations. First, image information from a single angle may not fully display the characteristics of the object, resulting in a lack of comprehensiveness in the results generated by the model; second, when processing multimodal information, the model may face conflicts or inconsistencies between different modal information, which in turn leads to the so-called "hallucination problem", that is, the model generates false and inaccurate outputs. The existence of the hallucination problem not only affects the performance of large multimodal models, but also limits their application in practical scenarios. In some high-risk scenarios, such as autonomous driving and medical image analysis, the hallucination problem may lead to serious consequences. For example, in autonomous driving, if the model incorrectly judges the number or location of pedestrians on the road, it may cause an accident; in medical diagnosis, the model's misjudgment of image data may lead to incorrect diagnostic results.

[0027] Optionally, the multimodal question and answer method based on multi-angle images of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the multimodal question and answer method based on multi-angle images in this embodiment being executed by a server.

[0028] Figure 1 is one of the flow charts of the multi-modal question answering method based on multi-angle images provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0029] Step 101: Acquire an original image and question text input by a user, and generate a plurality of multi-angle images based on the original image.

[0030] Among them, the multi-angle image is used to represent different angles of the same entity in the original image.

[0031] In some embodiments, when the user inputs only one original image, the model may ignore information that may be crucial but not obvious or sufficient when obtaining visual information because the information in the single image is not rich enough or the amount of information in detail is too little, which in turn causes the model to be unable to obtain effective information from the visual representation when making a response. In addition, for real three-dimensional objects, the image is only a two-dimensional projection of the object, which is insufficient to reflect the three-dimensional information of the object. The model cannot obtain the true situation of the part of the object that does not appear in the original image, and the loss of information will also lead to the appearance of model hallucinations.

[0032] In some embodiments, see Figure 3 , Figure 3 FIG. 3 is a flow chart of a multi-modal question-answering method based on multi-angle images provided by the present invention. Figure 3 As shown, the embodiment of the present invention expands a single original image input by a user into multiple images of different angles for representing the same entity in the original image through a multi-angle image generation module. The multi-angle image generation module can use the HFGI3D model, which is a high-fidelity 3D generative adversarial network (GAN) inversion framework that can retain specific details of the input original image while synthesizing realistic new views. For example, when processing an image containing facial features and background of a specific person, the HFGI3D model can generate 3D views from different perspectives based on this image, and details such as the facial features of the person and the background can be well preserved. The HFGI3D model uses pre-trained generators, visibility analysis, and potential optimization to achieve high-fidelity 3D rendering and editing, thereby retaining the details of the input image and synthesizing new views, and generating multiple images of the same entity from different perspectives.

[0033] In an embodiment of the present invention, a single input original image is processed by a multi-angle image generation module to generate multiple images observed from different perspectives, which not only improves the model's ability to understand objects, but also provides richer input for subsequent visual information extraction. In practical applications, the multi-angle image generation module can be used in various scenarios that require multi-perspective information. For example, in the field of autonomous driving, vehicles need to obtain information about the surrounding environment from different angles through multiple cameras in order to make more accurate decisions; in tasks such as robot navigation and object recognition, the multi-angle image generation module can help the model better understand the three-dimensional features of objects, thereby enhancing the richness of visual information.

[0034] In other embodiments, when the original image input by the user is multiple images of the same entity at different angles, there is no need to use a multi-angle image generation module to generate a multi-angle image based on the original image. The subsequent preprocessing steps can be directly performed on the multiple original images, thereby reducing additional computing overhead.

[0035] Step 102: pre-process the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively.

[0036] In some embodiments, step 102 can be specifically implemented in the following manner: performing image encoding processing on the multi-angle images respectively to obtain image embedding vectors corresponding to the multi-angle images respectively; performing mapping processing on the image embedding vectors to obtain embedded vectors corresponding to the multi-angle images respectively.

[0037] Specifically, see Figure 3 After obtaining multiple multi-angle images at different angles, the multi-angle images are input into the image encoder through the image encoder respectively, and the multi-angle images are encoded into a high-dimensional embedded representation. The input accepted by the image encoder is a single image, and the output is the embedded representation vector corresponding to the image, that is, the image embedding vector. Each multi-angle image at different angles is encoded by the image encoder into a high-dimensional embedded vector of the same dimension for subsequent further calculations.

[0038] Since the semantic spaces of language modality and image modality are different, after obtaining the embedded representation vectors corresponding to different multi-angle images, it is necessary to use the image mapping module to map the obtained image embedding vectors to the semantic space of the language modality, so that the subsequent language model can directly interact with the visual information contained in the image and generate text output. The input accepted by the image mapping module is the image embedding vector encoded by the image encoder, and the output is the text embedded representation vector of the same dimension, that is, the embedded vectors corresponding to the multi-angle images.

[0039] In some embodiments, the image encoder can use the Vision Transformer model, and the mapping module can use a fully connected network. Vision Transformer is an image encoder based on a self-attention mechanism that can extract global features of an image by dividing the image into multiple patches and processing these patches as input. Each multi-angle image encoded by ViT is converted into a high-dimensional embedded vector representation, which can retain the key features in the image and facilitate subsequent calculations and processing.

[0040] At the same time, the present invention uses a fully connected network as a mapping module to project the image embedding vector into the semantic space of the language model. Through this mapping process, the features between different modalities can be aligned in the same space, thereby facilitating the fusion and processing of multimodal information. During the training of the model, the image encoder does not participate in parameter updates, that is, the weights of the encoder part remain fixed, avoiding excessive adjustment of image feature extraction during training, and being able to fully utilize the visual information extraction capabilities of the pre-trained Vision Transformer model, which not only improves the stability of the model, but also reduces the computational overhead during training.

[0041] Step 103: Obtain a trainable soft prompt corresponding to each of the embedded vectors, and perform multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation.

[0042] In some embodiments, see Figure 4 , Figure 4 It is a structural schematic diagram of the multi-view feature sensor provided by the present invention, and the multi-view feature sensor includes a visual extraction module and a multi-head sampler, wherein the visual extraction module extracts visual information according to the embedded vector corresponding to the preprocessed multi-angle image, and the multi-head sampler calculates the importance of the visual information in different multi-angle images, and calculates the relative contribution of the visual information in the multi-angle image according to the importance, and the multi-view feature sensor accepts the embedded vector corresponding to the preprocessed multi-angle image and the trainable soft prompt corresponding to each embedded vector as input, and outputs a single high-dimensional embedded vector as the final representation containing multi-angle visual information, that is, a multi-angle visual feature representation.

[0043] Specifically, the multi-view feature perceptron interacts the embedded vector with trainable soft prompts through prompt learning and cross-attention mechanism to extract key visual information from multi-view images.

[0044] Among them, prompt learning is a technology that is gradually gaining attention in deep learning. It allows the model to receive prompt information before inputting data, thereby guiding the model's attention to focus on specific features. In the present invention, trainable soft prompts serve as a bridge for multi-angle visual feature identification extraction, helping the model to extract the most representative visual information from the embedded vectors corresponding to multi-angle images. Each trainable soft prompt represents the main features extracted from the multi-angle image input from a specific angle.

[0045] The cross-attention mechanism is another important component of the multi-view feature perceptron. This mechanism allows information between different modalities to interact, thereby enhancing the model's ability to process multi-modal information. By using the embedded vectors corresponding to multi-angle images as keys and values, and soft prompts as query vectors, the cross-attention mechanism can extract key information from multi-angle images and inject it into soft prompts, ultimately generating soft prompts containing visual information as output. The visual extraction module adopts a structure that stacks multiple transformer decoder blocks, each of which consists of a cross-attention layer, a Layernorm layer, and a fully connected layer of a feed-forward network. This design can make full use of the input information of multi-view images, aggregate and extract visual information from different angles, and ensure that the final output multi-angle visual feature representation (i.e. Figure 4 The final visual representation in ) can capture all important image feature information.

[0046] In some embodiments, see Figure 2 , Figure 2 This is the second flow chart of the multi-modal question-answering method based on multi-angle images provided by the present invention. Figure 1 Step 103 shown can be specifically performed by Figure 2 Steps 201 to 203 shown in FIG.

[0047] Step 201: Based on the cross-attention mechanism, the visual information contained in each of the embedded vectors is injected into the corresponding trainable soft prompt to obtain a soft prompt containing the visual information.

[0048] In some embodiments, step 201 can be specifically implemented in the following manner: for each of the embedded vectors, the corresponding trainable soft prompt is used as a query variable, the embedded vector is used as a key and a value, and cross-attention processing is performed based on the query variable, key and value to obtain the soft prompt containing visual information.

[0049] Specifically, see Figure 4 , Figure 4 The visual extraction module in accepts the preprocessed embedded vectors and trainable soft prompts as input and generates soft prompts containing visual information. Through the trainable soft prompts and cross-attention mechanism, the visual extraction module extracts the main visual information from the embedded vectors corresponding to the multi-view images for downstream tasks. The visual extraction module is a main information extractor of visual information and aligns the visual information in the embedded vectors with the semantic space of the language model. The output of the visual extraction module is a set of soft prompts containing visual information, and each output soft prompt contains the visual information corresponding to the trainable soft prompt in the embedded vector corresponding to the multi-view image.

[0050] The visual extraction module in the present invention is composed of a plurality of stacked Transformer decoder blocks, wherein each Transformer decoder block is composed of a cross-attention layer, a Layernorm layer and a fully connected layer of a feedforward network. The visual extraction module uses the input trainable soft prompt as the query in the attention mechanism, and the embedded vector corresponding to the multi-angle image as the key and value, and uses the cross-attention mechanism to inject the visual information contained in the embedded vector into the trainable soft prompt to form a soft prompt containing visual information.

[0051] Formally, the embedded vectors of multiple inputs are represented as ,in It is The embedded vectors of the input multi-angle images, is the total number of embedded vectors; the trainable soft hints are denoted as , each trainable soft cue contains Tokens, ,in is the model dimension of the language model. The trainable soft prompts here are intended to provide appropriate visual prompts for the downstream tasks that the language model needs to complete. The trainable soft prompts and embedded vectors are input into the cross-attention layer so that the soft prompts interact with the visual information. In the cross-attention mechanism, the trainable soft prompts are mapped to the Query, and the embedded vectors corresponding to the input multi-angle images from different angles are mapped to the Key and Value respectively. Since there are embedded vectors corresponding to multiple multi-angle images, the visual extraction module performs cross-attention on the embedded vector of each multi-angle image and its corresponding trainable soft prompt.

[0052] The output of the visual extraction module is represented as , the mapping matrix is ​​expressed as , and ,but It can be calculated as follows:

[0053] in, represents the output of the visual extraction module, , and Represent the mapping matrices corresponding to query, key, and value respectively, For trainable soft prompts, For the The embedded vectors corresponding to the input multi-angle images are Embedded vectors representing multiple inputs, is the model dimension of the language model, represents a real matrix.

[0054] Since different multi-angle input images have different contributions to the final output, the output The weights calculated by the multi-head sampler (i.e. Figure 4 In addition, the visual extraction module calculates each cross-attention output separately and in parallel, instead of taking the previous output in the above formula as the next Query. This is because in most cases, there is no sequential relationship between the input multi-angle images, so their relationship with the trainable soft prompts should be calculated separately and the visual information should be extracted.

[0055] Step 202: Calculate the attention weights of the embedded vectors based on the multi-head attention mechanism to obtain the weight values ​​corresponding to each of the embedded vectors.

[0056] In some embodiments, before step 202, the present invention further includes the following steps: for each of the embedded vectors, mapping the embedded vectors into multiple groups of vector representations of the same dimension.

[0057] Step 202 can be specifically implemented in the following manner: for each group of the vector representations, the corresponding trainable soft prompt is used as a query variable, the vector representation is used as a key, and multi-head attention processing is performed based on the query variable and the key to obtain a weight score corresponding to each attention head; the average value of the weight scores corresponding to each attention head is calculated, and the average value is used as the weight value corresponding to each embedded vector.

[0058] Step 203: Perform weighted sum processing on the soft prompts containing visual information based on the weight values ​​to obtain the multi-angle visual feature representation.

[0059] Specifically, images at different angles may contain different visual information, and the contribution of this visual information may also be different. For example, an image showing the front of an object may contain more key information than another image showing the back of the object. Therefore, considering that image inputs at different angles contain different visual information and therefore have different contributions to the final visual representation, a multi-head sampler is designed to calculate the relative contribution of the embedded vectors corresponding to input images at different angles to the final visual representation, and to measure the importance of the visual information corresponding to different multi-angle images in the form of weight values.

[0060] In some embodiments, see Figure 5 , Figure 5 It is a structural schematic diagram of the multi-head sampler provided by the present invention. The multi-head sampler includes a decoupling module for further decomposing the information from the embedded vector, and multiple attention heads for calculating the attention weights of different embedded vectors, and taking the average of the attention weights of each attention head as the output of the multi-head sampler.

[0061] The multi-head sampler is responsible for calculating the relative contribution of the embedded vectors corresponding to images at different angles to the final visual representation. The output of the multi-head sampler is used as the weight of the weighted sum of the visual information extracted by the visual extraction module. The weighted sum of the embedded representation vectors in . In the multi-head sampler, in order to further decompose the visual information of the input image, the present invention uses a decoupling module to further decouple the input embedded vector. The decoupling module is composed of a two-layer fully connected multi-layer perceptron (MLP), which accepts the [CLS] token in the embedded vector and maps it to multiple groups of vector representations of the same dimension. Then, the multi-head attention mechanism is used to calculate the attention weight for each group, and the mean of the attention weights of each group is used as the relative contribution of the multi-head sampler output.

[0062] The decoupling module in the multi-head sampler only takes the [CLS] token in the image embedded representation vector as input and maps it into multiple sets of different embedded vectors for further calculation. The two-layer fully connected network (MLP) in the decoupling module can further enhance the model's ability to handle fine-grained features. By mapping the [CLS] token in the embedded vector into multiple sets of vector representations, the decoupling module ensures that the features corresponding to images from different perspectives can be fully utilized at a fine-grained level. Each attention head calculates the contribution of different vector representations in its corresponding group, and the final weighted sum represents the visual information of multi-view input.

[0063] In the multi-head attention module, each group of vector representations is assigned a trainable attention head to calculate the relative contribution of different vector representations in the group. The purpose of this design is to introduce multiple experts in the form of attention heads to focus on different features in the image input, because only one attention head cannot fully capture the correlation between the coarse-grained embedded vectors and the trainable soft cues. Figure 5 As shown in Figure 1, the soft prompt can be trained as a query in the attention mechanism, and each group of vectors after decomposition is represented as a key for calculating the attention weight. In the multi-head attention module, only the attention weight is calculated, and the module uses the average value on the query dimension as the output of each attention head. Represented as the jth decomposition token of the i-th input image embedding, the output weight of each attention head is It can be calculated by the following formula:

[0064] in, is the model dimension, For trainable soft prompts, Indicates The embedded vectors of multiple inputs of decomposed tokens, For and The attention score calculation function, that is, the attention weight calculation process in each attention head. is the number of tokens in the trainable soft hint, Expressed as The first embedded image of the input decomposition token, Indicates The output weight of each attention head; Indicates The score matrix of the attention heads, represents a real matrix, Represents the dimension of the weight vector.

[0065] After each attention head calculates the attention weight score for each set of embedded representation vectors, the average of the weights of each attention head is used as the output of the multi-head sampler. :

[0066] In the above formula, The average value of the weights of each attention head is taken as the output of the multi-head sampler, Indicates The output weight of the attention head is represents a real matrix, represents the dimension of the weight vector, The number of groups into which the embedded vectors corresponding to the input multi-angle images are decoupled by the decoupling module is represented by multiple groups of vectors. It is also the number of attention heads in the multi-head attention module (one attention head is assigned to each group of embedded representation vectors in the multi-head attention module, so the number of attention heads is the same as the number of mapping groups in the decoupling module).

[0067] The multi-head sampler aims to further capture fine-grained visual features in the embedded vector of the input image by applying different attention heads after decomposing the embedded vector of the input. Different attention heads can focus on different aspects of the input and perceive their intrinsic relationships, further extract visual information between different images, and calculate relative contributions.

[0068] Furthermore, in the multi-view feature sensor, the visual extraction module extracts the visual information in different embedded vectors, and the multi-head sampler obtains the relative contribution of the visual information in different embedded vectors to the final visual representation. The final visual representation is obtained by performing weighted summation based on the relative contribution calculated by the multi-head sampler as the weight of the input of the visual extraction module. The calculation process can be expressed by the following formula:

[0069] in, for Middle The weight value corresponding to the input, represents the output of the visual extraction module, The average value representing the weights of each attention head is taken as the output of the multi-head sampler.

[0070] In the design of the multi-view feature sensor of the present invention, the number of image inputs is not limited, and this design enables the model architecture of the present invention to accept any number of image inputs. The output of the multi-view feature sensor adopts the form of weighted summation, which also ensures that the input order has no effect on the final output, making the model more stable and reliable in actual use.

[0071] Step 104: Perform prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0072] In some embodiments, step 104 can be specifically implemented in the following ways: performing feature extraction processing on the question text to obtain text features; splicing the multi-angle visual feature representation and the text features to obtain spliced ​​features; inputting the spliced ​​features into a pre-trained language model for prediction processing to obtain an answer corresponding to the question text.

[0073] Specifically, in the model architecture of the present invention, the language model has various forms, which can be various language models with certain language capabilities after pre-training, or language models with multimodal understanding and generation capabilities after multimodal alignment. After extracting visual information and generating final visual prompts (i.e., multi-angle visual features) through a multi-view feature sensor, feature extraction processing is performed on the question text input by the user to obtain text features, and the final visual prompts are spliced ​​before the text features and used as inputs of the language model for calculation. The language model completes text output based on the user text input and the visual prompts containing visual information, and obtains the answer corresponding to the original image input by the user and the question text.

[0074] By combining visual cues with the features of the text input by the user, the language model can fully consider the visual information when generating text, thereby improving the accuracy and logic of the output. This design enables the multimodal model architecture of the present invention to be widely used in various scenarios, including natural language generation, machine translation, question-answering systems and other tasks.

[0075] The multimodal question-answering device based on multi-angle images provided by the present invention is described below. The multimodal question-answering device based on multi-angle images described below and the multimodal question-answering method based on multi-angle images described above can be referred to each other.

[0076] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a multi-modal question-answering device based on multi-angle images provided by the present invention, such as Figure 6 As shown, the multimodal question-answering device based on multi-angle images provided by the present invention includes an acquisition module 601, a preprocessing module 602, a feature perception module 603 and a prediction module 604, wherein the acquisition module 601 is used to acquire the original image and question text input by the user, and generate a plurality of multi-angle images based on the original image, wherein the multi-angle image is used to represent different angles of the same entity in the original image; the preprocessing module 602 is used to preprocess the multi-angle images respectively to obtain the embedded vectors corresponding to the multi-angle images respectively; the feature perception module 603 is used to obtain the trainable soft prompts corresponding to each of the embedded vectors respectively, and perform multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; the prediction module 604 is used to perform prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0077] Specifically, the multimodal question and answer device based on multi-angle images provided by the present invention can implement all the method steps implemented by the multimodal question and answer method embodiment based on multi-angle images, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0078] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a multi-modal question-answering method based on multi-angle images, the method comprising: obtaining an original image and a question text input by a user, and generating a plurality of multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; pre-processing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; obtaining trainable soft prompts corresponding to each of the embedded vectors respectively, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; performing prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0079] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0080] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal question and answer method based on multi-angle images provided by the above methods, and the method includes: obtaining an original image and a question text input by a user, and generating a plurality of multi-angle images based on the original image, wherein the multi-angle image is used to represent different angles of the same entity in the original image; preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; obtaining trainable soft prompts corresponding to each of the embedded vectors respectively, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; performing prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0081] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal question-answering method based on multi-angle images provided by the above-mentioned methods, the method comprising: obtaining an original image and a question text input by a user, and generating a plurality of multi-angle images based on the original image, wherein the multi-angle image is used to represent different angles of the same entity in the original image; preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; obtaining trainable soft prompts corresponding to each of the embedded vectors respectively, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; performing prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal question answering method based on multi-angle images, characterized in that: include: Acquire an original image and a question text input by a user, and generate a plurality of multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; Preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; Obtaining trainable soft prompts corresponding to each of the embedded vectors, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representation; Prediction processing is performed based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

2. The multimodal question answering method based on multi-angle images according to claim 1, characterized in that: The performing multi-view feature perception processing based on the embedded vector and the trainable soft prompt to obtain a multi-angle visual feature representation includes: Injecting the visual information contained in each of the embedded vectors into the corresponding trainable soft prompt based on a cross attention mechanism to obtain a soft prompt containing the visual information; Calculate the attention weights of the embedded vectors based on the multi-head attention mechanism to obtain the weight values ​​corresponding to each of the embedded vectors; The soft prompts containing visual information are weighted and summed based on the weight values ​​to obtain the multi-angle visual feature representation.

3. The multimodal question answering method based on multi-angle images according to claim 2, characterized in that: The method of injecting the visual information contained in each of the embedded vectors into the corresponding trainable soft prompts based on the cross attention mechanism to obtain the soft prompts containing the visual information includes: For each of the embedded vectors, the corresponding trainable soft prompt is used as a query variable, the embedded vector is used as a key and a value, and cross-attention processing is performed based on the query variable, the key and the value to obtain the soft prompt containing visual information.

4. The multimodal question answering method based on multi-angle images according to claim 2, characterized in that: The method further comprises: For each of the embedded vectors, mapping the embedded vectors into multiple groups of vector representations of the same dimension; The method of calculating the attention weights of the embedded vectors based on the multi-head attention mechanism to obtain the weight values ​​corresponding to each of the embedded vectors includes: For each set of the vector representations, using the corresponding trainable soft prompt as a query variable and the vector representation as a key, and performing multi-head attention processing based on the query variable and the key to obtain a weight score corresponding to each attention head; Calculate the average value of the weight scores corresponding to each of the attention heads, and use the average value as the weight value corresponding to each of the embedded vectors.

5. The multi-modal question answering method based on multi-angle images according to claim 1, characterized in that: The preprocessing of the multi-angle images respectively to obtain the embedded vectors corresponding to the multi-angle images respectively includes: Performing image coding processing on the multi-angle images respectively to obtain image embedding vectors corresponding to the multi-angle images respectively; Mapping processing is performed on the image embedding vector to obtain embedded vectors corresponding to the multi-angle images respectively.

6. The multi-modal question answering method based on multi-angle images according to claim 1, characterized in that: The performing prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text includes: Performing feature extraction processing on the question text to obtain text features; Splicing the multi-angle visual feature representation with the text feature to obtain a spliced ​​feature; The concatenated features are input into a pre-trained language model for prediction processing to obtain an answer corresponding to the question text.

7. A multi-modal question-answering device based on multi-angle images, characterized in that: include: An acquisition module, used to acquire an original image and a question text input by a user, and generate a plurality of multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; A preprocessing module, used to preprocess the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; A feature perception module, configured to obtain a trainable soft prompt corresponding to each of the embedded vectors, and perform multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation; The prediction module is used to perform prediction processing based on the multi-angle visual feature representation and the question text to obtain an answer corresponding to the question text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal question answering method based on multi-angle images as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal question answering method based on multi-angle images as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal question answering method based on multi-angle images as claimed in any one of claims 1 to 6 is implemented.

Images