Multimodal question answering methods, devices, and electronic devices based on multi-angle images

Abstract

This invention provides a multimodal question-answering method, apparatus, and electronic device based on multi-angle images, comprising: acquiring an original image and question text input by a user, and generating several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; preprocessing the multi-angle images to obtain embedded vectors corresponding to each multi-angle image; acquiring trainable soft prompts corresponding to each embedded vector, and performing multi-view feature perception processing based on the embedded vectors and trainable soft prompts to obtain multi-angle visual feature representations; and performing prediction processing based on the multi-angle visual feature representations and question text to obtain the answer corresponding to the question text. By fully utilizing the visual information contained in different images, the model obtains richer and more comprehensive visual information when generating answers, thereby alleviating the illusion problem in the multimodal question-answering process and making the accuracy of the model's answer generation higher.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal question-answering method, apparatus, and electronic device based on multi-angle images. Background Technology

[0002] Large Language Models (LLMs) are natural language processing (NLP) models based on deep learning techniques. Through pre-training on massive amounts of text data, they are able to generate grammatically correct and context-sensitive language output. In recent years, with advancements in model architecture (such as Transformer) and training methods, LLMs have achieved remarkable success in language understanding and generation. LLMs have demonstrated outstanding performance in multiple fields, including text generation, machine translation, and automatic question answering, driving the development of the NLP field.

[0003] However, LLMs also face some challenges, including the hallucination problem: LLMs sometimes generate fictitious content that is irrelevant to actual semantics, a phenomenon known as "hallucination." For example, when answering questions, LLMs may generate facts that do not exist or unfounded inferences.

[0004] It is evident that the visual input information of multimodal large models in related technologies is too singular, which is prone to causing "illusion problems". Summary of the Invention

[0005] This invention provides a multimodal question-answering method, device, and electronic device based on multi-angle images to address the shortcomings of existing multimodal large models where the visual input information is too singular, easily leading to the "illusion problem." By fully utilizing the visual information contained in different images, the model obtains richer and more comprehensive visual information when generating answers, thereby alleviating the illusion problem in the multimodal question-answering process and making the model generate answers with higher accuracy.

[0006] This invention provides a multimodal question answering method based on multi-angle images, comprising the following steps: acquiring an original image and question text input by a user, and generating several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; preprocessing the multi-angle images to obtain embedded vectors corresponding to each multi-angle image; acquiring trainable soft prompts corresponding to each embedded vector, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representations; and performing prediction processing based on the multi-angle visual feature representations and the question text to obtain the answer corresponding to the question text.

[0007] According to the present invention, a multimodal question answering method based on multi-angle images is provided. The step of performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain a multi-angle visual feature representation includes: injecting the visual information contained in each embedded vector into the corresponding trainable soft prompt based on a cross-attention mechanism to obtain a soft prompt containing visual information; calculating the attention weights of the embedded vectors based on a multi-head attention mechanism to obtain a weight value corresponding to each embedded vector; and performing a weighted summation processing on the soft prompts containing visual information based on the weight values ​​to obtain the multi-angle visual feature representation.

[0008] According to the present invention, a multimodal question answering method based on multi-angle images is provided, wherein the visual information contained in each embedded vector is injected into the corresponding trainable soft cue based on a cross-attention mechanism to obtain a soft cue containing visual information, comprising: for each embedded vector, using the corresponding trainable soft cue as a query variable, using the embedded vector as a key and a value, and performing cross-attention processing based on the query variable, key and value to obtain the soft cue containing visual information.

[0009] According to the present invention, a multimodal question answering method based on multi-angle images is provided. The method further includes: for each embedded vector, mapping the embedded vector to multiple sets of vector representations of the same dimension; calculating the attention weights of the embedded vectors based on a multi-head attention mechanism to obtain the weight value corresponding to each embedded vector, including: for each set of vector representations, using the corresponding trainable soft prompt as a query variable, using the vector representation as a key, and performing multi-head attention processing based on the query variable and the key to obtain the weight score corresponding to each attention head; calculating the average value of the weight scores corresponding to each attention head, and using the average value as the weight value corresponding to each embedded vector.

[0010] According to the present invention, a multimodal question answering method based on multi-angle images is provided, wherein the preprocessing of the multi-angle images to obtain the embedded vectors corresponding to the multi-angle images includes: performing image encoding processing on the multi-angle images to obtain the image embedding vectors corresponding to the multi-angle images; and performing mapping processing on the image embedding vectors to obtain the embedded vectors corresponding to the multi-angle images.

[0011] According to the present invention, a multimodal question answering method based on multi-angle images is provided. The step of performing prediction processing based on the multi-angle visual feature representation and the question text to obtain the answer corresponding to the question text includes: performing feature extraction processing on the question text to obtain text features; concatenating the multi-angle visual feature representation and the text features to obtain concatenated features; and inputting the concatenated features into a pre-trained language model for prediction processing to obtain the answer corresponding to the question text.

[0012] This invention also provides a multimodal question-answering device based on multi-angle images, comprising the following modules: an acquisition module, used to acquire an original image and question text input by a user, and generate several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; a preprocessing module, used to preprocess the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; a feature perception module, used to acquire trainable soft prompts corresponding to each embedded vector respectively, and perform multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representations; and a prediction module, used to perform prediction processing based on the multi-angle visual feature representations and the question text to obtain the answer corresponding to the question text.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal question answering method based on multi-angle images as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal question-answering method based on multi-angle images as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal question answering method based on multi-angle images as described above.

[0016] This invention provides a multimodal question-answering method based on multi-angle images. It acquires the original image and question text input by the user, and generates several multi-angle images from different angles representing the same entity in the original image, thus fully utilizing the visual information contained in different images. Then, it performs multi-view feature perception processing based on the embedded vectors corresponding to each multi-angle image and trainable soft cues to obtain multi-angle visual feature representations. Finally, it performs prediction processing based on the multi-angle visual feature representations and the question text to obtain the answer corresponding to the question text. By fully utilizing the visual information contained in different images, the model obtains richer and more comprehensive visual information when generating the final text output. Image information from different angles further enhances the model's ability to perceive image details, thereby alleviating the illusion problem widely existing in the answer generation process and enabling the model to produce more reasonable and reliable answer outputs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the multimodal question-answering method based on multi-angle images provided by the present invention.

[0019] Figure 2 This is the second flowchart of the multimodal question-answering method based on multi-angle images provided by the present invention.

[0020] Figure 3 This is the third flowchart of the multimodal question-answering method based on multi-angle images provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the multi-view feature perceptron provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the multi-head sampler provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the multimodal question-answering device based on multi-angle images provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Traditional multimodal models mostly rely on visual input from a single angle, such as processing image information from only a single camera or a single viewpoint. However, this design has significant limitations. First, image information from a single angle may not fully represent the features of an object, resulting in a lack of comprehensiveness in the model's generated results. Second, when processing multimodal information, the model may face conflicts or inconsistencies between different modalities, leading to the so-called "illusion problem," where the model generates false or inaccurate outputs. The existence of the illusion problem not only affects the performance of large multimodal models but also limits their application in real-world scenarios. In certain high-risk scenarios, such as autonomous driving and medical image analysis, the illusion problem can lead to serious consequences. For example, in autonomous driving, if the model incorrectly judges the number or location of pedestrians on the road, it may cause an accident; in medical diagnosis, misinterpretation of image data by the model may lead to incorrect diagnostic results.

[0027] Optionally, the multimodal question-answering method based on multi-angle images in this application embodiment can be executed by a server, by a terminal device, or by both a server and a terminal device. Taking the execution of the multimodal question-answering method based on multi-angle images in this embodiment by a server as an example.

[0028] Figure 1 This is one of the flowcharts illustrating the multimodal question-answering method based on multi-angle images provided by this invention, such as... Figure 1 As shown, the method includes the following steps 101 to 104.

[0029] Step 101: Obtain the original image and question text input by the user, and generate several multi-angle images based on the original image.

[0030] Among them, multi-angle images are used to represent the same entity from different angles in the original image.

[0031] In some embodiments, when the user inputs only one original image, the information in a single image is insufficient or lacks detail, causing the model to overlook potentially crucial but not obvious or sufficient information when acquiring visual information. This results in the model being unable to extract effective information from the visual representation when making a response. Furthermore, for real-world 3D objects, images are merely two-dimensional projections of the object, insufficient to reflect its 3D information. The model cannot capture the true nature of parts of the object not present in the original image, and this information loss also leads to model illusions.

[0032] In some embodiments, see Figure 3 , Figure 3 This is the third flowchart of the multimodal question-answering method based on multi-angle images provided by this invention, as shown below. Figure 3 As shown, this embodiment of the invention uses a multi-angle image generation module to expand a single original image input by the user into multiple images from different angles representing the same entity in the original image. The multi-angle image generation module can use the HFGI3D model, a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize realistic new views while preserving specific details of the input original image. For example, when processing an image containing specific facial features and a background, the HFGI3D model can generate 3D views from different perspectives based on this image, and the facial features and background details are well preserved. The HFGI3D model utilizes a pre-trained generator, visibility analysis, and latent optimization to achieve high-fidelity 3D rendering and editing, thereby preserving the details of the input image and synthesizing new views, generating multiple images of the same entity viewed from different perspectives.

[0033] In this embodiment of the invention, a multi-angle image generation module processes a single original input image to generate multiple images observed from different perspectives. This not only improves the model's ability to understand objects but also provides richer input for subsequent visual information extraction. In practical applications, the multi-angle image generation module can be used in various scenarios requiring multi-view information. For example, in the field of autonomous driving, vehicles need to acquire information about their surrounding environment from different angles using multiple cameras to make more accurate decisions; in tasks such as robot navigation and object recognition, the multi-angle image generation module can help the model better understand the three-dimensional features of objects, thereby enhancing the richness of visual information.

[0034] In other embodiments, when the user inputs multiple images of the same entity from different angles, there is no need to use a multi-angle image generation module to generate multi-angle images based on the original images. Instead, the multiple original images are directly processed in subsequent preprocessing steps, thereby reducing additional computational overhead.

[0035] Step 102: Preprocess the multi-angle images to obtain the embedded vectors corresponding to the multi-angle images.

[0036] In some embodiments, step 102 can be implemented in the following way: performing image encoding processing on the multi-angle images respectively to obtain image embedding vectors corresponding to the multi-angle images respectively; performing mapping processing on the image embedding vectors to obtain embedded vectors corresponding to the multi-angle images respectively.

[0037] Specifically, see [link to relevant documentation] Figure 3 After obtaining multiple multi-angle images from different angles, the multi-angle images are input into the image encoder, which encodes the multi-angle images into high-dimensional embedded representations. The image encoder accepts a single image as input and outputs the embedded representation vector corresponding to that image, i.e., the image embedding vector. Each multi-angle image from a different angle is encoded by the image encoder into a high-dimensional embedded vector of the same dimension for further calculations.

[0038] Since the semantic spaces of the language modality and the image modality are different, after obtaining the embedded representation vectors corresponding to different multi-angle images, an image mapping module is needed to map the obtained image embedding vectors to the semantic space of the language modality. This allows the subsequent language model to directly interact with the visual information contained in the images and generate text output. The image mapping module accepts the image embedding vector encoded by the image encoder as input and outputs text embedded representation vectors of the same dimension, i.e., the embedded vectors corresponding to the multi-angle images respectively.

[0039] In some embodiments, the image encoder can use the Vision Transformer model, and the mapping module can use a fully connected network. The Vision Transformer is an image encoder based on a self-attention mechanism that extracts global features from an image by dividing it into multiple patches and processing these patches as input. Each multi-angle image encoded by ViT is converted into a high-dimensional embedded vector representation. These embedded vectors preserve key features in the image, facilitating subsequent computation and processing.

[0040] Meanwhile, this invention uses a fully connected network as a mapping module to project image embedding vectors onto the semantic space of the language model. Through this mapping process, features from different modalities can be aligned in the same space, facilitating the fusion and processing of multimodal information. During model training, the image encoder does not participate in parameter updates; that is, the weights of the encoder part remain fixed. This avoids over-adjustment of image feature extraction during training and fully utilizes the visual information extraction capabilities of the pre-trained Vision Transformer model, improving model stability and reducing computational overhead during training.

[0041] Step 103: Obtain the trainable soft cue corresponding to each of the embedded vectors, and perform multi-view feature perception processing based on the embedded vectors and the trainable soft cue to obtain multi-angle visual feature representation.

[0042] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the structure of the multi-view feature perceptron provided by the present invention. The multi-view feature perceptron includes a visual extraction module and a multi-head sampler. The visual extraction module extracts visual information based on the embedded vectors corresponding to the preprocessed multi-angle images. The multi-head sampler calculates the importance of visual information in different multi-angle images and calculates the relative contribution of visual information in multi-angle images based on importance. The multi-view feature perceptron accepts the embedded vectors corresponding to the preprocessed multi-angle images and the trainable soft cue corresponding to each embedded vector as input, and outputs a single high-dimensional embedded vector as the final representation containing multi-angle visual information, i.e., multi-angle visual feature representation.

[0043] Specifically, the multi-view feature perceptron interacts with trainable soft prompts by using prompt learning and cross-attention mechanisms to extract key visual information from multi-view images.

[0044] Cue learning is a technique gaining increasing importance in deep learning, allowing models to receive cues before input data, thereby guiding the model's attention to specific features. In this invention, trainable soft cues serve as a bridge for multi-angle visual feature extraction, helping the model extract the most representative visual information from the embedded vectors corresponding to multi-angle images. Each trainable soft cue represents the main features extracted from a multi-angle image input from a specific angle.

[0045] Cross-attention is another crucial component of the multi-view feature perceptron. This mechanism allows information to interact between different modalities, enhancing the model's ability to process multi-modal information. By using embedded vectors corresponding to multi-view images as keys and values, and soft cues as query vectors, the cross-attention mechanism extracts key information from multi-view images and injects it into the soft cues, ultimately generating soft cues containing visual information as output. The visual extraction module employs a stacked structure of multiple transformer decoder blocks, each consisting of a cross-attention layer, a Layernorm layer, and a fully connected feed-forward network. This design fully utilizes the input information from multi-view images, aggregating and extracting visual information from different angles, ensuring the final output's multi-view visual feature representation (i.e., Figure 4 The final visual representation in the image can encompass all important image feature information.

[0046] In some embodiments, see Figure 2 , Figure 2 This is the second flowchart illustrating the multimodal question-answering method based on multi-angle images provided by this invention. Figure 1 Step 103 shown can be specifically achieved through... Figure 2 Steps 201 to 203 shown in the figure are implemented.

[0047] Step 201: Based on the cross-attention mechanism, inject the visual information contained in each of the embedded vectors into the corresponding trainable soft cue to obtain a soft cue containing visual information.

[0048] In some embodiments, step 201 can be implemented in the following way: for each embedded vector, the corresponding trainable soft cue is used as a query variable, the embedded vector is used as a key and a value, and cross-attention processing is performed based on the query variable, key and value to obtain the soft cue containing visual information.

[0049] Specifically, see [link to relevant documentation] Figure 4 , Figure 4 The visual extraction module takes preprocessed embedded vectors and trainable soft cues as input and generates soft cues containing visual information. Through trainable soft cues and a cross-attention mechanism, the visual extraction module extracts key visual information from the embedded vectors corresponding to multi-view images for downstream tasks. The visual extraction module acts as the primary information extractor for visual information and aligns the visual information in the embedded vectors with the semantic space of the language model. The output of the visual extraction module is a set of soft cues containing visual information; each output soft cue contains the visual information from the embedded vectors corresponding to the trainable soft cue in the multi-view images.

[0050] In this invention, the visual extraction module consists of multiple stacked Transformer decoder blocks, each of which comprises a cross-attention layer, a Layernorm layer, and a fully connected feedforward network layer. The visual extraction module uses the input trainable soft cue as the Query in the attention mechanism, and the embedded vectors corresponding to the multi-angle images as the Key and Value. It then uses the cross-attention mechanism to inject the visual information contained in the embedded vectors into the trainable soft cue, forming a soft cue containing visual information.

[0051] Formally, the embedded vector of multiple inputs is represented as ,in It is the first Embedded vectors of multiple input multi-angle images, Let be the total number of embedded vectors; let the trainable soft cue be denoted as . Each trainable soft cue contains One token, ,in This refers to the model dimension of the language model. The trainable soft cues here aim to provide appropriate visual cues for the downstream tasks the language model needs to complete. The trainable soft cues and embedded vectors are input into a cross-attention layer, allowing the soft cues to interact with visual information. In the cross-attention mechanism, the trainable soft cues are mapped to the Query, and the embedded vectors corresponding to the multi-angle images from different angles are mapped to the Key and Value, respectively. Since there are multiple embedded vectors corresponding to multiple multi-angle images, the visual extraction module performs cross-attention on the embedded vector of each multi-angle image and its corresponding trainable soft cues.

[0052] The output of the visual extraction module is represented as The mapping matrix is ​​represented as , and ,but It can be calculated in the following ways:

[0053]

[0054]

[0055]

[0056] in, This represents the output of the visual extraction module. , and These represent the mapping matrices corresponding to the query, key, and value, respectively. For trainable soft cueing, For the first Embedded vectors corresponding to each input multi-angle image. Represents an embedded vector with multiple inputs. It is the model dimension of the language model. Represents a real matrix.

[0057] Because different multi-angle input images contribute differently to the final output, the output... The weights calculated based on the multi-head sampler (i.e. Figure 4 The relative contributions in each input image are weighted and summed. Furthermore, the visual extraction module computes each cross-attention output in parallel, rather than using the previous output as the next query. This is because, in most cases, there is no sequential relationship between the input multi-angle images; therefore, their relationships with trainable soft cues should be computed separately to extract visual information.

[0058] Step 202: Calculate the attention weights of the embedded vectors based on the multi-head attention mechanism to obtain the weight values ​​corresponding to each embedded vector.

[0059] In some embodiments, prior to step 202, the present invention further includes the following step: for each of the embedded vectors, mapping the embedded vectors to multiple sets of vector representations of the same dimension.

[0060] Step 202 can be implemented in the following way: for each group of vector representations, the corresponding trainable soft cue is used as a query variable, the vector representation is used as a key, and multi-head attention processing is performed based on the query variable and the key to obtain the weight score corresponding to each attention head; the average value of the weight score corresponding to each attention head is calculated, and the average value is used as the weight value corresponding to each embedded vector.

[0061] Step 203: Perform a weighted summation on the soft cue containing visual information based on the weight values ​​to obtain the multi-angle visual feature representation.

[0062] Specifically, images from different angles may contain different visual information, and the contribution of this visual information will vary. For example, an image showing the front of an object may contain more key information than an image showing the back of the object. Therefore, considering that input images from different angles contain different visual information and thus contribute differently to the final visual representation, a multi-head sampler is designed to calculate the relative contribution of the embedded vectors corresponding to the input images from different angles to the final visual representation, measuring the importance of the visual information from different multi-angle images in the form of weight values.

[0063] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the multi-head sampler provided by the present invention. The multi-head sampler includes a decoupling module for further decomposing information from embedded vectors, and multiple attention heads for calculating the attention weights of different embedded vectors. The average value of the attention weights of each attention head is used as the output of the multi-head sampler.

[0064] The multi-head sampler is responsible for calculating the relative contribution of the embedded vectors corresponding to images from different angles to the final visual representation. The output of the multi-head sampler is used as the weight of the weighted sum of visual information extracted by the visual extraction module. That is, it is used to calculate the output of the visual extraction module. The embedded representation vectors are weighted sums. In the multi-head sampler, to further decompose the visual information of the input image, this invention uses a decoupling module to further decouple the input embedded vectors. This decoupling module consists of a two-layer fully connected multilayer perceptron (MLP), which accepts the [CLS] token in the embedded vector and maps it to multiple sets of vector representations of the same dimension. Then, it uses a multi-head attention mechanism to calculate the attention weights for each set, and uses the mean of the attention weights of each set as the relative contribution of the multi-head sampler output.

[0065] The decoupling module in the multi-head sampler only requires the [CLS] token from the image embedded representation vector as input and maps it to multiple sets of different embedded vectors for further computation. The two-layer fully connected network (MLP) in the decoupling module further enhances the model's ability to process fine-grained features. By mapping the [CLS] token in the embedded vector to multiple vector representations, the decoupling module ensures that features corresponding to images from different viewpoints are fully utilized at a fine-grained level. Each attention head calculates the contribution of different vector representations in its corresponding set, and the resulting weighted sum represents the visual information from the multi-view input.

[0066] In a multi-head attention module, each set of vector representations is assigned a trainable attention head to compute the relative contributions of different vector representations within that set. The aim of this design is to introduce multiple experts to focus on different features in the image input in the form of attention heads, because a single attention head cannot adequately capture the correlation between coarse-grained embedded vectors and trainable soft cues. Figure 5 As shown, in the attention mechanism, trainable soft cues serve as the query, and each decomposed vector representation serves as the key for calculating attention weights. In the multi-head attention module, only the attention weights are calculated, and this module uses the average value across the query dimension as the output of each attention head. Let the j-th decomposed token be embedded in the i-th input image, then the output weight of each attention head is... It can be calculated using the following formula:

[0067]

[0068]

[0069]

[0070]

[0071] in, For model dimensions, For trainable soft cueing, Indicates the first An embedded vector of multiple inputs to a decomposed token. To and The attention score calculation function is the process of calculating the attention weight in each attention head. It is the number of tokens in the trainable soft hints. Represented as the first The first input image embedded in the first Decompose the token. Indicates the first The output weights of each attention head; Indicates the first The score matrix of each attention head. Represents a real matrix. This represents the dimension of the weight vector.

[0072] After calculating the attention weight score for each set of embedded representation vectors at each attention head, the average of the weights of each attention head is used as the output of the multi-head sampler. :

[0073]

[0074] In the above formula, The average weight of each attention head is used as the output of the multi-head sampler. Indicates the first The output weights of each attention head Represents a real matrix. This represents the dimension of the weight vector. The number of groups of embedded vectors corresponding to the input multi-angle image decoupled into multiple vector representations by the decoupling module is also the number of attention heads in the multi-head attention module (an attention head is assigned to each group of embedded representation vectors in the multi-head attention module, so the number of attention heads is the same as the number of mapping groups in the decoupling module).

[0075] Multi-head samplers aim to capture fine-grained visual features from the embedded vectors of an input image by applying different attention heads after decomposing the input's embedded vectors. Different attention heads can focus on different aspects of the input and perceive their inherent relationships, further extracting visual information between different images and calculating their relative contributions.

[0076] Furthermore, in the multi-view feature perceptron, the visual extraction module extracts visual information from different embedded vectors, and the multi-head sampler obtains the relative contribution of the visual information in different embedded vectors to the final visual representation. The final visual identifier is obtained by performing a weighted summation of the input of the visual extraction module with the relative contribution calculated by the multi-head sampler as weights. The calculation process can be represented by the following formula:

[0077]

[0078]

[0079] in, for The Middle The weight values ​​corresponding to each input. This represents the output of the visual extraction module. The average weight of each attention head is used as the output of the multi-head sampler.

[0080] In the design of the multi-view feature perceptron of this invention, the number of image inputs is unlimited, which allows the proposed model architecture to accept any number of image inputs. The weighted summation method used in the output of the multi-view feature perceptron also ensures that the input order does not affect the final output, making the model more stable and reliable in practical use.

[0081] Step 104: Based on the multi-angle visual feature representation and the question text, perform prediction processing to obtain the answer corresponding to the question text.

[0082] In some embodiments, step 104 can be implemented in the following ways: performing feature extraction processing on the question text to obtain text features; concatenating the multi-angle visual feature representation with the text features to obtain concatenated features; and inputting the concatenated features into a pre-trained language model for prediction processing to obtain the answer corresponding to the question text.

[0083] Specifically, in the model architecture of this invention, the language model takes various forms. It can be a language model with certain language capabilities after pre-training, or a language model with multimodal understanding and generation capabilities after multimodal alignment. After extracting visual information and generating the final visual cues (i.e., multi-angle visual features) through a multi-view feature perceptron, feature extraction processing is performed on the user-input question text to obtain text features. The final visual cues are then concatenated before the text features and used as input to the language model for computation. The language model completes the text output based on the user's text input and the visual cues containing visual information, obtaining the answer corresponding to the original image and question text input by the user.

[0084] By concatenating visual cues with features of the user-input text, the language model can fully consider visual information when generating text, thereby improving the accuracy and logical consistency of the output. This design enables the multimodal model architecture of this invention to be widely applied in various scenarios, including tasks such as natural language generation, machine translation, and question answering systems.

[0085] The following describes the multimodal question-answering device based on multi-angle images provided by the present invention. The multimodal question-answering device based on multi-angle images described below and the multimodal question-answering method based on multi-angle images described above can be referred to in correspondence.

[0086] See Figure 6 , Figure 6 This is a schematic diagram of the structure of the multimodal question-answering device based on multi-angle images provided by the present invention, as shown below. Figure 6 As shown, the multimodal question-answering device based on multi-angle images provided by the present invention includes an acquisition module 601, a preprocessing module 602, a feature perception module 603, and a prediction module 604. The acquisition module 601 acquires the original image and question text input by the user, and generates several multi-angle images based on the original image. These multi-angle images represent different angles of the same entity in the original image. The preprocessing module 602 preprocesses the multi-angle images to obtain embedded vectors corresponding to each image. The feature perception module 603 acquires trainable soft prompts corresponding to each embedded vector, and performs multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representations. The prediction module 604 performs prediction processing based on the multi-angle visual feature representations and the question text to obtain the answer corresponding to the question text.

[0087] Specifically, the multimodal question-answering device based on multi-angle images provided by the present invention can implement all the method steps implemented in the above-mentioned multimodal question-answering method embodiment based on multi-angle images, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0088] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a multimodal question-answering method based on multi-angle images. This method includes: acquiring an original image and question text input by a user, and generating several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; preprocessing the multi-angle images to obtain embedded vectors corresponding to each multi-angle image; acquiring trainable soft prompts corresponding to each embedded vector, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representations; and performing prediction processing based on the multi-angle visual feature representations and the question text to obtain the answer corresponding to the question text.

[0089] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal question-answering method based on multi-angle images provided by the above methods. The method includes: acquiring an original image and question text input by a user, and generating several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to the multi-angle images respectively; acquiring trainable soft prompts corresponding to each embedded vector respectively, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representations; and performing prediction processing based on the multi-angle visual feature representations and the question text to obtain an answer corresponding to the question text.

[0091] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multimodal question-answering method based on multi-angle images provided by the methods described above. This method includes: acquiring an original image and question text input by a user, and generating several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; preprocessing the multi-angle images respectively to obtain embedded vectors corresponding to each multi-angle image; acquiring trainable soft prompts corresponding to each embedded vector, and performing multi-view feature perception processing based on the embedded vectors and the trainable soft prompts to obtain multi-angle visual feature representations; and performing prediction processing based on the multi-angle visual feature representations and the question text to obtain an answer corresponding to the question text.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal question-answering method based on multi-angle images, characterized in that, include: Obtain the original image and question text input by the user, and generate several multi-angle images based on the original image, wherein the multi-angle images are used to represent the same entity in the original image from different angles; The multi-angle images are preprocessed to obtain the embedded vectors corresponding to the multi-angle images respectively; A trainable soft cue corresponding to each of the embedded vectors is obtained, and multi-view feature perception processing is performed based on the embedded vectors and the trainable soft cue to obtain a multi-angle visual feature representation. This includes: injecting the visual information contained in each of the embedded vectors into the corresponding trainable soft cue based on a cross-attention mechanism to obtain a soft cue containing visual information; calculating the attention weights of the embedded vectors based on a multi-head attention mechanism to obtain a weight value corresponding to each embedded vector; and performing a weighted summation process on the soft cue containing visual information based on the weight values ​​to obtain the multi-angle visual feature representation. Based on the multi-angle visual feature representation and the question text, a prediction process is performed to obtain the answer corresponding to the question text.

2. The multimodal question-answering method based on multi-angle images according to claim 1, characterized in that, The method of injecting the visual information contained in each embedded vector into the corresponding trainable soft cue based on the cross-attention mechanism to obtain a soft cue containing visual information includes: For each embedded vector, the corresponding trainable soft cue is used as a query variable, the embedded vector is used as a key and value, and cross-attention processing is performed based on the query variable, key and value to obtain the soft cue containing visual information.

3. The multimodal question-answering method based on multi-angle images according to claim 1, characterized in that, The method further includes: For each of the embedded vectors, the embedded vectors are mapped to multiple sets of vector representations of the same dimension; The step of calculating the attention weights of the embedded vectors based on the multi-head attention mechanism to obtain the weight value corresponding to each embedded vector includes: For each set of vector representations, the corresponding trainable soft cue is used as the query variable, the vector representation is used as the key, and multi-head attention processing is performed based on the query variable and the key to obtain the weight score corresponding to each attention head. Calculate the average of the weight scores corresponding to each attention head, and use the average as the weight value corresponding to each embedded vector.

4. The multimodal question-answering method based on multi-angle images according to claim 1, characterized in that, The step of preprocessing the multi-angle images to obtain embedded vectors corresponding to each multi-angle image includes: The multi-angle images are respectively processed by image encoding to obtain the image embedding vectors corresponding to the multi-angle images; The image embedding vector is mapped to obtain the embedding vectors corresponding to the multi-angle images.

5. The multimodal question-answering method based on multi-angle images according to claim 1, characterized in that, The prediction process based on the multi-angle visual feature representation and the question text to obtain the answer corresponding to the question text includes: The problem text is subjected to feature extraction processing to obtain text features; The multi-angle visual feature representation is concatenated with the text feature to obtain the concatenated feature; The concatenated features are input into a pre-trained language model for prediction processing to obtain the answer corresponding to the question text.

6. A multimodal question-answering device based on multi-angle images, characterized in that, include: The acquisition module is used to acquire the original image and question text input by the user, and generate several multi-angle images based on the original image, wherein the multi-angle images are used to represent different angles of the same entity in the original image; The preprocessing module is used to preprocess the multi-angle images respectively to obtain the embedded vectors corresponding to the multi-angle images respectively; A feature perception module is used to acquire trainable soft cues corresponding to each of the embedded vectors, and to perform multi-view feature perception processing based on the embedded vectors and the trainable soft cues to obtain a multi-angle visual feature representation. This includes: injecting the visual information contained in each of the embedded vectors into the corresponding trainable soft cues based on a cross-attention mechanism to obtain soft cues containing visual information; calculating the attention weights of the embedded vectors based on a multi-head attention mechanism to obtain weight values ​​corresponding to each embedded vector; and performing a weighted summation process on the soft cues containing visual information based on the weight values ​​to obtain the multi-angle visual feature representation. The prediction module is used to perform prediction processing based on the multi-angle visual feature representation and the question text to obtain the answer corresponding to the question text.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal question-answering method based on multi-angle images as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal question-answering method based on multi-angle images as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal question-answering method based on multi-angle images as described in any one of claims 1 to 5.

Images