Question and answer interaction method, electronic equipment and readable storage medium

Abstract

The invention discloses a question and answer interaction method, electronic equipment and a readable storage medium, and relates to the technical field of artificial intelligence. According to the specific technical scheme, the electronic equipment can input a question image and a question text into a trained question and answer interaction model, and through the question and answer interaction model, candidate image areas related to the question text in the question image are determined according to the question text; and through a question and answer interaction model, according to the candidate image region and the question text, outputting a question answer.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a question-and-answer interaction method, an electronic device, and a readable storage medium. Background Technology

[0002] Currently, in scenarios where users use Multimodal Large Language Model (MLLM) for text-image dialogue, users can trigger electronic devices to input questions and images into the MLLM. The MLLM then encodes the image and text to obtain the input sequence, and performs reasoning based on this input sequence to output the answer.

[0003] However, since the MLLM may take a long time to fully encode the image and text after the electronic device inputs the question and image, it takes a long time for the MLLM to output the answer, resulting in low efficiency of text-to-image dialogue. Summary of the Invention

[0004] The purpose of this application is to provide a question-and-answer interaction method, an electronic device, and a readable storage medium that can improve the efficiency of text-based dialogue while ensuring its performance.

[0005] In a first aspect, embodiments of this application provide a question-and-answer interaction method, which includes: an electronic device inputting a question image and question text into a trained question-and-answer interaction model; using the question-and-answer interaction model to determine candidate image regions in the question image that are related to the question text based on the question text; and using the question-and-answer interaction model to output the question answer based on the candidate image regions and the question text.

[0006] In some embodiments of this application, the question-answering interaction model includes a cross-modal fusion processor and a region proposal network. The process of determining candidate image regions related to the question text within a question image using the question-answering interaction model includes: the electronic device using the cross-modal fusion processor to determine an initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text; and using the region proposal network to determine candidate image regions based on the initial intent vector and the question image features.

[0007] In some embodiments of this application, the question-and-answer interaction model further includes a global image encoder; before the cross-modal fusion processor of the question-and-answer interaction model determines the initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text, the method further includes: the electronic device performs downsampling processing on the question image through the question-and-answer interaction model to obtain a thumbnail; and obtains the image features corresponding to the thumbnail through the global image encoder, and determines the image features corresponding to the thumbnail as the question image features.

[0008] In some embodiments of this application, determining the initial intent vector based on the problem image features corresponding to the problem image and the problem text features corresponding to the problem text includes: the electronic device calculating the initial intent vector based on the problem image features and the problem text features using an attention mechanism.

[0009] In some embodiments of this application, the above-mentioned determination of candidate image regions based on the initial intent vector and problem image features includes: the electronic device performing regression prediction based on the initial intent vector and problem image features to obtain at least one predicted location information and at least one confidence level, each confidence level being used to characterize the probability that a predicted location information is the location information of an image region in the problem image related to the problem text; performing non-maximum suppression deduplication processing on the at least one predicted location information; determining the candidate location information with the highest confidence level and an overlap rate with the historical exploration area below the overlap rate threshold from the deduplicated predicted location information; and determining the image region in the problem image corresponding to the candidate location information as the candidate image region.

[0010] In some embodiments of this application, the question-and-answer interaction model includes a local image encoder and an inference model; the above-mentioned outputting the question answer based on the candidate image region and the question text through the question-and-answer interaction model includes: the electronic device obtains the image region features corresponding to the candidate image region through the local image encoder; and infers the question answer based on the image region features and the question text features corresponding to the question text through the inference model.

[0011] In some embodiments of this application, the question-and-answer interaction model further includes a decision-maker; before the above-mentioned reasoning based on image region features and question text features corresponding to the question text, and outputting the question answer through the reasoning model, the method further includes: the electronic device performs decision evaluation based on image region features and question text features through the decision-maker, and outputs evaluation result information, which indicates whether the candidate image region is all image regions in the question image related to the question text; the above-mentioned reasoning based on image region features and question text features corresponding to the question text, and outputting the question answer through the reasoning model, includes: when the evaluation result information indicates that the candidate image region is all image regions in the question image related to the question text, the electronic device performs reasoning based on image region features and question text features through the reasoning model, and outputs the question answer.

[0012] In some embodiments of this application, the question-and-answer interaction model further includes a region proposal network; the method further includes: when the evaluation result information indicates that the candidate image region is not all image regions in the question image that are related to the question text, the electronic device generates an updated intent vector based on the image region features and the initial intent vector through the question-and-answer interaction model, the initial intent vector being used to characterize the question intent of the question text; and determines the remaining image regions in the question image that are related to the question text based on the updated intent vector and the question image features through the region proposal network.

[0013] In some embodiments of this application, before inputting the question image and question text into the trained question-answering interaction model, and determining the candidate image region related to the question text in the question image based on the question text through the question-answering interaction model, the method further includes: an electronic device acquiring a training sample set, which includes sample question images, sample question text, sample candidate image regions, and sample question answers, wherein the sample candidate image regions are image regions in the sample question images related to the sample question text; inputting the training sample set into a preset model, and determining the predicted candidate image regions related to the sample question text in the sample question image based on the sample question text through the preset model; and outputting the predicted question answer based on the predicted candidate image regions and the sample question text through the preset model; determining a first loss value based on the predicted candidate image regions and the sample candidate image regions, and determining a second loss value based on the predicted question answer and the sample question answer; and performing a first stage of training on the preset model based on the first loss value, and performing a second stage of training on the preset model after the first stage of training based on the second loss value, to obtain the trained question-answering interaction model.

[0014] In some embodiments of this application, the aforementioned preset model includes a preset fusion unit and a preset proposal network; the aforementioned method of determining the predicted candidate image region related to the sample question text in the sample question image based on the sample question text using the preset model includes: the electronic device determining an initial sample intent vector based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text using the preset fusion unit, the initial sample intent vector being used to characterize the question intent of the sample question text; and determining the predicted candidate image region based on the initial sample intent vector and the sample question image features using the preset proposal network; the aforementioned method of training the preset model in the first stage based on a first loss value includes: the electronic device training the preset fusion unit and the preset proposal network in the first stage based on the first loss value.

[0015] In some embodiments of this application, the aforementioned preset model further includes a preset global image encoder; before determining the initial intent vector of the sample using the preset fusion processor based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, the method further includes: the electronic device downsampling the sample question image using the preset model to obtain a sample thumbnail; and obtaining the image features corresponding to the sample thumbnail using the preset global image encoder, and determining the image features corresponding to the sample thumbnail as the sample question image features; the aforementioned training of the preset fusion processor and the preset proposal network in the first stage based on the first loss value includes: the electronic device training the preset fusion processor, the preset proposal network, and the preset global image encoder in the first stage based on the first loss value.

[0016] In some embodiments of this application, the aforementioned preset model includes a preset local image encoder and a preset inference model; the aforementioned outputting the predicted question answer based on the predicted candidate image region and sample question text through the preset model includes: the electronic device obtains the sample image region features corresponding to the predicted candidate image region through the preset local image encoder; and performs inference based on the sample image region features and sample question text features through the preset inference model to output the predicted question answer; the aforementioned performing the second stage training of the preset model after the first stage training based on the second loss value includes: the electronic device performing the second stage training of the preset local image encoder and the preset inference model based on the second loss value.

[0017] In some embodiments of this application, the learning rate of the first stage of training is greater than the learning rate of the second stage of training.

[0018] Secondly, embodiments of this application provide a question-and-answer interaction device, which includes: a processing module, configured to input a question image and a question text into a trained question-and-answer interaction model, determine candidate image regions in the question image related to the question text based on the question text through the question-and-answer interaction model, and output the question answer based on the candidate image regions and the question text through the question-and-answer interaction model.

[0019] In some embodiments of this application, the question-answering interaction model includes a cross-modal fusion processor and a region proposal network. Specifically, the processing module is used to determine an initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text through the cross-modal fusion processor. The initial intent vector is used to characterize the question intent of the question text. The region proposal network is used to determine candidate image regions based on the initial intent vector and the question image features.

[0020] In some embodiments of this application, the question-and-answer interaction model further includes a global image encoder; the processing module is further configured to perform downsampling processing on the question image through the question-and-answer interaction model to obtain a thumbnail before determining the initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text through the cross-modal fusion of the question-and-answer interaction model; and to obtain the image features corresponding to the thumbnail through the global image encoder, and determine the image features corresponding to the thumbnail as the question image features.

[0021] In some embodiments of this application, the above-described processing module is specifically used to calculate an initial intent vector based on the features of the problem image and the problem text, using an attention mechanism.

[0022] In some embodiments of this application, the above-described processing module is specifically configured to perform regression prediction based on the initial intent vector and the features of the problem image to obtain at least one predicted location information and at least one confidence level, wherein each confidence level is used to characterize the probability that a predicted location information is the location information of an image region in the problem image related to the problem text; and to perform non-maximum suppression deduplication processing on the at least one predicted location information; and to determine, from the deduplicated predicted location information, the candidate location information with the highest confidence level and an overlap rate with the historical exploration area lower than the overlap rate threshold; and to determine the image region in the problem image corresponding to the candidate location information as the candidate image region.

[0023] In some embodiments of this application, the question-and-answer interaction model includes a local image encoder and an inference model; the processing module is specifically used to obtain image region features corresponding to candidate image regions through the local image encoder; and to infer the answer to the question based on the image region features and the question text features corresponding to the question text through the inference model.

[0024] In some embodiments of this application, the question-and-answer interaction model further includes a decision-maker; the processing module is further configured to, before inferring and outputting the question answer through the inference model based on the image region features and the question text features corresponding to the question text, perform decision evaluation through the decision-maker based on the image region features and the question text features, and output evaluation result information, which indicates whether the candidate image region is all image regions in the question image related to the question text; specifically, the processing module is configured to, when the evaluation result information indicates that the candidate image region is all image regions in the question image related to the question text, perform inference and output the question answer through the inference model based on the image region features and the question text features.

[0025] In some embodiments of this application, the question-and-answer interaction model further includes a region proposal network; the processing module is further configured to, when the evaluation result information indicates that the candidate image region is not all image regions in the question image that are related to the question text, generate an updated intent vector based on the image region features and the initial intent vector through the question-and-answer interaction model, wherein the initial intent vector is used to characterize the question intent of the question text; and determine the remaining image regions in the question image that are related to the question text based on the updated intent vector and the question image features through the region proposal network.

[0026] In some embodiments of this application, the above-mentioned question-and-answer interaction device further includes: an acquisition module, configured to acquire a training sample set before the processing module inputs the question image and question text into the trained question-and-answer interaction model, and determines, based on the question text, a candidate image region in the question image related to the question text; the training sample set includes sample question images, sample question text, sample candidate image regions, and sample question answers, wherein the sample candidate image regions are image regions in the sample question images related to the sample question text; the processing module is further configured to input the training sample set acquired by the acquisition module into a preset model, and determine, based on the sample question text, a candidate image region in the question image related to the question text. The question-answering interaction device further includes: a training module, used to train the preset model in the first stage according to the first loss value determined by the processing module, and to train the preset model after the first stage of training in the second stage according to the second loss value determined by the processing module, to obtain the trained question-answering interaction model.

[0027] In some embodiments of this application, the aforementioned preset model includes a preset fusion unit and a preset proposal network; the aforementioned processing module is specifically used to determine an initial intent vector of a sample based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, wherein the initial intent vector is used to characterize the question intent of the sample question text; and to determine a predicted candidate image region based on the initial intent vector of the sample and the sample question image features through the preset proposal network; the aforementioned training module is specifically used to perform a first stage of training on the preset fusion unit and the preset proposal network based on a first loss value.

[0028] In some embodiments of this application, the aforementioned preset model further includes a preset global image encoder; the aforementioned processing module is further configured to, before determining the initial intent vector of the sample through the preset fusion device based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, perform downsampling processing on the sample question image through the preset model to obtain a sample thumbnail; and obtain the image features corresponding to the sample thumbnail through the preset global image encoder, and determine the image features corresponding to the sample thumbnail as the sample question image features; the aforementioned training module is specifically configured to perform a first stage of training on the preset fusion device, the preset proposal network and the preset global image encoder based on the first loss value.

[0029] In some embodiments of this application, the preset model includes a preset local image encoder and a preset inference model; the processing module is specifically used to obtain the sample image region features corresponding to the predicted candidate image region through the preset local image encoder; and to perform inference based on the sample image region features and sample question text features through the preset inference model, and output the predicted question answer; the training module is specifically used to perform a second stage of training on the preset local image encoder and the preset inference model based on the second loss value.

[0030] In some embodiments of this application, the learning rate of the first stage of training is greater than the learning rate of the second stage of training.

[0031] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions implementing the steps of the method as described in the first aspect when executed by the processor.

[0032] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0033] Fifthly, embodiments of this application provide a chip including a processor and a communication interface coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0034] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first aspect.

[0035] In this embodiment, the electronic device can input a question image and question text into a trained question-and-answer interaction model. The model then determines candidate image regions related to the question text within the question image, and outputs the answer based on these candidate image regions and the question text. Since the electronic device can use this model to select candidate image regions related to the question text—that is, select a portion of the image region related to the question text—after inputting the question image and question text into the trained model, rather than performing calculations based on the entire image region and the question text, the model avoids performing calculations on image regions unrelated to the question text. This reduces the computational load of the model and ensures that it operates based on image regions related to the question text. This improves the efficiency of outputting the answer while maintaining accuracy, thus enhancing the performance and efficiency of the text-based dialogue. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a question-and-answer interaction method provided in some embodiments of this application;

[0037] Figure 2 This is a flowchart illustrating a question-and-answer interaction method provided in some embodiments of this application;

[0038] Figure 3 This is a flowchart illustrating a question-and-answer interaction method provided in some embodiments of this application;

[0039] Figure 4 This is a flowchart illustrating a question-and-answer interaction method provided in some embodiments of this application;

[0040] Figure 5 These are schematic diagrams of the structure of a question-and-answer interaction device provided in some embodiments of this application;

[0041] Figure 6These are schematic diagrams of the hardware structure of electronic devices provided in some embodiments of this application;

[0042] Figure 7 These are schematic diagrams of the hardware structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0044] The following will explain the technical terms used in this application.

[0045] Large Language Models (LLMs) are deep learning models trained on large amounts of text data. They are able to understand the meaning of the input question text and output the corresponding answer. Currently, LMMs can be used in various natural language processing tasks, including but not limited to model dialogue, machine translation, and document writing.

[0046] MLLM (Multimodal Learning Model) is an ensemble of LLM (Multimodal Learning Model) that incorporates multimodal information. It can understand input question text, question images, question language, etc., and output corresponding answers. Currently, MLLM can be used in various multimodal understanding tasks, including but not limited to text-to-image dialogue, image recognition, and speech recognition.

[0047] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0048] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0049] The question-and-answer interaction method, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0050] It should be noted that the question-and-answer interaction method provided in this application can be executed by electronic devices such as mobile phones, tablets, laptops, PDAs, and in-vehicle electronic devices. Some embodiments of this application use electronic devices as the executing entity to illustrate the question-and-answer interaction method provided in this application.

[0051] The question-and-answer interaction method provided in this application embodiment can be applied to AI question-and-answer interaction scenarios.

[0052] One specific application scenario is when users use MLLM to ask questions about the details of the content in an image, and another specific application scenario is when users use MLLM to ask questions about the differences between different contents in an image.

[0053] Figure 1 A flowchart illustrating the question-and-answer interaction method provided in an embodiment of this application is shown. Figure 1 As shown, the question-and-answer interaction method provided in this application embodiment may include the following steps 101 and 102.

[0054] Step 101: The electronic device inputs the question image and question text into the trained question-answering interaction model. Based on the question text, the model determines the candidate image regions in the question image that are related to the question text.

[0055] In some embodiments of this application, the aforementioned problem image is an image input by the user, and the number of problem images can be at least one. The aforementioned problem text is text input by the user in response to the problem image, and the problem text includes at least one text unit, which may include, but is not limited to, Chinese characters, foreign languages, numbers, symbols, etc.

[0056] In some embodiments of this application, the question-and-answer interaction model described above may include, but is not limited to, a neural network model. For example, the question-and-answer interaction model may be an MLLM (Multi-Level Model). The question-and-answer interaction model may include at least one sub-model connected sequentially. The question-and-answer interaction model may be a model stored in an electronic device or a model on a cloud server.

[0057] In some embodiments of this application, when a user needs to ask a question in response to a question image, the user can trigger an electronic device to open an AI application and input the question image and question text into the interface of the AI ​​application. In this way, the electronic device can input the question image and question text into the trained question-and-answer interaction model.

[0058] In some examples, the aforementioned AI-type applications can be applications with AI capabilities, meaning that electronic devices can interact with question-and-answer interaction models through these AI-type applications.

[0059] In some examples, when the question-answering interaction model is a model stored in an electronic device, the electronic device can directly input the question image and question text into the question-answering interaction model; when the question-answering interaction model is a model in a cloud server, the electronic device can send a request message to the cloud server, which includes the question image and question text, so that the cloud server can input the question image and question text into the trained question-answering interaction model according to the request message.

[0060] For example, in one scenario, a user wants to ask a question about details in a specific area of ​​an image. The user can input the question image and question text into the interface of an AI application. The question image can be a city street view image with a resolution of 3840×2160, or it can be a 4K image. The city street view image includes streets, pedestrians, vehicles, and a shop sign in the distance. The shop sign is green and includes the phone number 555-0199. The question text is "Please tell me what the phone number on that green shop sign in the distance is?" The electronic device can then input the city street view image and the text "Please tell me what the phone number on that green shop sign in the distance is?" into a trained question-and-answer interaction model.

[0061] To illustrate further, in another scenario, if a user wants to ask a question about the differences between various elements in a question image, the user can input the question image and text into the interface of an AI application. The question image could be a front view of a supermarket shelf displaying dozens of different beverages, each with a price tag. For example, the price tag below the red bottle on the far left of the shelf might read "Price $2.5," while the price tag below the blue bottle on the far right might read "Price $1.8." The question text would be, "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" The electronic device can then input the front view of the supermarket shelf and the question "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" into a trained question-and-answer interaction model.

[0062] In some embodiments of this application, the aforementioned "candidate image region related to the question text" can be understood as: the question text is directed at the candidate image region or an object within the candidate image region. The number of the aforementioned candidate image regions can be at least one.

[0063] For example, in one of the scenarios described above, the candidate image region related to the text "Please tell me the phone number on that green sign in the distance?" is the image region where the shop sign is located in the city street view image. In this scenario, there is only one candidate image region.

[0064] To illustrate further, in the other scenario mentioned above, the candidate image regions related to the text "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" are the image regions containing the red bottle on the far left of the shelf in the front view of the supermarket shelf, and the image regions containing the blue bottle on the far right of the shelf in the front view of the supermarket shelf. In this scenario, there are two candidate image regions.

[0065] In some embodiments of this application, the electronic device can determine candidate image regions related to the question text in the question image based on the question text using at least one of the above-described sub-models.

[0066] In some embodiments of this application, the above-described question-and-answer interaction model includes a cross-modal fusion processor and a regional proposal network. Combined with... Figure 1 ,like Figure 2 As shown, step 101 can be implemented through steps 101a and 101b below.

[0067] Step 101a: The electronic device inputs the question image and question text into the trained question-answering interaction model. Through the cross-modal fusion device, the initial intent vector is determined based on the question image features corresponding to the question image and the question text features corresponding to the question text.

[0068] In some examples, the cross-modal fusion processor described above may include, but is not limited to, attention-based computational sub-models. The region proposal network described above may include, but is not limited to, location prediction sub-models.

[0069] In some examples, the aforementioned problem image features may be features of the problem image itself, or features of a thumbnail of the problem image.

[0070] In some examples, the aforementioned question text features can be features of the question text itself.

[0071] In some examples, the question-and-answer interaction model described above also includes a global image encoder and a text encoder. Thus, the electronic device can first extract features from the question image or a thumbnail of the question image using the global image encoder to obtain the features of the question image, and then extract features from the question text using the text encoder to obtain the features of the question text.

[0072] Optionally, the global image encoder described above may include, but is not limited to, a convolutional neural network sub-model, such as a small convolutional neural network like MobileNetV3. This global image encoder can perform convolution operations on the problem image or a thumbnail of the problem image to output problem image features containing global image information.

[0073] Optionally, the text encoder described above may include, but is not limited to, a language sub-model, such as the lightweight Transformer encoder DistilBERT. This text encoder can perform Transformer encoding on the question text and output the aforementioned question text features.

[0074] For example, in one of the scenarios described above, the electronic device can first perform convolution operations on the aforementioned city street view image or a thumbnail of the city street view image using a small convolutional neural network, MobileNetV3, to output 256×768-dimensional problem image features. The text "Please tell me the phone number on that green sign in the distance?" is encoded using the lightweight Transformer encoder DistilBERT, outputting L×768-dimensional question text features. L can be a positive integer, such as 10.

[0075] To illustrate further, in another scenario described above, the electronic device can first perform convolution operations on the front view of the supermarket shelf or a thumbnail of the front view of the supermarket shelf using a small convolutional neural network, MobileNetV3, to output 256×768-dimensional problem image features. The text "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" is encoded using the lightweight Transformer encoder DistilBERT, outputting L×768-dimensional question text features. L can be a positive integer, such as 10.

[0076] In this embodiment of the application, the initial intent vector is used to characterize the question intent of the question text.

[0077] In some examples, the initial intent vector described above mathematically represents "image global features weighted by the attention of the question text," which encodes a cross-modal fusion processor that locks in potential search directions in the image feature space based on the keywords of the question text.

[0078] In some examples, electronic devices can perform modal alignment based on an attention mechanism through a cross-modal fusion processor to output an initial intent vector.

[0079] In some examples, step 101a above can be specifically implemented by step 101a1 below.

[0080] Step 101a1: The electronic device inputs the question image and question text into the trained question-answering interaction model, and calculates the initial intent vector based on the features of the question image and the question text through a cross-modal fusion device based on the attention mechanism.

[0081] Optionally, the attention mechanism described above may include, but is not limited to, a cross-attention mechanism.

[0082] Alternatively, the electronic device can first use a cross-modal fusion processor, based on a learnable weight matrix, to... The query Q is determined based on the features of the question text, and then a learnable weight matrix is ​​used. , The key K and value V are determined by the image features and the query Q, respectively. Thus, the electronic device can calculate the initial intent vector based on the query Q, key K and value V through the cross-modal fusion processor.

[0083] Among them, the electronic device can use the first algorithm, based on the weight matrix. The query Q is calculated from the features of the question text. The first algorithm is as follows:

[0084] (1)

[0085] To query Q, Features of the problem text.

[0086] Among them, electronic devices can use the second algorithm, based on the weight matrix. The key K is calculated from the features of the problem image, and a third algorithm is used based on the weight matrix. The second algorithm calculates the value V from the features of the problem image, and is as follows:

[0087] (2)

[0088] For key K, Features of the problem image.

[0089] The third algorithm is as follows:

[0090] (3)

[0091] Let the value be V. Features of the problem image.

[0092] The electronic device can employ a fourth algorithm to calculate the initial intent vector based on the query Q, key K, and value V. This fourth algorithm is as follows:

[0093] (4)

[0094] For the initial intent vector, This is the transpose of key K. is the scaling factor for the feature dimension.

[0095] For example, in one of the scenarios described above, electronic devices can use a cross-modal fusion processor based on the aforementioned cross-attention mechanism, according to the features of the problem image. and question text features The initial intent vector is calculated. .

[0096] To illustrate further, in another scenario described above, electronic devices can use a cross-modal fusion processor based on the aforementioned cross-attention mechanism, according to the features of the problem image. and question text features The initial intent vector is calculated. .

[0097] Thus, it can be seen that since electronic devices can use a cross-modal fusion processor based on an attention mechanism to calculate an initial intent vector based on the features of the problem image and the problem text, the cross-modal fusion processor can focus on the most relevant parts of the problem image and problem text features during the determination of the initial intent vector. Therefore, the initial intent vector can accurately represent the problem intent of the problem text, thereby improving the accuracy of the determined initial intent vector.

[0098] Step 101b: The electronic device determines candidate image regions based on the initial intent vector and the features of the problem image through the region proposal network.

[0099] In some examples, electronic devices can use a region proposal network to perform regression prediction based on an initial intent vector and features of the problem image to determine the location information of candidate image regions in the problem image. Thus, electronic devices can use the region proposal network to determine candidate image regions based on the location information of the candidate image regions in the problem image.

[0100] The aforementioned location information may include, but is not limited to, at least one of the following: coordinate information of the region center, and coordinate information of the region bounding box. This coordinate information may be coordinate information in a target coordinate system, which may have one edge of the problem image as the x-axis and another edge of the problem image intersecting that edge as the y-axis.

[0101] Optionally, the shape of the aforementioned region boundary box may include, but is not limited to, any of the following: polygon, circle, ellipse, or irregular shape. Wherein, when the shape of the region boundary box is a polygon, for example, when the shape of the region boundary box is a quadrilateral, the coordinate information of the aforementioned region boundary box may be the coordinate information of at least some of the four corners, for example, the coordinate information of two corners, which may be the upper left corner and the lower right corner.

[0102] Optionally, when there are at least two candidate image regions, the electronic device may first determine a candidate image region based on the initial intent vector and the problem image features, then update the initial intent vector based on the candidate image region, and determine another candidate image region based on the updated initial intent vector and the problem image features, and so on, until all candidate image regions are determined.

[0103] In some examples, step 101b above can be specifically implemented through steps 101b1 to 101b4 below.

[0104] Step 101b1: The electronic device performs regression prediction based on the initial intent vector and problem image features through the regional proposal network to obtain at least one predicted location information and at least one confidence level.

[0105] Optionally, the predicted location information can be predicted coordinate information, such as the coordinate information of the predicted bounding box. It should be noted that the description of the predicted coordinate information can be found in the specific description in the above embodiments, and will not be repeated here.

[0106] In this embodiment of the application, each of the at least one confidence level is used to characterize the probability that a predicted location information is the location information of an image region in the problem image that is related to the problem text.

[0107] For example, in one of the scenarios described above, electronic devices can use a regional proposal network based on the initial intent vector. and problem image features Regression prediction is performed to obtain at least one predicted location information and at least one confidence level, wherein the at least one predicted location information includes predicted location information [ , , , ], The x-coordinate of the top-left corner of the predicted candidate image region. The y-coordinate value of the top-left corner of the predicted candidate image region. The x-coordinate of the lower right corner of the predicted candidate image region. The predicted location information is the y-coordinate of the lower right corner of the predicted candidate image region. , , , ], The x-coordinate of the top-left corner of the predicted candidate image region. The y-coordinate value of the top-left corner of the predicted candidate image region. The x-coordinate of the lower right corner of the predicted candidate image region. The y-axis coordinate value of the lower right corner of the predicted candidate image region, wherein the at least one confidence level includes an 80% confidence level and a 70% confidence level, and the 80% confidence level is used to characterize the predicted location information. , , , The confidence level of 70% represents the probability of a location region in a city street view image being associated with the text "Please tell me the phone number on that green sign in the distance?". , , , [The probability of the location information of the image region associated with the text "Please tell me the phone number on that green sign in the distance?" in a city street view image.]

[0108] To illustrate further, in another scenario described above, electronic devices can utilize a regional proposal network based on the initial intent vector. and problem image features Regression prediction is performed to obtain at least one predicted location information and at least one confidence level, wherein the at least one predicted location information includes predicted location information [ , , , ], The x-coordinate of the top-left corner of the predicted candidate image region. The y-coordinate value of the top-left corner of the predicted candidate image region. The x-coordinate of the lower right corner of the predicted candidate image region. The predicted location information is the y-coordinate of the lower right corner of the predicted candidate image region. , , , ], The x-coordinate of the top-left corner of the predicted candidate image region. The y-coordinate value of the top-left corner of the predicted candidate image region. The x-coordinate of the lower right corner of the predicted candidate image region. The y-coordinate value of the lower right corner of the predicted candidate image region, wherein the at least one confidence level includes a 90% confidence level and a 60% confidence level, and the 90% confidence level is used to characterize the predicted location information. , , , The confidence level of 60% represents the probability of the location information of the image region associated with the text "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" in a frontal photo of a supermarket shelf. , , , The probability of the location information of the image region associated with the text "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" in a frontal photo of a supermarket shelf.

[0109] Step 101b2: The electronic device performs non-maximum suppression deduplication processing on at least one predicted location information through the regional proposal network.

[0110] It is understandable that by using non-maximum suppression deduplication, duplicate predicted location information can be removed from at least one predicted location information, so that each predicted location information in the deduplicated predicted location information is different.

[0111] It should be noted that for the explanation of non-maximum suppression deduplication processing, please refer to the specific description in the relevant technology, and the embodiments of this application will not be repeated here.

[0112] Step 101b3: The electronic device uses the regional proposal network to determine the candidate location information with the highest confidence and an overlap rate with the historical exploration area below the overlap rate threshold from the deduplicated predicted location information.

[0113] Alternatively, "less than" can be understood as any of the following: less than, less than or equal to.

[0114] Optionally, the aforementioned historical exploration area can be a candidate image area determined in the previous instance based on the initial intent vector and the features of the problem image.

[0115] For example, when there are at least two candidate image regions, if the electronic device determines a candidate image region, the electronic device can use a region proposal network to determine another candidate location information from at least one predicted location information, which corresponds to the region with the highest confidence and whose overlap rate with the location information of the candidate image region is less than or equal to the overlap rate threshold.

[0116] For example, in one of the above scenarios, the electronic device uses a regional proposal network to determine, from at least one predicted location information, the candidate location information with the highest confidence level and an overlap rate below the overlap rate threshold with the historically explored area. This means the predicted location information […]. , , , [Identified as candidate location information] .

[0117] To illustrate further, in another scenario described above, the electronic device uses a regional proposal network to determine, from at least one predicted location information, the candidate location information with the highest confidence level and an overlap rate below the overlap rate threshold with the historically explored area. This means the predicted location information […]. , , , [Identified as candidate location information] .

[0118] Step 101b4: The electronic device uses the region proposal network to determine the image region in the problem image that corresponds to the candidate location information as the candidate image region.

[0119] Optionally, when the number of candidate image regions is at least two, the electronic device may perform steps 101b1 to 101b3 at least twice to determine at least two candidate image regions.

[0120] In the case where there are at least two candidate image regions, in step 101b4 above, the electronic device can use the region proposal network to determine the image region in the problem image that corresponds to the other candidate location information as another candidate image region, and so on.

[0121] It is understood that when there are at least two candidate image regions, the overlap rate between each candidate image region and the image regions other than each candidate image region is less than or equal to the overlap rate threshold.

[0122] Thus, it can be seen that since the electronic device can use the region proposal network to perform regression prediction based on the initial intent vector and the features of the problem image, obtain at least one predicted location information and at least one confidence level, and perform non-maximum suppression deduplication on the at least one predicted location information, and determine the candidate location information with the highest confidence level and an overlap rate with the historical exploration area below the overlap rate threshold from the deduplicated predicted location information, that is, determine the candidate location information of the image region most likely to be related to the problem text in the problem image, the electronic device can determine the image region in the problem image corresponding to the candidate location information, that is, the image region in the problem image most likely to be related to the problem text in the problem image, as the candidate image region, thereby improving the accuracy of the determined candidate image region.

[0123] As can be seen from the above, since electronic devices can determine the initial intent vector for representing the question intent of the question text by using a cross-modal fusion processor based on the question image features corresponding to the question image and the question text features corresponding to the question text, the electronic devices can accurately determine the candidate image regions that match the question intent of the question text by using a region proposal network based on the initial intent vector and the question image features. Therefore, the accuracy of the determined candidate image regions can be improved.

[0124] In some examples, the question-and-answer interaction model described above also includes a global image encoder. Optionally, prior to step 101a above, the question-and-answer interaction method provided in this application embodiment may further include steps 201 and 202 as described below.

[0125] Step 201: The electronic device downsamples the question image using a question-and-answer interaction model to obtain a thumbnail.

[0126] It should be noted that the description of the global image encoder can be found in the specific description in the above embodiments, and will not be repeated here in the embodiments of this application.

[0127] For example, in one of the scenarios described above, electronic devices can use a question-and-answer interaction model to downsample city street view images, resulting in a low-resolution global thumbnail of 256×256. .

[0128] To illustrate further, in one of the aforementioned scenarios, electronic devices can use a question-and-answer interaction model to downsample a frontal photo of a supermarket shelf, resulting in a low-resolution global thumbnail of 256×256. .

[0129] Step 202: The electronic device obtains the image features corresponding to the thumbnail through the global image encoder, and determines the image features corresponding to the thumbnail as the problem image features.

[0130] For example, in one of the scenarios described above, electronic devices can use a small convolutional neural network, MobileNetV3, to process global thumbnails. Perform convolution operations to output 256×768 dimensional problem image features. .

[0131] To illustrate further, in one of the aforementioned scenarios, electronic devices can use a small convolutional neural network, MobileNetV3, to process global thumbnails. Perform convolution operations to output 256×768 dimensional problem image features. .

[0132] Thus, it can be seen that since electronic devices can downsample the question image through a question-and-answer interaction model to obtain a thumbnail with less data, and then obtain the image features corresponding to the thumbnail through a global image encoder, that is, obtain image features with less data, and determine the image features corresponding to the thumbnail as the question image features, the amount of data for the question image features is reduced. Therefore, the amount of calculation in subsequent steps can be simplified and the calculation efficiency can be improved.

[0133] Furthermore, downsampling the problem image does not affect the global information of the problem image. In other words, the electronic device can still accurately determine the candidate image region in subsequent steps. Therefore, while ensuring the accuracy of the determined candidate image region, the computational load in subsequent steps can be simplified and the computational efficiency can be improved.

[0134] Step 102: The electronic device outputs the answer to the question based on the candidate image region and the question text through the question-and-answer interaction model.

[0135] In some embodiments of this application, an electronic device can use a question-and-answer interaction model to perform model inference based on candidate image regions and output a question answer, which is the answer to the question text.

[0136] In some embodiments of this application, the electronic device can first crop out the candidate image region from the question image through a question-and-answer interaction model, and then output the question answer based on the candidate image region and the question text.

[0137] It is understandable that electronic devices can first use a question-and-answer interaction model to losslessly crop candidate image regions (i.e. local patches) from the question image (i.e., the original high-resolution image), and then output the answer to the question based on the candidate image regions and the question text.

[0138] For example, in one of the scenarios described above, electronic devices can use a question-and-answer interaction model to determine the candidate location information. Candidate image regions were cropped from the aforementioned city street view images. The city street view image can be understood as the original high-resolution image, and the candidate image region... This can be understood as a local patch that is cropped without loss of quality, and the candidate image region is then... Divide the image into a 16×16 tile sequence, and output the answer to the question based on the 16×16 tile sequence and the question text.

[0139] To illustrate further, in another scenario described above, electronic devices can use a question-and-answer interaction model to determine the candidate location information. Candidate image areas were cropped from the front photos of the aforementioned supermarket shelves. The photograph of the front of the supermarket shelf can be understood as the original high-resolution image, and this candidate image area... This can be understood as a local patch that is cropped without loss of quality, and the candidate image region is then... Divide the image into a 16×16 tile sequence, and output the answer to the question based on the 16×16 tile sequence and the question text.

[0140] In some embodiments of this application, the question-and-answer interaction model described above includes a local image encoder and an inference model. Combined with... Figure 1 ,like Figure 3 As shown, step 102 can be implemented through steps 102a and 102b below.

[0141] Step 102a: The electronic device obtains the image region features corresponding to the candidate image region through a local image encoder.

[0142] In some examples, the local encoder described above may include, but is not limited to, the standard visual Transformer sub-model. The inference model described above may include, but is not limited to, MLLM, such as the pre-trained multimodal large model architecture LLAVA-1.5.

[0143] In some examples, electronic devices can extract deep features from candidate image regions using a local image encoder to obtain image region features.

[0144] For example, in one of the scenarios described above, the electronic device can select candidate image regions. A sequence of 16×16 image patches is input into a local image encoder, which performs depth feature extraction to obtain image region features. The features of this image region For one or a group of regions that can characterize the candidate image region The visual feature vector of refined information, with dimensions that can be , Candidate image regions of high, Candidate image regions The width of the image region features This includes the clear texture details on the aforementioned shop signs.

[0145] To illustrate further, in another scenario described above, the electronic device can select candidate image regions. A sequence of 16×16 image patches is input into a local image encoder, which performs depth feature extraction to obtain image region features. The features of this image region For one or a group of regions that can characterize the candidate image region The visual feature vector of refined information, with dimensions that can be , Candidate image regions of high, Candidate image regions The width of the image region features This includes clear texture details on the price tags below the red bottled beverage on the far left of the aforementioned shelf and the price tags below the blue bottled beverage on the far right of the aforementioned shelf.

[0146] Step 102b: The electronic device uses an inference model to infer based on the features of the image region and the features of the question text corresponding to the question text, and outputs the answer to the question.

[0147] In some examples, electronic devices can first concatenate image region features and question text features through a question-and-answer interaction model to obtain an input sequence, and then input this input sequence into an inference model. The inference model then infers based on this input sequence and outputs the answer to the question.

[0148] It should be noted that for the explanation of reasoning based on the input sequence through the reasoning model, please refer to the specific description in the related technology, and the embodiments of this application will not be repeated here.

[0149] For example, in one of the scenarios described above, electronic devices can use a question-and-answer interaction model to extract image region features. and question text features These features are integrated into a unified, highly condensed input sequence, which is then fed into the core reasoning part of the inference model. The model then performs reasoning to output the answer to the question, such as the text "The phone number is 555-01999".

[0150] To illustrate further, in another scenario described above, electronic devices can use a question-and-answer interaction model to extract image region features. and question text features These features are integrated into a unified, highly condensed input sequence, which is then fed into the core reasoning part of the inference model. The model then performs reasoning and outputs the answer to the question, such as the text "Yes, the drink in the red bottle (priced at $2.5) is more expensive than the drink in the blue bottle (priced at $1.8)".

[0151] Understandably, after receiving these highly relevant image region features that have been filtered and refined, the inference model can focus on performing complex cognitive tasks that it excels at, such as deep logical reasoning, relational analysis, and comparative judgment. Ultimately, it generates fluent and accurate natural language answers, i.e., question answers, through autoregression, as the final response to the user's question text.

[0152] Therefore, since electronic devices can obtain image region features corresponding to candidate image regions through local image encoders, and the data volume of these candidate image region features is much smaller than that of the image features corresponding to the question image, the computational load of the electronic device's inference model can be reduced, allowing the inference model to quickly output the question answer. Furthermore, since the candidate image regions are image regions related to the question text, the electronic device can accurately infer the question answer through the inference model. Thus, the efficiency of text-based dialogue can be improved while ensuring its performance.

[0153] In some examples, the question-and-answer interaction model described above also includes a decision-maker. Optionally, prior to step 102b above, the question-and-answer interaction method provided in this application embodiment further includes step 301 below, and step 102b above can be specifically implemented through step 102b1 below.

[0154] Step 301: The electronic device, through the decision-maker, makes a decision evaluation based on the image region features and the problem text features, and outputs the evaluation result information.

[0155] Optionally, the decision-maker may include, but is not limited to, a neural network binary classifier based on fully connected layers.

[0156] In this embodiment of the application, the above evaluation result information indicates whether the candidate image region is all image regions in the problem image that are related to the problem text.

[0157] Optionally, when there are at least two candidate image regions and the electronic device determines one candidate image region, the electronic device can add the image region features corresponding to the one candidate image region into the "evidence pool" E, and use the decision-maker to concatenate the features in the "evidence pool" E with the problem text features, and perform decision evaluation based on the concatenated features, and output the evaluation result information.

[0158] For example, in one of the scenarios described above, the electronic device can select candidate image regions. Corresponding image region features Add the features from the "evidence pool" E, and then use the decision-maker to add the features from the "evidence pool" E. and question text features The data is spliced ​​together, and a decision evaluation is performed based on the spliced ​​features, outputting the evaluation results.

[0159] Step 102b1: If the evaluation result information indicates that the candidate image region is all image regions in the question image that are related to the question text, the electronic device uses the reasoning model to reason based on the image region features and the question text features, and outputs the question answer.

[0160] For example, in one of the scenarios described above, the evaluation result information indicates candidate image regions. If the entire image region in the city street view image is related to the text "Please tell me the phone number on that green sign in the distance?", then the electronic device can directly use the inference model to determine the relationship between the image regions and their features. and question text features Perform reasoning and output the answer to the question, namely the text "The phone number is 555-01999".

[0161] Thus, it can be seen that since the electronic device can make decisions and evaluate based on the features of the image region and the features of the question text through the decision-maker, and output evaluation result information, it can know whether the obtained candidate image region is all the image regions in the question image that are related to the question text. In this way, when the evaluation result information indicates that the candidate image region is all the image regions in the question image that are related to the question text, that is, when the question answer can be accurately output, the electronic device can make inferences based on the features of the image region and the features of the question text through the inference model. Therefore, the accuracy of the output question answer can be ensured.

[0162] In some examples, the question-and-answer interaction model described above also includes a regional proposal network. The question-and-answer interaction method provided in this application embodiment further includes steps 401 and 402 as described below.

[0163] Step 401: If the evaluation result information indicates that the candidate image region is not all image regions in the question image that are related to the question text, the electronic device generates an updated intent vector based on the image region features and the initial intent vector through a question-and-answer interaction model.

[0164] In this embodiment of the application, the initial intent vector is used to characterize the question intent of the question text.

[0165] Optionally, the electronic device can concatenate image region features and the initial intent vector to generate an updated intent vector.

[0166] It is understandable that electronic devices can incorporate information from the acquired candidate image regions into the initial intent vector to generate an updated intent vector.

[0167] For example, in another scenario described above, the electronic device can select candidate image regions. Corresponding image region features Add the features from the "evidence pool" E, and then use the decision-maker to add the features from the "evidence pool" E. and question text features The images are stitched together, and a decision evaluation is performed based on the stitched features. The evaluation result information is output, indicating that the candidate image region is not one of all image regions in the problem image that are related to the problem text, such as the candidate image region mentioned above. If the image only includes the price tag below the red bottled beverage on the far left of the shelf, then the electronic device can use a question-and-answer interaction model based on the features of the image region. and initial intent vector Generate updated intent vector .

[0168] Step 402: The electronic device uses the regional proposal network to determine the remaining image regions in the question image that are related to the question text, based on the updated intent vector and the features of the question image.

[0169] It should be noted that the description of how an electronic device determines the remaining image regions related to the question text in the question image based on the updated intent vector and question image features through the region proposal network can be found in the above embodiment, which describes how the electronic device determines candidate image regions based on the initial intent vector and question image features through the region proposal network. This embodiment will not repeat the description here.

[0170] For example, in another scenario described above, electronic devices can use a regional proposal network to update the intent vector. and problem image features Identify the remaining image areas in the front view of the supermarket shelf that relate to the text "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?" The remaining image area This includes the price tag below the blue bottled beverage on the far right of the shelf mentioned above.

[0171] It is understandable that after determining the remaining image regions mentioned above... Then, the electronic device can crop candidate image regions from the aforementioned city street view images. This allows the candidate image region to be selected. The image is divided into a 16×16 tile sequence, and candidate image regions are selected based on this tile sequence. Given a sequence of partitioned tiles and the text "Is the drink in the red bottle on the far left of the shelf more expensive than the drink in the blue bottle on the far right?", output the answer to the question.

[0172] Optionally, after determining the remaining image regions in the problem image that are related to the problem text, the electronic device can add the image region features corresponding to the remaining image regions to the "evidence pool" E, and then, through the decision-maker, perform decision evaluation based on the image region features in the "evidence pool" E and the problem text, output evaluation result information. If the evaluation result information indicates that the candidate image regions are all image regions in the problem image that are related to the problem text, the electronic device can use the reasoning model to reason based on the image region features and the problem text features, and output the problem answer; or, if the evaluation result information indicates that the candidate image regions are not all image regions in the problem image that are related to the problem text, repeat steps 401 and 402 above, and so on.

[0173] Thus, even when the evaluation result indicates that the candidate image regions are not all image regions in the question image that are related to the question text, i.e., when the electronic device has not yet obtained the required candidate image regions, the electronic device can still generate an updated intent vector based on the image region features and the initial intent vector through the question-answering interaction model. Then, through the region proposal network, it can determine the remaining image regions in the question image that are related to the question text based on the updated intent vector and the question image features, thereby obtaining all the required candidate image regions. Therefore, it can be ensured that when the evaluation result indicates that the candidate image regions are all image regions in the question image that are related to the question text, i.e., when the question answer can be accurately output, the electronic device can perform inference based on the image region features and the question text features through the inference model. Therefore, the accuracy of the output question answer can be ensured.

[0174] The following two examples illustrate the question-and-answer interaction method provided in the embodiments of this application.

[0175] Example 1: A scenario where a user uses MLLM to ask questions about the details of the content in an image.

[0176] In this example, the user needs to extract fine-grained information from a city street view.

[0177] Users input an image into the interface of an AI-based application: a high-resolution photo of a city street scene, including streets, pedestrians, vehicles, and shop signs in the distance, and text: "Please tell me the phone number on that green sign in the distance?"

[0178] Execution process:

[0179] 1. Overall Overview: The question-answering interaction model first downsamples the input 4K image to... The low-resolution thumbnail is combined with the input text encoding. At this point, due to resolution limitations, the image features corresponding to the 4K image cannot be parsed to extract the phone number, but based on the phrase "green signboard" in the input text, an initial intent vector for finding the green rectangular region is generated.

[0180] 2. First iteration (localization): The region proposal network uses the initial intent vector to locate the green sign area of ​​a distant store in the coordinate system of the 4K image and uses it as a candidate image region.

[0181] 3. Local Focus: The question-and-answer interaction model losslessly crops candidate image regions of the sign from the 4K image. At this point, the text on the sign is clearly visible. The question-and-answer interaction model performs fine encoding on the candidate image regions, extracting image region features containing digital textures, and stores them in the "evidence pool".

[0182] 4. Decision and Termination: The decision-maker analyzes the "evidence pool" and finds that clear numerical features have been extracted (such as "555-0199"). It judges that the information is sufficient and decides to terminate the search.

[0183] 5. Final reasoning: The image region features in the "evidence pool" and the input text are fed into the reasoning model, and the answer is directly output: "The phone number is 555-0199".

[0184] Example 2: A scenario where a user asks a question based on the differences between different contents in an MLLM image.

[0185] In this example, the user needs to compare prices of goods on supermarket shelves, which means the user needs to solve the problem of "integrating scattered information from multiple objectives".

[0186] Users input an image into the interface of an AI-based application: a front view of a supermarket shelf filled with dozens of different beverages, each with a price tag below it, and the text "Is the red bottle on the far left of the shelf more expensive than the blue bottle on the far right?"

[0187] Execution process:

[0188] 1. Global Overview: The question-answering interaction model uses low-resolution full-image features and question encoding to understand that the task requires comparing information from two objects at different locations, thus obtaining the initial intent vector.

[0189] 2. First iteration (search target A): Based on the initial intent vector of "leftmost red bottle", the question-answering interaction model locates and crops high-resolution images of the leftmost red bottle and its price tag from the input image, extracts image region features, and stores them in the "evidence pool". At this time, the decision module judges that there is insufficient information because there is a lack of comparison objects, and updates the intent vector to "find rightmost blue bottle".

[0190] 3. Second iteration (search target B): Based on the updated intent vector, the question-answering interaction model locates and crops high-resolution images of the blue bottle on the right and its price tag from the input image, extracts image region features, and adds them to the "evidence pool".

[0191] 4. Decision and Termination: The decision-maker detects that the evidence pool contains price information for two target objects, which satisfies the comparison conditions, and terminates the search.

[0192] 5. Final reasoning: The reasoning model reads the characteristics of the two sets of goods in the evidence pool, performs logical comparison, and outputs the answer: "Yes, the red bottle (2.5) is more expensive than the blue bottle (1.8)".

[0193] This application provides a question-and-answer interaction method. An electronic device can input a question image and question text into a trained question-and-answer interaction model. The model determines candidate image regions related to the question text within the question image, and then outputs the answer based on these candidate image regions and the question text. Since the electronic device can use this model to select candidate image regions related to the question text—that is, select a portion of the image region related to the question text—after inputting the question image and question text into the trained model, the electronic device can perform calculations based on this portion of the image region and the question text, rather than on the entire image region and the question text. This avoids the model performing calculations on image regions unrelated to the question text, reducing the computational load on the model and ensuring that it operates based on image regions related to the question text. This improves the efficiency of outputting the answer while maintaining accuracy, thus improving both the performance and efficiency of the text-based dialogue.

[0194] Furthermore, this application's embodiments introduce an iterative "observation-focusing-reasoning" workflow, designing an efficient intelligent perception front-end for multimodal large models, significantly improving the model's performance and practicality. Its core beneficial effect lies in the fact that, through a dynamic processing mode of "first global overview, then local focus," computational redundancy is greatly reduced, achieving efficient and low-cost reasoning. This characteristic allows multimodal large models, originally limited by high computing resources, to be applied in a wider range of scenarios, such as real-time multi-detail question answering of complex product images on smartphones, or rapid identification and response to dynamic traffic signs and signals in in-vehicle assisted driving systems, providing strong technical support for terminal-side deployment and real-time interactive applications. In addition, this application's embodiments significantly enhance the model's reasoning reliability in complex scenarios by iteratively collecting multiple dispersed visual evidences. In applications requiring comparative or correlation analysis, such as in the field of security monitoring where the behavior of multiple targets is correlated, this invention ensures that key information is not overlooked, thereby obtaining more accurate analysis results.

[0195] In some embodiments of this application, the electronic device can first train a trained question-and-answer interaction model, and then input the question image and question text into the trained question-and-answer interaction model so that the electronic device can perform the above steps 101 and 102.

[0196] In some examples, electronic devices can directly train a pre-trained question-and-answer interaction model. Alternatively, electronic devices can request a cloud server to train a pre-trained question-and-answer interaction model.

[0197] In some embodiments of this application, combined with Figure 1 ,like Figure 4 As shown, before step 101 above, the question-and-answer interaction method provided in this application embodiment may further include steps 501 to 504 below.

[0198] Step 501: The electronic device acquires the training sample set.

[0199] In this embodiment of the application, the training sample set includes sample question images, sample question text, sample candidate image regions, and sample question answers. The sample candidate image regions are image regions in the sample question images that are related to the sample question text.

[0200] For example, the training sample set can be a quadruple (I,T,A,B), where I is the sample question image (i.e., the original high-resolution sample image), T is the sample question text (i.e., the question text for the sample question image), A is the sample question answer (i.e., the standard text answer), and B is the sample candidate image region. The sample candidate image region can be composed of one or more sets of location information, such as the coordinate information of the bounding box of the sample candidate image region.

[0201] In some examples, the number of training sample sets mentioned above can be at least one.

[0202] It should be noted that the descriptions of the sample question image, sample question text, sample candidate image region, and sample question answer can be found in the specific descriptions of the question image, question text, candidate image region, and question answer in the above embodiments, and will not be repeated here in the embodiments of this application.

[0203] Step 502: The electronic device inputs the training sample set into the preset model. Based on the sample question text, the preset model determines the prediction candidate image region in the sample question image that is related to the sample question text.

[0204] It should be noted that, regarding the explanation of how electronic devices determine the predicted candidate image regions related to the sample question text in the sample question image based on the sample question text using a preset model, the specific description of how electronic devices determine the candidate image regions related to the question text in the question image based on the question text using a question-and-answer interaction model in the above embodiments can be referred to, and will not be repeated here in the embodiments of this application.

[0205] Step 503: The electronic device outputs the predicted question answer based on the predicted candidate image region and sample question text using a preset model.

[0206] It should be noted that, regarding the explanation of how electronic devices output predicted question answers based on predicted candidate image regions and sample question text using a preset model, the specific description of how electronic devices output question answers based on candidate image regions and question text using a question-and-answer interaction model in the above embodiments can be referred to, and will not be repeated here in the embodiments of this application.

[0207] Step 504: The electronic device determines a first loss value based on the predicted candidate image region and the sample candidate image region, and determines a second loss value based on the predicted question answer and the sample question answer.

[0208] In some examples, the electronic device may employ a first predetermined loss function to determine a first loss value based on the predicted candidate image region and the sample candidate image region. This first predetermined loss function may include, but is not limited to, a smooth L1 loss function.

[0209] For example, suppose the location information of the predicted candidate image region is the coordinate information of the bounding box of the predicted candidate image region. =( , , , The location information of the candidate image region is the coordinate information of the bounding box of the candidate image region. =( , , , Then the first loss value is defined as:

[0210] (5)

[0211] in, The first loss value, To predict the i-th coordinate information in the bounding box coordinate information of the candidate image region, The i-th coordinate information in the coordinate information of the bounding box of the candidate image region is given. This is the deviation value.

[0212] When the absolute value of the deviation hour, ;otherwise, .

[0213] Assuming the coordinates of the bounding box of the above sample candidate image region are [1460, 780, 1890, 1060], and the coordinates of the bounding box of the above predicted candidate image region are [1486, 768, 1904, 1050], then the absolute value of the above deviation is: , , , Since all the above deviations are greater than 1, according to the formula logic (approximately linear part), the total loss value is (26-0.5)+(12-0.5)+(14-0.5)+(10-0.5)=60.

[0214] In some examples, the electronic device may employ a second predetermined loss function to determine a second loss value based on the predicted question answer and the sample question answer. This second predetermined loss function may include, but is not limited to, the cross-entropy loss function.

[0215] For example, the second loss value is defined as:

[0216] (6)

[0217] in, t is the second loss value, and t is the probability that the answer to the predicted question is the same as the answer to the sample question.

[0218] Assuming the answer to the sample problem The first word is "5" (corresponding to index 501 in the vocabulary). After receiving the concatenated sequence, the preset model uses the output layer Softmax to calculate the predicted probability of the token with ID 501. Substitute the values ​​into the loss function formula to calculate the loss value. If the preset model makes an incorrect prediction, for example, assuming the probability of "5" is only 0.1, then the calculated loss value is the preset value, for example... High loss values ​​will prompt the preset model to adjust its parameters significantly.

[0219] Step 505: The electronic device performs a first stage of training on the preset model based on the first loss value, and performs a second stage of training on the preset model after the first stage of training based on the second loss value, to obtain the trained question-and-answer interaction model.

[0220] In some examples, the learning rate of the first stage of training is greater than the learning rate of the second stage of training.

[0221] Optionally, the learning rate for the first stage of training can be a constant, such as 1 / 10, and the learning rate for the second stage of training can be a constant, such as 1×10. -5 .

[0222] Thus, since the learning rate of the first stage of training is greater than that of the second stage of training, the interference of the second stage of training on the sub-models in the preset model after the first stage of training can be reduced, thereby improving the accuracy of the preset model obtained through training.

[0223] In some examples, the electronic device may first train a portion of the sub-models in the preset model in the first stage according to a first loss value, and then train at least a portion of the sub-models of the preset model after the first stage of training according to a second loss value.

[0224] Thus, it can be seen that since electronic devices can acquire training sample sets and input them into a preset model, and through this preset model, determine the predicted candidate image regions related to the sample question text in the sample question image based on the sample question text in the training sample set, and output the predicted question answer based on the predicted candidate image regions and the sample question text, electronic devices can accurately determine the first loss value and the second loss value, and accurately train the preset model based on the first loss value and the second loss value, thereby improving the accuracy of training the question-answering interaction model.

[0225] In some examples, the aforementioned preset model includes a preset fusion processor and a preset proposal network. Step 502 can be specifically implemented through steps 502a and 502b below, and step 505 can be specifically implemented through step 505a below.

[0226] Step 502a: The electronic device determines the initial intent vector of the sample by using a preset fusion processor based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text.

[0227] In this embodiment of the application, the above-mentioned initial intent vector of the sample is used to characterize the question intent of the sample question text.

[0228] It should be noted that, regarding the explanation of how electronic devices determine the initial intent vector of a sample based on the sample problem image features corresponding to the sample problem image and the sample problem text features corresponding to the sample problem text through a preset fusion processor, the specific description of how electronic devices determine the initial intent vector based on the problem image features corresponding to the problem image and the problem text features corresponding to the problem text in the above embodiment can be referred to, and will not be repeated here in the embodiments of this application.

[0229] Step 502b: The electronic device determines the predicted candidate image region based on the initial intent vector of the sample and the features of the sample question image through a preset proposal network.

[0230] It should be noted that the description of how electronic devices determine the predicted candidate image region based on the initial intent vector and the features of the sample question image through a preset proposal network can be found in the above embodiment, which describes how electronic devices determine the candidate image region based on the initial intent vector and the features of the question image through a region proposal network. This embodiment will not repeat the description here.

[0231] Step 505a: The electronic device performs a first-stage training on the preset fusion network and the preset proposal network based on the first loss value, and performs a second-stage training on the preset model after the first-stage training based on the second loss value, to obtain the trained question-and-answer interaction model.

[0232] Thus, it can be seen that the electronic device can determine the predicted candidate image region through the preset fusion unit and the preset proposal network. Therefore, the electronic device can perform the first stage of training on the preset fusion unit and the preset proposal network based on the first loss value, thereby improving the accuracy of training the preset fusion unit and the preset proposal network.

[0233] Optionally, the preset model further includes a preset global image encoder. Before step 502a, the question-and-answer interaction method provided in this application embodiment may also include steps 601 and 602 as described below, and step 505a can be specifically implemented through step 505a1 as described below.

[0234] Step 601: The electronic device performs downsampling processing on the problem image using a preset model to obtain a sample thumbnail.

[0235] It should be noted that, regarding the explanation of how electronic devices downsample problem images using a preset model to obtain sample thumbnails, the above embodiment can be referenced to the electronic device using a question-and-answer interaction model.

[0236] Step 602: The electronic device obtains the image features corresponding to the sample thumbnail through a preset global image encoder, and determines the image features corresponding to the sample thumbnail as the sample problem image features.

[0237] It should be noted that, for the explanation of how electronic devices obtain image features corresponding to sample thumbnails through a preset global image encoder, please refer to the specific description of how electronic devices obtain image features corresponding to thumbnails through a global image encoder in the above embodiments. This application will not repeat the description here.

[0238] Step 505a1: The electronic device performs the first stage of training on the preset fusion network, the preset proposal network, and the preset global image encoder based on the first loss value.

[0239] Thus, it can be seen that the electronic device can determine the predicted candidate image region through the preset fusion unit, the preset proposal network and the preset global image encoder. Therefore, the electronic device can perform the first stage of training on the preset fusion unit, the preset proposal network and the preset global image encoder according to the first loss value, thereby improving the accuracy of training the preset fusion unit and the preset proposal network.

[0240] In some examples, the preset model includes a preset local image encoder and a preset inference model. Step 503 can be implemented through steps 503a and 503b below, and step 505 can be implemented through step 505b below.

[0241] Step 503a: The electronic device obtains the sample image region features corresponding to the predicted candidate image region through a preset local image encoder.

[0242] It should be noted that, for the explanation of how electronic devices obtain sample image region features corresponding to predicted candidate image regions through a preset local image encoder, please refer to the specific description of the electronic device's local image encoder obtaining image region features corresponding to candidate image regions in the above embodiments. This application will not repeat the details here.

[0243] Step 503b: The electronic device uses a preset reasoning model to reason based on the features of the sample image region and the features of the sample question text, and outputs the predicted question answer.

[0244] It should be noted that, regarding the explanation of how electronic devices use a preset inference model to infer based on the features of sample image regions and the features of sample question text, and output the predicted question answer, please refer to the specific description of how electronic devices use an inference model to infer based on the features of image regions and the features of question text, and output the question answer in the above embodiments. This application embodiment will not repeat the details here.

[0245] Step 505b: The electronic device performs a first-stage training on the preset model based on the first loss value, and performs a second-stage training on the preset local image encoder and the preset inference model based on the second loss value.

[0246] Thus, it can be seen that the electronic device can output the predicted answer to the question through the preset local image encoder and the preset inference model. Therefore, the electronic device can perform a second stage of training on the preset local image encoder and the preset inference model based on the second loss value, thereby improving the accuracy of training the preset local image encoder and the preset inference model.

[0247] The following example illustrates the process of training a question-and-answer interaction model on an electronic device.

[0248] 1. Construct a multi-sub-model neural network architecture and complete parameter initialization.

[0249] This step aims to build the basic framework of the core algorithm of this application. Specifically, this step constructs an end-to-end composite neural network model, namely the preset model in the above embodiments. This model is not a loose collection of components, but a holistic architecture tightly connected by data flow, specifically composed of the following sub-network modules connected together, and finally outputs a trainable model instance with completed parameter initialization:

[0250] Global Image Encoder: As a convolutional neural network submodule (in this embodiment, a small convolutional neural network MobileNetV3 is used), it is responsible for receiving downsampled low-resolution images as input, performing convolution operations, and outputting a global context feature vector containing global image information.

[0251] Text Encoder: As a language model submodule (in this implementation, the lightweight Transformer encoder DistilBERT is used), it is responsible for receiving the user's natural language text question, encoding it with Transformer, and outputting the text feature vector.

[0252] Cross-modal fusion module: As an attention calculation submodule, it is responsible for receiving the global context feature vector and text feature vector mentioned above, performing modal alignment through a cross-attention mechanism, and outputting an "intent vector" that guides the direction of subsequent visual search.

[0253] Region Proposal Network: As a location prediction submodule, it is responsible for performing regression prediction in the original image coordinate system based on the "intent vector" and image features, and outputting the bounding box coordinates of the candidate regions most likely to contain answer information.

[0254] Local encoder: As a standard visual Transformer submodule, it is responsible for receiving high-resolution local image patches cropped according to the above coordinates, extracting depth features, and outputting high-expressive local visual feature vectors.

[0255] Reasoning Model: As the core sub-module of the system's reasoning, it is responsible for receiving the integrated local visual evidence and textual questions, performing logical reasoning, and outputting the final natural language answer text. In this implementation, the existing pre-trained multimodal large model architecture LLAVA-1.5, which has powerful reasoning capabilities, can be used.

[0256] After completing the construction of the above network architecture, except for the parameters of the regional proposal network and the cross-modal fusion module which are randomly initialized, the weights of the other sub-modules are loaded from the pre-trained weights on a large public dataset, thus completing the initialization of the model as a whole.

[0257] 2. Obtain the training sample set

[0258] 3. Perform the first stage of training on the aforementioned preset fusion network, preset proposal network, and preset global image encoder.

[0259] 4. End-to-end Fine-tuning: After the aforementioned preset model has reliable visual localization capabilities, the goal of this stage is to streamline the entire process, training the model to learn how to perform comprehensive reasoning and generate the correct answer after finding the correct region. The specific implementation process is as follows:

[0260] Model state settings: Set the parameters of all sub-models in the preset model to a trainable state. At the same time, to prevent the loss of knowledge pre-trained on a large dataset, set a learning rate much smaller than that of the first training stage.

[0261] Determine the answer to the predicted question, and then determine the second loss value based on the predicted answer and the sample question answer.

[0262] The second stage of training is performed on all sub-models in the preset model based on the second loss value.

[0263] It is understandable that after completing the second stage of training, the pre-set model can obtain a complete system that can actively perform visual search and reasoning based on the question, namely the question-answering interaction model, which can be used in actual prediction (or reasoning) scenarios, so that users can use the question-answering interaction model to interact through electronic devices.

[0264] In summary, a unified architecture based on an "encoder-large language model" is commonly adopted in related technologies. Its typical workflow is as follows: First, a powerful visual encoder, typically a visual Transformer, processes the complete high-resolution input image. This visual encoder segments the image into a grid of numerous tiles, encoding each tile into a high-dimensional visual feature vector. Subsequently, this sequence of visual tokens, representing all the details of the entire image, is concatenated with the user-input text question token sequence to form a unified, longer input sequence. Finally, this long sequence, incorporating all visual and textual information, is fed into the LLM (Large Language Model). The LLM performs end-to-end analysis and reasoning on the entire sequence through its internal self-attention mechanism, generating the final text answer in an autoregressive manner. This paradigm of "flattening" the image into a unified sequence and processing it all at once is the fundamental technical solution for achieving multimodal understanding. However, while multimodal large models based on the integrated "encoding-fusion-inference" paradigm have achieved significant success, their inherent "one-step" workflow, which processes all image information at once, exposes two core flaws in practical applications. First, there is computational redundancy and resource waste. The biggest flaw of this paradigm lies in its "static" and "indiscriminate" processing method. Regardless of whether the user's question concerns the overall style of an image or only a trivial detail (e.g., "What is the text on the road sign in the upper left corner of the image?"), the model must divide the entire high-resolution image into thousands of tiles and perform complete encoding and attention calculations. For most tasks that only require attention to local information, the computation of massive amounts of irrelevant background areas (such as sky, ground, and pedestrians) is a huge waste of resources. This redundant computation not only significantly increases the model's inference latency but also places stringent demands on the hardware of the deployment environment (such as video memory), severely hindering its application on resource-constrained devices such as mobile devices. Secondly, the model's complex reasoning ability is limited by information dilution. For complex questions that require integrating information from multiple scattered areas in an image (such as "Compare the prices of the products on the left and right sides of the shelf"), the "one-step" approach proves inadequate. In this model, all key visual evidence (such as the price tags of the products on the left and right sides) is "diluted" in a massive information stream composed of thousands of visual tokens, resulting in an extremely low signal-to-noise ratio. The model needs to capture and associate all key information simultaneously amidst a deluge of irrelevant information through a single self-attention computation, posing a significant challenge to its reasoning ability. Therefore, when handling complex tasks requiring multi-step, focused observation, the model's reliability decreases significantly, often resulting in problems such as "missing" key details or failing to establish correct associations.

[0265] This application overcomes the problems of computational redundancy and insufficient complex reasoning ability in multimodal large-scale models when processing visual information in related technologies, and provides a more efficient, intelligent, and human-friendly multimodal understanding method. To achieve the above objectives, the improvement of this application lies in proposing a query-guided active visual search mechanism, which completely changes the static paradigm of traditional models performing one-time, global image processing. The main improvements of this application are as follows:

[0266] 1. Introducing an iterative, coarse-to-fine dynamic processing flow: This application avoids the brute-force "one-size-fits-all" encoding of the entire high-resolution image. Instead, it first understands the task intent by rapidly co-encoding the low-resolution image and the problem. Then, based on this intent, it actively and iteratively locates and analyzes the most relevant local regions in the original high-resolution image. This dynamic flow of "global overview first, local focus later" significantly reduces unnecessary computation and greatly improves inference efficiency.

[0267] 2. Constructing a proactive, "on-demand" visual information acquisition mechanism: This application endows the model with the ability to proactively "observe" and "focus." The model can autonomously decide "where to look" and "how much detail to look" based on the needs of the specific question, acquiring the key visual evidence necessary to answer the question on demand. This not only avoids interference from massive amounts of irrelevant information but also makes the integration and reasoning of multiple scattered pieces of evidence clearer and more reliable.

[0268] 3. Constructing an efficient information preprocessing front-end for multimodal large-scale models: This application does not aim to replace existing multimodal large-scale models, but rather to serve as their efficient "intelligent perception front-end." Through an active search mechanism, the most crucial and refined visual evidence is extracted from the original large-scale image and packaged into a short sequence with extremely high information density before being submitted to the large-scale model for final deep reasoning. This allows the large-scale model to leverage its strengths and avoid its weaknesses, focusing on its core competencies of logical analysis and language generation, thereby achieving a dual improvement in both performance and efficiency overall.

[0269] As can be seen from the above, in the embodiments of this application, the electronic device can first determine a portion of the image region, i.e., the candidate image region, from the complete question image through the question-answering interaction model. In this way, when outputting the question answer through the question-answering interaction model, the question-answering interaction model only needs to reason about the candidate image region with a small amount of data that is related to the question text. This can improve the efficiency of outputting the question answer while ensuring the accuracy of the output question answer. Thus, the efficiency of text-image dialogue can be improved while ensuring the performance of text-image dialogue.

[0270] It should be noted that each of the above method embodiments, or various possible implementations of each method embodiment, can be executed individually or in combination of any two or more. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0271] The question-and-answer interaction method provided in this application can be executed by a question-and-answer interaction device. This application uses a question-and-answer interaction device executing the question-and-answer interaction method as an example to illustrate the question-and-answer interaction device provided in this application.

[0272] Figure 5 This is a schematic diagram of the structure of a question-and-answer interaction device provided in an embodiment of this application. Figure 5 As shown, the question-and-answer interaction device 500 includes a processing module 501.

[0273] The processing module 501 is used to input the question image and question text into the trained question-answering interaction model, and through the question-answering interaction model, determine the candidate image region in the question image that is related to the question text based on the question text; and through the question-answering interaction model, output the question answer based on the candidate image region and the question text.

[0274] This application provides a question-and-answer interaction device. After the question-and-answer interaction device inputs the question image and question text into the trained question-and-answer interaction model, the device can use the model to determine candidate image regions related to the question text in the question image. In other words, it selects a portion of the image region related to the question text from the question image. This allows the device to perform calculations based on this portion of the image region and the question text, rather than on the entire image region and the question text. Therefore, it avoids the question-and-answer interaction model performing calculations on image regions unrelated to the question text. This reduces the computational load of the question-and-answer interaction model and ensures that the model operates based on the portion of the image region related to the question text. This improves the efficiency of outputting the question answer while ensuring its accuracy. Thus, it improves the efficiency of text-based dialogue while maintaining its performance.

[0275] In one possible implementation, the question-answering interaction model includes a cross-modal fusion processor and a regional proposal network. Specifically, the processing module 501 is used to determine an initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text through the cross-modal fusion processor. The initial intent vector is used to characterize the question intent of the question text. The regional proposal network is used to determine candidate image regions based on the initial intent vector and the question image features.

[0276] In one possible implementation, the question-and-answer interaction model further includes a global image encoder; the processing module 501 is further configured to perform downsampling processing on the question image through the question-and-answer interaction model to obtain a thumbnail before determining the initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text through the cross-modal fusion of the question-and-answer interaction model; and to obtain the image features corresponding to the thumbnail through the global image encoder, and determine the image features corresponding to the thumbnail as the question image features.

[0277] In one possible implementation, the aforementioned processing module 501 is specifically used to calculate an initial intent vector based on the features of the problem image and the problem text, using an attention mechanism.

[0278] In one possible implementation, the processing module 501 is specifically configured to perform regression prediction based on the initial intent vector and the features of the problem image to obtain at least one predicted location information and at least one confidence level, wherein each confidence level is used to characterize the probability that a predicted location information is the location information of an image region in the problem image related to the problem text; and to perform non-maximum suppression deduplication processing on the at least one predicted location information; and to determine the candidate location information with the highest confidence level and an overlap rate with the historical exploration area below the overlap rate threshold from the deduplicated predicted location information; and to determine the image region in the problem image corresponding to the candidate location information as the candidate image region.

[0279] In one possible implementation, the question-answering interaction model includes a local image encoder and an inference model; the processing module 501 is specifically used to obtain the image region features corresponding to the candidate image region through the local image encoder; and to infer the question answer based on the image region features and the question text features corresponding to the question text through the inference model.

[0280] In one possible implementation, the question-and-answer interaction model further includes a decision-maker; the processing module 501 is further configured to, before inferring and outputting the answer through the inference model based on the image region features and the question text features corresponding to the question text, perform decision evaluation through the decision-maker based on the image region features and the question text features, and output evaluation result information, which indicates whether the candidate image region is all image regions in the question image that are related to the question text; specifically, when the evaluation result information indicates that the candidate image region is all image regions in the question image that are related to the question text, the processing module 501 is configured to, through the inference model, perform inference based on the image region features and the question text features, and output the answer to the question.

[0281] In one possible implementation, the question-and-answer interaction model further includes a region proposal network; the processing module 501 is further configured to, when the evaluation result information indicates that the candidate image region is not all the image regions in the question image that are related to the question text, generate an updated intent vector based on the image region features and the initial intent vector through the question-and-answer interaction model, wherein the initial intent vector is used to characterize the question intent of the question text; and determine the remaining image regions in the question image that are related to the question text based on the updated intent vector and the question image features through the region proposal network.

[0282] In one possible implementation, the question-and-answer interaction device 500 may further include: an acquisition module, configured to acquire a training sample set before the processing module 501 inputs the question image and question text into the trained question-and-answer interaction model, and determines, based on the question text, a candidate image region in the question image related to the question text, by means of the question-and-answer interaction model. The training sample set includes sample question images, sample question text, sample candidate image regions, and sample question answers, wherein the sample candidate image regions are image regions in the sample question image related to the sample question text. The processing module 501 is further configured to input the training sample set acquired by the acquisition module into a preset model, and determine, based on the sample question text, a candidate image region in the question image related to the question text by means of the preset model. The question-answering interaction device 500 further includes: a training module, used to train the preset model in the first stage according to the first loss value determined by the processing module 501, and to train the preset model after the first stage of training in the second stage according to the second loss value determined by the processing module 501, to obtain the trained question-answering interaction model.

[0283] In one possible implementation, the aforementioned preset model includes a preset fusion processor and a preset proposal network; the aforementioned processing module 501 is specifically used to determine an initial intent vector of a sample based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, the initial intent vector of which is used to characterize the question intent of the sample question text; and to determine a predicted candidate image region based on the initial intent vector of the sample and the sample question image features through the preset proposal network; the aforementioned training module is specifically used to perform a first stage of training on the preset fusion processor and the preset proposal network based on a first loss value.

[0284] In one possible implementation, the aforementioned preset model further includes a preset global image encoder; the aforementioned processing module 501 is further configured to, before determining the initial intent vector of the sample through the preset fusion processor based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, perform downsampling processing on the sample question image through the preset model to obtain a sample thumbnail; and obtain the image features corresponding to the sample thumbnail through the preset global image encoder, and determine the image features corresponding to the sample thumbnail as the sample question image features; the aforementioned training module is specifically configured to perform a first stage of training on the preset fusion processor, the preset proposal network and the preset global image encoder based on the first loss value.

[0285] In one possible implementation, the preset model includes a preset local image encoder and a preset inference model; the processing module 501 is specifically used to obtain the sample image region features corresponding to the predicted candidate image region through the preset local image encoder; and to perform inference based on the sample image region features and sample question text features through the preset inference model, and output the predicted question answer; the training module is specifically used to perform a second stage of training on the preset local image encoder and the preset inference model based on the second loss value.

[0286] In one possible implementation, the learning rate of the first stage of training is greater than the learning rate of the second stage of training.

[0287] The question-and-answer interaction device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not impose specific limitations.

[0288] The question-and-answer interaction device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0289] The question-and-answer interaction device provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0290] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described question-and-answer interaction method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0291] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0292] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0293] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0294] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0295] The processor 810 is used to input the question image and question text into the trained question-answering interaction model, and through the question-answering interaction model, determine the candidate image region in the question image that is related to the question text based on the question text; and through the question-answering interaction model, output the question answer based on the candidate image region and the question text.

[0296] This application provides an electronic device. After the electronic device inputs a question image and question text into a trained question-and-answer interaction model, the electronic device can use the question-and-answer interaction model to determine candidate image regions in the question image that are related to the question text, i.e., select a portion of the image region in the question image that is related to the question text. In this way, the electronic device can perform calculations based on this portion of the image region and the question text, instead of performing calculations based on the entire image region of the question image and the question text. Therefore, the question-and-answer interaction model can avoid performing calculations on image regions in the question image that are unrelated to the question text. This reduces the computational load of the question-and-answer interaction model and ensures that the model operates based on the portion of the image region that is related to the question text. This improves the efficiency of outputting the question answer while ensuring the accuracy of the output. Thus, it improves the efficiency of text-based dialogue while ensuring its performance.

[0297] In some embodiments of this application, the question-answering interaction model includes a cross-modal fusion processor and a region proposal network. Specifically, the processor 810 is used to determine an initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text, using the cross-modal fusion processor. This initial intent vector represents the question intent of the question text. The processor also uses the region proposal network to determine candidate image regions based on the initial intent vector and the question image features.

[0298] In some embodiments of this application, the question-and-answer interaction model further includes a global image encoder; the processor 810 is further configured to perform downsampling processing on the question image through the question-and-answer interaction model to obtain a thumbnail before determining the initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text through the cross-modal fusion processor of the question-and-answer interaction model; and to obtain the image features corresponding to the thumbnail through the global image encoder, and determine the image features corresponding to the thumbnail as the question image features.

[0299] In some embodiments of this application, the processor 810 is specifically used to calculate an initial intent vector based on the features of the problem image and the features of the problem text, using an attention mechanism.

[0300] In some embodiments of this application, the processor 810 is specifically configured to perform regression prediction based on an initial intent vector and problem image features to obtain at least one predicted location information and at least one confidence level, wherein each confidence level is used to characterize the probability that a predicted location information is the location information of an image region in the problem image related to the problem text; and to perform non-maximum suppression deduplication processing on the at least one predicted location information; and to determine, from the deduplicated predicted location information, the candidate location information with the highest confidence level and an overlap rate with the historical exploration area lower than the overlap rate threshold; and to determine the image region in the problem image corresponding to the candidate location information as the candidate image region.

[0301] In some embodiments of this application, the question-and-answer interaction model includes a local image encoder and an inference model; the processor 810 is specifically used to obtain image region features corresponding to candidate image regions through the local image encoder; and to infer the answer to the question based on the image region features and the question text features corresponding to the question text through the inference model.

[0302] In some embodiments of this application, the question-and-answer interaction model further includes a decision unit; the processor 810 is further configured to, before inferring and outputting the question answer through the inference model based on the image region features and the question text features corresponding to the question text, perform decision evaluation through the decision unit based on the image region features and the question text features, and output evaluation result information, which indicates whether the candidate image region is all image regions in the question image related to the question text; specifically, the processor 810 is configured to, when the evaluation result information indicates that the candidate image region is all image regions in the question image related to the question text, perform inference and output the question answer through the inference model based on the image region features and the question text features.

[0303] In some embodiments of this application, the question-and-answer interaction model further includes a region proposal network; the processor 810 is further configured to, when the evaluation result information indicates that the candidate image region is not all image regions in the question image that are related to the question text, generate an updated intent vector based on the image region features and the initial intent vector through the question-and-answer interaction model, wherein the initial intent vector is used to characterize the question intent of the question text; and determine the remaining image regions in the question image that are related to the question text based on the updated intent vector and the question image features through the region proposal network.

[0304] In some embodiments of this application, the processor 810 is further configured to: obtain a training sample set, which includes sample question images, sample question text, sample candidate image regions, and sample question answers, before inputting the question image and question text into the trained question-answering interaction model and determining, based on the question text, candidate image regions in the question image related to the question text; input the training sample set into a preset model, which determines, based on the sample question text, predicted candidate image regions in the sample question image related to the sample question text; output a predicted question answer based on the predicted candidate image regions and sample question text; determine a first loss value based on the predicted candidate image regions and sample candidate image regions, and determine a second loss value based on the predicted question answer and sample question answer; and perform a first stage of training on the preset model based on the first loss value, and perform a second stage of training on the preset model after the first stage of training based on the second loss value, thereby obtaining a trained question-answering interaction model.

[0305] In some embodiments of this application, the preset model includes a preset fusion unit and a preset proposal network; the processor 810 is specifically used to determine an initial intent vector of a sample based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, the initial intent vector of which is used to characterize the question intent of the sample question text; and to determine a predicted candidate image region based on the initial intent vector of the sample and the sample question image features through the preset proposal network; the processor 810 is specifically used to perform a first stage of training on the preset fusion unit and the preset proposal network based on a first loss value.

[0306] In some embodiments of this application, the preset model further includes a preset global image encoder; the processor 810 is further configured to, before determining the initial intent vector of the sample by the preset fusion device based on the sample question image features corresponding to the sample question image and the sample question text features corresponding to the sample question text, perform downsampling processing on the sample question image through the preset model to obtain a sample thumbnail; and obtain the image features corresponding to the sample thumbnail through the preset global image encoder, and determine the image features corresponding to the sample thumbnail as the sample question image features; the processor 810 is further configured to perform a first stage of training on the preset fusion device, the preset proposal network and the preset global image encoder based on the first loss value.

[0307] In some embodiments of this application, the preset model includes a preset local image encoder and a preset inference model; the processor 810 is specifically used to obtain the sample image region features corresponding to the predicted candidate image region through the preset local image encoder; and to perform inference based on the sample image region features and sample question text features through the preset inference model, and output the predicted question answer; the processor 810 is specifically used to perform a second stage of training on the preset local image encoder and the preset inference model based on the second loss value.

[0308] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0309] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0310] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0311] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described question-and-answer interaction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0312] The processor mentioned above is the processor in the electronic device described in the above embodiments. The readable storage medium mentioned above includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0313] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described question-and-answer interaction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0314] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0315] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the question-and-answer interaction method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0316] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0317] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0318] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A question-and-answer interaction method, characterized in that, include: The question image and question text are input into the trained question-answering interaction model. The question-answering interaction model determines candidate image regions in the question image that are related to the question text based on the question text. The question-and-answer interaction model outputs the answer to the question based on the candidate image region and the question text.

2. The method according to claim 1, characterized in that, The question-and-answer interaction model includes a cross-modal fusion engine and a regional proposal network; The step of determining candidate image regions related to the question text in the question image based on the question text through the question-answering interaction model includes: The cross-modal fusion processor determines an initial intent vector based on the problem image features corresponding to the problem image and the problem text features corresponding to the problem text. The initial intent vector is used to characterize the problem intent of the problem text. The candidate image regions are determined using the region proposal network based on the initial intent vector and the problem image features.

3. The method according to claim 2, characterized in that, The question-and-answer interaction model also includes a global image encoder; Before determining the initial intent vector based on the question image features corresponding to the question image and the question text features corresponding to the question text using the cross-modal fusion processor of the question-answering interaction model, the method further includes: The question image is downsampled using the question-and-answer interaction model to obtain a thumbnail; The global image encoder obtains the image features corresponding to the thumbnail and determines the image features corresponding to the thumbnail as the problem image features.

4. The method according to claim 2, characterized in that, The step of determining the initial intent vector based on the problem image features corresponding to the problem image and the problem text features corresponding to the problem text includes: Based on the attention mechanism, the initial intent vector is calculated according to the features of the question image and the features of the question text.

5. The method according to claim 2, characterized in that, The step of determining the candidate image region based on the initial intent vector and the problem image features includes: Regression prediction is performed based on the initial intent vector and the features of the problem image to obtain at least one predicted location information and at least one confidence level. Each confidence level is used to characterize the probability that a predicted location information is the location information of an image region in the problem image that is related to the problem text. The at least one predicted location information is subjected to non-maximum suppression and deduplication processing; From the deduplicated predicted location information, the candidate location information with the highest confidence and an overlap rate with the historical exploration area below the overlap rate threshold is determined. The image region in the problem image that corresponds to the candidate location information is determined as the candidate image region.

6. The method according to claim 1, characterized in that, The question-and-answer interaction model includes a local image encoder and an inference model; The step of outputting the question answer based on the candidate image region and the question text through the question-answering interaction model includes: The image region features corresponding to the candidate image region are obtained through the local image encoder. The reasoning model is used to infer the answer to the question based on the features of the image region and the features of the question text.

7. The method according to claim 6, characterized in that, The question-and-answer interaction model also includes a decision-maker; Before the step of reasoning based on the image region features and the question text features corresponding to the question text using the reasoning model to output the answer to the question, the method further includes: The decision-maker performs decision evaluation based on the image region features and the question text features, and outputs evaluation result information. The evaluation result information indicates whether the candidate image region is all image regions in the question image that are related to the question text. The step of inferring the answer to the question using the inference model based on the image region features and the question text features corresponding to the question text includes: If the evaluation result information indicates that the candidate image region is all image regions in the question image that are related to the question text, the reasoning model is used to infer the answer to the question based on the features of the image region and the features of the question text.

8. The method according to any one of claims 1 to 7, characterized in that, Before inputting the question image and question text into the trained question-answering interaction model, and determining, through the question-answering interaction model, candidate image regions in the question image related to the question text based on the question text, the method further includes: Obtain a training sample set, which includes sample question images, sample question text, sample candidate image regions, and sample question answers, wherein the sample candidate image regions are image regions in the sample question images that are related to the sample question text. The training sample set is input into a preset model, and the preset model determines the predicted candidate image region in the sample question image that is related to the sample question text based on the sample question text. Based on the preset model, the predicted question answer is output according to the predicted candidate image region and the sample question text; A first loss value is determined based on the predicted candidate image region and the sample candidate image region, and a second loss value is determined based on the predicted question answer and the sample question answer; The preset model is trained in the first stage based on the first loss value, and then trained in the second stage based on the second loss value to obtain the trained question-answering interaction model.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 8.

Images