Question answering methods and training methods for question answering models

Abstract

Embodiments of this disclosure provide a question answering method and a question answering model training method, the question answering method comprising the steps of: obtaining question information to be answered; and inputting the question information into a question answering model to obtain answer information corresponding to the question information, the question answering model being obtained by training on predicted answer information, the predicted answer information being obtained by predicting answer information corresponding to a question sample using a pre-trained question answering model, the pre-trained question answering model being obtained by training on predicted image-text matching results, the predicted image-text matching results being obtained by matching a training target image with target semantic information using the pre-trained question answering model, and the target semantic information being generated based on an image description text corresponding to the training target image and a dialogue text associated with the training target image.

Description

Technical Field

[0001] (Cross - reference to related applications) This disclosure claims the priority of a Chinese patent application filed with the Chinese Patent Office on April 11, 2023, with the application number 202310412430.3 and the application title "Question - Answering Method and Training Method for Question - Answering Model", and all of its contents are incorporated into this disclosure by reference.

[0002] Embodiments of this disclosure relate to the technical field of computers, and in particular, to question - answering methods.

Background Art

[0003] With the enrichment of multimedia data, task - oriented dialogue systems, such as intelligent customer service in the field of e - commerce, have attracted wide attention. A task - oriented dialogue system can help users obtain answers to questions through self - service. As the requirements of users for task - oriented dialogue systems increase, the dialogue system needs to determine whether to return an answer to the user in a modal, that is, whether to answer with an image or text, and then present the answer content. However, due to the difficulty of collecting multi - modal dialogue data, the training data is scarce, the training of the dialogue generation model is difficult, and the application effect of the dialogue generation model is low.

Summary of the Invention

[0004] In view of this, embodiments of this disclosure provide a question - answering method. To solve the technical drawbacks existing in the prior art, one or more embodiments of this disclosure relate to another question - answering method, a training method for a question - answering model, a text question - answering method, a question - answering device, a computing device, a computer - readable storage medium, and a computer program.

[0005] According to the first aspect of the embodiments of this disclosure, the step of obtaining question information to be answered, The steps include inputting the aforementioned question information into a question answering model to obtain answer information corresponding to the aforementioned question information, The present invention provides a question answering method in which the question answering model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question answering model, the pre-trained question answering model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the image to be trained with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained.

[0006] According to a second embodiment of the embodiments of this disclosure, another question answering method applied to a cloud-side device, The steps include receiving a question response request containing question information sent from the terminal device, The steps include: inputting the aforementioned question information into a question answering model to obtain answer information corresponding to the aforementioned question information; The step includes transmitting the aforementioned response information to the terminal device, The present invention provides a question answering method in which the question answering model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question answering model, the pre-trained question answering model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the image to be trained with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained.

[0007] According to a third aspect of the embodiments of this disclosure, a method for training a question answering model applied to a cloud-side device, A step of obtaining a pre-training sample that includes an image to be trained, an image description text corresponding to the image to be trained, and dialogue text associated with the image to be trained. A step of training the pre-trained question answering model based on the predicted image-text matching results corresponding to the pre-trained samples, wherein the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text and the dialogue text. The steps include obtaining a training image and a sample question containing dialogue text associated with the training image, The steps include: inputting the aforementioned question sample into the pre-trained question-answering model, obtaining predicted answer information output from the pre-trained question-answering model, and training the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model; The present invention provides a method for training a question-answering model, which includes the step of transmitting the model parameters of the question-answering model to a terminal-side device.

[0008] According to a fourth aspect of the embodiments of this disclosure, a text question answering method applied to a terminal device, Steps include receiving text question information, A step of inputting the aforementioned text question information into a question answering model to obtain text answer information corresponding to the aforementioned text question information, wherein the question answering model is obtained by the training method of the question answering model, The present invention provides a text question answering method, which includes the step of displaying the aforementioned text answer information.

[0009] According to a fifth aspect of the embodiments of this disclosure, A question acquisition module configured to obtain information about the question to be answered, The present invention provides a question answering device comprising: an answer acquisition module configured to input the aforementioned question information into a question answering model to obtain answer information corresponding to the aforementioned question information, wherein the question answering model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question answering model, the pre-trained question answering model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the image to be trained with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained.

[0010] According to a sixth aspect of the embodiments of this disclosure, Equipped with memory and a processor, The present invention provides a computing device in which the memory is configured to store commands that can be executed by a computer, the processor is configured to execute commands that can be executed by the computer, and when a command that can be executed by the computer is executed by the processor, the steps of the above method are realized.

[0011] According to a seventh embodiment of the present disclosure, a computer-readable storage medium is provided which stores a computer-executable command that, when executed by a processor, enables the steps of the above method to be realized.

[0012] According to an eighth aspect of the embodiments of this disclosure, a computer program is provided that, when executed on a computer, causes the computer to perform the steps of the above method.

[0013] In a question answering method provided in one embodiment of the present disclosure, question information to be answered is acquired, the question information is input to a question answering model to obtain answer information corresponding to the question information, the question answering model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question answering model, the pre-trained question answering model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the image to be trained with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained.

[0014] In the embodiments of this disclosure, answer information is obtained by inputting question information to be answered into a question-answering model, and the question-answering model can be realized by training a pre-trained question-answering model, which predicts answer information corresponding to a question sample, and trains the question-answering model with the predicted answer information, and the pre-trained question-answering model is obtained by training based on the predicted image-text matching result, and the predicted image-text matching result is obtained by matching the training target image with target semantic information using the pre-trained question-answering model, where the target semantic information is generated based on the image description text corresponding to the training target image and the dialogue text associated with the training target image. With this method, it is possible to train the question-answering model to learn the target semantic text more favorably using a small amount of dialogue text and training target images, and further train the question-answering model to output dialogue content according to the dialogue text, so that the question-answering model can learn semantic knowledge in the dialogue text more favorably without using a large amount of dialogue text as training data, thereby making the answer information of the trained question-answering model more accurate and improving the application effect of the model. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram illustrating an application scenario of the question answering method provided in one embodiment of the present disclosure. [Figure 2] This is a flowchart of a question answering method provided in one embodiment of the present disclosure. [Figure 3a] This is a schematic diagram of the first stage of training of a question-answering model in a question-answering method provided in one embodiment of the present disclosure. [Figure 3b] This is a schematic diagram of the second stage of training of the question answering model in a question answering method provided in one embodiment of the present disclosure. [Figure 3c]It is a schematic diagram of the third-stage training of a question-and-answer model in the question-and-answer method provided in an embodiment of the present disclosure. [Figure 4] It is a flowchart of another question-and-answer method provided in an embodiment of the present disclosure. [Figure 5] It is a flowchart of a training method of a question-and-answer model provided in an embodiment of the present disclosure. [Figure 6] It is a flowchart of a text question-and-answer method provided in an embodiment of the present disclosure. [Figure 7] It is a schematic diagram of the configuration of a question-and-answer device provided in an embodiment of the present disclosure. [Figure 8] It is a block diagram of the configuration of a computing device provided in an embodiment of the present disclosure.

Embodiments for Carrying Out the Invention

[0016] To fully understand the present disclosure, many specific details are set forth in the following description. However, the present disclosure can be implemented in many other forms different from those described herein, and those skilled in the art can similarly generalize without departing from the spirit of the present disclosure. Therefore, the present disclosure is not limited by the specific implementations disclosed below.

[0017] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a kind", "the" and "said" used in one or more embodiments of the present disclosure and the appended claims are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be understood that the term "and / or" used in one or more embodiments of the present disclosure means any or all possible combinations of one or more related listed items.

[0018] In one or more embodiments of this disclosure, terms such as "First," "Second," etc., may be used to describe various types of information, but it should be understood that this information is not limited to these terms. These terms are used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this disclosure, "First" may be called "Second," and similarly, "Second" may be called "First." Depending on the context, the word "case" as used herein may be interpreted as "at the time," "on the occasion," or "according to the decision."

[0019] Furthermore, it should be noted that all user information (including, but not limited to, user device information, user personal information, etc.) and data (including, but not limited to, data for analysis, stored data, displayed data, etc.) relating to one or more embodiments of this disclosure are information and data authorized by the user or fully authorized by each party concerned, and the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding entry points for the user to choose to authorize or deny access will be provided.

[0020] First, we will interpret any nouns relating to one or more embodiments of this disclosure.

[0021] In the field of artificial intelligence, "multimodal" generally refers to perceived information such as images, text, and audio.

[0022] With the increasing richness of multimedia data, conversational communication on the internet is no longer limited to pure text format. Therefore, introducing multimodal dialogue will be an unstoppable trend for future intelligent customer service. In multimodal dialogue, the model is required to first determine whether to respond modally to the user, i.e., with an image or text; secondly, to have the ability to search for contextually appropriate answers in both image and text searches; simultaneously, to have user state tracking capabilities to track user needs; and finally, to have text generation capabilities to produce smooth and natural answers. However, in today's data-driven world, multimodal dialogue data is difficult to collect due to a series of issues such as privacy, resulting in relatively small quantities and making model training difficult. On the other hand, multimodal non-dialogue data is available in large quantities. Therefore, the embodiments of this disclosure propose an incremental combinatorial expert multimodal dialogue pre-training model that utilizes multimodal non-dialogue data to model the identification and matching capabilities of text experts and image experts. By inheriting the above expert capabilities and introducing multimodal dialogue data, a dialogue expert model is constructed, thereby achieving a gradual improvement in model capabilities.

[0023] This disclosure provides a question answering method, and this disclosure also relates to other question answering methods, methods for training question answering models, text question answering methods, question answering devices, computing devices, and computer-readable storage media, which are described in detail one by one in the following embodiments.

[0024] Please refer to Figure 1, which shows a schematic diagram of an application scenario for the question-answering method according to one embodiment of this disclosure.

[0025] It should be explained that the question-answering method provided in this embodiment can be applied to various application scenarios such as question-answering in intelligent customer service, and the specific application scenarios can be adapted to different project needs, but this embodiment is not limited to these. For ease of understanding, an example of an application scenario in this embodiment where a user uses intelligent customer service will be described.

[0026] In actual application, the user sends question information requiring an answer to an intelligent customer service, where the question information may be understood as information in different application scenarios and includes, but is not limited to, text information, image information, etc. Since the intelligent customer service uses the question answering model in the question answering method provided in this embodiment, the question answering model can output answer information corresponding to the question information after receiving the question information sent by the user, where the answer information may be understood as the content of the response that has been fed back to the question information and includes, but is not limited to, text information, image information, etc. Furthermore, the user can receive the content of the response that has been fed back to the intelligent customer service. It should be noted that the question answering model needs to be trained in advance before use, and specifically, in order to improve the training effect of the question answering model, in this embodiment the model can be trained in stages, which is achieved by combining different expert networks, constructing the question answering model by introducing multimodal dialogue data, and achieving a gradual improvement in the model's capabilities.

[0027] Referring to Figure 2, which shows a flowchart of a question answering method according to one embodiment of the present disclosure, the method specifically includes the following steps 202 and 204.

[0028] In step 202, obtain the question information to be answered.

[0029] In actual application, the implementing entity can obtain question information to be answered. This question information can be understood as the content of a user's question awaiting an answer, and includes, but is not limited to, text content, image content, text and image content, etc.

[0030] In step 204, the question information is input into the question answer model to obtain answer information corresponding to the question information.

[0031] The question-answering model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question-answering model, the pre-trained question-answering model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question-answering model, and the target semantic information is generated based on the image description text corresponding to the training target image and the dialogue text associated with the training target image.

[0032] In actual application, the implementing entity can input question information into a question-answering model to obtain corresponding answer information. Here, the answer information may be understood as corresponding answer content, including text content and / or image content, determined for the question information. It should be explained that the question-answering model provided in this embodiment can be realized by training a pre-trained question-answering model, which is trained using predicted answer information output by the pre-trained question-answering model based on question samples. Furthermore, the pre-trained question-answering model can be obtained by training with a training target image and target semantic information. Here, the target semantic information may be understood as being generated from the image description text corresponding to the training target image and the dialogue text associated with the training target image.

[0033] To facilitate understanding, the training process of the question-answering model will be described in the following embodiment. The training process of the question-answering model may include three stages of training. That is, first, the initial question-answering model is trained in the first stage using images and image annotation text, then the pre-trained question-answering model is trained in the second stage using images, image annotation text and dialogue text, and finally, the question-answering model is trained in the third stage using images and dialogue text. It should be noted that the initial question-answering model, the pre-trained question-answering model, and the question-answering model all refer to the same question-answering model, and it may be understood that the model output capabilities and effects of the question-answering model differ at different training stages. The training process provided in this embodiment first trains the pre-trained question-answering module, then trains the question-answering model, i.e., performs the second and third stages of the training process. Specifically, before the step of inputting the question information into the question-answering model to obtain answer information corresponding to the question information, A step of obtaining a pre-training sample that includes an image to be trained, an image description text corresponding to the image to be trained, and dialogue text associated with the image to be trained. A step of training the pre-trained question answering model based on the predicted image-text matching results corresponding to the pre-trained samples, wherein the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text and the dialogue text. The steps include obtaining a training image and a sample question containing dialogue text associated with the training image, The method further includes the steps of inputting the aforementioned question sample into the pre-trained question-answering model, obtaining predicted answer information output from the pre-trained question-answering model, and training the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model.

[0034] Here, the pre-training sample may be understood as sample data for training a pre-trained question-answering model, and includes the image to be trained, the image description text corresponding to the image to be trained, and the dialogue text associated with the image to be trained. The image to be trained may be understood as any type of training image, and is not limited to any particular type of image. The image description text corresponding to the image to be trained may be understood as text information describing the image content in the image to be trained, for example, a description of the object content and state in the image. The dialogue text associated with the image to be trained may be understood as relevant dialogue information considered regarding the image to be trained, and this dialogue information may be understood as dialogue data between different subjects, but this embodiment does not limit the form of the dialogue content or the subjects of the dialogue.

[0035] In actual application, a pre-trained question-answering model can be trained using the predicted image-text matching results corresponding to pre-training samples. Here, the predicted image-text matching results can be obtained by matching the training target image with the target semantic information using the information matching layer (FFN) in the pre-trained question-answering model. It should be explained that the target semantic information may be understood as obtained by performing vector matching between the image description text and the dialogue text, that is, the image description text allows the pre-trained question-answering model to better understand the semantic information in the dialogue text. Furthermore, after the training of the pre-trained question-answering model is completed, more question samples can be obtained. These question samples may be understood as training samples used in the question-answering model, and include the training target image and the dialogue text associated with the training target image. Next, the question samples can be input into the pre-trained question-answering model to obtain predicted answer information output from the information generation module in the pre-trained question-answering model. The pre-trained question-answering model can then be continued to be trained using the predicted answer information, and finally, a question-answering model with excellent training effectiveness can be obtained.

[0036] In the question-answering model training method according to this embodiment, a pre-trained question-answering model is trained, and then the pre-trained question-answering model is continued to be trained in the next stage, thereby giving the trained question-answering model a stepwise learning ability.

[0037] Furthermore, in the third stage of training the question-answering model, training can be performed based on predicted answer information output from the pre-trained question-answering model, and this embodiment can perform model training using an autoregressive method. Specifically, the step of training the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model is: The steps include: calculating the autoregressive loss value based on the aforementioned predicted response information; The process includes the step of training the pre-trained question answering model based on the autoregressive loss value to obtain the question answering model.

[0038] In actual application, the implementing entity can obtain a question-answering model by calculating an autoregressive loss value using the predicted answer information output from the pre-trained question-answering model, and then training the pre-trained question-answering model with the autoregressive loss value. In this embodiment, the determination of the autoregressive loss value can be made by referring to Equation 1 below.

number

number

[0039] Furthermore, in the stage of training the pre-trained question-answering model, i.e., the second stage, it is necessary to train the model to be able to match images with dialogue text, and furthermore, it is necessary to train the image-text alignment layer (FFN) in the pre-trained question-answering model. Specifically, the step of training the pre-trained question-answering model based on the predicted image-text matching results corresponding to the pre-trained samples is as follows: The steps include: 1. Using the pre-trained question-answering model, image processing is performed on the image to be trained in the pre-trained sample to obtain an image vector. The steps include: using the pre-trained question-answering model to process the dialogue text associated with the image to be trained in the pre-training sample to obtain a dialogue vector; A step of generating a predicted image-text matching result based on the image vector and the dialogue vector, The process includes the steps of determining an image-text alignment loss value based on the predicted image-text matching result, and training the pre-trained question-answering model based on the image-text alignment loss value.

[0040] In actual application, during the training phase of the pre-trained question-answering model, the target image from the pre-training samples is input to the image processing layer (FFN) of the pre-trained question-answering model to obtain an image vector. Next, the dialogue text associated with the target image from the pre-training samples is input to the dialogue processing layer (FFN) of the pre-trained question-answering model to obtain a dialogue vector. After performing vector processing on both the image and the text, the image vector and the dialogue vector are matched to generate a predicted image-text matching result. The loss value of the image-text alignment layer (FFN) is determined based on the predicted image-text matching result, and the pre-trained question-answering model can be trained using this loss value.

[0041] This method utilizes multimodal data of images and dialogues to train a model on matching information between images and dialogue text, thereby giving the pre-trained question-answering model the ability to match images and dialogues.

[0042] Furthermore, in the stage of training the pre-trained question-answering model, in order to improve the training effect of the pre-trained question-answering model, in addition to training using image-text alignment loss values, loss values ​​corresponding to the images to be trained and loss values ​​corresponding to the dialogue text may also be referenced. Specifically, the step of training the pre-trained question-answering model based on the image-text alignment loss values ​​is: The steps include: calculating the dialogue annotation alignment loss value based on the image vector, and calculating the image dialogue loss value based on the dialogue vector; The process includes the step of training the pre-trained question-answering model based on the dialogue annotation alignment loss value, the image dialogue loss value, and the image text alignment loss value.

[0043] In actual application, the implementing entity can calculate the corresponding dialogue annotation alignment loss value using the image vector obtained by encoding the training target image with the image processing module, and further, it can calculate the corresponding image dialogue loss value using the dialogue vector obtained by encoding the dialogue text with the dialogue processing module. Finally, the pre-trained question answering model is continuously trained using the dialogue annotation alignment loss value, image dialogue loss value, and image text alignment loss value. In this embodiment, Equation 2 below can be used as an example training objective.

number

number

number

number

number

number

number

number

[0044] Furthermore, before training the pre-trained question-answering model, a first stage may be added to train the initial question-answering model, thereby giving the initial question-answering model the ability to match images and text. Specifically, before the aforementioned step of obtaining pre-training samples, The steps include obtaining an initial training sample that includes an image to be trained and image annotation text corresponding to the image to be trained, The steps include: obtaining an image vector by processing the training target image using an initial question response model; The steps include: using the initial question response model to process the image annotation text corresponding to the image to be trained to obtain a text annotation vector; The steps include determining a predicted image description text based on the image vector and the text annotation vector, The further step includes training the initial question-answering model based on the predicted image description text.

[0045] In actual application, initial training samples are obtained, and an initial question-answering model is trained based on these initial training samples. Specifically, the images to be trained in the initial training samples are input to the image processing layer (FFN) of the initial question-answering model to obtain image vectors. Next, the image annotation text in the initial training samples is input to the image annotation layer (FFN) of the initial question-answering model to obtain text annotation vectors. Furthermore, the predicted image description text is determined using the image vectors and text annotation vectors, the model is trained to match images and text, and then the initial question-answering model is trained using the predicted image description text. It should be noted that the image annotation text may be understood as manual image annotation information for the images to be trained, or as information from a basic image annotation model, and this embodiment is not limited to this.

[0046] Furthermore, this embodiment further provides that the loss value is determined by the predicted image description text and the initial question answering model is trained. Specifically, the step of training the initial question answering model based on the predicted image description text is: The steps include: calculating an image text annotation loss value based on the image vector and the text annotation vector; The steps include: calculating an image-text alignment loss value based on the predicted image description text; The process includes the step of training the initial question response model using the image-text annotation loss value and the image-text alignment loss value.

[0047] In actual application, the image processing module and image annotation module in the initial question answering model can encode the image and text respectively, and then calculate the image-text annotation loss value and image-text alignment loss value, respectively. Furthermore, the initial question answering model is trained using the image-text annotation loss value and image-text alignment loss value. In this embodiment, the following training objectives can be referenced.

number

number

number

number

[0048] Based on this, in the first stage of training the question-answering model, by calculating the image-text annotation loss value and the image-text alignment loss value to train the initial question-answering model, it becomes easier to continue training the question-answering model's ability to match multimodal data of dialogue text and images based on the initial question-answering model, thereby improving the question-answering model's multimodal data processing capability.

[0049] Furthermore, after obtaining a question-answering model through three stages, the implementing entity can obtain user feedback on the answer information by sending the answer information corresponding to the question information output from the question-answering model to the user, thereby enabling a process of fine-tuning the question-answering model. Specifically, after the step of obtaining the answer information corresponding to the question information, The steps include sending the aforementioned response information to the user, The steps include receiving the user's feedback result regarding the response information, The further step includes adjusting the model parameters in the question answering model based on the feedback results.

[0050] In actual application, the implementing entity can send response information to the user, and after receiving the response information, the user can provide corresponding feedback or evaluation based on the specific content of the response information and send the feedback results to the implementing entity. The implementing entity can then fine-tune the model parameters in the question answering model based on the received feedback results, thereby obtaining a question answering model with better application effectiveness.

[0051] In summary, the question answering method provided in this embodiment completes the answer to the question information using a question answering model. In the training process of the question answering model, a three-stage training process is employed to continuously learn image-text matching ability, image-dialogue matching ability, and answer generation ability from the question answering model. In the specific training process, each processing module in the question answering model is trained using an incremental combinatorial expert network method, introducing the ability to model dialogue data in a multimodal dialogue, and achieving a gradual improvement in the model's capabilities.

[0052] Next, refer to Figures 3a, 3b, and 3c, which show schematic diagrams of the training of each stage of the question answering model in the question answering method provided in one embodiment of the present disclosure.

[0053] It should be explained that in each stage of training, the question-answering model includes an L-layer configuration, which may include five different expert networks in each layer, such as image annotation, dialogue context, image, image-text alignment, and text generation, and each layer also has one shared multi-head self-attention mechanism layer.

[0054] Figure 3a shows a schematic diagram of the first stage of training of a question answering model in a question answering method provided in one embodiment of the present disclosure.

[0055] In actual application, in the first stage of the pre-training process shown in Figure 3a, the image and the image annotation text are encoded and input into the subsequent LF layer to complete the matching of the image annotation text and the image. Furthermore, in the F layer, an image-text alignment network is used to realize the first stage of the pre-trained question answering model.

[0056] Figure 3b shows a schematic diagram of the second stage of training of the question answering model in a question answering method provided in one embodiment of the present disclosure.

[0057] In actual application, in the second stage of the pre-training process shown in Figure 3b, the model's inputs are images, image annotation text, and dialogue text. First, these are input to the LF layer to complete the matching ability between image annotation text and dialogue text to obtain the target semantic text. Then, these are input to the F layer of the model to match with the images, completing the image-text alignment training process.

[0058] Figure 3c shows a schematic diagram of the third stage of training of a question answering model in a question answering method provided in one embodiment of the present disclosure.

[0059] In actual application, in the third stage of the pre-training process shown in Figure 3c, the model's input consists of dialogue text and images. Furthermore, since the question-answering model has acquired the ability to match dialogue and images in the previous two stages of training, in the third stage of pre-training, two networks in the LF layer, such as dialogue context and image, are used to understand the matching between dialogue and images. Additionally, the text generation network in the F layer of the model can be trained to improve the question-answering model's ability to generate answer information.

[0060] It should be explained that when applying a question-answering model to a multimodal dialogue task, it may include a variety of execution tasks, such as: 1. Multimodal intent recognition, specifically predicting whether or not image sharing is necessary in the current round. 2. Multimodal search, searching for appropriate images and text from the dialogue history and sending replies. 3. Multimodal dialogue tracking, tracking the user's essential dialogue needs. 4. Multimodal text generation, generating appropriate answers from the dialogue history.

[0061] Based on this, the training process for the question-answering model provided in this embodiment divides each layer of the model into multiple expert networks, and combines different expert networks to perform specific tasks. By dividing the model into smaller modules, the training of the model becomes more controllable and its interpretability is strengthened. At the same time, since the parameters of expert networks not used in specific tasks are not activated, the inference speed of the model can be greatly accelerated and the parameter utilization rate can be increased. Furthermore, by assigning different semantic knowledge to each expert network, each modal and each data type has traceability in the flow within the model, strengthening the interpretability and generalization ability of the model.

[0062] Referring to Figure 4, which shows a flowchart of another question answering method according to one embodiment of the present disclosure, the following steps 402, 404 and 406 are specifically included.

[0063] It should be explained that the question answering method provided in this embodiment is applied to the cloud-side device; that is, after receiving question information from the terminal-side device, the question answering model on the cloud-side device can provide an answer to the question information.

[0064] In step 402, the terminal device receives a question response request containing question information.

[0065] In actual application, the cloud-side device can receive question-answer requests sent from the terminal-side device and obtain the question information contained in those requests. It should be noted that the specific content and format of the question information are not limited to this embodiment.

[0066] In step 404, the question information is input into the question answer model to obtain answer information corresponding to the question information.

[0067] Here, the question answering model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question answering model, the pre-trained question answering model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text corresponding to the training target image and the dialogue text associated with the training target image.

[0068] In actual application, the cloud-side device inputs question information into the question-answering model and obtains answer information output from the question-answering model. Here, the training process for the question-answering model on the cloud-side device can be referred to the training process described in the above embodiment, i.e., the three-stage training of the question-answering model, and a detailed explanation is omitted here.

[0069] In step 406, the response information is transmitted to the terminal device.

[0070] Furthermore, after obtaining the response information, the cloud-side device can feed that information back to the terminal-side device.

[0071] Based on this, this embodiment provides a question answering method applicable to the cloud-side device, enabling question answers to be provided for question information transmitted from the terminal-side device. At the same time, since the question answering model is obtained by training on the cloud-side device, the computational resources and memory space of the terminal-side device can be saved, providing users with a more favorable usage effect.

[0072] Referring to Figure 5, which shows a flowchart of a training method for a question-answering model according to one embodiment of the present disclosure, the method specifically includes the following steps 502, 504, 506, 508, and 510.

[0073] It should be noted that the training method for the question-answering model provided in this embodiment may be applied to terminal-side devices or cloud-side devices, and this embodiment is not limited to either.

[0074] In step 502, a pre-training sample is obtained that includes the image to be trained, the image description text corresponding to the image to be trained, and the dialogue text associated with the image to be trained.

[0075] In step 504, the pre-trained question-answering model is trained based on the predicted image-text matching results corresponding to the pre-trained samples. The predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question-answering model, and the target semantic information is generated based on the image description text and the dialogue text.

[0076] In step 506, a sample question is obtained that includes the image to be trained and the dialogue text associated with the image to be trained.

[0077] In step 508, the sample questions are input into the pre-trained question-answering model, predicted answer information is obtained from the pre-trained question-answering model, and the pre-trained question-answering model is trained based on the predicted answer information to obtain a question-answering model.

[0078] In step 510, the model parameters of the question answering model are transmitted to the terminal device.

[0079] Here, the pre-training sample may be understood as sample data for training a pre-trained question-answering model, and includes the image to be trained, the image description text corresponding to the image to be trained, and the dialogue text associated with the image to be trained. The image to be trained may be understood as any type of training image, and is not limited to any particular type of image. The image description text corresponding to the image to be trained may be understood as text information describing the image content in the image to be trained, for example, a description of the object content and state in the image. The dialogue text associated with the image to be trained may be understood as relevant dialogue information considered regarding the image to be trained, and this dialogue information may be understood as dialogue data between different subjects, but this embodiment does not limit the form of the dialogue content or the subjects of the dialogue.

[0080] In actual application, a pre-trained question-answering model can be trained using the predicted image-text matching results corresponding to pre-training samples. Here, the predicted image-text matching results can be obtained by matching the training target image with the target semantic information using the information matching layer (FFN) in the pre-trained question-answering model. It should be explained that the target semantic information may be understood as obtained by performing vector matching between the image description text and the dialogue text. That is, the image description text can be understood as enabling the pre-trained question-answering model to better understand the semantic information in the dialogue text. Furthermore, after the training of the pre-trained question-answering model is completed, more question samples can be obtained. These question samples may be understood as training samples used in the question-answering model, and include the training target image and the dialogue text associated with the training target image. Next, the question samples can be input into the pre-trained question-answering model to obtain predicted answer information output from the information generation layer (FFN) in the pre-trained question-answering model. The pre-trained question-answering model can then be continued to be trained using the predicted answer information, and finally, a question-answering model with excellent training effectiveness can be obtained. Next, by transmitting the model parameters of the question-answering model to the terminal device, the terminal device can construct a question-answering model with superior usability based on the model parameters, thereby increasing the effectiveness of using the question-answering model on the terminal device.

[0081] Based on this, a pre-trained question-answering model that matches dialogue text with images can be realized through training. Furthermore, by training this pre-trained question-answering model, image-text alignment between dialogue text and images can be achieved, and answer information can be generated and answers can be presented. The question-answering model's answer effectiveness can be improved by training its dialogue generation capabilities using multimodal data.

[0082] Referring to Figure 6, which shows a flowchart of a text question answering method according to one embodiment of the present disclosure, the method specifically includes the following steps 602, 604, and 606.

[0083] To clarify and make it easier to understand, the text question answering method provided in this embodiment is applicable to intelligent customer service scenarios and can provide text answers to text questions presented by the user. Here, the automated answer text of the intelligent customer service is an answer output using a question answering model.

[0084] In step 602, text question information is received.

[0085] Here, text question information may be understood as a text question submitted by a user to the intelligent customer service, and the text question relates to a project on the platform where the intelligent customer service is currently located, and this embodiment is not limited thereto.

[0086] In step 604, the text question information is input into the question answer model to obtain text answer information corresponding to the text question information.

[0087] In actual application, after inputting text question information into a question-answering model, the model can output text answer information corresponding to the text question information; that is, the intelligent customer service can be understood as having returned a text answer to a text question presented by the user. It should be noted that the question-answering model provided in this embodiment can be obtained by referring to the training method described in the above embodiment, and without going into detail here, it includes a question-answering model trained using a cloud-side device.

[0088] Furthermore, in this embodiment, the question answering model is obtained by training on a terminal device and is further provided to be applied to the terminal device. Specifically, before the step of inputting the text question information into the question answering model and obtaining text answer information corresponding to the text question information, The steps include obtaining the model parameters of the question answering model on the cloud-side device, Steps to obtain text question samples, The process includes the steps of: constructing a question-answering model to be trained based on the model parameters; and training the question-answering model to be trained based on the text question samples to obtain a question-answering model.

[0089] In actual application, the terminal device can obtain model parameters of the question-answering model in the cloud device. Here, the process for obtaining these model parameters can be found in the process for training the question-answering model in the cloud device described above, and a detailed explanation is omitted here. Furthermore, the terminal device can obtain text question samples, construct a question-answering model to be trained using the model parameters of the question-answering model obtained from the cloud device, and then train the question-answering model to be trained using the text question samples in the terminal device to obtain a question-answering model. This makes the question-answering model trained in the terminal device applicable locally.

[0090] Regarding the process of training the model parameters of the question-answering model on the cloud-side device, you can refer to the description of the question-answering model training in the above embodiment, which includes a three-stage training process for the question-answering model, and a detailed explanation is omitted here.

[0091] In step 606, the text response information is displayed.

[0092] In actual application, the terminal device can provide answers to textual question information using a question-answering model, and then display the textual answer information on the terminal device's display screen.

[0093] In summary, the text question answering method provided in this embodiment automatically provides answers to text question information using a question answering model. At the same time, the question answering model may be one trained on a cloud-side device or one trained on a terminal-side device. In either case, it is possible to provide quick and rational answers to text question information, thereby increasing the effectiveness of text answers for users.

[0094] Corresponding to the above-described embodiment of the method, the present disclosure further provides an embodiment of a question answering device, and Figure 7 shows a schematic diagram of the configuration of a question answering device provided in one embodiment of the present disclosure. As shown in Figure 7, the device is A question acquisition module 702 is configured to acquire question information to be answered, The system includes an answer acquisition module 704 configured to input the aforementioned question information into a question answer model to obtain answer information corresponding to the aforementioned question information, wherein the question answer model is obtained by training on predicted answer information, the predicted answer information is obtained by predicting answer information corresponding to a question sample using a pre-trained question answer model, the pre-trained question answer model is obtained by training on predicted image-text matching results, the predicted image-text matching results are obtained by matching the image to be trained with target semantic information using the pre-trained question answer model, and the target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained.

[0095] Selectively, the device, Obtain a pre-training sample that includes the image to be trained, the image description text corresponding to the image to be trained, and the dialogue text associated with the image to be trained. The pre-trained question-answering model is trained based on the predicted image-text matching results corresponding to the pre-trained samples, wherein the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question-answering model, and the target semantic information is generated based on the image description text and the dialogue text. Obtaining a training image and a sample question containing dialogue text associated with the training image, The system further includes a question-answering model training module configured to input the aforementioned question samples into the pre-trained question-answering model, obtain predicted answer information output from the pre-trained question-answering model, and train the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model.

[0096] Selectively, the question-answering model training module further, The autoregressive loss value is calculated based on the aforementioned predicted response information, The system is configured to perform the following actions: train the pre-trained question-answering model based on the autoregressive loss value to obtain the question-answering model.

[0097] Selectively, the question-answering model training module further, The process involves using the pre-trained question-answering model to perform image processing on the training target images in the pre-training samples to obtain image vectors, The dialogue text associated with the training target image in the pre-training sample is processed using the pre-trained question-answering model to obtain a dialogue vector. The process involves generating a predicted image-text matching result based on the image vector and the dialogue vector, The system is configured to determine an image-text alignment loss value based on the predicted image-text matching result, and to train the pre-trained question-answering model based on the image-text alignment loss value.

[0098] Selectively, the question-answering model training module further, The process involves calculating the dialogue annotation alignment loss value based on the aforementioned image vector, and calculating the image dialogue loss value based on the aforementioned dialogue vector. The system is configured to train the pre-trained question-answering model based on the dialogue annotation alignment loss value, the image dialogue loss value, and the image text alignment loss value.

[0099] Selectively, the question-answering model training module further, Obtain an initial training sample containing the image to be trained and the corresponding image annotation text. The process involves using an initial question response model to perform image processing on the training target image to obtain an image vector, The image annotation text corresponding to the image to be trained is processed using the initial question response model to obtain a text annotation vector. The predicted image description text is determined based on the image vector and the text annotation vector, The system is configured to train the initial question response model based on the predicted image description text.

[0100] Selectively, the question-answering model training module further, The image text annotation loss value is calculated based on the image vector and the text annotation vector. The image text alignment loss value is calculated based on the predicted image description text, The system is configured to train the initial question response model using the image-text annotation loss value and the image-text alignment loss value.

[0101] Selectively, the device, Sending the aforementioned response information to the user, Receiving the user's feedback results regarding the aforementioned response information, The system further includes a model tuning module configured to adjust the model parameters in the question answering model based on the feedback results.

[0102] In summary, the question answering device provided in the embodiment of this disclosure obtains answer information output from a question answering model by inputting question information to be answered into the question answering model. The question answering model can be realized by training a pre-trained question answering model. The pre-trained question answering model predicts answer information corresponding to a question sample, and the question answering model is trained using the predicted answer information. The pre-trained question answering model is obtained by training based on the predicted image-text matching result, and the predicted image-text matching result is obtained by matching the training target image with target semantic information using the pre-trained question answering model. The target semantic information is generated based on the image description text corresponding to the training target image and the dialogue text associated with the training target image. With this method, it is possible to train the question answering model to learn the target semantic text more effectively using a small amount of dialogue text and training target images, and further train the question answering model to output dialogue content according to the dialogue text. In this way, even without using a large amount of dialogue text as training data, the question answering model can learn semantic knowledge in the dialogue text more effectively, thereby making the answer information of the trained question answering model more accurate and improving the application effect of the model.

[0103] The above is a general overview of the solution for the question-answering device of this embodiment. It should be noted that the technical solution for this question-answering device belongs to the same concept as the technical solution for the question-answering method described above, and for details not described in detail in the technical solution for the question-answering device, refer to the description of the technical solution for the question-answering method described above.

[0104] Figure 8 shows a block diagram of the configuration of a computing device 800 according to one embodiment of the present disclosure. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 and the memory 810 are connected via a bus 830, and a database 850 is configured to store data.

[0105] The computing device 800 further comprises an access device 840 that enables the computing device 800 to communicate over one or more networks 860. Examples of these networks include a combination of communication networks such as a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth® interface, or Near Field Communication (NFC).

[0106] In one embodiment of the present disclosure, the components of the computing device 800 and other components not shown in Figure 8 may be connected to each other, for example, via a bus. It should be understood that the block diagram of the computing device configuration shown in Figure 8 is illustrative and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

[0107] The computing device 800 may be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablets, personal digital assistants, laptop computers, notebooks, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.), or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 may also be a mobile or stationary server.

[0108] Here, the processor 820 is configured to execute computer-executable commands, which, when executed by the processor, realize the steps of the above-described question answering method, question answering model training method, and text question answering method.

[0109] The above is a general overview of the computing device of this embodiment. It should be noted that the technical solution for this computing device belongs to the same concept as the technical solution for the question answering method, the question answering model training method, and the text question answering method described above. For details not described in detail in the technical solution for this computing device, refer to the descriptions of the technical solution for the question answering method, the question answering model training method, and the text question answering method described above.

[0110] One embodiment of the present disclosure further provides a computer-readable storage medium that stores computer-executable commands, which, when executed by a processor, realize the steps of the above-described question answering method, question answering model training method, and text question answering method.

[0111] The above is a general overview of the solution for the computer-readable storage medium in this embodiment. It should be noted that the technical solution for the storage medium belongs to the same concept as the technical solutions for the question answering method, the question answering model training method, and the text question answering method described above. For details not described in detail in the technical solution for the storage medium, refer to the descriptions of the technical solutions for the question answering method, the question answering model training method, and the text question answering method.

[0112] One embodiment of the present disclosure further provides a computer program that, when executed on a computer, causes the computer to perform the steps of the above-described question answering method, question answering model training method, and text question answering method.

[0113] The above is a general overview of the solution provided by the computer program in this embodiment. It should be noted that the technical solution provided by this computer program belongs to the same concept as the technical solution provided by the question answering method, the question answering model training method, and the text question answering method described above. For details not described in detail in the technical solution provided by this computer program, refer to the descriptions of the technical solution provided by the question answering method, the question answering model training method, and the text question answering method described above.

[0114] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the attached claims. Desired results may be achieved by performing the operations or steps described in the claims in a different order than those described in the embodiments. The processes described in the drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In some embodiments, multitasking or parallel processing may be possible or beneficial.

[0115] The computer command includes computer program code, which may be in source code format, object code format, executable file format, or some intermediate format. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. It should be noted that the contents of the computer-readable medium may be increased or decreased as appropriate according to the requirements of patent practice, for example, in some regions, patent practice stipulates that the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0116] It should be noted that, for the sake of simplicity, the embodiments of each method described above have been presented as a series of operational combinations. However, those skilled in the art should know that, according to the embodiments of this disclosure, some steps may be performed in a different order or simultaneously, and therefore the embodiments of this disclosure are not limited to the operational order described. Furthermore, those skilled in the art should know that all embodiments described in this disclosure are preferred embodiments, and the relevant operations and modules are not necessarily essential to the embodiments of this disclosure.

[0117] In the above embodiments, the descriptions of each embodiment focus on different aspects. For parts not described in detail in one embodiment, refer to the relevant descriptions in other embodiments.

[0118] The preferred embodiments of the Disclosure disclosed above are provided solely to aid in illustrating the Disclosure. The selectable embodiments do not detail all aspects, nor do they limit the invention to the specific embodiments described above. Naturally, many modifications and changes can be made based on the embodiments of the Disclosure. The purpose of selecting and describing these embodiments in the Disclosure is to better interpret the principles and practical applications of the embodiments of the Disclosure so that those skilled in the art may favorably understand and utilize the Disclosure. The Disclosure is limited to the claims and their entirety and their equivalents. (Industrial applicability)

[0119] The training method, text generation method, and apparatus for a text generation model provided in the embodiments of this disclosure are applied to a scenario in which text generation is trained. Question information to be answered is input to a question-answering model that can be realized by training a pre-trained question-answering model, thereby obtaining answer information output from the question-answering model. The pre-trained question-answering model predicts answer information corresponding to the question sample, and the question-answering model is trained with the predicted answer information. The pre-trained question-answering model is obtained by training based on the predicted image-text matching results. Simultaneously, the predicted image-text matching results are obtained by matching the training target image with the target semantic information using the pre-trained question-answering model. The target semantic information is generated based on the image description text corresponding to the training target image and the dialogue text associated with the training target image. With this method, it is possible to train the question-answering model to learn the target semantic text more effectively using a small amount of dialogue text and training target images, and further train the question-answering model to output dialogue content according to the dialogue text. In this way, even without using a large amount of dialogue text as training data, the question-answering model can learn semantic knowledge in the dialogue text more effectively, thereby making the answer information of the trained question-answering model more accurate and improving the application effect of the model.

Claims

1. Steps to obtain the question information to be answered, The steps include: inputting the aforementioned question information into a question answering model to obtain answer information corresponding to the aforementioned question information; Includes, The aforementioned question answering model is obtained by training it based on predicted response information. The aforementioned predicted response information is obtained by predicting response information corresponding to a question sample using a pre-trained question answering model. The aforementioned pre-trained question answering model is obtained by training it based on the predicted image-text matching results. The predicted image-text matching results are obtained by matching the training target image with the target semantic information using the pre-trained question-answering model. The target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained. Question and answer method.

2. Before the step of inputting the aforementioned question information into a question answering model to obtain answer information corresponding to the aforementioned question information, A step of obtaining a pre-training sample that includes an image to be trained, an image description text corresponding to the image to be trained, and dialogue text associated with the image to be trained. A step of training the pre-trained question-answering model based on the predicted image-text matching results corresponding to the pre-trained samples, wherein the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question-answering model, and the target semantic information is generated based on the image description text and the dialogue text, The steps include obtaining a training image and a sample question containing dialogue text associated with the training image, The steps include: inputting the aforementioned question sample into the pre-trained question-answering model, obtaining predicted answer information output from the pre-trained question-answering model, and training the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model; This also includes, The method according to claim 1.

3. The step of training the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model is: The steps include: calculating the autoregressive loss value based on the aforementioned predicted response information; The steps include: training the pre-trained question-answering model based on the autoregressive loss value to obtain the question-answering model; including, The method according to claim 2.

4. The step of training the pre-trained question answering model based on the predicted image-text matching results corresponding to the pre-trained samples is: The steps include:

1. Using the pre-trained question-answering model, image processing is performed on the image to be trained in the pre-trained sample to obtain an image vector. The steps include: using the pre-trained question-answering model to process the dialogue text associated with the training target image in the pre-training sample to obtain a dialogue vector; A step of generating a predicted image-text matching result based on the image vector and the dialogue vector, The steps include determining an image-text alignment loss value based on the predicted image-text matching result, and training the pre-trained question-answering model based on the image-text alignment loss value, including, The method according to claim 2 or claim 3.

5. The step of training the pre-trained question response model based on the image-text alignment loss value is: The steps include: calculating the dialogue annotation alignment loss value based on the image vector, and calculating the image dialogue loss value based on the dialogue vector; The steps include training the pre-trained question-answering model based on the dialogue annotation alignment loss value, the image dialogue loss value, and the image text alignment loss value, including, The method according to claim 4.

6. Before the aforementioned step of obtaining pre-training samples, The steps include obtaining an initial training sample that includes the image to be trained and the image annotation text corresponding to the image to be trained, The steps include: processing the image to be trained using an initial question response model to obtain an image vector; The steps include: using the initial question response model to process the image annotation text corresponding to the image to be trained to obtain a text annotation vector; The steps include determining a predicted image description text based on the image vector and the text annotation vector, A step of training the initial question response model based on the predicted image description text, This also includes, The method according to claim 2.

7. The step of training the initial question response model based on the predicted image description text is: The steps include: calculating an image text annotation loss value based on the image vector and the text annotation vector; The steps include: calculating an image-text alignment loss value based on the predicted image description text; The steps include training the initial question response model using the image-text annotation loss value and the image-text alignment loss value, including, The method according to claim 6.

8. After the step of obtaining answer information corresponding to the aforementioned question information, The steps include sending the aforementioned response information to the user, The steps include receiving the user's feedback result regarding the response information, The steps include adjusting the model parameters in the question answering model based on the feedback results, This also includes, The method according to claim 1.

9. A question answering method applied to a cloud-side device, The steps include receiving a question response request containing question information sent from the terminal device, The steps include: inputting the aforementioned question information into a question answering model to obtain answer information corresponding to the aforementioned question information; The steps include: transmitting the aforementioned response information to the terminal device; Includes, The aforementioned question answering model is obtained by training it based on predicted response information. The aforementioned predicted response information is obtained by predicting response information corresponding to a question sample using a pre-trained question answering model. The aforementioned pre-trained question answering model is obtained by training it based on the predicted image-text matching results. The predicted image-text matching results are obtained by matching the training target image with the target semantic information using the pre-trained question-answering model. The target semantic information is generated based on the image description text corresponding to the image to be trained and the dialogue text associated with the image to be trained. Question and answer method.

10. A method for training a question answering model applied to a cloud-side device, A step of obtaining a pre-training sample that includes an image to be trained, an image description text corresponding to the image to be trained, and dialogue text associated with the image to be trained. A step of training the pre-trained question answering model based on the predicted image-text matching results corresponding to the pre-trained samples, wherein the predicted image-text matching results are obtained by matching the training target image with target semantic information using the pre-trained question answering model, and the target semantic information is generated based on the image description text and the dialogue text. The steps include obtaining a training image and a sample question containing dialogue text associated with the training image, The steps include: inputting the aforementioned question sample into the pre-trained question-answering model, obtaining predicted answer information output from the pre-trained question-answering model, and training the pre-trained question-answering model based on the predicted answer information to obtain a question-answering model; The steps include transmitting the model parameters of the aforementioned question answering model to the terminal device, including, Training methods for question-answering models.

11. A text-based question answering method applied to a terminal device, Steps include receiving text question information, A step of inputting the text question information into a question answer model to obtain text answer information corresponding to the text question information, wherein the question answer model is obtained by the method described in claim 10, The steps include: displaying the aforementioned text response information, including, Text-based question and answer method.

12. Before the step of inputting the aforementioned text question information into a question answering model to obtain text answer information corresponding to the aforementioned text question information, The steps include obtaining the model parameters of the question answering model on the cloud-side device, Steps to obtain text question samples, The steps include constructing a question-answering model to be trained based on the aforementioned model parameters, and training the question-answering model based on the aforementioned text question samples to obtain a question-answering model, including, The method according to claim 11.

13. Equipped with memory and a processor, The memory is configured to store commands that can be executed by the computer. The processor is configured to execute commands that can be executed by the computer, When a command executable by the computer is executed by the processor, the steps of the method according to any one of claims 1 to 12 are realized. Computing device.

14. A computer-executable command is stored which, when executed by a processor, realizes a step of the method according to any one of claims 1 to 12. Computer-readable storage medium.

Images