Question-answering method and question-answering model training method

Abstract

Embodiments of the present specification provide a question and answer method and a question and answer model training method, wherein the question and answer method comprises: obtaining question information to be replied; inputting the question information into a question and answer model to obtain reply information corresponding to the question information; wherein the question and answer model is obtained by training based on predicted reply information, the predicted reply information is obtained by predicting reply information corresponding to a question sample by using a pre-trained question and answer model, the pre-trained question and answer model is obtained by training based on a predicted image-text matching result, the predicted image-text matching result is obtained by matching a to-be-trained image and target semantic information by using the pre-trained question and answer model, and the target semantic information is generated based on an image description text corresponding to the to-be-trained image and a dialogue text associated with the to-be-trained image, so that the reply information of the trained question and answer model is more accurate, and the application effect of the model is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a question and answer method. BACKGROUND

[0002] With the richness of multimedia data, task-oriented dialogue systems have attracted extensive attention, for example, intelligent customer service in the e-commerce field. Task-oriented dialogue systems can help users to obtain replies to problems by themselves. With the increasing requirements of users on task-oriented dialogue systems, the dialogue system needs to judge the mode of replying to the user first, that is, whether to reply to the picture or the text, and then complete the reply of the content. However, due to the difficulty in collecting multi-modal dialogue data, the training data is less, the dialogue generation model is difficult to train, and the application effect of the dialogue generation model is poor. SUMMARY

[0003] Therefore, the embodiments of the present specification provide a question and answer method. One or more embodiments of the present specification also relate to another question and answer method, a question and answer model training method, a text question and answer method, a question and answer device, a computing device, a computer readable storage medium, and a computer program to solve the technical defects in the prior art.

[0004] According to a first aspect of the embodiments of the present specification, a question and answer method is provided, comprising:

[0005] obtaining problem information to be replied;

[0006] inputting the problem information into a question and answer model to obtain reply information corresponding to the problem information;

[0007] wherein the question and answer model is obtained by training based on predicted reply information, the predicted reply information is obtained by predicting reply information corresponding to a problem sample by using a pre-trained question and answer model, the pre-trained question and answer model is obtained by training based on a predicted image-text matching result, the predicted image-text matching result is obtained by matching a to-be-trained image and target semantic information by using the pre-trained question and answer model, and the target semantic information is generated based on an image description text corresponding to the to-be-trained image and dialogue text associated with the to-be-trained image.

[0008] According to a second aspect of the embodiments of the present specification, another question and answer method is provided, applied to a cloud-side device, comprising:

[0009] receiving a question and answer request sent by an end-side device, wherein the question and answer request carries problem information;

[0010] inputting the problem information into a question and answer model to obtain reply information corresponding to the problem information;

[0011] The question and answer model is obtained based on predicted reply information, the predicted reply information is obtained by predicting reply information corresponding to a question sample by using a pre-trained question and answer model, the pre-trained question and answer model is obtained based on predicted image-text matching results, the predicted image-text matching results are obtained by matching a to-be-trained image and target semantic information by using the pre-trained question and answer model, and the target semantic information is generated based on image description text corresponding to the to-be-trained image and dialogue text associated with the to-be-trained image.

[0012] The reply information is sent to the terminal device.

[0013] According to a third aspect of the embodiments of the present specification, a question and answer model training method is provided, applied to a cloud side device, including:

[0014] Obtaining pre-training samples, wherein the pre-training samples include a to-be-trained image, image description text corresponding to the to-be-trained image, and dialogue text associated with the to-be-trained image.

[0015] Training the pre-trained question and answer model based on predicted image-text matching results corresponding to the pre-training samples, wherein the predicted image-text matching results are obtained by matching the to-be-trained image and target semantic information by using the pre-trained question and answer model, and the target semantic information is generated based on the image description text and the dialogue text.

[0016] Obtaining question samples, wherein the question samples include a to-be-trained image and dialogue text associated with the to-be-trained image.

[0017] Inputting the question samples into the pre-trained question and answer model, obtaining predicted reply information output by the pre-trained question and answer model, and training the pre-trained question and answer model based on the predicted reply information, to obtain a question and answer model.

[0018] Sending model parameters of the question and answer model to a terminal device.

[0019] According to a fourth aspect of the embodiments of the present specification, a text question and answer method is provided, applied to a terminal device, including:

[0020] Receiving text question information.

[0021] Inputting the text question information into a question and answer model to obtain text reply information corresponding to the text question information, wherein the question and answer model is obtained according to the question and answer model training method.

[0022] Displaying the text reply information.

[0023] According to a fifth aspect of the embodiments of the present specification, a question and answer device is provided, including:

[0024] a question obtaining module configured to obtain question information to be replied to;

[0025] a reply obtaining module configured to input the question information into a question and answer model to obtain reply information corresponding to the question information, wherein the question and answer model is obtained based on predicted reply information, the predicted reply information is obtained by predicting reply information corresponding to a question sample by using a pre-trained question and answer model, the pre-trained question and answer model is obtained based on a predicted image-text matching result, the predicted image-text matching result is obtained by matching a to-be-trained image and target semantic information by using the pre-trained question and answer model, and the target semantic information is generated based on image description text corresponding to the to-be-trained image and dialogue text associated with the to-be-trained image.

[0026] According to a sixth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0027] a memory and a processor;

[0028] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the above method.

[0029] According to a seventh aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions, when executed by a processor, implement the steps of the above method.

[0030] According to an eighth aspect of an embodiment of the present specification, a computer program is provided, and when the computer program is executed in a computer, the computer program causes the computer to execute the steps of the above method.

[0031] The question and answer method provided by one embodiment of the present specification comprises: obtaining question information to be replied to; inputting the question information into a question and answer model to obtain reply information corresponding to the question information; wherein the question and answer model is obtained based on predicted reply information, the predicted reply information is obtained by predicting reply information corresponding to a question sample by using a pre-trained question and answer model, the pre-trained question and answer model is obtained based on a predicted image-text matching result, the predicted image-text matching result is obtained by matching a to-be-trained image and target semantic information by using the pre-trained question and answer model, and the target semantic information is generated based on image description text corresponding to the to-be-trained image and dialogue text associated with the to-be-trained image.

[0032] In an embodiment of the present specification, by inputting the to-be-answered question information into the question and answer model, the reply information output by the question and answer model is obtained, wherein the question and answer model can be realized by training a pre-trained question and answer model, the pre-trained question and answer model is trained according to the predicted reply information of the question sample corresponding to the reply information, and the pre-trained question and answer model is trained based on the predicted image-text matching result, and the predicted image-text matching result is obtained according to the pre-trained question and answer model matching the to-be-trained image and the target semantic information. In addition, the target semantic information is generated based on the image description text corresponding to the to-be-trained image and the dialogue text associated with the to-be-trained image. This method can train the question and answer model to better learn the target semantic text by using a small amount of dialogue text and to-be-trained image, and then train the question and answer model to output dialogue content according to the dialogue text. In this way, without a large amount of dialogue text as training data, the question and answer model can better learn the semantic knowledge in the dialogue text, so that the reply information of the trained question and answer model is more accurate, and the application effect of the model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is an application scenario diagram of a question and answer method provided by an embodiment of the present specification;

[0034] Figure 2 is a flowchart of a question and answer method provided by an embodiment of the present specification;

[0035] Figure 3a is a first stage training diagram of a question and answer model in a question and answer method provided by an embodiment of the present specification;

[0036] Figure 3b is a second stage training diagram of a question and answer model in a question and answer method provided by an embodiment of the present specification;

[0037] Figure 3c is a third stage training diagram of a question and answer model in a question and answer method provided by an embodiment of the present specification;

[0038] Figure 4 is a flowchart of another question and answer method provided by an embodiment of the present specification;

[0039] Figure 5 is a flowchart of a question and answer model training method provided by an embodiment of the present specification;

[0040] Figure 6 is a flowchart of a text question and answer method provided by an embodiment of the present specification;

[0041] Figure 7 is a structural diagram of a question and answer device provided by an embodiment of the present specification;

[0042] Figure 8 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0043] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it can be apparent to those skilled in the art that the present specification can be practiced without the specific details. In other instances, well-known methods, procedures, components, and networks have not been described in detail so as not to unnecessarily obscure aspects of the present specification.

[0044] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It will be understood that, although the terms first, second, etc. can be employed in one or more embodiments of the present specification, these terms are used to distinguish one information from another and are not intended to signify or imply these information by their use. For example, a first can be termed a second, and, similarly, a second can be termed a first, without departing from the scope of one or more embodiments of the present specification. The phrase, as used herein, can be interpreted to mean that if the condition is met, then the action will be performed. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0046] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0047] First, the terms involved in one or more embodiments of the present specification are explained.

[0048] Multi-modal: In the field of artificial intelligence, it often refers to perception information such as images, texts, voices, etc.

[0049] With the richness of multimedia data, the dialogue communication on the Internet is no longer limited to the pure text mode. Therefore, the introduction of multi-modal dialogue will become an unstoppable trend for future intelligent customer service. Multi-modal dialogue requires the model to first judge the user's reply mode, that is, whether to reply to the picture or text. Secondly, it should have the ability to retrieve replies that meet the context, including image retrieval and text retrieval. At the same time, it should have the ability to track user state to track user needs. Finally, it should have the ability to generate text to generate smooth and natural replies. However, in today's data-driven world, multi-modal dialogue data is difficult to collect due to privacy and other issues, resulting in relatively small quantities, making it difficult for the model to train. Non-dialogue data of multi-modal has a large scale. Therefore, an incremental combined expert multi-modal dialogue pre-training model is proposed in the embodiments of the present specification, which uses multi-modal non-dialogue data to model the recognition and matching capabilities of text experts and image experts. Through the inheritance of the above expert capabilities, multi-modal dialogue data is introduced to model the dialogue expert to realize the step-by-step improvement of the model capability.

[0050] In the present specification, a question and answer method is provided, and the present specification also relates to another question and answer method, a question and answer model training method, a text question and answer method, a question and answer device, a computing device, and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0051] Referring to Figure 1 , Figure 1 A schematic diagram of an application scenario of a question and answer method according to an embodiment of the present specification is shown.

[0052] It should be noted that the question and answer method provided in the present embodiment can be applied to the question and answer reply scene of intelligent customer service, and the specific application scenario can be adapted to different needs according to different project needs, which is not limited in the present embodiment. In order to facilitate understanding, the application scenario of the user using the intelligent customer service is taken as an example for description in the present embodiment.

[0053] In actual application, a user sends question information requiring a reply to an intelligent customer service. The question information can be understood as information in different application scenarios, including but not limited to text information, image information, etc. Since the intelligent customer service uses the question and answer model in the question and answer method provided in this embodiment, after the question and answer model receives the question information sent by the user, the question and answer model can output the reply information corresponding to the question information. The reply information can be understood as the reply content fed back to the question information, including but not limited to text information, image information, etc. Then, the user can receive the reply content fed back by the intelligent customer service. It should be noted that before the question and answer model is put into use, the question and answer model needs to be pre-trained. Specifically, in order to improve the training effect of the question and answer model, the model can be trained in stages in this embodiment, implemented by combining different expert networks, multi-modal dialogue data is introduced to model the question and answer model, and the model capacity is improved in steps.

[0054] Referring to Figure 2 , Figure 2 A flowchart of a question and answer method according to one embodiment of the present specification is shown, specifically including the following steps.

[0055] Step 202: Obtain question information requiring a reply.

[0056] In actual application, the subject can obtain question information requiring a reply. The question information can be understood as the question content requiring a reply by the user, including but not limited to text content, image content, text and image content, etc.

[0057] Step 204: Input the question information into the question and answer model to obtain the reply information corresponding to the question information.

[0058] The question and answer model is obtained by training based on predicted reply information. The predicted reply information is obtained by predicting the reply information corresponding to the question sample by using a pre-trained question and answer model. The pre-trained question and answer model is obtained by training based on a predicted image-text matching result. The predicted image-text matching result is obtained by matching the to-be-trained image and target semantic information by using the pre-trained question and answer model. The target semantic information is generated based on the image description text corresponding to the to-be-trained image and the dialogue text associated with the to-be-trained image.

[0059] In actual application, the execution subject can input the question information into the question and answer model to obtain the reply information corresponding to the question information. The reply information can be understood as the corresponding reply content determined for the question information, including text content and / or image content, etc. It should be noted that the question and answer model provided in the embodiment can be implemented by training a pre-trained question and answer model, and the pre-trained question and answer model is trained according to the predicted reply information output by the pre-trained question and answer model according to the question sample. Further, the pre-trained question and answer model can also be trained according to the to-be-trained image and the target semantic information, where the target semantic information can be understood as the image description text corresponding to the to-be-trained image, or the dialogue text generated in association with the to-be-trained image.

[0060] For ease of understanding, the following embodiments introduce the training process of the question and answer model, which can include three stages of training, that is, first training an initial question and answer model in the first stage through images and image annotation texts, then training a pre-trained question and answer model in the second stage through images, image annotation texts and dialogue texts, and finally training a question and answer model in the third stage according to images and dialogue texts. It should be noted that the initial question and answer model, the pre-trained question and answer model and the question and answer model all refer to the same question and answer model, which can be understood as having different model output capabilities and effects in different training stages. The training process provided in the embodiment is to first train a pre-trained question and answer module and then train a question and answer model, that is, the training process in the second stage and the third stage. Specifically, before the question information is input into the question and answer model to obtain the reply information corresponding to the question information, the following steps are further included:

[0061] Obtaining pre-training samples, wherein the pre-training samples include to-be-trained images, image description texts corresponding to the to-be-trained images, and dialogue texts associated with the to-be-trained images;

[0062] Training the pre-trained question and answer model based on the predicted image-text matching result corresponding to the pre-training samples, wherein the predicted image-text matching result is obtained by matching the to-be-trained image and the target semantic information using the pre-trained question and answer model, and the target semantic information is generated based on the image description text and the dialogue text;

[0063] Obtaining question samples, wherein the question samples include to-be-trained images and dialogue texts associated with the to-be-trained images;

[0064] Inputting the question samples into the pre-trained question and answer model to obtain predicted reply information output by the pre-trained question and answer model, and training the pre-trained question and answer model based on the predicted reply information to obtain a question and answer model.

[0065] The pre-training sample can be understood as a sample data for training the pre-training question and answer model, and includes a to-be-trained image, an image description text corresponding to the to-be-trained image, and a dialogue text associated with the to-be-trained image. The to-be-trained image can be understood as any type of training image, and the image category is not limited. The image description text corresponding to the to-be-trained image can be understood as text information describing the image content in the to-be-trained image, such as describing the object content and state in the image. The dialogue text associated with the to-be-trained image can be understood as dialogue information discussing the to-be-trained image. The dialogue information can be understood as dialogue data between different subjects, but the form of the dialogue content and the dialogue subject are not limited in this embodiment.

[0066] In actual application, the pre-training question and answer model can be trained according to a predicted image-text matching result corresponding to the pre-training sample. The predicted image-text matching result can be obtained by matching the to-be-trained image and target semantic information based on an information matching layer (FFN) in the pre-training question and answer model. It should be noted that the target semantic information can be obtained by vector matching between the image description text and the dialogue text, that is, the pre-training question and answer model can better understand the semantic information in the dialogue text according to the image description text. Further, after the training of the pre-training question and answer model is completed, a question sample can be obtained. The question sample can be understood as a training sample applied to the question and answer model, and includes a to-be-trained image and a dialogue text associated with the to-be-trained image. The question sample is input into the pre-training question and answer model, the predicted reply information output by an information generation module in the pre-training question and answer model is obtained, the pre-training question and answer model is further trained according to the predicted reply information, and finally, a question and answer model with better training effect is obtained.

[0067] The training method of the question and answer model in this embodiment adopts the method of first training the pre-training question and answer model, and then continuing to train the pre-training question and answer model in the next stage, so that the question and answer model obtained by training has a step-by-step learning ability.

[0068] Further, in the third stage of training the question and answer model, the pre-training question and answer model can be trained based on the predicted reply information output by the pre-training question and answer model. In this embodiment, the model training can be performed in a self-recurrent manner. Specifically, the pre-training question and answer model is trained based on the predicted reply information to obtain a question and answer model, including:

[0069] calculating a self-recurrent loss value based on the predicted reply information;

[0070] training the pre-training question and answer model based on the self-recurrent loss value to obtain the question and answer model.

[0071] In actual application, the execution subject can calculate the autoregressive loss value by using the predicted reply information output by the pre-trained question and answer model, and train the pre-trained question and answer model according to the autoregressive loss value, thereby obtaining the question and answer model. The determination of the autoregressive loss value in this embodiment can refer to the following formula 1:

[0072]

[0073] wherein, represents the loss value of the third stage, C represents the dialogue history, V represents the image, and p represents the predicted result of the model. * represents the model prediction result.

[0074] Further, for the stage of training the pre-trained question and answer model, i.e., the second stage, the model needs to learn the matching ability between the image and the dialogue text, and then the image-text alignment layer (FFN) in the pre-trained question and answer model needs to be trained. Specifically, the pre-trained question and answer model is trained based on the predicted image-text matching result corresponding to the pre-training sample, which comprises:

[0075] For the training image in the pre-training sample, the pre-trained question and answer model is used to perform image processing on the training image to obtain an image vector.

[0076] For the dialogue text associated with the training image in the pre-training sample, the pre-trained question and answer model is used to perform text processing on the dialogue text to obtain a dialogue vector.

[0077] Based on the image vector and the dialogue vector, a predicted image-text matching result is generated.

[0078] Based on the predicted image-text matching result, an image-text alignment loss value is determined, and the pre-trained question and answer model is trained based on the image-text alignment loss value.

[0079] In actual application, in the stage of training the pre-trained question and answer model, the training image in the pre-training sample can be input into the image processing layer (FFN) in the pre-trained question and answer model to obtain an image vector. Then, the dialogue text associated with the training image in the pre-training sample is input into the dialogue processing layer (FFN) in the pre-trained question and answer model to obtain a dialogue vector. After the image and the text are processed into vectors respectively, the image vector and the dialogue vector can be matched to generate a predicted image-text matching result, and the loss value of the image-text alignment layer (FFN) is determined according to the predicted image-text matching result, and the pre-trained question and answer model is trained according to the loss value.

[0080] In this way, by using the multi-modal data of the image and the dialogue, the model learns the matching information between the image and the dialogue text, so that the pre-trained question and answer model has the ability of image and dialogue matching.

[0081] Further, in the stage of training the pre-training question and answer model, in order to improve the training effect of the pre-training question and answer model, in addition to training according to the image-text alignment loss value, the loss value corresponding to the to-be-trained image and the loss value corresponding to the dialogue text can also be referred to; specifically, the pre-training question and answer model is trained based on the image-text alignment loss value, including:

[0082] The dialogue annotation alignment loss value is calculated based on the image vector, and the image-dialogue loss value is calculated based on the dialogue vector;

[0083] The pre-training question and answer model is trained based on the dialogue annotation alignment loss value, the image-dialogue loss value and the image-text alignment loss value.

[0084] In actual application, the execution subject can also calculate the corresponding dialogue annotation alignment loss value according to the image vector obtained by encoding the to-be-trained image by the image processing module, and further calculate the corresponding image-dialogue loss value according to the dialogue vector obtained by encoding the dialogue text by the dialogue processing module; finally, the pre-training question and answer model is continuously trained according to the dialogue annotation alignment loss value, the image-dialogue loss value and the image-text alignment loss value; in this embodiment, Formula 2 below can be referred to as an exemplary training target:

[0085]

[0086] Wherein, Specifically, Formula 3-Formula 5 below can be referred to:

[0087]

[0088]

[0089]

[0090] Wherein, D represents the loss value of the second stage, D d represents dialogue multi-modal data, represents the masked character, y * represents the correct value, represents the intermediate representation of the model L-F layer.

[0091] Further, before training the pre-training question and answer model, it also includes a training initial question and answer model stage, i.e. a first stage, which can enable the initial question and answer model to have the ability to match images and texts; specifically, before obtaining the pre-training sample, it also includes:

[0092] Obtain an initial training sample, wherein the initial training sample includes a to-be-trained image and an image annotation text corresponding to the to-be-trained image;

[0093] For the to-be-trained image, an initial question and answer model is used for image processing on the to-be-trained image to obtain an image vector;

[0094] For the image annotation text corresponding to the to-be-trained image, the initial question and answer model is used for text processing on the image annotation text to obtain a text annotation vector;

[0095] Based on the image vector and the text annotation vector, a predicted image description text is determined;

[0096] The initial question and answer model is trained based on the predicted image description text.

[0097] In actual application, initial training samples are obtained, and the initial question and answer model is trained based on the initial training samples. Specifically, the to-be-trained image in the initial training sample is input into an image processing layer (FFN) in the initial question and answer model to obtain an image vector, and then the image annotation text in the initial training sample is input into an image annotation layer (FFN) in the initial question and answer model to obtain a text annotation vector. Then, according to the image vector and the text annotation vector, a predicted image description text is determined, so that the model learns the matching ability between the image and the text. Then, the initial question and answer model is trained according to the predicted image description text. It should be noted that the image annotation text can be understood as artificial image annotation information of the to-be-trained image, or can be understood as being realized by using a basic image annotation model. This embodiment does not limit this.

[0098] Further, the embodiment also provides that a loss value is determined according to the predicted image description text to train the initial question and answer model. Specifically, the training of the initial question and answer model based on the predicted image description text comprises:

[0099] Based on the image vector and the text annotation vector, a text-image annotation loss value is calculated;

[0100] Based on the predicted image description text, a text-image alignment loss value is calculated;

[0101] According to the text-image annotation loss value and the text-image alignment loss value, the initial question and answer model is trained.

[0102] In actual application, the image processing module and the image annotation module in the initial question and answer model can respectively calculate a text-image annotation loss value and a text-image alignment loss value after respectively encoding processing the image and the text. Then, according to the text-image annotation loss value and the text-image alignment loss value, the initial question and answer model is trained. In this embodiment, the following training target can be referred to:

[0103]

[0104]

[0105]

[0106] wherein, represents the loss value of the first stage, D n represents the non-dialogue multi-modal data (image description text), and CE represents the cross-entropy calculation.

[0107] Based on this, in the first stage of training the question and answer model, the initial question and answer model is trained by calculating the image-text annotation loss value and the image-text alignment loss value, so as to facilitate subsequent training of the matching ability of the question and answer model between the dialogue text and the image in the initial question and answer model, so as to improve the processing ability of the question and answer model to the multi-modal data.

[0108] In addition, after the execution subject obtains the question and answer model through the three stages, the reply information corresponding to the question information output by the question and answer model can be sent to the user to obtain the feedback result of the user to the reply information, so as to realize the fine-tuning process of the question and answer model. Specifically, after obtaining the reply information corresponding to the question information, the method further comprises:

[0109] sending the reply information to the user;

[0110] receiving the feedback result of the user to the reply information;

[0111] adjusting the model parameters in the question and answer model based on the feedback result.

[0112] In actual application, the execution subject can send the reply information to the user, and the user can give corresponding feedback or evaluation according to the specific content of the reply information after receiving the reply information, and send the feedback result to the execution subject. The execution subject can fine-tune the model parameters in the question and answer model according to the received feedback result, so as to obtain a question and answer model with better application effect.

[0113] In summary, the question and answer method provided in the embodiment uses a question and answer model to complete the reply to the question information, wherein the training process of the question and answer model also adopts a three-stage training process, so that the question and answer model continuously learns the image-text matching ability, the image-dialogue matching ability and the reply generation ability. The specific training process adopts an incremental combination of expert networks, trains each processing module in the question and answer model, introduces the modeling dialogue ability of multi-modal dialogue data, and realizes the step-by-step improvement of the model ability.

[0114] The following describes the embodiment in conjunction with the accompanying Figure 3a , Figure 3b and Figure 3cFIG. 1 shows a training schematic diagram of each stage of a question and answer model in a question and answer method according to an embodiment of the present specification.

[0115] It should be noted that in the training of each stage, the question and answer model includes an L-layer structure, and each layer can include five different expert networks, namely the image annotation, dialogue context, image, image-text alignment and text generation network layers, and in addition, each layer has a shared multi-head self-attention mechanism layer.

[0116] Figure 3a FIG. 1 shows a training schematic diagram of each stage of a question and answer model in a question and answer method according to an embodiment of the present specification.

[0117] In practical applications, Figure 3a The one-stage pre-training process in FIG. 1 uses the encoding of the image and the image annotation text as input to the subsequent L-F layer to complete the matching between the image annotation text and the image, and then uses the image-text alignment network in the F layer to realize the one-stage pre-training question and answer model.

[0118] Figure 3b FIG. 2 shows a training schematic diagram of each stage of a question and answer model in a question and answer method according to an embodiment of the present specification.

[0119] In practical applications, Figure 3b The two-stage pre-training process in FIG. 2 uses the image, image annotation text and dialogue text as input to the L-F layer to complete the matching between the image annotation text and the dialogue text, and then inputs the target semantic text into the F layer of the model to complete the image-text alignment training process.

[0120] Figure 3c FIG. 3 shows a training schematic diagram of each stage of a question and answer model in a question and answer method according to an embodiment of the present specification.

[0121] In practical applications, Figure 3c The three-stage pre-training process in FIG. 3 uses the dialogue text and image as input to the L-F layer, since the question and answer model has the ability to match the dialogue and image in the previous two stages of training, and then uses the dialogue context and image in the L-F layer to match and understand the dialogue and image in the three-stage pre-training, and then trains the text generation network in the F layer of the model to train the reply information generation ability of the question and answer model.

[0122] It should be noted that when the question and answer model is applied to a multi-modal dialogue task, it can include multiple types of execution tasks: 1. Multi-modal intent recognition, specifically predicting whether the current round needs to share the image; 2. Multi-modal retrieval, retrieving appropriate images and text for reply according to the dialogue history; 3. Multi-modal dialogue tracking, tracking the user's key dialogue needs; 4. Multi-modal text generation, generating appropriate replies according to the dialogue history.

[0123] Based on this, the question and answer model training process provided in this embodiment divides each layer in the model into multiple expert networks, and different expert networks are combined when completing specific tasks to achieve task processing. Due to the division of small modules, the training of the model becomes more controllable, and the interpretability is also enhanced. At the same time, the parameters of the expert networks that are not used in specific tasks will not be activated, which can greatly speed up the inference speed of the model and improve the parameter utilization rate. In addition, different semantic knowledge is given to each expert network, so that each modality and each data type has a trace in the flow of the model, enhancing the robustness and generalization ability of the model.

[0124] Referring to Figure 4 , Figure 4 A flowchart of another question and answer method provided according to an embodiment of the present specification is shown, specifically including the following steps.

[0125] It should be noted that the question and answer method provided in this embodiment is applied to a cloud-side device, that is, after receiving the question information sent by the terminal-side device, the question and answer model in the cloud-side device can be used to reply to the question information.

[0126] Step 402: receiving a question and answer request sent by a terminal-side device, wherein the question and answer request carries question information.

[0127] In actual application, the cloud-side device can receive the question and answer request sent by the terminal-side device, and can obtain the question information carried in the question and answer request; it should be noted that the specific content and form of the question information are not limited in this embodiment.

[0128] Step 404: inputting the question information into a question and answer model to obtain reply information corresponding to the question information.

[0129] The question and answer model is obtained by training based on predicted reply information, the predicted reply information is obtained by predicting the reply information corresponding to the question sample by using a pre-trained question and answer model, the pre-trained question and answer model is obtained by training based on a predicted image-text matching result, the predicted image-text matching result is obtained by matching the target semantic information with the to-be-trained image by using the pre-trained question and answer model, and the target semantic information is generated based on the image description text corresponding to the to-be-trained image and the dialogue text associated with the to-be-trained image.

[0130] In actual applications, the cloud-side device can input the question information into the question-answering model and obtain the reply information output by the question-answering model. The training process of the question-answering model in the cloud-side device can refer to the training process described in the above embodiments, that is, the three-stage question-answering model training, which will not be described in detail here.

[0131] Step 406: sending the reply information to the terminal-side device.

[0132] Further, after obtaining the reply information, the cloud-side device can feed back the reply information to the terminal-side device.

[0133] Based on this, the embodiment provides a question-answering method applied to a cloud-side device, which can realize question reply to the question information sent by the terminal-side device. Meanwhile, the question-answering model is trained in the cloud-side device, which can save the computing resources and storage space of the terminal-side device and provide better use effect for users.

[0134] Referring to Figure 5 , Figure 5 A flowchart of a question-answering model training method according to one embodiment of the present specification is shown, which specifically includes the following steps.

[0135] It should be noted that the question-answering model training method provided in the embodiment can be applied to a terminal-side device or a cloud-side device, which is not limited in the embodiment.

[0136] Step 502: obtaining pre-training samples, wherein the pre-training samples include a to-be-trained image, an image description text corresponding to the to-be-trained image, and a dialogue text associated with the to-be-trained image.

[0137] Step 504: training the pre-training question-answering model based on a predicted image-text matching result corresponding to the pre-training samples, wherein the predicted image-text matching result is obtained by matching the to-be-trained image and target semantic information using the pre-training question-answering model, and the target semantic information is generated based on the image description text and the dialogue text.

[0138] Step 506: obtaining question samples, wherein the question samples include a to-be-trained image and a dialogue text associated with the to-be-trained image.

[0139] Step 508: inputting the question samples into the pre-training question-answering model, obtaining predicted reply information output by the pre-training question-answering model, and training the pre-training question-answering model based on the predicted reply information to obtain a question-answering model.

[0140] Step 510: sending model parameters of the question-answering model to a terminal-side device.

[0141] The pre-training sample can be understood as a sample data for training the pre-training question and answer model, and includes a to-be-trained image, an image description text corresponding to the to-be-trained image, and a dialogue text associated with the to-be-trained image. The to-be-trained image can be understood as any type of training image, and the category of the image is not limited. The image description text corresponding to the to-be-trained image can be understood as text information describing the image content in the to-be-trained image, such as describing the object content and state in the image. The dialogue text associated with the to-be-trained image can be understood as dialogue information discussing the to-be-trained image. The dialogue information can be understood as dialogue data between different subjects, but the form of the dialogue content and the dialogue subject are not limited in this embodiment.

[0142] In actual application, the pre-training question and answer model can be trained according to a predicted image-text matching result corresponding to the pre-training sample. The predicted image-text matching result can be obtained by matching the to-be-trained image and target semantic information based on an information matching layer (FFN) in the pre-training question and answer model. It should be noted that the target semantic information can be obtained by vector matching between the image description text and the dialogue text, that is, the pre-training question and answer model can better understand the semantic information in the dialogue text according to the image description text. Further, after the training of the pre-training question and answer model is completed, a question sample can be obtained. The question sample can be understood as a training sample applied to the question and answer model, and includes a to-be-trained image and a dialogue text associated with the to-be-trained image. The question sample is input into the pre-training question and answer model, the predicted reply information output by an information generation layer (FFN) in the pre-training question and answer model is obtained, the pre-training question and answer model is further trained according to the predicted reply information, and finally, a question and answer model with better training effect can be obtained. The model parameters in the question and answer model are sent to the terminal device, so that the terminal device can construct a question and answer model with better use effect according to the model parameters, and the use effect of the question and answer model in the terminal device is improved.

[0143] Therefore, the pre-training question and answer model that can match the dialogue text and the image is trained, and further, the pre-training question and answer model is trained to realize the image-text alignment of the dialogue text and the image, and then generate and reply the reply information. The question and answer model trains the dialogue generation capability by using the multi-modal data, so as to improve the reply effect of the question and answer model.

[0144] Referring to Figure 6 , Figure 6 A flowchart of a text question and answer method according to one embodiment of the present specification is shown, and specifically includes the following steps.

[0145] It should be noted that, in order to facilitate understanding, the text question and answer method provided in the embodiment can be applied to the application scene of intelligent customer service, and a text reply is given to the text question raised by the user, wherein the automatic reply text of the intelligent customer service is the output reply of the question and answer model.

[0146] Step 602: receiving text question information.

[0147] The text question information can be understood as a text question raised by a user to an intelligent customer service, and the text question is associated with a project in a platform where the current intelligent customer service is located, which is not limited in the embodiment.

[0148] Step 604: inputting the text question information into a question and answer model to obtain text reply information corresponding to the text question information.

[0149] In actual application, after the text question information is input into the question and answer model, the question and answer model can output the text reply information corresponding to the text question information, that is, the text reply of the intelligent customer service to the text question raised by the user can be understood, and it should be noted that the question and answer model provided in the embodiment can be obtained by referring to the training method described in the above embodiment, and will not be described in detail here, including the question and answer model trained by the cloud side device.

[0150] In addition, the question and answer model is also provided in the embodiment to be trained in the end side device and applied in the end side device. Specifically, before the text question information is input into the question and answer model to obtain the text reply information corresponding to the text question information, the method comprises the following steps.

[0151] Obtaining model parameters of the question and answer model in the cloud side device;

[0152] Obtaining a text question sample;

[0153] Based on the model parameters, constructing a question and answer model to be trained, training the question and answer model to be trained based on the text question sample, and obtaining a question and answer model.

[0154] In actual application, the end side device can obtain the model parameters of the question and answer model in the cloud side device, wherein the process of obtaining the model parameters can refer to the process of training the question and answer model in the cloud side device, which will not be described here; further, the end side device can also obtain the text question sample, and construct a question and answer model to be trained by using the model parameters of the question and answer model obtained from the cloud side device, and then train the question and answer model to be trained according to the text question sample in the end side device to obtain a question and answer model; further, the question and answer model trained in the end side device can be applied locally.

[0155] It should be noted that the process of training the model parameters of the question and answer model by the cloud-side device can refer to the description of the training of the question and answer model in the above embodiments, that is, the process of training the question and answer model through three stages, which will not be repeated here.

[0156] Step 606: Display the text reply information.

[0157] In actual application, after the end-side device replies to the text question information by using the question and answer model, the text reply information can be displayed in the display interface of the end-side device.

[0158] To sum up, the text question and answer method provided in this embodiment automatically replies to the text question information by using the question and answer model. The question and answer model can be a question and answer model trained by a cloud-side device or a question model trained by an end-side device, both of which can quickly and reasonably reply to the text question information, thereby improving the use effect of the text reply.

[0159] Corresponding to the method embodiments described above, the present specification also provides question and answer device embodiments, Figure 7 Fig. 1 shows a structural schematic diagram of a question and answer device according to an embodiment of the present specification. As shown in the figure, Figure 7 The device comprises:

[0160] The question obtaining module 702 is configured to obtain question information to be replied to.

[0161] The reply obtaining module 704 is configured to input the question information into a question and answer model to obtain reply information corresponding to the question information. The question and answer model is obtained by training based on predicted reply information. The predicted reply information is obtained by predicting the reply information corresponding to the question sample by using a pre-trained question and answer model. The pre-trained question and answer model is obtained by training based on a predicted image-text matching result. The predicted image-text matching result is obtained by matching a to-be-trained image and target semantic information by using the pre-trained question and answer model. The target semantic information is generated based on an image description text corresponding to the to-be-trained image and a dialogue text associated with the to-be-trained image.

[0162] Optionally, the device further comprises:

[0163] The question and answer model training module is configured to obtain pre-training samples, wherein the pre-training samples comprise a to-be-trained image, an image description text corresponding to the to-be-trained image, and a dialogue text associated with the to-be-trained image.

[0164] training the pre-training question and answer model based on a predicted image-text matching result corresponding to the pre-training sample, wherein the predicted image-text matching result is obtained by matching the to-be-trained image and target semantic information using the pre-training question and answer model, and the target semantic information is generated based on the image description text and the dialogue text;

[0165] obtaining a question sample, wherein the question sample includes a to-be-trained image and dialogue text associated with the to-be-trained image;

[0166] inputting the question sample into the pre-training question and answer model, obtaining predicted reply information output by the pre-training question and answer model, and training the pre-training question and answer model based on the predicted reply information to obtain a question and answer model.

[0167] Optionally, the question and answer model training module is further configured to:

[0168] calculating an autoregressive loss value based on the predicted reply information;

[0169] training the pre-training question and answer model based on the autoregressive loss value to obtain the question and answer model.

[0170] Optionally, the question and answer model training module is further configured to:

[0171] performing image processing on the to-be-trained image in the pre-training sample using the pre-training question and answer model to obtain an image vector;

[0172] performing text processing on the dialogue text associated with the to-be-trained image in the pre-training sample using the pre-training question and answer model to obtain a dialogue vector;

[0173] generating a predicted image-text matching result based on the image vector and the dialogue vector;

[0174] determining an image-text alignment loss value based on the predicted image-text matching result, and training the pre-training question and answer model based on the image-text alignment loss value.

[0175] Optionally, the question and answer model training module is further configured to:

[0176] calculating a dialogue label alignment loss value based on the image vector, and calculating an image dialogue loss value based on the dialogue vector;

[0177] training the pre-training question and answer model based on the dialogue label alignment loss value, the image dialogue loss value, and the image-text alignment loss value.

[0178] Optionally, the question and answer model training module is further configured to:

[0179] obtaining an initial training sample, wherein the initial training sample includes a to-be-trained image and image annotation text corresponding to the to-be-trained image;

[0180] performing image processing on the to-be-trained image by using an initial question and answer model to obtain an image vector;

[0181] performing text processing on the image annotation text corresponding to the to-be-trained image by using the initial question and answer model to obtain a text annotation vector;

[0182] determining a predicted image description text based on the image vector and the text annotation vector;

[0183] training the initial question and answer model based on the predicted image description text.

[0184] Optionally, the question and answer model training module is further configured to:

[0185] calculating an image-text annotation loss value based on the image vector and the text annotation vector;

[0186] calculating an image-text alignment loss value based on the predicted image description text;

[0187] training the initial question and answer model according to the image-text annotation loss value and the image-text alignment loss value.

[0188] Optionally, the apparatus further comprises:

[0189] a model adjustment module configured to send the reply information to a user;

[0190] receiving a feedback result of the user for the reply information;

[0191] adjusting a model parameter in the question and answer model based on the feedback result.

[0192] To sum up, the question and answer device provided by the embodiments of the present specification inputs the question information to be replied into a question and answer model to obtain the reply information output by the question and answer model, wherein the question and answer model can be realized by training a pre-trained question and answer model, the pre-trained question and answer model is trained according to the predicted reply information corresponding to the question sample, and the pre-trained question and answer model is obtained based on the predicted image-text matching result, the predicted image-text matching result is obtained according to the matching of the pre-trained question and answer model to the to-be-trained image and the target semantic information, in addition, the target semantic information is generated based on the image description text corresponding to the to-be-trained image and the dialogue text associated with the to-be-trained image. In this way, the question and answer model can be trained to better learn the target semantic text through a small part of dialogue text and the to-be-trained image, and then the question and answer model can output dialogue content according to the dialogue text, so that the question and answer model can better learn the semantic knowledge in the dialogue text without a large amount of dialogue text as training data, so as to realize more accurate reply information of the trained question and answer model and improve the application effect of the model.

[0193] The above is a schematic scheme of the question and answer device of the present embodiment. It should be noted that the technical scheme of the question and answer device belongs to the same concept as the technical scheme of the question and answer method described above, and the details of the technical scheme of the question and answer device that are not described in detail can be referred to the description of the technical scheme of the question and answer method.

[0194] Figure 8 A structural block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 through a bus 830, and a database 850 is used to save data.

[0195] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 840 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, Near Field Communication (NFC).

[0196] In one embodiment of the present specification, the above-mentioned components of the computing device 800 and other components not shown in the Figure 8 may be connected to each other, such as through a bus. It should be understood that Figure 8 The computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0197] The computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 can also be a mobile or stationary server.

[0198] Among them, the processor 820 is used to execute the following computer executable instructions, which are executed by the processor to implement the steps of the above question and answer method, question and answer model training method, text question and answer method.

[0199] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the question and answer method, the question and answer model training method, and the text question and answer method belong to the same concept. Details of the technical scheme of the computing device that are not described in detail can be seen from the description of the technical scheme of the question and answer method, the question and answer model training method, and the text question and answer method.

[0200] An embodiment of the present specification further provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the question and answer method, the question and answer model training method, and the text question and answer method.

[0201] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the question and answer method, the question and answer model training method, and the text question and answer method belong to the same concept. Details of the technical scheme of the storage medium that are not described in detail can be seen from the description of the technical scheme of the question and answer method, the question and answer model training method, and the text question and answer method.

[0202] An embodiment of the present specification further provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the question and answer method, the question and answer model training method, and the text question and answer method.

[0203] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program and the technical schemes of the question and answer method, the question and answer model training method, and the text question and answer method belong to the same concept. Details of the technical scheme of the computer program that are not described in detail can be seen from the description of the technical scheme of the question and answer method, the question and answer model training method, and the text question and answer method.

[0204] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0205] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or deletions according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0206] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present specification.

[0207] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0208] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their entire scope and equivalents.

Claims

1. A question-and-answer method, comprising: Get information on questions that are yet to be answered; Input the question information into the question-answering model to obtain the corresponding answer information; The question-answering model is trained based on predicted response information, which is obtained by predicting the response information corresponding to the question sample using the pre-trained question-answering model. The pre-trained question-answering model is trained based on predicted image-text matching results, which are obtained by matching the training image and target semantic information using the pre-trained question-answering model. The target semantic information is generated based on the image description text corresponding to the training image and the dialogue text associated with the training image.

2. The method according to claim 1, before inputting the question information into the question-answering model to obtain the answer information corresponding to the question information, further comprising: Obtain pre-training samples, wherein the pre-training samples include a training image, image description text corresponding to the training image, and dialogue text associated with the training image; The pre-trained question answering model is trained based on the predicted image-text matching results corresponding to the pre-trained samples, wherein the predicted image-text matching results are obtained by the pre-trained question answering model matching the image to be trained and the target semantic information, and the target semantic information is generated based on the image description text and the dialogue text; Obtain problem samples, wherein the problem samples include a training image and dialogue text associated with the training image; The question sample is input into the pre-trained question answering model to obtain the predicted response information output by the pre-trained question answering model, and the pre-trained question answering model is trained based on the predicted response information to obtain the question answering model.

3. The method according to claim 2, wherein training the pre-trained question-answering model based on the predicted response information to obtain the question-answering model includes: Calculate the autoregressive loss value based on the predicted response information; The pre-trained question-answering model is trained based on the autoregressive loss value to obtain the question-answering model.

4. The method according to claim 2 or 3, wherein training the pre-trained question-answering model based on the predicted image-text matching result corresponding to the pre-trained sample includes: For the training images in the pre-training samples, the pre-trained question-answering model is used to perform image processing on the training images to obtain image vectors; For the dialogue text associated with the image to be trained in the pre-training samples, the pre-trained question answering model is used to process the dialogue text to obtain dialogue vectors; Based on the image vector and the dialogue vector, a predicted image-text matching result is generated; The image-text alignment loss value is determined based on the predicted image-text matching result, and the pre-trained question answering model is trained based on the image-text alignment loss value.

5. The method according to claim 4, wherein training the pre-trained question-answering model based on the image-text alignment loss value comprises: The dialogue annotation alignment loss value is calculated based on the image vector, and the image dialogue loss value is calculated based on the dialogue vector. The pre-trained question answering model is trained based on the dialogue annotation alignment loss value, the image dialogue loss value, and the image-text alignment loss value.

6. The method according to claim 2, further comprising, before obtaining the pre-training samples: Obtain initial training samples, wherein the initial training samples include the image to be trained and the image annotation text corresponding to the image to be trained; For the image to be trained, the initial question-answering model is used to process the image to obtain an image vector; For the image annotation text corresponding to the image to be trained, the initial question-answering model is used to process the image annotation text to obtain the text annotation vector; Based on the image vector and the text annotation vector, determine the predicted image description text; The initial question-answering model is trained based on the predicted image description text.

7. The method according to claim 6, wherein training the initial question-answering model based on the predicted image description text comprises: Calculate the image and text annotation loss value based on the image vector and the text annotation vector; Based on the predicted image description text, calculate the image-text alignment loss value; The initial question-answering model is trained based on the image and text annotation loss value and the image and text alignment loss value.

8. The method according to claim 1, further comprising, after obtaining the response information corresponding to the question information: The reply information will be sent to the user; Receive feedback from the user regarding the response information; The model parameters in the question-answering model are adjusted based on the feedback results.

9. A question-answering method applied to cloud-side devices, comprising: The receiving end-side device sends a question-and-answer request, wherein the question-and-answer request carries question information; Input the question information into the question-answering model to obtain the corresponding answer information; The question-answering model is trained based on predicted response information, which is obtained by predicting the response information corresponding to the question sample using the pre-trained question-answering model. The pre-trained question-answering model is trained based on predicted image-text matching results, which are obtained by matching the training image and target semantic information using the pre-trained question-answering model. The target semantic information is generated based on the image description text corresponding to the training image and the dialogue text associated with the training image. The response information is sent to the end-side device.

10. A question-answering model training method, applied to cloud-side devices, comprising: Obtain pre-training samples, wherein the pre-training samples include a training image, image description text corresponding to the training image, and dialogue text associated with the training image; A pre-trained question answering model is trained based on the predicted image-text matching results corresponding to the pre-trained samples. The predicted image-text matching results are obtained by matching the image to be trained and the target semantic information using the pre-trained question answering model. The target semantic information is generated based on the image description text and the dialogue text. Obtain problem samples, wherein the problem samples include a training image and dialogue text associated with the training image; The question sample is input into the pre-trained question answering model to obtain the predicted answer information output by the pre-trained question answering model, and the pre-trained question answering model is trained based on the predicted answer information to obtain the question answering model; The model parameters of the question-and-answer model are sent to the edge device.

11. A text-based question-answering method, applied to an edge device, comprising: Receive text error message; The text question information is input into the question-answering model to obtain the text answer information corresponding to the text question information, wherein the question-answering model is obtained according to the method described in claim 10 above; The text response information will be displayed.

12. The method according to claim 11, before inputting the text question information into the question-answering model to obtain the text response information corresponding to the text question information, includes: Obtain the model parameters of the question-answering model in the cloud-side device; Obtain text question samples; A question-answering model to be trained is constructed based on the model parameters, and the question-answering model to be trained is trained based on the text question samples to obtain the question-answering model.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images