Question and answer method and apparatus based on large model, device and storage medium

Abstract

The present application discloses a question and answer method and apparatus based on a large model, a device and a storage medium. The method comprises: obtaining session content of a question and answer process; when the session content comprises non-text content, determining content relevancy between text content and the non-text content in the session content; when the content relevancy meets a preset relevancy condition, determining, from a candidate reply model set, a target reply model matched with the content modality of the non-text content; and on the basis of the session content, obtaining reply content of a question in the session content by means of the target reply model.

Description

Large model-based question and answer method, device and equipment and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410652183.9, filed on May 24, 2024, and entitled "Large model-based question and answer method, device and equipment and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computers, in particular to the technical field of large models, and provides a large model-based question and answer method, device and equipment and storage medium.

[0003] BACKGROUND

[0004] With the development of artificial intelligence (AI) technology, large language models (LLM) are gradually being widely used. A large language model is a relatively complex AI model that can understand and generate human language. The large language model is characterized by a large size, with a scale of ten billion parameters or a scale of one hundred billion parameters. The large language model is often applied to scenarios such as text summarization, question and answer, and translation. For example, in the question and answer scenario, the question text can be input in the form of a conversation. The input question text is provided as a prompt to the large language model, and the large language model can generate the reply content corresponding to the current question text according to the prompt. Since the large language model is a deep learning model trained based on a large amount of text data, it can greatly improve the accuracy of the reply content in the question and answer scenario. SUMMARY

[0005] The embodiments of the present application provide a large model-based question and answer method, device and equipment and storage medium, which are used to enrich the use scenarios of the large model question and answer and improve the accuracy of the reply content.

[0006] The embodiments of the present application provide a large model-based question and answer method, which comprises:

[0007] Obtaining the conversation content of the question and answer process;

[0008] When the conversation content includes non-text content, determining the content correlation degree between the text content and the non-text content in the conversation content;

[0009] When the content correlation degree meets a preset correlation degree condition, determining a target reply model from a candidate reply model set, the target reply model being adapted to the content mode of the non-text content;

[0010] According to the conversation content, the reply content of a question in the conversation content is obtained through the target reply model.

[0011] The embodiment of the application further provides a large model-based question and answer device, comprising:

[0012] An input unit is configured to obtain conversation content of a question and answer process;

[0013] A multi-modal intention recognition unit is configured to determine content correlation between text content and non-text content in the conversation content when the conversation content comprises the non-text content;

[0014] A reply model determination unit is configured to determine, when the content correlation meets a preset correlation condition, a target reply model from a candidate reply model set, the target reply model being adapted to a content mode of the non-text content;

[0015] A reply execution unit is configured to obtain, according to the conversation content, reply content of a question in the conversation content through the target reply model.

[0016] The embodiment of the application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements steps of any of the above methods when executing the computer program.

[0017] The embodiment of the application further provides a computer storage medium, which stores a computer program, and the computer program is executable by a processor to implement steps of any of the above methods.

[0018] The embodiment of the application further provides a computer program product, comprising a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program, so that the computer device executes steps of any of the above methods.

[0019] In the embodiments of the present application, the content input form can support multiple content modalities in the question and answer process, and is no longer limited to a single text dimension. For example, the question and answer process can be based on pictures, videos, or audio and other non-text content modalities. The content input form is more diverse, and the use scenarios of the question and answer process are expanded. Moreover, the introduction of multi-modal content can express information in different content forms through different content modalities. The information expression method is more diverse. For example, pictures or videos can express information that text cannot express. Therefore, more diverse information can be provided to the large model. Compared with the related art, which can only express information through text, the method of the embodiments of the present application helps to improve the accuracy of the large model in understanding the input content, helps the large model to more accurately understand the user's intention, and thus selects a more accurate target reply model to generate more accurate reply content, thereby improving the accuracy of the reply content.

[0020] In the question and answer process, when the conversation content includes non-text content, the method of the embodiments of the present application performs multi-modal intent recognition on the conversation content and the non-text content. The multi-modal intent recognition result indicates the content relevance between the conversation content and the non-text content. Therefore, a target reply model that is suitable can be selected according to the content relevance, the accuracy of the selected reply model is improved, and thus more accurate reply content can be generated for the user, thereby improving the accuracy of the reply content.

[0021] BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art descriptions. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without any creative effort.

[0023] FIG. 1 is a schematic diagram of an application scenario provided by the embodiments of the present application;

[0024] FIG. 2 is a flowchart of a question and answer method based on a large model provided by the embodiments of the present application;

[0025] FIGS. 3A and 3B are example diagrams of a question and answer conversation interface provided by the embodiments of the present application;

[0026] FIGS. 4A to 4C are example diagrams of content relevance provided by the embodiments of the present application;

[0027] FIG. 5 is a logic diagram for determining actual input content provided by the embodiments of the present application;

[0028] FIG. 6A is an example diagram of a target prompt template provided by the embodiments of the present application;

[0029] FIG. 6B is an example diagram of target prompt content provided by an embodiment of the present application;

[0030] FIG. 7 is a flow diagram of determining a multi-modal intent recognition result provided by an embodiment of the present application;

[0031] FIG. 8 is a model structure diagram of a Transformer provided by an embodiment of the present application;

[0032] FIG. 9 is a flow diagram of an image encoding process provided by an embodiment of the present application;

[0033] FIG. 10 is an example diagram of iterative text prediction provided by an embodiment of the present application;

[0034] FIG. 11 is a flow diagram of a training method provided by an embodiment of the present application;

[0035] FIG. 12A is an example diagram of a Prompt template in a training stage provided by an embodiment of the present application;

[0036] FIG. 12B is another example diagram of a Prompt instruction in a training stage provided by an embodiment of the present application;

[0037] FIG. 13 is a flow diagram of another large model-based question and answer method provided by an embodiment of the present application;

[0038] FIG. 14 is a flow diagram of a model training process of a text intent recognition model provided by an embodiment of the present application;

[0039] FIG. 15A is an example diagram of a Prompt template in a training stage provided by an embodiment of the present application;

[0040] FIG. 15B is another example diagram of a Prompt instruction in a training stage provided by an embodiment of the present application;

[0041] FIG. 16A is a processing flow diagram of a question and answer process provided by an embodiment of the present application;

[0042] FIG. 16B is an example diagram of a question and answer process provided by an embodiment of the present application;

[0043] FIG. 17 is a structure diagram of a video frame encoding and decoding apparatus provided by an embodiment of the present application;

[0044] FIG. 18 is a component structure diagram of a computer device provided by an embodiment of the present application;

[0045] FIG. 19 is a component structure diagram of another computer device to which an embodiment of the present application is applied.

[0046] Manner of implementing the present application

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. The embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily without conflict. Moreover, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that here.

[0048] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, some key terms used in the embodiments of the present application are explained first.

[0049] Language Model: a model used for modeling natural language, whose purpose is to predict the next word or character of a given text sequence. Language models can be used for various natural language processing tasks, such as semantic extraction of text, text generation, machine translation, speech recognition, etc. Currently, pre-trained language models (PLM) based on Transformer are commonly used in various tasks of natural language processing and can usually achieve good results. For example, commonly used pre-trained language models include Bidirectional Encoder Representation from Transformers (Bert) and Generative Pre-Trained Transformer (GPT).

[0050] Large Language Model: also known as large model, natural language model or large-scale language model, etc. It refers to a natural language processing model with large-scale parameters and training data. The training process of a large language model usually adopts an unsupervised learning method, i.e. training the model through a large-scale text corpus to learn the probability distribution and language rules of language. In the training process, the large language model usually uses a language model (Language Model) as the objective function, i.e. optimizing the model parameters by maximizing the prediction probability of the next word. For example, the GPT series model based on the Transformer model structure is trained on a large-scale corpus and can generate high-quality natural language text such as articles and dialogues.

[0051] Content modality: In the embodiments of the present application, each medium form or bearing method of information can be referred to as a modality, for example, audio, video, image and text can be referred to as a modality. In the field of AI, multi-modal means that multiple different medium forms (also referred to as input or output methods) can be combined, that is, the AI system can understand and process multiple types of data.

[0052] Content relevance: In the embodiments of the present application, the content relevance refers to the relevance between the conversation content in the question and answer process and the non-text content involved, for example, the relevance between the text content and the non-text content. The relevance can represent whether the non-text content is related to the conversation content, or can be understood as the degree of dependence on the non-text content when answering the last round of questions. When the degree of dependence is high, it means that the answer to the question must depend on the non-text content, and then subsequent processing needs to be based on the non-text content. But when the degree of dependence is low, it means that whether the non-text content has little influence on the answer to the question.

[0053] Pretrained model: refers to a model trained on a large corpus, usually using unsupervised learning methods such as autoencoder, language model, etc. The basic idea of the pre-trained model is to use a large corpus to learn a large amount of general knowledge and rules through unsupervised learning methods, so as to serve as a basic model for various natural language processing tasks.

[0054] Transformer: a common deep learning model architecture, widely used in natural language processing, computer vision (CV) and speech processing. The Transformer was first proposed as a sequence-to-sequence model architecture for machine translation, consisting of an encoder and a decoder, both of which are composed of a series of identical Transformer blocks. Each Transformer block is composed of at least a multi-head self-attention layer and a feed-forward neural network layer. Currently, the Transformer has become a common architecture in natural language processing and is often used as a pre-trained model. In addition to language-related applications, the Transformer has also been applied to computer vision, audio processing and other fields.

[0055] Prompt Learning: A natural language generation technique based on human prompts to better use the knowledge of pre-trained language models. It adds additional text to the input segment to improve the quality and efficiency of the final generated model. The basic idea is to give a human prompt or partial generated result during the generation process, called prompt token or prompt text, which represents commands or instructions for the large language model to perform actions or generate outputs, so that the model can continue to generate text based on this. Prompt learning can be applied to various natural language generation tasks such as text summarization, machine translation, and dialogue generation. Prompt learning can help the model better understand the context and context of the generation task, generating more accurate and natural text, thus improving the quality and efficiency of the generation model, while also reducing the generation space of the generation model, thus improving the efficiency and speed of generation.

[0056] Fine-tuning: Also known as fine-tuning training, it refers to further training a pre-trained model using a specific data set to adapt to a specific task or field, so it can also be called adaptive training. Ultimately, it can improve the performance and effectiveness of the model on specific tasks. The process of prompt learning usually involves fine-tuning the model to better understand the prompt text and the context and context of the generation task, thus generating more accurate and natural text. For example, one way to fine-tune the model is supervised fine-tuning (SFT).

[0057] Attention mechanism: a way of measuring the intermediate features in the network by using advanced information, so that the network pays attention to the part of the information that helps to judge the image and ignores irrelevant information. The essence of attention mechanism comes from human visual attention mechanism. When people perceive things, they generally do not look at everything from beginning to end every time, but often observe and pay attention to a specific part according to their needs. When people find that a scene often appears in a part where they want to observe, they will learn to put their attention on that part in the future when similar scenes appear. Therefore, attention mechanism is essentially a means of filtering high-value information from a large amount of information. In a large amount of information, different information is of different importance to the result, which can be reflected by assigning different weights. In other words, attention mechanism can be understood as a rule of allocating weights when synthesizing multiple sources. It is usually used to solve the problem that it is difficult to obtain a reasonable vector representation when the input sequence of the model is long. The method is to retain the intermediate results of the model, learn them with a new model, and associate them with the output, so as to achieve the purpose of information filtering. Attention mechanism includes attention mechanism, self-attention mechanism, single-head attention mechanism and multi-head attention mechanism, etc.

[0058] The scheme provided by the embodiments of the present application relates to the generation of reply content for a question raised in a question and answer process. In the process of generating reply content, the scheme provided by the embodiments of the present application allows users to add non-text content such as images when raising a question, so as to enrich the information contained in the input content. Therefore, before generating the reply content, it is necessary to determine whether the reply to the current question needs to rely on these non-text content according to the technical scheme provided by the embodiments of the present application, that is, to determine the content relevance between the question and the non-text content, so as to determine the actual input content input into the large model to generate the reply content. And it is also necessary to determine the target reply model needed to be used according to the current content intention, which is used to generate the reply content. When determining the content relevance and judging the target reply model, a deep learning-based artificial neural network model needs to be used. Taking the determination of content relevance as an example, the embodiments of the present application use machine learning and the method of prompt learning for pre-trained models to obtain a multi-modal intention recognition model capable of determining content relevance. The multi-modal intention recognition model realizes the ability to process and understand the relevance and semantic relationship between the conversation content and the non-text content in the question and answer scene based on machine learning, to determine the content relevance between the conversation content and the non-text content.

[0059] The determination of the content relevance and the judgment of the target reply model in the embodiments of the present application can be divided into two parts, including a training part and an application part. The training part relates to the technical field of machine learning. In the training part, the pre-training language model is prompted to learn by using the training data in the question and answer field, and the model parameters are continuously adjusted by an optimization algorithm until the model converges, so that the fine-tuned pre-training language model can adapt to the question and answer field and better understand the context and context of the task of determining the content relevance and the judgment of the target reply model in the question and answer field, thereby generating more accurate task results. The application part is used to use the artificial neural network model trained in the training part to perform multi-modal intent recognition on the text content and non-text content input in the actual use process, and determine the target reply model used finally according to the multi-modal intent recognition result, so as to generate reply content through the target reply model. In addition, it should be noted that the artificial neural network model in the embodiments of the present application can be online training or offline training, which is not limited here. In this paper, offline training is taken as an example for illustration.

[0060] The related technology provides a model framework for pure text intent recognition. In this architecture, the input content of a round of question and answer process is only text content, so that the text content can only be used as a prompt text to generate corresponding reply content. In the question and answer scene, the prompt text is the main basis for generating reply content, so the more accurate and rich the information provided by the prompt text is, the higher the accuracy of the reply content generated by the large language model is. However, in the related technology, the prompt text can only be limited to a single dimension of text, and the text dimension can only provide limited information, which limits the accuracy of the reply content.

[0061] Based on this, the embodiments of the present application provide a question and answer method based on a large model. In this method, it is no longer limited to a single text dimension, and multiple content modalities can be supported in the question and answer process, so that the question and answer process can be performed based on non-text content modalities such as pictures, videos or audio, thereby expanding the use scenarios of the question and answer process. Moreover, the introduction of multi-modal content can express different forms of information through different content modalities, and the information expression is more rich, so that more rich information can be provided to the large model, which helps to improve the accuracy of the large model in understanding the input content, thereby generating more accurate reply content and improving the accuracy of the reply content. In the question and answer process, when the conversation content includes non-text content, the method of the embodiments of the present application performs multi-modal intent recognition on the conversation content and the non-text content, and the multi-modal intent recognition result indicates the content relevance between the conversation content and the non-text content, so that a target reply model that is suitable can be selected according to the content relevance, thereby improving the accuracy of the selected reply model and further improving the accuracy of the reply content.

[0062] In addition, in the embodiments of the present application, the supervised fine-tuning training method is used to fine-tune the pre-trained model, so that the final model can better adapt to the current question and answer scene tasks, such as multi-modal intent recognition tasks and text intent recognition tasks, etc., to improve the accuracy of the task results in the question and answer scene, and finally improve the accuracy of the reply content.

[0063] The application scenarios to which the technical solutions of the embodiments of the present application can be applied will be briefly introduced below. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0064] The scheme provided by the embodiments of the present application can be applied to a question and answer scene. As shown in FIG. 1, it is a schematic diagram of an application scenario provided by the embodiments of the present application, which can include a terminal device 101 and a server 102 in the scene.

[0065] The terminal device 101 can be, for example, a mobile phone, a tablet computer (PAD), a notebook computer, a desktop computer, a smart home appliance (such as a smart television), a smart vehicle-mounted device, a smart wearable device, and an aircraft, etc. The server 102 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, but is not limited thereto.

[0066] The method of the embodiments of the present application can be performed by the terminal device 101 or the server 102 alone, or by the server 102 and the terminal device 101 together. When performed by the terminal device 101 or the server 102 alone, both the model training and the application process can be implemented by the terminal device 101 or the server 102 alone. For example, the pre-trained large language model can be fine-tuned and trained by using the collected question and answer sample set on the terminal device 101, to obtain the trained multi-modal intent recognition model and the text intent recognition model. Correspondingly, after training, the terminal device 101 can use the trained multi-modal intent recognition model to recognize the content relevance between the text content and the non-text content in the last round of question and answer process, and thus perform the last round of question and answer process according to the content relevance. In some other embodiments, the above process can also be performed by the server 102. When performed by the server 102 and the terminal device 101 together, the server 102 can train the multi-modal intent recognition model, and then deploy the trained multi-modal intent recognition model to the terminal device 101, so that the terminal device 101 implements the question and answer process. In some other embodiments, part of the model training or application process can be implemented by the terminal device 101, and part of the model training or application process can be implemented by the server 102, and the two cooperate to implement the model training or application process. In actual application, specific configurations can be made according to the situation, which is not limited in the present application.

[0067] The server 102 and the terminal device 101 can each include one or more processors, memories, and I / O interfaces for interaction, etc. In addition, the server 102 can also be configured with a database, which can be used to store the model parameters obtained by training, etc. The memories of the server 102 and the terminal device 101 can also store program instructions required for respective execution in the large model based question and answer method provided by the embodiments of the present application, which can be used to implement the model training process or the large model based question and answer process provided by the embodiments of the present application when executed by the processor.

[0068] It should be noted that when the large model based question and answer method provided by the embodiments of the present application is performed by the server 102 or the terminal device 101 alone, the application scenarios described above can only include the server 102 or the terminal device 101 alone, or the server 102 and the terminal device 101 can be considered as the same device. Of course, in actual application, when the large model based question and answer method provided by the embodiments of the present application is performed by the server 102 and the terminal device 101 together, the server 102 and the terminal device 101 can also be the same device, i.e., the server 102 and the terminal device 101 can be different functional modules of the same device, or virtual devices virtualized by the same physical device.

[0069] For example, the user can provide input content through the terminal device 101 to initiate a round of question and answer process, so that the server 102 can receive the input content provided by the user, and then generate reply content of the last round of question and answer process according to the conversation content by using the method of the embodiment of the application, and return to the terminal device 101 for presentation.

[0070] In the embodiment of the application, the terminal device 101 and the server 102 can be directly or indirectly connected in communication through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a wireless fidelity (WIFI) network. Of course, other possible networks can also be used, and the embodiment of the application does not limit this. It should be noted that the number of terminal devices and servers is not limited in practice, and is not specifically limited in the embodiment of the application.

[0071] FIG. 2 is a flowchart of a large model-based question and answer method provided by the embodiment of the application. The method can be executed by a computer device. The computer device can be, for example, a terminal device or a server as shown in FIG. 1. The method can include the following steps.

[0072] Step 201: Obtain conversation content of at least one round of question and answer process, the conversation content at least including: text content recording a question of a last round of question and answer process.

[0073] The method in the embodiment of the application can be applied to a question and answer scenario. In the question and answer scenario, the user can input a question, and the large model can provide corresponding reply content to the user according to the question. After the user submits a question of a round of question and answer process, conversation content of at least one round of question and answer process including the round of question and answer process can be obtained. The at least one round of question and answer process can refer to the last round of question and answer process, or can include the last round of question and answer process and at least one previous round of question and answer process. The question and answer scenario can be any scenario of providing a question and providing reply content according to the question, such as a retrieval scenario, a knowledge question and answer scenario, a question and answer robot scenario, or an AI assistant scenario.

[0074] FIGS. 3A and 3B are schematic diagrams of a question and answer session interface according to an embodiment of the present application. In the question and answer process, the user can provide input content through the session interface to interact with the large model. If the method shown in FIG. 2 is performed by the terminal device, after the user provides input content through the session interface on the terminal device, the terminal device can execute the subsequent method process according to the input content to obtain the corresponding reply content, and present the reply content on the session interface. If the method shown in FIG. 2 is performed by the server, after the user provides input content through the session interface on the terminal device, the terminal device can send the input content to the server, the server can execute the subsequent method process according to the input content to obtain the corresponding reply content, and return the reply content to the terminal device, so that the terminal device presents the reply content on the session interface.

[0075] FIG. 3A illustrates a session interface in a pure text scenario. In the pure text scenario, the input content of the user only includes text content, and all information comes from the text content. However, the information expressed by the text content is limited, which limits the application of the question and answer scenario. Therefore, in the present application, a question and answer method supporting non-text content is provided. As shown in FIG. 3B, the user can input non-text content in the session interface. For example, in FIG. 3B, an image is taken as an example. The image can vividly describe the content that cannot be expressed by the text content, so that the input content covers more information, which helps the large model to provide more accurate reply content.

[0076] In the present application, the session content includes input content and reply content involved in at least one round of question and answer process. For example, as shown in FIGS. 3A and 3B, the input content in the present application can include but is not limited to the following content.

[0077] (1) Text content, which records a question in one round of question and answer process. Of course, in addition to the question, other description content can also be included as auxiliary content for answering the question. For example, as shown in FIG. 3A, “A city weather today how” and “tomorrow” are text content, and as shown in FIG. 3B, “what is the plant in the picture” and “what color” are also text content.

[0078] In the present application, the question is mainly provided in the form of text. In actual scenarios, the question can also be provided in other ways, such as voice or video, which is not limited in the present application.

[0079] (2) Non-text content, which refers to content expressed in other content modalities in addition to text content. Other content modalities refer to content modalities in non-text modalities, such as images, audio or video, etc.

[0080] In the embodiments of the present application, when replying, the reply is usually made to the last round of questions. However, for the last round of question-answering process, the non-text content can be provided in the round of question-answering process or in the historical question-answering process before the round of question-answering process, that is, whether the non-text content is provided in the last round of question-answering process is not limited. As shown in FIG. 3B, in the first round of question-answering process (i.e., the round of “what is the plant in the picture”), an image is uploaded, and thus the input content of the round can include the image. In the second round of question-answering process (i.e., the round of “what color is it?”), although no image is uploaded, the image uploaded in the first round of question-answering process can still be used as the non-text content of the second round of question-answering process.

[0081] In some embodiments, the number of rounds of non-text content that can be traced back for each round of question-answering process can be preset, and the non-text content within the number of rounds can be used as the non-text content of the last round of question-answering process. For example, if the number of rounds is set to 5, the non-text content uploaded in the 5 rounds of question-answering process before the last round of question-answering process is still valid for the last round of question-answering process.

[0082] In the embodiments of the present application, one question and the corresponding reply obtained according to the question can be considered as one round of question-answering process.

[0083] In step 202, when the conversation content further includes non-text content, multi-modal intent recognition is performed on the conversation content and the non-text content, and a corresponding multi-modal intent recognition result is obtained.

[0084] In this step, when the conversation content further includes non-text content, the content correlation between the text content and the non-text content in the conversation content is determined.

[0085] In the embodiments of the present application, although the non-text content is supported as input content, it is considered that in actual scenarios, the non-text content uploaded by the user can not be related to the last round of question-answering process, and thus such non-text content does not bring beneficial influence to the subsequent reply process. On the contrary, since more data is processed, the processing burden of the model can also be increased. Therefore, in the embodiments of the present application, by performing multi-modal intent recognition on the conversation content and the non-text content, the multi-modal intent recognition result is used to indicate the content correlation between the conversation content (e.g., the text content) and the non-text content. For example, the multi-modal intent recognition result can indicate whether the multi-round conversation is related to the non-text content, or the multi-modal intent recognition result can indicate the relevant degree type or level to which the multi-round conversation is related to the non-text content, or the multi-modal intent recognition result can indicate the correlation value between the multi-round conversation and the non-text content.

[0086] In the multi-modal intent recognition of the conversation content and the non-text content, the conversation content can refer to the text content contained therein, which can include question text content and reply text content involved in the at least one round of question and answer process.

[0087] In some embodiments, the content relevance can also represent the degree of dependence of the non-text content when the question mentioned in the reply text content. For example, the content relevance refers to the degree of dependence of the last round of question and answer process on the non-text content in the input content. For example, one expression of the degree of dependence can be "dependent (or yes)" or "not dependent (or no)", dependent means that the non-text content needs to be dependent when the question mentioned in the reply text content, and not dependent means that the non-text content does not need to be dependent when the question mentioned in the reply text content. Accordingly, the content relevance is relevant or irrelevant. For another example, another expression of the degree of dependence can be expressed by a numerical value, for example, a numerical range of 0-1, the closer to 0, the lower the degree of dependence, and the lower the content relevance, the closer to 1, the higher the degree of dependence, and the higher the content relevance.

[0088] In some embodiments, the content relevance can also be understood as the relevance of the non-text content to the at least one round of question and answer process.

[0089] In some embodiments, the expression of the content relevance can be a pre-defined set of relevance types, and then the content relevance between the text content and the non-text content can be attributed to a certain relevance type. For example, one relevance type can be represented by a category text, different relevance types can be represented by different category texts, and one category text can represent one relevance range, and each category text can include at least one character position, which can be in the form of letters, symbols, numbers, etc.

[0090] For example, the relevance type can be defined as "yes" and "no". "Yes" represents that the conversation content and the non-text content are relevant, and the non-text content needs to be dependent when replying to the question. "No" represents that the conversation content and the non-text content are not relevant, and the non-text content does not need to be dependent when replying to the question.

[0091] For example, the relevance type can be defined as "completely relevant", "basically irrelevant", and "completely irrelevant". The completely relevant type represents a high relevance between the conversation content and the non-text content, and a high dependence on the non-text content when answering the question. The basically irrelevant type represents a low relevance between the conversation content and the non-text content, and the question can be answered without dependence on the non-text content. The completely irrelevant type represents a very low relevance between the conversation content and the non-text content, and the question can be answered without dependence on the non-text content. The reason for setting the basically irrelevant type is that in practice, if only two extreme types of "completely relevant (or yes)" and "completely irrelevant (or no)" are defined, classification errors may occur for some boundary questions. By defining three types and adding the basically irrelevant type, the classification error can be reduced, the accuracy of the content relevance can be improved, and the accuracy of the reply content can be improved.

[0092] Referring to FIGS. 4A-4C, examples of different content relevance between text content and non-text content provided by embodiments of the present application are shown. Here, the non-text content is taken as an example of a picture. In FIG. 4A, the text content is "describe this picture", and the intent of the current question is to describe the content in the picture. Obviously, the question cannot be answered without the current uploaded picture, and therefore, the answer to the question is obviously completely relevant to the picture content. In FIG. 4B, the current uploaded picture contains a picture of a walnut, and the text content is "where are walnuts usually used in cooking or recipes?". Both contain "walnuts", but the question can be answered without the current picture content. Therefore, the text content and the non-text content are related, but the answer to the question is obviously basically irrelevant to the picture content. In FIG. 4C, the current uploaded picture contains two bees, and the text content is "what is the weather like today?". There is no correlation between the two, and therefore, the answer to the question obviously does not depend on the current picture content, and the answer to the question is obviously completely irrelevant to the picture content.

[0093] In some embodiments, the content relevance can also be expressed by a numerical value, for example, a numerical range of 0-1. The closer to 0, the lower the content relevance, and the closer to 1, the higher the content relevance.

[0094] In embodiments of the present application, when determining the content relevance between the text content and the non-text content, the content relevance can be determined according to the conversation content (for example, the text content) and the non-text content of at least one round of question and answer process. That is, the multi-modal intent recognition can be performed by comprehensively considering the content and the non-text content involved in the multi-round dialogue process to obtain the corresponding content relevance.

[0095] In actual scenarios, the user can upload non-text content of multiple content modalities in the question-answering process, that is, the non-text content can include content corresponding to at least one content modality, for example, can include pictures, videos, or audio, and the like.

[0096] In some embodiments, when the content relevance is determined, the content relevance can be determined for the content of each content modality respectively, and then the multi-modal intent recognition result can include the content relevance corresponding to at least one content modality, that is, the content relevance between the content corresponding to each content modality in the at least one content modality and the text content. In this way, in the subsequent step, at least one target content modality can be determined from the at least one content modality, in which the content relevance meets a preset relevance condition, and the target reply model that is adapted to the at least one target content modality can be determined from the candidate reply model set.

[0097] For example, for the content of each content modality, the content relevance between the conversation content and the content of the content modality can be determined by performing multi-modal intent recognition on the text content in the conversation content and the content of the content modality.

[0098] In some embodiments, when the content relevance is determined, the content corresponding to at least one content modality can be taken as a whole to determine the content relevance with the conversation content.

[0099] In the embodiments of the present application, when the number of non-text content is multiple, the sub-relevance of each non-text content can be determined respectively when the content relevance is determined, and then the content relevance includes the sub-relevance corresponding to each non-text content. For example, the non-text content includes two images, and the sub-relevance between the two images and the text content can be determined respectively.

[0100] Step 203: determining a target reply model adapted from the candidate reply model set according to the multi-modal intent recognition result.

[0101] In this step, when the content relevance meets the preset relevance condition, the target reply model adapted to the content modality of the non-text content is determined from the candidate reply model set.

[0102] In the embodiments of the present application, the candidate reply model set includes multiple candidate reply models, and each candidate reply model corresponds to an intent type. In this step, the target reply model adapted to the intent can be selected according to the intent recognition result.

[0103] In some embodiments, the multimodal intent recognition result indicates a content relevance between the conversation content and the non-text content. If the content relevance indicated by the multimodal intent recognition result satisfies a preset relevance condition, a target reply model that is adapted to the intent type of the conversation content is determined from a candidate reply model subset that is related to the content modality of the non-text content and that is included in the candidate reply model set.

[0104] For example, the candidate reply models in the candidate reply model set can be divided into different candidate reply model subsets according to whether the preset relevance condition is satisfied and the content modality of the non-text content. When the preset relevance condition is satisfied, the candidate reply model subset is determined according to the content modality of the non-text content. When the relevance condition is not satisfied, the corresponding candidate reply model subset is a candidate reply model subset related to the text modality. Different candidate reply models in the same candidate reply model subset can correspond to different intent types, so that reply content can be generated according to the corresponding intent type.

[0105] For example, when the content relevance is “completely relevant”, the corresponding candidate reply model subset is a candidate reply model subset related to the content modality of the non-text content. When the content relevance is “basically irrelevant” or “completely irrelevant”, the corresponding candidate reply model subset is a candidate reply model subset related to the text modality.

[0106] The preset relevance condition can be set according to the actual scene requirements, and the specific content of the embodiments of the present application is not limited.

[0107] For example, when the content relevance is represented by a numerical value, the preset relevance condition can be a set relevance threshold, such as 50% or 80%, and whether the non-text content is used as the actual input content can be determined by comparing the content relevance with the preset relevance threshold. For example, when the content relevance is not less than the preset relevance threshold, it is considered that the preset relevance condition is satisfied. Conversely, when the content relevance is less than the preset relevance threshold, it is considered that the preset relevance condition is not satisfied.

[0108] For another example, when the content relevance is represented by a category text, the preset relevance condition can be that the content relevance is a specified category text. For example, when the content relevance is completely relevant, it is considered that the preset relevance condition is satisfied. Conversely, when the content relevance is other category text (such as basically irrelevant or completely irrelevant), it is considered that the preset relevance condition is not satisfied.

[0109] When the multimodal intent recognition result indicates that the content relevance between the conversation content and the non-text content is high, the user intent obtains information in the non-text content or obtains reply content according to the information in the non-text content, and then a model related to the content mode of the non-text content can be used to generate the corresponding reply content. When the content relevance is low, the user intent is not to obtain information in the non-text content or to obtain reply content according to the information in the non-text content, and then the reply content can be obtained only according to the text content, so a model related to the text mode can be used to generate the corresponding reply content.

[0110] For example, when the non-text content is an image, if the multimodal intent recognition result indicates that the conversation content is completely related to the image, it is determined to call a look-at-picture-to-speak model to generate the reply content. When the multimodal intent recognition result indicates that the conversation content is basically irrelevant or completely irrelevant to the image, a text-related model can be called to generate the reply content.

[0111] In some embodiments, the multimodal intent recognition result indicates the content relevance between the conversation content and the non-text content, and the content relevance can represent the degree of dependence on the non-text content when answering the question in the reply text content. The higher the degree of dependence, the lower the probability of answering the question without the non-text content, and thus the actual input content input into the reply model can be determined according to the content mode involved in the actual input content, and then the target reply model suitable for the content mode can be selected.

[0112] In some embodiments, the higher the degree of dependence on the non-text content when answering the question in the reply text content, the more the non-text content needs to be used as the actual input content, for example, the last round of text content and non-text content are used as the actual input content, or the last round of text content, non-text content and historical question and answer records are used as the actual input content. On the contrary, the lower the degree of dependence on the non-text content when answering the question in the reply text content, the higher the probability of answering the question without the non-text content, and thus the non-text content can not be used as the actual input content to some extent, for example, only the last round of text content is used as the actual input content, or the last round of text content and historical question and answer records are used as the actual input content.

[0113] As shown in FIG. 5, a logic diagram for determining actual input content is provided in the embodiments of the present application. The content relevance can be determined to see if it meets a preset relevance condition. If the content relevance does not meet the preset relevance condition, it indicates that the reply to the current question does not need to rely on the non-text content, and the non-text content does not need to be used as input content. The intent type of the text content can be identified, and a target reply model that is adapted to the intent type of the text content can be determined from a candidate reply model subset related to the text modality included in the candidate reply model set. In some embodiments, the question in the last round of the question and answer process can be rewritten according to the context information in the conversation content, the intent type of the rewritten text content can be identified, and a target reply model adapted to the intent type can be determined from a candidate reply model subset related to the text modality included in the candidate reply model set. That is, the rewritten text content is used as actual input content. In this way, the non-text content does not need to be processed, so as to reduce the processing burden of the model. If the content relevance meets the preset relevance condition, it indicates that the reply to the current question needs to rely on the non-text content, and the non-text content needs to be used as input content. Therefore, the non-text content and the rewritten text content can be used as actual input content. Of course, in addition to these contents, the historical question and answer records can also be used as actual input content, which is not limited in the embodiments of the present application.

[0114] In this implementation, the candidate reply model set includes a plurality of candidate reply models. Each candidate reply model can process content of at least one content modality, and can output reply content of the last round of the question and answer process according to the content of the at least one content modality. The content modalities that can be processed by different candidate reply models can be the same or different. In some examples, the candidate reply model can also be referred to as a plug-in.

[0115] In actual application, the actual input content can include at least one content modality, such as one or more of a text modality, an image modality, a video modality, and an audio modality. In order to successfully process the actual input content, the selected target reply model needs to be able to understand the content of various modalities included in the actual input content. Then, the target reply model that is adapted to the content modality included in the actual input content can be selected from the candidate reply model set. For example, the candidate reply model A can generate corresponding reply content according to text content, the candidate reply model B can generate corresponding reply content according to text content and image content, and the candidate reply model C can generate corresponding reply content according to image content. When the actual input content only includes text content, the candidate reply model A can be selected as the target reply model.

[0116] In some embodiments, the candidate reply model set can be divided into different candidate reply model subsets according to different content modalities that can be processed. The candidate reply models in the same candidate reply model subset can process the same content modality. At least one content modality corresponding to any two candidate reply model subsets is different. Different candidate reply models in the same candidate reply model subset can correspond to different intent types, so that reply content can be generated according to the corresponding intent types. For example, for a candidate reply model subset that processes text content, it can include a weather plug-in, a calendar plug-in, a map plug-in, and a text-to-image plug-in. The weather plug-in is used to generate weather-related reply content when it is identified that the current text content is used to inquire about weather-related content. The calendar plug-in is used to generate calendar-related reply content when it is identified that the current text content is used to inquire about calendar-related content, and so on.

[0117] According to at least one content modality included in the actual input content, a target reply model subset that is adapted to the at least one content modality can be determined from a plurality of candidate reply model subsets included in the candidate reply model set. In addition, intent type identification can also be performed on the actual input content. For example, text intent identification can be performed according to text content in the actual input content to obtain a corresponding text intent identification result, and then a target reply model that is adapted to the intent type of the actual input content can be determined from the target reply model subset. For example, if the content intent identification result indicates that a video is described, a target reply model that can describe the video can be selected; or if the content intent identification result indicates that the weather is described, a target reply model related to the weather, such as the weather plug-in described above, can be selected.

[0118] Through this embodiment, the target reply model can be selected by comprehensively considering the content modality and the content intent, so that the selected target reply model can process the current input content and be adapted to the content intent, thereby generating more accurate reply content to improve the accuracy of the reply content and improve the user experience.

[0119] In the embodiments of the present application, when the non-text content includes content corresponding to at least one content modality, the multi-modal intent identification result can include content relevance of the content corresponding to at least one content modality.

[0120] In some embodiments, when the target reply model is determined, the target content modality that satisfies a preset relevance condition in terms of content relevance can be determined from at least one content modality according to the multi-modal intent identification result, and then the target reply model that is adapted to the target content modality can be determined from the candidate reply model subset related to the target content modality.

[0121] In some embodiments, if the actual input content is to be determined, the actual input content can be obtained from the content of the content modality that meets the preset relevance condition in terms of content relevance. That is, when there are multiple non-text contents of different content modalities, the non-text content of the content modality that meets the preset relevance condition can be selected as the actual input content.

[0122] Through this implementation, the non-text content with high content relevance between the text content in the current round of text question and answer process can be selected from the non-text contents of multiple content modalities, and the non-text content is taken as the actual input content, which helps to refer to the non-text content when answering the question in the text content, and improves the accuracy of the answer content.

[0123] When the non-text content includes multiple contents, the content that meets the preset relevance condition can also be selected from the multiple contents after determining the content relevance of each of the multiple contents, respectively, to take the content as the actual input content.

[0124] Step 204: obtaining, by the target reply model, the reply content of the question in the last round of question and answer process according to the conversation content and the intent type corresponding to the target reply model.

[0125] In this step, the reply content of the question in the last round of question and answer process is obtained by the target reply model according to the conversation content.

[0126] In the embodiments of the present application, each candidate reply model in the candidate reply model set is a trained neural network model that can generate reply content meeting the demand according to the intent type corresponding to itself. Then, after the target reply model is selected, the actual input content can be taken as input to call the target reply model for forward inference to obtain the reply content output by the target reply model. For example, the candidate reply model can be a natural language model, and the actual input content can be taken as a prompt text to generate subsequent content, which is taken as the reply content. Of course, the candidate reply model can also be other types of models, and the embodiments of the present application do not limit this.

[0127] Next, the process of performing multi-modal intent recognition on the conversation content and the non-text content to obtain a multi-modal intent recognition result is introduced. In the embodiments of the present application, the content relevance can be identified by the following several implementation manners.

[0128] In some embodiments, a multi-modal intent recognition model can be trained by a model training method, and then the conversation content and the non-text content can be taken as model input to perform forward inference by the trained multi-modal intent recognition model to obtain the multi-modal intent recognition result output by the multi-modal intent recognition model.

[0129] For example, when the category text is used as the content relevance classification, the multi-modal intent recognition model can be implemented by using the multi-modal intent recognition model obtained based on the prompt learning method. The multi-modal intent recognition model obtained based on the prompt learning method is a generative natural language model, which is used to predict the subsequent content of the prompt information according to the input prompt instruction. Then, the conversation content and the non-text content can be used as the prompt information to generate the category text.

[0130] The text content and the non-text content in the conversation content can be organized into a prompt instruction according to a preset prompt content template, and the content relevance output by the multi-modal intent recognition model can be obtained by inputting the prompt instruction into the multi-modal intent recognition model. For example, before the prompt information is input into the multi-modal intent recognition model, the corresponding target prompt content needs to be generated according to the conversation content and the non-text content according to the preset prompt content template. The prompt content template is a template type that can be recognized by the multi-modal intent recognition model, which is used to help the multi-modal intent recognition model understand the semantic information of each part of the content.

[0131] In the embodiments of the present application, the positions of each item of content and the order between them are preset in the prompt content template, and the corresponding target prompt content can be generated according to the prompt content template. For example, when generating the target prompt content, attribute labels and round labels can be added to the text content in the conversation content according to the order of at least one round of question and answer process. The attribute label represents that the corresponding text content is question content or reply content, to assist the model in distinguishing the attributes of the text content. The round label represents the question and answer round corresponding to the content, to assist the model in distinguishing the round to which the content belongs. In addition, for the non-text content, a corresponding modality label can also be added to the non-text content, which represents the content modality corresponding to the non-text content, to assist the model in distinguishing the content modality to which the non-text content belongs. Further, according to the preset content order in the prompt content template, the labeled contents are combined, and the target prompt content can be obtained as the prompt instruction.

[0132] In the embodiments of the present application, by adding attribute labels, round labels and modality labels and other information in the prompt template, the model can more accurately distinguish each item of content, so as to more accurately understand these contents and obtain more accurate content relevance. It should be noted that the process of adding labels can also be considered as the process of filling content in the target prompt template, that is, the target prompt template already contains each label, and the content can be filled in the corresponding position according to the label corresponding to each content.

[0133] FIG. 6A is an example of a target prompt template provided by an embodiment of the present application. In this example, the non-text content is an image. The target prompt template includes a “multi-round conversation”, an “upload image”, and a prompt content “Refer to the multi-round conversation above and the provided image, and determine whether the last question and the given image are related:”. The “multi-round conversation” is used to fill in the text content involved in the multi-round question and answer process, and the “upload image” is used to fill in the image. In the “multi-round conversation”, “question 1” represents the first question in the multi-round conversation, “answer 1” represents the reply content of the first question in the multi-round conversation, “question 2” represents the second question in the multi-round conversation, and so on until the last question in the multi-round conversation. In the “upload image”, the image content to be asked is input.

[0134] For example, FIG. 6B shows an example of a target prompt content. As shown in FIG. 6B, “question 1” represents that the content belongs to the text content including the question and belongs to the first round of the question and answer process, “answer 1” represents that the content belongs to the reply content and belongs to the first round of the question and answer process, and so on. “Upload image” represents that the corresponding content is an image modality. Of course, when the non-text content is of other modalities, corresponding labels can be used to represent them, and the specific content of the labels is not limited in the embodiments of the present application.

[0135] After obtaining the target prompt content, the target prompt content can be used as a prompt instruction and input into the trained multi-modal intent recognition model to obtain a multi-modal intent recognition result output by the multi-modal intent recognition model.

[0136] Through this embodiment, the content relevance is generated by the trained multi-modal intent recognition model in the embodiments of the present application in a content generation manner, which helps to improve the efficiency of multi-modal intent recognition.

[0137] In some embodiments, the text content (which can include question content and reply content) in the historical question and answer record and the text content of the last round of question and answer process can be subjected to feature extraction, and the non-text content can be subjected to feature extraction, and then the multi-modal intent recognition result can be determined according to the similarity between the features. In the process of feature extraction of the two parts, the text content and the non-text content can be mapped into the same feature space for comparison, so that the more similar the two are, the higher the relevance between them is.

[0138] In some embodiments, the text content (which can include question text content and reply text content) in the historical question and answer record can be subjected to word segmentation, and then each word segmentation can be matched with the non-text content, and the multi-modal intent recognition result can be obtained by comprehensively matching the degrees of matching between each word segmentation and the non-text content. It can be understood that the higher the matching degree between the word segmentation in the text content and the non-text content, and the more word segmentations with higher matching degrees, the higher the content relevance is.

[0139] In actual application, considering that the text content of the last round of the question and answer process has a higher degree of influence on the reply content, different weights can be set for the text content of different question and answer processes, for example, the weight of the segmentation of the text content of the last round of the question and answer process in calculating the content correlation is larger, and the weight of the segmentation of the text content of the question and answer process earlier in time in calculating the content correlation is smaller.

[0140] In the embodiments of the present application, when the content correlation is determined by using the multi-modal intent recognition model, the multi-modal intent recognition model can use an artificial neural network model based on deep learning. The model can be any possible model, for example, a generative pre-trained (GPT) series model, a Transformer, an encoder-decoder architecture model, a recurrent neural network (RNN), a long short-term memory (LSTM), or a variant model of the foregoing models.

[0141] FIG. 7 is a flowchart of a process of determining a multi-modal intent recognition result by using a multi-modal intent recognition model according to an embodiment of the present application. In the process, the multi-modal intent recognition result can be obtained by performing the following steps by using the trained multi-modal intent recognition model.

[0142] Step 701: encode the text content in the target prompt content by using a text encoding manner to obtain text encoding features.

[0143] In the embodiments of the present application, the text content in the target prompt content can be encoded respectively to obtain text encoding features of each text content, and the obtained text encoding features of each text content are spliced to obtain final text encoding features. The text content in the target prompt content can also be spliced according to a certain format and then encoded as a whole to obtain text encoding features. The text encoding features can also be referred to as text representation, text representation, or text representation vector, etc.

[0144] The text encoding can be implemented by a text encoder. The text encoder can adopt any possible encoder structure, such as an encoding part in a GPT model, a Transformer, an encoder-decoder, an RNN, or an LSTM. Taking the Transformer as an example, the Transformer can more efficiently capture long-distance dependencies of sequence data. Referring to FIG. 8, which is a schematic diagram of a model structure of the Transformer. The Transformer is composed of multiple encoders and decoders, and the multiple encoders are in a cascaded structure. The embodiment of the present application can adopt a one-layer or multi-layer encoder structure, that is, the encoder structure shown in the dashed box in FIG. 8, to perform basic feature extraction on each text unit (such as a segmented word or a word) in the text content and then superimpose the position information corresponding to the text unit. After the superimposed content feature vector is input to the encoder, the content feature vector is processed by an attention (Attention) layer, then processed by a residual & normalization (add & Norm) layer, and then input to a feed forward network (Feed Forword) layer, and then processed by a residual & normalization layer to obtain the final text encoding feature.

[0145] The Attention layer can be a multi-attention (Multi-Attention) layer or a self-attention (Self-attention) layer, and the feed forward network layer can be a position-wise feed-forward neural network (Position-wise Feed-forward Neural Networks) or a multilayer perceptron (Multilayer Perceptron, MLP), and the multiple attention layers and feed forward network layers are stacked together through residual connections (Residual Connections) and layer normalization (Layer Normalization). The Self-attention layer captures the context-related information in the sequence through a self-attention mechanism. The calculation of the self-attention involves three weight matrices (query matrix Q, key matrix K, and value matrix V), and the final attention weight is calculated through dot product, scaling, Softmax activation, and weighted summation. The Layer Normalization is used to accelerate the convergence of the model, and is used to normalize the output of each layer to alleviate the problems of gradient vanishing and explosion in the network.

[0146] Step 702: encoding the non-text content by using a target encoding mode corresponding to the content mode of the non-text content to obtain non-text encoding features.

[0147] For example, when the non-text content is a video, a video feature encoding mode is used to obtain corresponding video encoding features; when the non-text content is an image, an image feature encoding mode is used to obtain corresponding image encoding features. In the following, taking the non-text content as an image as an example, the encoding process of the non-text content is introduced.

[0148] In the embodiment of the present application, image encoding can be implemented by using a Vision Transformer (VIT), or can also use the encoding part in other possible models, which is not limited in the embodiment of the present application.

[0149] Referring to FIG. 9, it is a flowchart of an image encoding process provided by the embodiment of the present application. After inputting an image, first, the image is subjected to image segmentation processing to obtain a plurality of image blocks, and the image blocks are input to a vectorization layer, which is used to respectively encode the plurality of image blocks to obtain corresponding image block encoding features. As shown in FIG. 9, the vectorization layer can be a Liner projection of flattened patches, to convert the image from a three-dimensional matrix to a two-dimensional matrix. Assuming that the dimension of the input picture is (224, 224, 3), the picture is divided according to the size of 16*16, and after division, 196 patches can be obtained, each patch having a size of 16*16. According to the positions of the plurality of image blocks in the image, corresponding position encoding features are obtained, and then the obtained image block encoding features can be subjected to serialization encoding processing according to the obtained position encoding features to obtain corresponding image encoding features. As shown in FIG. 9, 1-9, for each image block, a position feature can be added to the corresponding image block feature according to its position in the input image, and finally the image block feature with the added position feature is serialized and encoded in the encoding layer. In addition, in order to facilitate and distinguish, in FIG. 9, an English classification category token, i.e., “0*” shown in FIG. 9, is added before the sequence, which is a trainable parameter. The encoding layer can use, for example, the transformer encoder shown in FIG. 8, or can use other possible encoder structures, which is not limited in the embodiment of the present application.

[0150] Through the embodiment, the image is converted from a three-dimensional feature to a two-dimensional feature, so that it can be combined with the text feature to perform subsequent processes.

[0151] Step 703: combining the text encoding features and the non-text encoding features to obtain input encoding features.

[0152] For example, the text encoding features and the non-text encoding features can be spliced to obtain the input encoding features.

[0153] Step 704: Perform multiple rounds of iterative text prediction based on the input encoding features to obtain the intent recognition result. In one round of text prediction, the output result of the current round of text prediction is obtained based on the input encoding features and the output result of the previous round.

[0154] In this embodiment, the input encoded features can be understood as a feature sequence composed of a token sequence. These input encoded features can be used as prompts to generate subsequent text sequences, which are the category texts used to characterize content relevance. The iterative text prediction process is a process of decoding the input encoded features; therefore, it can be implemented using a decoder, such as the decoder part of a Transformer.

[0155] In some embodiments, multiple text predictions can be performed using an autoregressive approach. That is, during a text prediction, the output of the current prediction is obtained based on the input encoded features and the output of the previous prediction process. For the initial text prediction, prediction is performed directly based on the input encoded features. After each prediction, the result is appended to the input features used in the current prediction for the next prediction, until the iteration stopping condition is met.

[0156] Figure 10 shows an example of iterative text prediction. Here, the input encoded features can be represented as a token sequence t1…t k-1 This corresponds to the token sequence after encoding, as shown in Figure 6B. The first prediction is made using a multimodal intent recognition model, based on t1…t as shown in Figure 6B. k-1 Generate the next token t k Adding this to the generated token sequence will give you a new token sequence t1…t k Then, the multimodal intent recognition model will determine the token sequence t1…t based on the generated token sequence. k Perform a second prediction to generate the next token t. k+1 Adding this to the generated token sequence will give you a new token sequence t1…t k+2 This process continues until the iteration termination condition is met.

[0157] In some embodiments, the iterative prediction termination condition in the embodiments of this application includes at least one of the following conditions:

[0158] (1) Stop iterative prediction when the length of the predicted token sequence reaches its upper limit.

[0159] (2) After the termination symbol is decoded or predicted, the iteration of the prediction ends, and the termination symbol represents the end of a sequence.

[0160] In the embodiments of the present application, the intention recognition result can be, for example, "completely relevant", "basically irrelevant", and "completely irrelevant", so as to represent whether the reply in the current round needs to rely on non-text content.

[0161] In the embodiments of the present application, before the above-mentioned process is performed based on the multi-modal intention recognition model, the multi-modal intention recognition model needs to be trained by a training method. Next, the training process of the multi-modal intention recognition model will be introduced. Referring to FIG. 11, it is a flowchart of a training method provided in the embodiments of the present application.

[0162] Step 1101: Obtain a set of question and answer samples, each question and answer sample in the set of question and answer samples including the conversation content of at least one round of question and answer process, the conversation content including text content and non-text content, and each question and answer sample being labeled with the actual relevance between the conversation content and the non-text content.

[0163] In some embodiments, the multi-modal intention recognition model of the embodiments of the present application can be based on a large language model after pre-training (Pretrain) by a large-scale corpus, and then SFT is performed by small-scale fine-labeled data to obtain a model for a specific task, that is, the multi-modal intention recognition model of the embodiments of the present application.

[0164] Before SFT is performed, small-scale fine-labeled training data, that is, a set of question and answer samples, needs to be prepared in advance. Each question and answer sample in the set of question and answer samples can include the conversation content of at least one round of question and answer process, the conversation content including text content and non-text content, the content in the question and answer process can be the question and answer record collected in the actual application, and each question and answer sample is labeled with the actual relevance between the conversation content and the non-text content.

[0165] In an implementation, the question and answer sample in the set of question and answer samples can also use a target prompt template to obtain corresponding target prompt content, so that the actual relevance of a text sample can be considered as a category text generated for the target prompt content as prompt information. That is, each question and answer sample can be a (multi-round conversation, relevance label) pair, the multi-round conversation representing the content involved in the question and answer process, and the relevance label representing the actual relevance, which can be understood as the expectation of the multi-modal intention recognition result.

[0166] Referring to FIG. 12A, an example of a prompt template in a training stage is shown, wherein the "multi-turn conversation" and the content in the uploaded picture are the content in the question-answering process, serving as the input of the model during training, and the "relevance label" serves as the output of the model. For example, the relevance label can include two types of "yes" and "no", or the relevance label can include three categories of "completely relevant", "basically irrelevant", and "completely irrelevant", and the embodiments of the present application do not make any limitation in this regard.

[0167] For example, as shown in FIG. 12B, the content in the question-answering process is spliced to obtain each question-answering sample, and the SFT is performed on the large language model. The content in the question-answering process in FIG. 12B is the input of the model, wherein "question 1" represents that the content belongs to the text content containing the question and belongs to the first round of the question-answering process, "answer 1" represents that the content belongs to the answer content and belongs to the first round of the question-answering process, and the rest is similar. The "uploaded picture" represents that the corresponding content is the image modality. The relevance label is the output of the model.

[0168] Step 1102: using the question-answering sample set, fine-tuning the pre-trained large language model until the training termination condition is met, to obtain the trained multi-modal intent recognition model.

[0169] For example, the fine-tuning training is an iterative training process, and the model parameters are continuously optimized so that the model can meet the accuracy requirement. When the current iteration is the first iteration, the model used in the current iteration is the initial model, i.e., the pre-trained large language model. When the current iteration is not the first iteration, the model used in the current iteration is the model after the last parameter adjustment. In each iteration training, part or all of the question-answering samples in the question-answering sample set can be input into the model used in the current iteration. For example, a random selection method can be used, or the question-answering sample set can be pre-divided into batches, and each time a batch of question-answering samples is input, the current training is based on the input question-answering samples, i.e., the model forward inference is performed on each question-answering sample by the model used in the current iteration.

[0170] For example, in the embodiments of the present application, the model structure used can be any model structure capable of realizing serialization processing, such as a transformer model, a recurrent neural network (RNN) model, an encoder-decoder architecture model, or a GPT series model, so as to fine-tune the model structure based on such pre-trained model to adapt to the above multi-modal recognition task.

[0171] In the embodiments of the present application, a prompt learning method is used for model training. The main idea of prompt learning is to input certain prompt content (or prompt instruction) to generate subsequent or remaining text. The prompt content in the embodiments of the present application is the content described above except the relevance label.

[0172] Since each iteration process is similar, the following describes one iteration process as an example. In each iteration process of the fine-tuning training process, the following steps are performed:

[0173] Step 11021: Perform multi-modal intent recognition on the conversation content and non-text content in each question and answer sample input in the current iteration process by using the large language model used in the current round, to obtain the predicted relevance between the conversation content and the non-text content of each question and answer sample.

[0174] During training, the training method of the autoregressive language model is adopted, and the goal is to predict the next token by using the current token and the tokens before it. Since the relevance label is included in each question and answer sample, when predicting the predicted relevance, the model needs to be able to perceive only the content in the question and answer sample except the relevance label and the already predicted content. The content in the relevance label that has not been predicted needs to be masked during calculation.

[0175] The process of obtaining the predicted relevance is a decoding process of the input content, and therefore a decoder can be used to implement it, such as the decoder module of the GPT or Transformer model. Taking the Transformer as an example, as shown in FIG. 8, the decoder includes a plurality of cascaded decoding block stacks with a mask mechanism in series, and the plurality of decoding block stacks of the Transformer are stacked to build a deeper decoder to improve the performance and generalization ability of the model. As shown in FIG. 8, each decoding block includes the following parts:

[0176] (1) Masked Multi-Head Self-Attention layer with a mask mechanism, which captures the context-related information in the sequence by performing multi-head self-attention calculation on the input. The calculation of self-attention involves three weight matrices (query matrix Q, key matrix K, and value matrix V), and the final attention weight is calculated by dot product, scaling, Softmax activation, and weighted summation. The mask mechanism is used to avoid using future information in the self-attention mechanism, that is, using a mask to mask the information after the current token in the self-attention mechanism, to ensure that the prediction is based only on the information of the previous token.

[0177] (2) Feedforward Neural Network layer, which enhances the feature expression ability of each position by performing full connection layer calculation on the representation of each position.

[0178] (3) Residual connection layer and layer normalization, which accelerates the training of the model and improves the performance of the model by connecting the output of the previous two parts with the input sequence in residual connection and normalizing the result after residual connection, and normalizing the output of each layer after layer normalization to alleviate the gradient vanishing / explosion problem in the network.

[0179] In an example, the parameters of the Transformer decoder can be set as follows:

[0180] wherein the number of layers is 40, indicating that the number of data processing layers of the decoder is 40 layers, the dimension of the hidden layer is 5120, and so on.

[0181] It should be noted that the prediction relevance is the same concept as the aforementioned content relevance, which is only used to distinguish the training stage and the application stage. The process of obtaining the prediction relevance by forward inference of the model is the same as the process of obtaining the content relevance described above, so reference can be made to the foregoing description, and will not be described again here.

[0182] Step 11022: According to the obtained respective prediction relevance and the corresponding actual relevance, determine the model loss value of the large language model used in this round.

[0183] In some embodiments, when the relevance label is a category text, a category text represents a range of relevance, such as a category text of "completely relevant" representing a higher range of relevance, a category text of "basically irrelevant" representing a range of relevance lower than "completely relevant", and a category text of "completely irrelevant" representing a range of relevance lower than "basically irrelevant". Each category text can include at least one character position, and the prediction relevance includes at least one character position respectively corresponding to a probability value set, and each probability value set contains a probability value of the corresponding character position pair for each candidate character in the candidate character set.

[0184] Then, when calculating the model loss value, the prediction relevance corresponding to each question and answer sample can be calculated. For example, for a prediction relevance, a prediction relevance corresponding to a sub-loss value can be determined based on the probability value corresponding to the true character obtained from each probability value set, and the true character is the character of the corresponding character position in the relevance label. Then, the model loss value can be determined according to each prediction relevance corresponding to each sub-loss value.

[0185] For example, the loss function of the SFT stage keeps consistent with the pre-training stage, for example, a language model can be used to minimize the maximum likelihood function. A possible expression form is as follows:

[0186] wherein U represents a training data set, P represents a probability of predicting the i-th token using the first k tokens, and k represents a window size. Therefore, the target of the embodiments of the present application is to maximize the product of the conditional probabilities.

[0187] In the embodiments of the present application, the model loss value can also be calculated by using other possible loss functions, for example, a cross entropy loss function, a mean squared error (MSE) loss function, a squared absolute error loss function, a likelihood loss (LHL) function, etc. Of course, other possible loss functions can also be used, and the embodiments of the present application do not limit this.

[0188] Step 11023: determining whether the large language model used in the current round reaches an iteration termination condition.

[0189] In the embodiments of the present application, the iteration termination condition can include at least one of the following conditions:

[0190] (1) the number of iterations reaches a set number threshold.

[0191] (2) the model loss value is less than a set loss threshold, or whether the conditional probability is a maximum value for multiple times in succession, etc.

[0192] Step 11024: if the determination process of step 11023 is no, then the large language model used in the current round is adjusted according to the model loss value, and the next iteration training is entered based on the large language model after the adjustment.

[0193] In the embodiments of the present application, when the number of iterations does not exceed the pre-set number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 11023 is no, that is, it is considered that the current model does not meet the iteration termination condition, and then the model parameters need to be adjusted for further training. After the parameter adjustment, the next iteration training process is entered, that is, jumping to step 11021.

[0194] In some embodiments, when the model still does not meet the convergence condition, the model weight parameters can be updated by using an optimization algorithm such as a gradient descent method or a stochastic gradient descent algorithm to minimize the above-mentioned objective function, and the training is continued with the updated model weight parameters, so that the model can better predict the next token.

[0195] When the number of iterations has exceeded the preset number threshold or the model loss value is less than the set loss threshold, the determination process of step 11023 is yes, that is, it is considered that the current model has met the convergence condition, the model training is ended, and the trained multi-modal model is obtained.

[0196] It should be noted that the above adjustment method of the model is only a basic adjustment method for the autoregressive pre-training model, and other possible fine-tuning methods of the large model can also be used, for example, p-tuning, LORA, etc. In actual application, a suitable fine-tuning method can be used according to the actual scene, model size, learning ability and other factors.

[0197] In the embodiment of the application, referring to FIG. 13, a flowchart of another large model-based question and answer method provided by the embodiment of the application is shown.

[0198] Step 1301: Obtain the conversation content of at least one round of question and answer process.

[0199] The conversation content can include the content input in the question and answer process, and the conversation content at least includes the text content recording the question of the last round of question and answer process. When the last round of question and answer process is the first round of question and answer process, the conversation content only includes the input content of the last round of question and answer process. When the last round of question and answer process is not the first round of question and answer process, the conversation content can include the text content recording the question input in the last round of question and answer process, and the input content and reply content involved in the historical question and answer record.

[0200] Step 1302: Determine whether the conversation content contains non-text content.

[0201] In the embodiment of the application, the conversation content can contain non-text content in addition to the text content, that is, the content expressed by other modalities in addition to the non-text modality, such as images, videos or audios, etc. Therefore, it is necessary to determine whether the conversation content contains non-text content. For example, after obtaining the conversation content, it can be determined whether the conversation content contains images.

[0202] Step 1303: If the determination result of step 1302 is yes, perform multi-modal intent recognition on the conversation content and the non-text content to obtain the corresponding multi-modal intent recognition result.

[0203] For example, the multi-modal intent recognition model trained by the foregoing training method can be called to obtain a corresponding multi-modal intent recognition result. The multi-modal intent recognition result is used to indicate the content correlation between the conversation content and the non-text content, that is, the multi-modal intent recognition result can be used to represent the correlation between the multi-turn conversation and the non-text content, such as whether they are related or belong to which correlation type, and the like. Alternatively, the multi-modal intent recognition result can also be understood as representing the degree of dependence of the question and answer process in the last round on the non-text content, for example, a high correlation degree represents a high degree of dependence of the question and answer process on the non-text content, and a low correlation degree represents a low degree of dependence of the question and answer process on the non-text content.

[0204] Step 1304: According to the multi-modal intent recognition result, it is determined whether a preset correlation condition is met. For example, it is determined whether the correlation degree is greater than a preset correlation threshold, or it is determined whether the multi-turn conversation and the non-text content are related, or it is determined whether it belongs to a specified correlation type, and the like.

[0205] Step 1305: If the determination result of step 1304 is yes, that is, the correlation degree is greater than the preset correlation threshold, or the multi-turn conversation and the non-text content are related, or the multi-modal intent recognition result indicates that it belongs to the “complete correlation” type, a target answer model that is adapted is determined from the candidate answer model subset that is related to the modality of the non-text content.

[0206] Step 1306: According to the context information in the conversation content, the text content is rewritten to obtain rewritten text content. If the determination result of step 1302 is no, step 1306 can also be directly executed.

[0207] In actual application, the question expressed by the text content input by the user is not comprehensive, or the question may have multiple ambiguities, and the actual meaning can be expressed only by combining the context. Therefore, after obtaining the input content, the text content can be rewritten to make the rewritten text content more consistent with the reading logic or express the certain meaning. The question rewriting function can also be a multi-turn rewriting function, which refers to a process of rewriting the question of the current conversation in combination with the multi-turn dialogue content. The process can be implemented by a multi-turn rewriting (QueryRewrite) component, which is a component for processing multi-turn dialogue and query rewriting, and is mainly used to understand and optimize the interaction process between the user and the robot, perform reference resolution and omission completion, and the like.

[0208] For example, the text content can be rephrased according to the historical question and answer records to obtain the rephrased text content. As shown in FIG. 5, the text content input by the user is “what color”, which lacks a subject and thus cannot fully express the content of the question. However, in combination with the context, it can be seen that the user intends to inquire about the color of the plant in the picture, and thus the current text content can be rephrased as “what color is the plant in the picture”, which clearly expresses the object to which the question is directed, facilitating the model to understand the question and helping to improve the accuracy of the reply content.

[0209] Step 1307: performing text intent recognition on the conversation content and the rephrased text content to obtain a corresponding text intent recognition result.

[0210] That is, for the text content involved in the conversation content, a text intent recognition method is used to obtain a text intent recognition result. For example, the intent of the question in the last round of the question and answer process is to obtain weather, calendar, map, or other information, or to generate an image according to the text, so as to call a corresponding reply model to generate reply content.

[0211] Step 1308: If the determination result of step 1304 is no, then a target reply model that is suitable is determined from a candidate reply model subset related to the text content according to the text intent recognition result.

[0212] For example, when the degree of relevance is not greater than a preset degree of relevance threshold, or the multi-round conversation is not related to the non-text content, or the multi-modal intent recognition result indicates that it belongs to the “basically irrelevant” or “completely irrelevant” type, it is determined that the preset degree of relevance condition is not met, and then a target reply model used finally can be selected according to the text intent recognition result.

[0213] Step 1309: obtaining the reply content of the question through the target reply model according to the conversation content and the intent label corresponding to the target reply model.

[0214] It should be noted that some steps in the method shown in FIG. 13 are similar to those in the method shown in FIG. 2, and thus the same or similar steps can be referred to the foregoing description, which will not be repeated here.

[0215] In the embodiment of the present application, the process of text intent recognition can be performed by using a trained text intent recognition model, and thus the text intent recognition model needs to be trained before text intent recognition. The process of model training of the text intent recognition model will be introduced below. As shown in FIG. 14, it is a flowchart of the process of model training of the text intent recognition model.

[0216] Step 1401: obtaining a question and answer sample set, each question and answer sample in the question and answer sample set including conversation content of at least one round of question and answer process, the conversation content including text content, and each question and answer sample being labeled with an actual intent type.

[0217] In some embodiments, the text intent recognition model of the embodiments of the present application can be based on a large language model after pre-training (Pretrain), and then SFT is performed on small-scale fine-labeled data to obtain a model for a specific task, i.e., the text intent recognition model of the embodiments of the present application.

[0218] Before SFT, small-scale fine-labeled training data, i.e., a question and answer sample set, needs to be prepared in advance. Each question and answer sample in the question and answer sample set can include conversation content of at least one round of question and answer process, the conversation content including only text content, the content in the question and answer process can be question and answer records collected in actual application, and each question and answer sample is labeled with a corresponding actual intent type.

[0219] In an implementation, similar to the question and answer samples of the multi-modal intent recognition model, the question and answer samples can also use a prompt template to obtain corresponding target prompt content, so that the actual intent type of a question and answer sample can be considered as a category text generated for the target prompt content as prompt information. That is, each question and answer sample can be a (multi-round conversation, intent type label) pair, the multi-round conversation representing the content involved in the question and answer process, and the intent type label representing the actual intent type, which can be understood as an expectation of the text intent recognition result.

[0220] Referring to FIG. 15A, an example of a prompt template in the training stage provided by the embodiments of the present application is shown, in which “multi-round conversation” is the content in the question and answer process as the input of the model during training, and “intent type label” is the output of the model. For example, the intent type label can include “weather”, “map”, “calendar”, or “text-to-image”, etc. In the application stage of the model, the prompt content can also be generated according to such a prompt template, the difference being that the prompt template in the application stage does not contain the intent type label.

[0221] For example, as shown in FIG. 15B, the content in the question and answer process is spliced to obtain each question and answer sample to perform SFT on the large language model. The content in the question and answer process in FIG. 12B is the input of the model, in which “question 1” represents that the content belongs to text content including a question and belongs to the first round of question and answer process, “answer 1” represents that the content belongs to reply content and belongs to the first round of question and answer process, and the rest are similar. The intent type label is the output of the model.

[0222] Step 1402: Fine-tuning training is performed on the pre-trained large language model using the question and answer sample set until the training termination condition is met, and a trained text intent recognition model is obtained.

[0223] The fine-tuning training of the text intent recognition model is similar to the fine-tuning training of the multi-modal intent recognition model described above, and therefore reference can be made to the description in the foregoing part, which will not be described again here.

[0224] For example, in an embodiment of the present application, the model structure used by the text intent recognition model can be any model structure capable of realizing serialization processing, such as a transformer model, a recurrent neural network (RNN) model, an encoder-decoder architecture model, or a GPT series model, etc., so as to fine-tune the model structure on the basis of such pre-trained models to adapt to the multi-modal recognition task described above.

[0225] Since each iteration process is similar, the following will be described by taking one iteration process as an example. In each iteration process of the fine-tuning training process, the following steps are performed:

[0226] Step 14021: The text content of each question and answer sample input in the current iteration process is subjected to text intent recognition by the large language model used in the current round, and the predicted intent type of each question and answer sample is obtained respectively.

[0227] When training, the training method of the autoregressive language model is used, and the goal is to predict the next token by the current token and the token before it. Since the actual intent type is included in each question and answer sample, when predicting the predicted intent type, the model needs to be able to perceive only the content in the question and answer sample other than the actual intent type and the content that has been predicted, and the content in the actual intent type that has not been predicted needs to be masked when calculating.

[0228] Among them, the process of obtaining the actual intent type is a process of decoding the input content, and therefore a decoder can be used to implement it, such as the decoder module of the GPT or Transformer model. Taking the Transformer as an example, its model structure can be seen from the structure shown in FIG. 8, and therefore will not be described again.

[0229] In one example, the parameters of the text intent recognition model when using the Transformer decoder can be set as follows:

[0230] Step 14022: According to the obtained respective predicted intent types and the corresponding actual intent types, determine the model loss value of the large language model used in this round.

[0231] In some embodiments, when the actual intent type is a category text, each category text includes at least one character position, then the predicted intent type can be represented by at least one probability value set corresponding to each character position, and each probability value set contains the probability value of each candidate character in the candidate character set corresponding to the corresponding character position.

[0232] Then when calculating the model loss value, the predicted intent type corresponding to each question and answer sample can be used for calculation. For example, for a predicted intent type, a sub-loss value corresponding to the predicted intent type can be determined based on the probability value corresponding to the true character in each obtained probability value set, and the true character is the character in the intent type label corresponding to the corresponding character position. Then, the model loss value can be determined according to the respective sub-loss values corresponding to each predicted intent type.

[0233] For example, the loss function of the SFT stage is consistent with the pre-training stage, for example, the language model can use the maximum likelihood function to minimize. A possible expression form is as follows:

[0234] Where U represents the training data set, P represents the probability of predicting the i-th token using the first k tokens, and k represents the window size. Then, the goal of the embodiments of the present application is to maximize the product of the conditional probability.

[0235] In the embodiments of the present application, the model loss value can also be calculated using other possible loss functions, such as the cross entropy loss function (Cross Entropy Loss Function), the mean squared error (Mean Squared Error, MSE) loss function, the square absolute error loss function, the maximum likelihood loss (Likelihood Loss, LHL) function, etc. Of course, other possible loss functions can also be used, and the embodiments of the present application do not limit this.

[0236] Step 14023: Determine whether the large language model used in this round meets the iteration termination condition.

[0237] In the embodiments of the present application, the iteration termination condition can include at least one of the following conditions:

[0238] (1) The number of iterations reaches a set number threshold.

[0239] (2) The model loss value is less than a set loss threshold, or whether the conditional probability is a maximum value for multiple consecutive times, etc.

[0240] Step 14024: If the determination process of step 14023 is no, the large language model used this time is adjusted according to the model loss value, and the next iteration training is entered based on the large language model after adjustment.

[0241] In the embodiments of the present application, when the number of iterations does not exceed the pre-set number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 14023 is no, that is, it is considered that the current model does not satisfy the iteration termination condition, and then the model parameters need to be adjusted for further training. After adjusting the parameters, the next iteration training process is entered, that is, jumping to step 14021.

[0242] In some embodiments, when the model still does not satisfy the convergence condition, the model weight parameters can be updated by gradient descent method, stochastic gradient descent algorithm and other optimization algorithms to minimize the above-mentioned objective function, and the training is continued with the updated model weight parameters, so that the model can better predict the next token.

[0243] When the number of iterations has exceeded the pre-set number threshold, or the model loss value is less than the set loss threshold, the determination process of step 14023 is yes, that is, it is considered that the current model has satisfied the convergence condition, and the model training is ended, obtaining the trained multi-modal model.

[0244] In summary, in the question and answer method provided by the embodiments of the present application, on the pure text intent recognition framework, it is judged whether the non-text content is related to the text content by the multi-modal intent recognition method, the multi-modal intent recognition capability is compatible without affecting the effective result of the pure text intent recognition, the accuracy of the reply content is improved, and more use scenarios are compatible.

[0245] Next, an example is used to introduce the technical solutions of the embodiments of the present application. Here, taking the content mode of non-text content as an image as an example, referring to FIG. 16A, it is a processing flow diagram of the question and answer process. First, step S1 is executed to determine whether the user inputs a picture. When the user only inputs pure text, the related process of pure text link is executed, that is, the process steps in the right frame of FIG. 16A, to output the type of the plug-in to which the pure text link belongs. Here, the multi-classification is judged by the text intent recognition model, that is, which one of the weather plug-in, calendar plug-in, …, map plug-in, and text-to-image plug-in is used to reply to the question. Among them, the pure text link includes step S2 of calling the multi-round rewriting plug-in to optimize and rewrite the current question text. After rewriting, step S3 is executed, that is, generating a prompt instruction according to a pure text prompt template, then step S4 is executed, that is, inputting the prompt instruction to the text intent recognition model to obtain the output text intent recognition result.

[0246] Further, when the determination in step S1 is yes, i.e. the user inputs an image, then the relevant processes of the pure text link and the multimodal link are executed in parallel. Among them, the relevant processes of the multimodal link, i.e. the process steps in the left box in FIG. 16A, include step S5, i.e. performing multimodal intent recognition according to the conversation content and the image by a multimodal intent recognition model to obtain a multimodal intent recognition result output by the multimodal intent recognition model. Then step S6 is executed to determine whether the multimodal intent recognition result indicates that the conversation content is related to the image, for example, when indicating “completely related”, step S7 of outputting the “look at the picture and speak” plug-in is directly executed. When the multimodal intent recognition result indicates that the determination is “basically irrelevant” or “completely irrelevant”, step S8 is executed to determine the corresponding plug-in type according to the text intent recognition result.

[0247] Finally, step S9 is executed to generate the reply content according to the output plug-in.

[0248] For example, as shown in FIG. 16B, in the first round of question and answer process, the image is contained, and the image and the text content “what is the plant in the picture” are obviously completely related, so the “look at the picture and speak” plug-in shown in FIG. 16B is finally output, and the reply content is generated using the “look at the picture and speak” plug-in, such as the reply content “there are several corn cobs in the picture, and their green leaf parts have been stripped off” in FIG. 16B. In the second round of question and answer process, since the image is input in the first round, the conversation content still contains the image, and the image and the text content “what color is it” are also completely related, so the “look at the picture and speak” plug-in shown in FIG. 16B is finally output, and the reply content is generated using the “look at the picture and speak” plug-in, such as the reply content “the color of the corn cobs is mainly yellow, and the corn kernels are arranged closely, showing a typical yellow and white color” in FIG. 16B. In the third round of question and answer process, the image and the text content “what is the weather like in A city today” are completely irrelevant, so the text intent recognition can be performed according to the text content to determine that the weather plug-in shown in FIG. 16B is finally output, and the reply content is generated using the weather plug-in, such as the reply content “A city weather today: thunderstorm, highest temperature: 32℃, lowest temperature: 26℃, relative humidity: 81%, light wind, wind force: 2 level to 1 level” in FIG. 16B. In the third round of question and answer process, the image and the text content “what about tomorrow” are completely irrelevant, so the text intent recognition can be performed according to the text content to determine that the weather plug-in shown in FIG. 16B is finally output, and the reply content is generated using the weather plug-in.

[0249] In the embodiments of the present application, the above-mentioned text link and multimodal link are divided into different process procedures, and different models are used to perform intent recognition, but in actual scenarios, the text intent recognition and the multimodal intent recognition can also be fused into one model for processing, which is not limited in the embodiments of the present application.

[0250] Please refer to FIG. 17, based on the same inventive concept, the embodiment of the present application also provides a large model-based question and answer device 170, which comprises:

[0251] an input unit 1701 configured to obtain conversation content of at least one round of question and answer process, the conversation content at least comprising: text content recording a question of the last round of question and answer process;

[0252] a multi-modal intent recognition unit 1702 configured to, when the conversation content further comprises non-text content, perform multi-modal intent recognition on the conversation content and the non-text content, to obtain a corresponding multi-modal intent recognition result, the multi-modal intent recognition result indicating: content correlation between the conversation content and the non-text content;

[0253] a reply model determination unit 1703 configured to, according to the multi-modal intent recognition result, determine a target reply model from a candidate reply model set, each candidate reply model in the candidate reply model set corresponding to an intent type;

[0254] a reply execution unit 1704 configured to, through the target reply model, obtain reply content of the question according to the conversation content and the intent type corresponding to the target reply model.

[0255] In some embodiments, the device further comprises a text intent recognition unit 1705 configured to:

[0256] rewrite the text content according to context information in the conversation content, to obtain rewritten text content;

[0257] perform text intent recognition on the conversation content and the rewritten text content, to obtain a corresponding text intent recognition result;

[0258] Then, the reply model determination unit 1703 is specifically configured to: if the content correlation indicated by the multi-modal intent recognition result does not satisfy a preset correlation condition, determine the target reply model from a candidate reply model sub-set related to a text modality included in the candidate reply model set, according to the text intent recognition result.

[0259] In some embodiments, the reply model determination unit 1703 is specifically configured to:

[0260] if the content correlation indicated by the multi-modal intent recognition result satisfies the preset correlation condition, determine the target reply model from a candidate reply model sub-set related to a content modality of the non-text content included in the candidate reply model set.

[0261] In some embodiments, the non-text content comprises at least one content modality each corresponding to content, and the multi-modal intent recognition result indicates: content correlation between each of the at least one content modality and the conversation content;

[0262] The answer model determining unit 1703 is specifically configured to:

[0263] According to the multi-modal intention recognition result, a target content modality satisfying a preset relevance condition in terms of content relevance is determined from at least one content modality.

[0264] A target answer model that is adapted is determined from a candidate answer model subset related to the target content modality.

[0265] In some embodiments, the multi-modal intention recognition unit 1702 is specifically configured to:

[0266] According to a preset prompt content template, corresponding target prompt content is generated according to the text content and the non-text content in the conversation content.

[0267] The target prompt content is taken as a prompt instruction and is input into the trained multi-modal intention recognition model to obtain a multi-modal intention recognition result output by the multi-modal intention recognition model.

[0268] In some embodiments, the multi-modal intention recognition unit 1702 is specifically configured to:

[0269] According to a sequence of at least one round of question and answer processes, an attribute label and a round label are added to the text content in the conversation content, the attribute label representing that the corresponding text content is question content or answer content, and the round label representing a question and answer round corresponding to the corresponding text content.

[0270] The non-text content is added with a corresponding modality label, the modality label representing a content modality corresponding to the non-text content.

[0271] According to a preset content sequence in the prompt content template, the labeled contents are combined to obtain the target prompt content.

[0272] In some embodiments, the multi-modal intention recognition unit 1702 is specifically configured to:

[0273] The multi-modal intention recognition model is used to perform the following steps:

[0274] A text encoding manner is used to perform encoding processing on the text content in the target prompt content to obtain text encoding features.

[0275] A target encoding manner corresponding to the content modality of the non-text content is used to perform encoding processing on the non-text content to obtain non-text encoding features.

[0276] The text encoding features and the non-text encoding features are combined to obtain input encoding features.

[0277] According to the input encoded features, multi-round iterative text prediction is performed to obtain an intent recognition result, wherein, in one round of text prediction, according to the input encoded features and the output result of the last round, an output result of the current round of text prediction is obtained.

[0278] In some embodiments, the non-text content is an image; then the multi-modal intent recognition unit 1702 is specifically configured to:

[0279] perform image segmentation processing on the image to obtain a plurality of image blocks;

[0280] perform image encoding processing on the plurality of image blocks respectively to obtain corresponding image block encoded features;

[0281] According to the positions of the plurality of image blocks in the image, corresponding position encoded features are obtained;

[0282] According to the obtained position encoded features, the obtained image block encoded features are sequentially encoded to obtain image encoded features of the image.

[0283] In some embodiments, the apparatus further comprises a model training unit 1706 configured to train the multi-modal intent recognition model through the following training process:

[0284] obtain a set of question and answer samples, each question and answer sample in the set of question and answer samples comprising the conversation content of at least one round of question and answer process, the conversation content comprising text content and non-text content, and each question and answer sample being labeled with the actual correlation between the conversation content and the non-text content;

[0285] using the set of question and answer samples, fine-tuning the pre-trained large language model until the training termination condition is met, wherein in each iteration process of the fine-tuning training, the following steps are performed:

[0286] using the large language model used in the current round, performing multi-modal intent recognition on the conversation content and the non-text content in each question and answer sample input in the current iteration process to obtain the predicted correlation between the conversation content and the non-text content of each question and answer sample respectively;

[0287] According to the obtained each predicted correlation and the corresponding actual correlation, the model loss value of the large language model used in the current round is determined, and after the large language model used in the current round is adjusted according to the model loss value, the adjusted large language model is used to enter the next iteration training.

[0288] In some embodiments, the training label is a correlation category text, the correlation category text comprises at least one character position, the predicted correlation comprises at least one probability value set corresponding to each character position, and each probability value set contains the probability value of the corresponding character position to each candidate character in the candidate character set;

[0289] The model training unit 1706 is specifically configured to:

[0290] For one prediction relevance, the following steps are performed:

[0291] Based on the obtained probability value corresponding to each probability value set corresponding to the true character, a sub-loss value corresponding to one prediction relevance is determined, and the true character is the character at the corresponding character position in the relevance label;

[0292] According to the sub-loss value corresponding to each prediction relevance, the model loss value is determined.

[0293] In some embodiments, the reply model determination unit 1703 is configured to:

[0294] According to the content relevance and the conversation content, the actual input content of the last round of question and answer process is obtained;

[0295] From the candidate reply model set, a candidate reply model subset adapted to at least one content modality included in the actual input content is determined;

[0296] From the candidate reply model subset, a target reply model adapted to the actual input content is determined

[0297] Through the above device, the content input of multiple content modalities can be supported in the question and answer process, the expression of information is more rich, so that more rich information can be provided to the large model, which helps to improve the accuracy of the large model in understanding the input content, so as to generate more accurate reply content, and the accuracy of the reply content is improved.

[0298] The device can be used to execute the method shown in the embodiments of the present application, therefore, the functions and the like that can be achieved by each functional module of the device can refer to the description of the foregoing embodiments, and will not be described in detail.

[0299] Please refer to FIG. 18, based on the same technical concept, the embodiments of the present application further provide a computer device. In one embodiment, the computer device can be the server shown in FIG. 1 or the device corresponding to the cloud, and the computer device is shown in FIG. 18, which includes a memory 1801, a communication module 1803 and one or more processors 1802.

[0300] The memory 1801 is used to store the computer program executed by the processor 1802. The memory 1801 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, and the programs required for running the instant messaging function and the like; the data storage area can store various instant messaging information and operation instruction sets and the like.

[0301] The memory 1801 can be a volatile memory, such as a random-access memory (RAM); the memory 1801 can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or the memory 1801 can be any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but not limited to the above. The memory 1801 can be a combination of the above memories.

[0302] The processor 1802 can include one or more central processing units (CPUs) or digital processing units, etc. The processor 1802 is used to implement the above-mentioned large model-based question and answer method when invoking the computer program stored in the memory 1801.

[0303] The communication module 1803 is used to communicate with terminal devices and other servers.

[0304] The specific connection medium between the above-mentioned memory 1801, communication module 1803, and processor 1802 is not limited in the embodiments of the present application. In FIG. 18, the memory 1801 and the processor 1802 are connected through a bus 1804, which is described by a thick line in FIG. 18. The connection mode between other components is only schematically described, and is not limited. The bus 1804 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, only one thick line is used to describe the bus 1804 in FIG. 18, but it is not described that there is only one bus or only one type of bus.

[0305] The memory 1801 stores a computer storage medium, and the computer storage medium stores computer executable instructions. The computer executable instructions are used to implement the large model-based question and answer method of the embodiments of the present application. The processor 1802 is used to execute the large model-based question and answer method of each embodiment.

[0306] In another embodiment, the computer device can also be a terminal device, such as the terminal device shown in FIG. 1. In this embodiment, the structure of the computer device can be as shown in FIG. 19, which includes a communication component 1910, a memory 1920, a display unit 1930, a camera 1940, a sensor 1950, an audio circuit 1960, a Bluetooth module 1970, a processor 1980, and the like.

[0307] The communication component 1910 is configured to communicate with a server. In some embodiments, a wireless fidelity (WiFi) module can be included, which belongs to a short-range wireless transmission technology. The computer device can help users send and receive information through the WiFi module.

[0308] The memory 1920 can be used to store software programs and data. The processor 1980 performs various functions and data processing of the terminal device by running the software programs or data stored in the memory 1920. The memory 1920 can include a high-speed random access memory, and can also include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. The memory 1920 stores an operating system that enables the terminal device to operate. In this application, the memory 1920 can store an operating system and various application programs, and can also store codes for executing the question and answer method based on a large model according to the embodiments of the present application.

[0309] The display unit 1930 can also be used to display information input by a user or information provided to a user, as well as a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 1930 can include a display screen 1932 arranged on the front of the terminal device. The display screen 1932 can be configured in the form of a liquid crystal display, a light-emitting diode, etc. The display unit 1930 can be used to display a conversation page in the embodiments of the present application.

[0310] The display unit 1930 can also be used to receive input digital or character information, and generate signal input related to user settings and function control of the terminal device. Specifically, the display unit 1930 can include a touch screen 1931 arranged on the front of the terminal device, which can collect touch operations of a user thereon or adjacent thereto, such as clicking buttons, dragging scroll boxes, etc.

[0311] The touch screen 1931 can be overlaid on the display screen 1932, or the touch screen 1931 and the display screen 1932 can be integrated to realize the input and output functions of the terminal device. After integration, it can be referred to as a touch display screen. In this application, the display unit 1930 can display application programs and corresponding operation steps.

[0312] The camera 1940 can be used to capture still images, which can be published by the user through an application with a comment. The camera 1940 can be one or more. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the processor 1980 to convert into a digital image signal.

[0313] The terminal device can also include at least one sensor 1950, such as an acceleration sensor 1951, a distance sensor 1952, a fingerprint sensor 1953, a temperature sensor 1954. The terminal device can also be configured with a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, a motion sensor, and other sensors.

[0314] The audio circuit 1960, the speaker 1961, and the microphone 1962 can provide an audio interface between the user and the terminal device. The audio circuit 1960 can convert the received audio data into an electrical signal and transmit it to the speaker 1961, which converts it into a sound signal for output. The terminal device can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1962 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1960 and converted into audio data, which is then output to the communication component 1910 for transmission to another terminal device, for example, or to the storage 1920 for further processing.

[0315] The Bluetooth module 1970 is used to interact with other Bluetooth devices with Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable computer device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1970, thereby interacting with data.

[0316] The processor 1980 is the control center of the terminal device, connects all parts of the terminal through various interfaces and lines, and performs various functions of the terminal device and processes data by running or executing software programs stored in the memory 1920 and calling data stored in the memory 1920. In some embodiments, the processor 1980 can include one or more processing units; the processor 1980 can also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application program, and the baseband processor mainly processes wireless communication. It can be understood that the above-mentioned baseband processor can also not be integrated into the processor 1980. The processor 1980 in the present application can run an operating system, an application program, a user interface display and a touch response, and a large model-based question and answer method of the embodiments of the present application. In addition, the processor 1980 is coupled with the display unit 1930.

[0317] Based on the same inventive concept, the embodiments of the present application also provide a storage medium storing a computer program, which, when executed by a computer, causes the computer to perform the steps of the large model-based question and answer method described above according to various exemplary embodiments of the present application.

[0318] In some possible implementations, various aspects of the large model-based question and answer method provided by the present application can also be implemented in the form of a computer program product, which includes a computer program for causing a computer device to execute the steps of the large model-based question and answer method described above according to various exemplary embodiments of the present application, for example, the computer device can execute the steps of the embodiments.

[0319] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More examples of readable storage media (non-exhaustive list) include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0320] The program product of the embodiments of the present application can employ a portable compact disc read only memory (CD-ROM) and include a computer program, and can be executed on a computer device. However, the program product of the present application is not limited thereto, and in the present application, the readable storage medium can be any tangible medium containing or storing a program, which includes a computer program that can be used by or in conjunction with a command execution system, device, or apparatus.

[0321] The readable signal medium can include a data signal propagated in a baseband or propagated as a carrier wave in a propagation medium, in which the readable computer program is embodied. Such a propagated signal can take a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a storage medium or that can be read by the command execution system, device, or apparatus using or in conjunction with a program.

[0322] The computer program embodied on the readable medium can be transmitted using any suitable medium, including, but not limited to, wireless, wired, optical fiber, RF, and the like, or any suitable combination thereof.

[0323] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

[0324] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by multiple units.

[0325] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined, performed simultaneously, and / or performed in a different order.

[0326] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0327] Although the preferred embodiments of the application have been described, those skilled in the art will understand that they can be subject to many changes and modifications without departing from the spirit and scope of the application. Accordingly, the appended claims are intended to embrace all such changes and modifications that fall within the scope of the application.

[0328] Obviously, many modifications and changes can be made to the present application without departing from the spirit and scope of the application. It is understood that the application is not to be limited to the particular embodiments described, but it is to be given broad scope to cover all falling within the scope of the claims and their equivalents.

Claims

1. A large model-based question answering method, executed by a computer device, comprising: obtaining session content of a question answering process; when the session content comprises non-text content, determining a content correlation between text content in the session content and the non-text content; when the content correlation meets a preset correlation condition, determining, from a candidate reply model set, a target reply model that is adapted to a content modality of the non-text content; obtaining, according to the session content, reply content of a question in the session content by the target reply model.

2. The method of claim 1, further comprising: identifying an intent type of the text content; when the content correlation does not meet the correlation condition, determining, from a candidate reply model subset related to a text modality included in the candidate reply model set, a target reply model that is adapted to the intent type of the text content.

3. The method of claim 1 or 2, wherein: the determining the content correlation between the text content in the session content and the non-text content comprises: when the non-text content comprises at least one content modality each corresponding to content, determining a content correlation between the content corresponding to each of the at least one content modality and the text content; the determining, from the candidate reply model set, the target reply model that is adapted to the content modality of the non-text content comprises: from the at least one content modality, determining at least one target content modality whose content correlation meets a preset correlation condition; from the candidate reply model set, determining the target reply model that is adapted to the at least one target content modality.

4. The method of claim 1 or 3, wherein, the determining, from the candidate reply model set, the target reply model that is adapted to the content modality of the non-text content comprises: from a candidate reply model subset related to the content modality of the non-text content included in the candidate reply model set, determining the target reply model that is adapted to an intent type of the session content.

5. The method of any one of claims 1-4, wherein, the determining the content correlation between the text content in the session content and the non-text content comprises: according to a preset prompt content template, organizing the text content in the session content and the non-text content into a prompt instruction, inputting the prompt instruction into a multi-modal intent recognition model, and obtaining the content correlation output by the multi-modal intent recognition model.

6. The method of claim 5, wherein, the organizing the text content in the session content and the non-text content into the prompt instruction according to the preset prompt content template comprises: according to a sequence between the question answering processes, adding an attribute label and a round label to the text content in the session content, the attribute label representing that the corresponding text content is question content or reply content, and the round label representing a question answering round corresponding to the corresponding text content; adding a corresponding modality label to the non-text content, the modality label representing a content modality corresponding to the non-text content; combining each content after the labels are added according to a preset content order in the prompt content template, to obtain the prompt instruction.

7. The method of claim 5 or 6, wherein, The prompt instruction is input into the multi-modal intent recognition model, and the multi-modal intent recognition result output by the multi-modal intent recognition model is obtained, which includes: The following steps are performed by the multi-modal intent recognition model: The text content in the target prompt content is encoded into text encoding features in a text encoding manner; The non-text content is encoded into non-text encoding features in an encoding manner corresponding to the content modal of the non-text content; The text encoding features and the non-text encoding features are combined into input encoding features; According to the input encoding features, multi-round iterative text prediction is performed to obtain the content relevance, wherein in one round of text prediction, the output result of the current round of text prediction is obtained according to the input encoding features and the output result of the previous round.

8. The method of claim 7, wherein, The non-text content is an image; The non-text content is an image; The image is divided into a plurality of image blocks; Each image block in the plurality of image blocks is encoded into an image block encoding feature; According to the position of each image block in the image, a position encoding feature of each image block is obtained; According to the position encoding feature, the image block encoding feature is serialized to obtain the image encoding feature of the image.

9. The method of any one of claims 5-8, wherein, The multi-modal intent recognition model is trained through the following training process: A question and answer sample set is obtained, each question and answer sample in the question and answer sample set including the conversation content of at least one round of question and answer process, the conversation content including text content and non-text content, and each question and answer sample being labeled with the actual relevance between the conversation content and the non-text content; The pre-trained large language model is fine-tuned using the question and answer sample set until the training termination condition is met, wherein in each iteration process of the fine-tuning training, the following steps are performed: Through the large language model, the predicted relevance between the text content and the non-text content in each question and answer sample input in the current iteration process is identified; According to the obtained each predicted relevance and the corresponding actual relevance, a loss value is determined, and the large language model is adjusted according to the loss value.

10. The method of claim 9, wherein, The training label is a relevance category text, the relevance category text includes at least one character position, the predicted relevance includes a probability value set corresponding to each of the at least one character position, and each probability value set contains the probability value of the corresponding character position to each candidate character in the candidate character set; According to the obtained each predicted relevance and the corresponding actual relevance, a loss value is determined, and the large language model is adjusted according to the loss value. For one predicted relevance, the following steps are performed: Based on the probability value corresponding to the true character of each probability value set, a sub-loss value corresponding to the one predicted relevance is determined, and the true character is the character in the corresponding character position in the relevance label; According to the sub-loss value corresponding to each predicted relevance, the model loss value is determined.

11. The method of any one of claims 1-4, wherein, determining a target reply model from the candidate reply model set, the target reply model being adapted to a content modality of the non-text content, comprises: obtaining input content associated with a last round of question and answer process in the conversation content according to the content relevance and the conversation content; determining a candidate reply model subset from the candidate reply model set, the candidate reply model subset being adapted to at least one content modality included in the input content; determining the target reply model from the candidate reply model subset.

12. A large model based question and answer device, comprising: an input unit configured to obtain conversation content of a question and answer process; a multi-modal intention recognition unit configured to determine a content relevance between text content and non-text content in the conversation content when the conversation content includes the non-text content; a reply model determination unit configured to determine a target reply model from a candidate reply model set when the content relevance satisfies a preset relevance condition, the target reply model being adapted to a content modality of the non-text content; a reply execution unit configured to obtain reply content of a question in the conversation content by the target reply model according to the conversation content.

13. The device of claim 12, wherein: the multi-modal intention recognition unit is configured to recognize an intention type of the text content; the reply model determination unit is configured to determine a target reply model from a candidate reply model subset included in the candidate reply model set and related to a text modality when the content relevance does not satisfy the relevance condition, the target reply model being adapted to the intention type of the text content.

14. The device of claim 12 or 13, wherein: the multi-modal intention recognition unit is configured to determine a content relevance between content corresponding to each of at least one content modality and the text content when the non-text content includes the content corresponding to each of the at least one content modality; the reply model determination unit is configured to determine at least one target content modality from the at least one content modality, the at least one target content modality satisfying a preset relevance condition; the target reply model is determined from the candidate reply model set and adapted to the at least one target content modality.

15. The device of claim 12 or 14, wherein: the reply model determination unit is configured to determine the target reply model from a candidate reply model subset included in the candidate reply model set and related to a content modality of the non-text content, the target reply model being adapted to an intention type of the conversation content.

16. The device of any one of claims 12-15, wherein: the multi-modal intention recognition unit is configured to organize the text content and the non-text content in the conversation content into a prompt instruction according to a preset prompt content template, input the prompt instruction into a multi-modal intention recognition model, and obtain the content relevance output by the multi-modal intention recognition model; 17. The apparatus of any one of claims 12-15, wherein, the reply model determination unit is configured to: obtain input content associated with a last round of question and answer process in the conversation content according to the content relevance and the conversation content; From the candidate reply model set, a candidate reply model subset adapted to at least one content modality included in the input content is determined; From the candidate reply model subset, a target reply model adapted is determined. 18.A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, causes the processor to implement the method of any one of claims 1 to 11. 19.A computer storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 11. 20.A computer program product, comprising a computer program, wherein the computer program, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 11.

Images