Question and answer model training method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a question and answer model training method and device, electronic equipment and a storage medium, and aims to improve the efficiency of acquiring required information by an object through a question and answer model. The method comprises the steps that non-text content contained in sample data is replaced with corresponding text content, and a sample text is obtained; based on the sample text and a preset prompt content template, constructing sample prompt content, and based on the sample text, obtaining a reference answer of the sample prompt content; and performing iterative training on the initial model by adopting the prompt content of each sample to obtain a question and answer model. According to the method and the device, the non-text content is replaced by the text content to obtain the sample text, so that the model can understand the meaning of the non-text content in training, the non-text content is fully combined when the answer is generated, and the image-text combined answer is output, so that the object can quickly obtain the required information from the answer, and the efficiency of obtaining the required information is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method, apparatus, electronic device, and storage medium for a question-answering model. Background Technology

[0002] In recent years, with the development of natural language processing technology, question answering models have received considerable attention, leading to the emergence of a series of enterprise-level intelligent question answering models. High-quality question answering models can provide 24 / 7 uninterrupted service, helping to solve common questions and reducing manual labor costs.

[0003] After receiving a question, the question-answering model analyzes and understands it, then returns the answer. This allows users to directly access the answer without having to search through a list of documents returned by a search engine, thus improving efficiency.

[0004] Currently, the interaction method of question answering models is still focused on text information. The user inputs a natural language sentence to ask a question into the question answering model. The question answering model performs question classification, keyword extraction and keyword expansion on the obtained natural language sentence to determine the user's question answering intention, and finally finds the corresponding text answer and outputs it.

[0005] However, presenting answers in text form is not intuitive or visual enough. When the recipient does not have a deep understanding of the relevant knowledge, it is difficult for them to quickly understand the answer and extract the necessary information. This increases the time required for the recipient to solve the problem and reduces the efficiency of obtaining the required information.

[0006] For example, if a user's question is "How do I turn the turn signals on or off?", and the output text answer is "Moving the light control lever to the second up / second down position will turn on the right / left turn signal, and the corresponding turn signal will remain on. Moving the light control lever to the first up / second down position or returning the steering wheel to its original position will turn off the turn signals." Based solely on this text, the user cannot quickly understand how to operate the light control lever in terms of "second up," "second down," "first up," and "next down." This results in the user spending a significant amount of time understanding how to operate the light control lever, increasing time consumption and reducing information acquisition efficiency.

[0007] Therefore, a new question-answering model needs to be designed to overcome the above-mentioned shortcomings. Summary of the Invention

[0008] This application provides a training method, apparatus, electronic device, and storage medium for a question-answering model to improve the efficiency of objects in obtaining the information they need.

[0009] This application provides a method for training a question-answering model, comprising:

[0010] Perform the following operations for each sample data:

[0011] Replace the non-text content in the sample data with the corresponding text content to obtain the sample text;

[0012] The text content includes at least the text description of the non-text content;

[0013] Based on the obtained sample text and the preset prompt content template, sample prompt content corresponding to the sample text is constructed. The sample prompt content includes the text content and the instruction content, which is used to instruct the initial model to output the answer based on the text content.

[0014] Based on the sample text, a reference answer to the sample prompt is obtained, wherein the reference answer includes the text content and sample answers related to the text content;

[0015] Using the constructed sample prompts, the initial model is iteratively trained to obtain a question-answering model. In each iteration, the following steps are performed: the initial model predicts the answer based on a sample prompt, and the model parameters are adjusted based on the difference between the predicted answer and the reference answer of the sample prompt.

[0016] This application provides a method for applying a question-answering model, including:

[0017] Get the problem text;

[0018] Based on the question text and the preset prompt content template, construct the prompt content;

[0019] The prompt content is input into the question-answering model to obtain the response answer output by the question-answering model. The question-answering model is obtained based on the training method of the question-answering model described above. The response answer includes: the answer text for the question text, and the text content of non-textual content related to the answer text. The text content includes at least the text description content of the non-textual content.

[0020] This application provides a training device for a question-answering model, comprising:

[0021] The processing unit is used to perform the following operations for each sample data:

[0022] Replace the non-text content in the sample data with the corresponding text content to obtain the sample text;

[0023] The text content includes at least the text description of the non-text content;

[0024] Based on the obtained sample text and the preset prompt content template, sample prompt content corresponding to the sample text is constructed. The sample prompt content includes the text content and the instruction content, which is used to instruct the initial model to output the answer based on the text content.

[0025] Based on the sample text, a reference answer to the sample prompt is obtained, wherein the reference answer includes the text content and sample answers related to the text content;

[0026] The training unit is used to iteratively train the initial model using the constructed sample prompts to obtain a question-answering model. In each iteration, the following steps are performed: the initial model predicts the answer based on a sample prompt, and the model parameters are adjusted based on the difference between the predicted answer and the reference answer of the sample prompt.

[0027] Optionally, the processing unit is specifically used to obtain the text description content through at least one of the following methods:

[0028] The non-text content is subjected to character recognition, the character information contained in the non-text content is extracted, and the text description content is obtained based on the character information;

[0029] Feature extraction is performed on the non-text content to obtain a content vector of the non-text content; feature reconstruction is performed based on the content vector to obtain text description information of the non-text content, and text description content is obtained based on the text description information.

[0030] Optionally, the prompt content template includes a question-and-answer pair prompt template, and the question-and-answer pair prompt template includes first instruction content;

[0031] The processing unit is specifically used for:

[0032] Based on the sample text and the question-and-answer pair prompt template, a first sample prompt content is constructed. The first sample prompt content includes the first instruction content and the text content. The first instruction content is used to instruct the initial model to output a question-and-answer pair based on the text content.

[0033] Optionally, the prompt content template includes a reading comprehension prompt template, which includes second instruction content;

[0034] The processing unit is specifically used for:

[0035] Based on the sample text and the reading comprehension prompt template, a second sample prompt content is constructed. The second sample prompt content includes the second instruction content, the sample question, and the text content. The second instruction content is used to instruct the initial model to output the predicted answer to the sample question based on the text content.

[0036] Optionally, the second sample prompt content is constructed based on the sample text and the reading comprehension prompt template, and the processing unit is specifically used for:

[0037] Obtain the aforementioned sample problem;

[0038] Retrieve reference knowledge text related to the sample question from the sample text and sample question-answer pairs;

[0039] Based on the reference knowledge text, the sample question, and the reading comprehension prompt template, the second sample prompt content is constructed.

[0040] Optionally, the text content may also include link text to the non-text content, the link text being used to link to the non-text content; the sample answer is related to the link text and the non-text content.

[0041] This application provides an application device for a question-answering model, comprising:

[0042] The acquisition unit is used to acquire the question text;

[0043] The construction unit is used to construct prompt content based on the question text and a preset prompt content template;

[0044] An extraction unit is used to input the prompt content into the question-answering model to obtain the response answer output by the question-answering model. The question-answering model is obtained based on the training method of the question-answering model described above. The response answer includes: the answer text for the question text, and the text content of non-textual content related to the answer text. The text content includes at least the text description content of the non-textual content.

[0045] Optionally, the text content may also include link text for linking to the non-text content;

[0046] The device also includes a display unit for:

[0047] Based on the linked text, obtain the non-text content;

[0048] Display the non-text content and the answer text.

[0049] Optionally, the prompt content template includes a reading comprehension prompt template, which includes second instruction content;

[0050] The building unit is specifically used for:

[0051] Based on the question text and the reading comprehension prompt template, a second prompt content is constructed. The second prompt content includes the question text, the second instruction content, and the text content. The text content includes at least the text description of the non-text content. The second instruction content is used to instruct the question-answering model to output the answer to the question text based on the text content.

[0052] Optionally, the building unit is specifically used for:

[0053] From candidate texts and candidate question-answer pairs, retrieve reference knowledge texts related to the question text, wherein the candidate texts contain relevant knowledge in the target domain, and the reference knowledge texts include the text content;

[0054] Based on the reference knowledge text, the question text, and the reading comprehension prompt template, the second prompt content is constructed.

[0055] Optionally, the prompt content template includes a question-and-answer pair prompt template, and the question-and-answer pair prompt template includes first instruction content;

[0056] The candidate question-answer pairs are obtained in the following manner:

[0057] Based on the candidate text and the question-answer pair prompt template, a first prompt content is constructed. The first prompt content includes the first instruction content and the text content. The first instruction content is used to instruct the question-answering model to output a question-answer pair based on the text content.

[0058] The first prompt is input into the question-answering model to obtain the candidate question-answer pair.

[0059] Optionally, the text description content can be obtained through at least one of the following methods:

[0060] The non-text content is subjected to character recognition, the character information included in the non-text content is extracted, and the text description content is obtained based on the character information;

[0061] Feature extraction is performed on the non-text content to obtain a content vector of the non-text content; feature reconstruction is performed based on the content vector to obtain text description information of the non-text content, and text description content is obtained based on the text description information.

[0062] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described question-answering model training methods.

[0063] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described question-answering model training methods.

[0064] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described question-answering model training methods.

[0065] The beneficial effects of this application are as follows:

[0066] This application provides a training method, apparatus, electronic device, and storage medium for a question-answering model. First, the non-text content in the sample data is replaced with corresponding text content to obtain sample text. Then, based on the obtained sample text and a preset prompt content template, corresponding sample prompt content is constructed. Based on the sample text, a reference answer for the sample prompt content is obtained. Finally, the initial model is iteratively trained using each sample prompt content to obtain a question-answering model.

[0067] To help the initial model better understand the images and output answers that combine text and images, this application replaces non-text content with text content when constructing sample text. This allows the initial model to understand non-text content through text content, thus enabling it to fully integrate non-text content when generating answers and output answers that combine text and images. This ensures the image output rate and the relevance between text and images in the answers. Consequently, when reading the answers, users can quickly understand them, acquire the necessary knowledge, reduce the time spent solving problems, and improve information acquisition efficiency.

[0068] Furthermore, this application uses a preset prompt content template of the sample text to generate sample prompt content for iterative training. Training is carried out through prompt learning, which can achieve good training results without a large amount of training data, reducing the resource consumption of constructing training data and improving resource utilization.

[0069] In addition, this application modifies the sample data to iteratively train the initial model. The model only needs to process text input, which can effectively reduce the complexity of the model and improve its computational efficiency and generalization ability.

[0070] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0071] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0072] Figure 1 This is an optional schematic diagram of an application scenario in an embodiment of this application;

[0073] Figure 2 This is a schematic diagram illustrating an intelligent customer service implementation in one embodiment of this application;

[0074] Figure 3 This is a question-and-answer interaction diagram in an embodiment of this application;

[0075] Figure 4 This is a flowchart illustrating the implementation of a training method for a question-answering model in an embodiment of this application.

[0076] Figure 5A This is a schematic diagram of non-text content in an embodiment of this application;

[0077] Figure 5B This is a schematic diagram of another non-text content in an embodiment of this application;

[0078] Figure 6 This is a schematic diagram illustrating the process of generating sample text in an embodiment of this application;

[0079] Figure 7 This is a schematic diagram illustrating the text content acquisition process in an embodiment of this application;

[0080] Figure 8 This is a schematic diagram of a position vector in an embodiment of this application;

[0081] Figure 9 This is a schematic diagram of the loss function calculation process in the embodiments of this application;

[0082] Figure 10 This is a flowchart illustrating the implementation of an application method for a question-answering model proposed in this application.

[0083] Figure 11 This is a schematic diagram of the framework of the question-answering system in the embodiments of this application;

[0084] Figure 12 This is a logical schematic diagram of a training method for a question-answering model in an embodiment of this application;

[0085] Figure 13 This is a schematic diagram of the structure of the training device for the question-answering model in the embodiments of this application;

[0086] Figure 14 This is a schematic diagram of the structure of the application device for the question-answering model in the embodiments of this application;

[0087] Figure 15 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0088] Figure 16 This is a schematic diagram of the hardware structure of another electronic device using an embodiment of this application. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0090] The following describes some of the concepts involved in the embodiments of this application.

[0091] Large Language Models (MLMs) are deep learning-based natural language processing models that learn the syntax and semantics of natural language, enabling them to generate human-readable text. A "language model" is an artificial intelligence (AI) model specifically designed to process language (or symbol systems), discover patterns, and automatically generate content that conforms to those patterns based on prompts. Large language models typically use a language model as the objective function, optimizing model parameters by maximizing the prediction probability of the next word. They are trained on large-scale corpora and can generate high-quality natural language text, such as articles and dialogues.

[0092] The initial model in this application can be a large language model. Current large language models can only process text content in the form of language and text. However, in this application, non-text content is replaced with corresponding text content. Therefore, the large language model can understand the meaning of non-text content through text content and output a combined text and image answer. In this way, the question-answering model trained by the method in the embodiments of this application can still output a combined text and image answer when the object asks a question in plain text, which makes it easier for the object to obtain the required information. The following mainly uses the large language model as the initial model for explanation.

[0093] Reading comprehension tasks: One of the tasks achieved through large language models, which can be used to understand and interpret text content. For example, large language models can be used to answer questions about a given text or generate summaries of a given text. This is because large language models are able to capture semantic and contextual information in the text during the learning process, thereby understanding the meaning of the text.

[0094] Question-Answer Pair Generation is a task in the field of Natural Language Processing, which can also be achieved using large language models. Its goal is to automatically generate relevant question-and-answer pairs (QPs) based on given text or context. This process typically involves extracting key information from the text, constructing questions related to that information, and providing corresponding answers.

[0095] Optical Character Recognition (OCR) is the process of converting text images into machine-readable text format. For example, if you scan a form or receipt, the computer saves the scan as an image file, which cannot be edited, searched, or counted using a text editor. However, OCR can be used to convert the image into a text document and store the content as text data.

[0096] Prompt learning unifies all downstream tasks into a pre-training task. Using a specific template, it transforms the data from downstream tasks into natural language, fully leveraging the capabilities of the pre-trained model. Essentially, it designs a template that fits the upstream pre-training task, unlocking the potential of the upstream model and enabling it to perform downstream tasks effectively with minimal need for labeled data. Prompt learning can be applied to various natural language generation tasks, such as text summarization, machine translation, and dialogue generation. It helps the model better understand the context and language of the generation task, resulting in more accurate and natural text. This improves the quality and efficiency of the generation model while reducing its generation space, thus increasing efficiency and speed.

[0097] Fine-tuning, also known as fine-tuning training, refers to further training a pre-trained model using a specific dataset to adapt it to a particular task or domain. This is also called adaptive training, ultimately improving the model's performance on specific tasks. The process of prompt learning often involves fine-tuning the model to better understand the prompt text and the context of the generation task, thus producing more accurate and natural text. For example, one method of fine-tuning is supervised fine-tuning (SFT).

[0098] Retrieval-Augmented Generation (RAG) models combine semantic models and information retrieval techniques, and are a common implementation structure for question-answering systems. When the model needs to generate text or answer a question, it first retrieves relevant information from a large collection of documents, and then uses the retrieved information to guide text generation, thereby improving the quality and accuracy of predictions. RAG models can be fine-tuned and their internal knowledge can be effectively modified without retraining the entire model. RAG models accept input and retrieve a set of relevant documents from a given source. These documents, acting as context, are connected to the original input prompts and fed into a text generator that produces the final output. This allows RAG models to adapt to situations where facts may change over time.

[0099] The embodiments of this application relate to artificial intelligence and machine learning (ML) technologies, and are mainly designed based on machine learning and natural language processing technologies in artificial intelligence.

[0100] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0101] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0102] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP deals with natural language, the language people use in daily life, and is closely related to linguistics; it also involves computer science and mathematics. Pre-trained models, a crucial technique for model training in artificial intelligence, evolved from large language models in NLP. After fine-tuning, these large language models can be widely applied to downstream tasks. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0103] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0104] Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0105] Artificial Neural Networks (ANNs) abstract the neural network of the human brain from an information processing perspective, establishing a simple model and forming different networks with different connection methods. A neural network is a computational model composed of a large number of interconnected nodes (or neurons). Each node represents a specific output function called the activation function. The connection between any two nodes represents a weighted value for the signal passing through that connection, called a weight. This is equivalent to the memory of the artificial neural network. The network's output varies depending on the connection method, weight values, and activation functions. The network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0106] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0107] The solution provided in this application relates to the generation of answers to questions raised during the question-and-answer process in the question-and-answer domain. It is used to train a question-and-answer model to generate answers. When training the question-and-answer model, machine learning and methods related to prompting learning for the pre-trained model are used to obtain the question-and-answer model based on a large language model.

[0108] Specifically, the training of the question-answering model involves the technical field of machine learning. In this embodiment, training data from the question-answering domain is used to learn prompts from a pre-trained large language model. The model parameters are continuously adjusted through optimization algorithms until the model converges, enabling the fine-tuned pre-trained language model to adapt to the question-answering domain and improve the image-text relevance and image-to-answer ratio. Furthermore, it should be noted that the artificial neural network model in this embodiment can be trained online or offline, without specific limitations. This paper uses offline training as an example.

[0109] The design concept of the embodiments of this application will be briefly introduced below:

[0110] In recent years, with the development of natural language processing technology, question answering models have received considerable attention, leading to the emergence of a series of enterprise-level intelligent question answering models. High-quality question answering models can provide 24 / 7 uninterrupted service, helping to solve common questions and reducing manual labor costs.

[0111] After receiving a question, the question-answering model analyzes and understands it, then returns the answer. This allows users to directly access the answer without having to search through a list of documents returned by a search engine, thus improving efficiency.

[0112] Currently, the interaction method of question answering models is still focused on text information. The user inputs a natural language sentence to ask a question into the question answering model. The question answering model performs question classification, keyword extraction and keyword expansion on the obtained natural language sentence to determine the user's question answering intention, and finally finds the corresponding text answer and outputs it.

[0113] However, presenting answers in text form is not intuitive or visual enough. When the recipient does not have a deep understanding of the relevant knowledge, it is difficult for them to quickly understand the answer and extract the necessary information. This increases the time required for the recipient to solve the problem and reduces the efficiency of obtaining the required information.

[0114] For example, if a user's question is "How do I turn the turn signals on or off?", and the output text answer is "Moving the light control lever to the second up / second down position will turn on the right / left turn signal, and the corresponding turn signal will remain on. Moving the light control lever to the first up / second down position or returning the steering wheel to its original position will turn off the turn signals." Based solely on this text, the user cannot quickly understand how to operate the light control lever in terms of "second up," "second down," "first up," and "next down." This results in the user spending a significant amount of time understanding how to operate the light control lever, increasing time consumption and reducing information acquisition efficiency.

[0115] In view of this, embodiments of this application provide a training method, apparatus, electronic device, and storage medium for a question-answering model. First, the non-text content contained in the sample data is replaced with corresponding text content to obtain sample text. Then, based on the obtained sample text and a preset prompt content template, corresponding sample prompt content is constructed, and based on the sample text, a reference answer for the sample prompt content is obtained. Finally, the initial model is iteratively trained using each sample prompt content to obtain a question-answering model.

[0116] To help the initial model better understand the images and output answers that combine text and images, this application replaces non-text content with text content when constructing sample text. This allows the initial model to understand non-text content through text content, thus enabling it to fully integrate non-text content when generating answers and output answers that combine text and images. This ensures the image output rate and the relevance between text and images in the answers. Consequently, when reading the answers, users can quickly understand them, acquire the necessary knowledge, reduce the time spent solving problems, and improve information acquisition efficiency.

[0117] Furthermore, this application uses a preset prompt content template of the sample text to generate sample prompt content for iterative training. Training is carried out through prompt learning, which can achieve good training results without a large amount of training data, reducing the resource consumption of constructing training data and improving resource utilization.

[0118] In addition, this application modifies the sample data to iteratively train the initial model. The model only needs to process text information, thus effectively reducing the complexity of the model and improving its computational efficiency and generalization ability.

[0119] In existing technologies, images are typically introduced into the system only through Uniform Resource Locators (URLs) or placeholders. In LLM systems, this can lead to bias in the model's answer generation, resulting in images not being displayed or generating unrelated images and text. Examples of incorrectly generated answers include:

[0120] Q: How does the automatic high beam system work?

[0121] A: The automatic high beam system uses a camera to detect the taillights of vehicles ahead, the headlights of oncoming vehicles, or ambient light levels. When the system detects high levels of light, it automatically suppresses the high beams, effectively turning them off. When the ambient light dims and there are no vehicles ahead, the suppression is lifted, and the high beams are back on. <Warning Image>

[0122] Existing models append a warning image to the above answer. This image is meaningless in question-answer generation or reading comprehension and should not be generated. However, because previous framework models cannot understand images and can only judge the likelihood of them being included based on the context of the text, they cannot guarantee that incorrect image links will not be included in the generated results. But with the solution in this application, by introducing non-textual text content and then retraining the LLM model through secondary fine-tuning, the model understands the meaning of the non-textual image through the text content and can determine whether the image is related to the surrounding text. Therefore, it can significantly improve the image inclusion rate and image-text relevance.

[0123] The solution provided in this application can be applied to question-and-answer scenarios. For example... Figure 1 The diagram shown is an application scenario provided by an embodiment of this application. In this scenario, a terminal device 101 and a server 102 may be included.

[0124] Terminal device 101 can be, for example, a mobile phone, tablet computer (PAD), laptop computer, desktop computer, smart home appliance (such as smart TV), smart in-vehicle device, smart wearable device, and aircraft. Server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, i.e., content delivery network (CDN), and big data and artificial intelligence platforms, but is not limited to these.

[0125] It should be noted that the method in this embodiment can be executed by either terminal device 101 or server 102 alone, or by both server 102 and terminal device 101. When executed by either terminal device 101 or server 102 alone, the model training and application processes can be implemented independently. For example, on terminal device 101, the initial model can be iteratively trained using the constructed sample prompts to obtain a trained question-answering model. After training, terminal device 101 can use the trained question-answering model to predict the answer to the question text. Alternatively, server 102 can execute the above process. When executed by both server 102 and terminal device 101, server 102 can train the initial model and then deploy the trained question-answering model to terminal device 101, where terminal device 101 implements the question-answering process. Alternatively, part of the model training or application process can be implemented by terminal device 101, and part by server 102, with both collaborating to achieve the model training or application process. In practical applications, specific configurations can be made according to the circumstances, and this application does not impose specific limitations here.

[0126] Both server 102 and terminal device 101 may include one or more processors, memory, and interactive I / O interfaces. Furthermore, server 102 may be configured with a database to store trained model parameters. The memory of both server 102 and terminal device 101 may also store program instructions required for execution in the question-answering model training method provided in this embodiment. These program instructions, when executed by the processor, can be used to implement the model training process or the question-answering process based on a large model provided in this embodiment.

[0127] It should be noted that when the question-answering model training method provided in this application embodiment is executed by either server 102 or terminal device 101 alone, the above application scenario may also include only a single device, either server 102 or terminal device 101. Alternatively, server 102 and terminal device 101 may be considered as the same device. Of course, in practical applications, when the question-answering model training method provided in this application embodiment is executed by both server 102 and terminal device 101, server 102 and terminal device 101 may also be the same device. That is, server 102 and terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0128] In one possible implementation, the object can provide question text through terminal device 101, and then server 102 can receive the question text provided by the object, and then use the method of this application embodiment to generate an answer based on the question text, and return it to terminal device 101 for presentation.

[0129] In this embodiment, the terminal device 101 and the server 102 can communicate directly or indirectly through one or more networks 103. The network 103 can be a wired network or a wireless network; for example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network, and of course, it can be other possible networks. This embodiment does not limit the types of networks used. It should be noted that... Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0130] The following describes the training method of the question-answering model provided by the exemplary embodiments of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0131] The question-answering model in this application embodiment can be applied to question-answering scenarios. The question-answering scenario can be any scenario in which a question is provided and an answer is provided based on the question, such as search, knowledge question answering, question answering robot, intelligent customer service or AI assistant scenarios.

[0132] Taking the application of this application's embodiments in an intelligent customer service scenario as an example, such as Figure 2As shown, intelligent customer service is implemented through the following process: First, knowledge documents, web pages, news, and other material data provided by the product provider are imported. After parsing the material data, it is sent to a large language model to generate question-and-answer pairs. The generated question-and-answer pairs undergo a knowledge verification process, and the question-and-answer pairs that are selected and added to the retrieval database are then selected. When a question is received from an individual, the retrieval system retrieves reference question-and-answer pairs from the retrieval database, constructs prompts, and sends them to the large language model to generate answers, enabling a real-time dialogue experience.

[0133] Specifically, such as Figure 3 As shown, when the question text "Introducing the 2.5L four-wheel drive - 'Five Killer Features' + E-FOUR Selling Points" is received, the retrieval system recalls the question. It retrieves relevant reference knowledge text from sources such as question-and-answer pairs, document slicing, and search engines. This information is then concatenated and fed into the question-and-answer model for reading comprehension and answering. Taking intelligent customer service for automobiles as an example, the retrieval system recalls the question and generates the following prompt:

[0134] Based on the following known information, answer the question concisely and professionally. If the answer cannot be found from the text, ignore the text and answer the question in Chinese.

[0135] Known information:

[0136] <Source> Product Selling Points Section 2

[0137] 2.5L Four-Wheel Drive - "Five Killer Features" + E-FOUR

[0138] 1. Rugged design: polygonal design elements, 18-inch alloy wheels

[0139] 2. Thoughtful Features: LED headlights, 7-inch LCD instrument panel, 10.1-inch central control screen, PVW holographic imaging, 8-way power-adjustable driver's seat with 2-way power lumbar support, sliding panoramic sunroof, and heated front seats.

[0140] 3. Flexible Space: 12 storage spaces inside the vehicle, 580L trunk, and flexible seat folding modes.

[0141] 4. Leading power: THS11, combined fuel consumption 5.0L, 3 driving modes

[0142] 5. Top-tier safety: All models come standard with TSS (Level 2 autonomous driving), 7 SRS airbags, and have received the TOPSAFETY PICK+ crash test certification.

[0143] 6. E-FOUR: Front-to-rear axle torque distribution of 100:0-20:80, AUTO LSD limited-slip differential comprehensively improves vehicle handling stability and enhances off-road capability.

[0144] <Image>

[0145] <Source> Product Selling Points Section 2

[0146] 2.0L HIGH 4WD - "Five Killer Features" + DTV 4WD

[0147] 1. Rugged design: polygonal design elements, 19-inch large wheels

[0148] 2. Thoughtful Features: LED headlights, 7-inch LCD instrument panel, 10.1-inch central control screen, PVW holographic imaging, 8-way power-adjustable driver's seat with 2-way power lumbar support, sliding panoramic sunroof, and heated front seats.

[0149] 3. Flexible Space: 12 storage spaces inside the vehicle, 580L trunk, and flexible seat folding modes.

[0150] 4. Leading power: 40% high thermal efficiency engine, world's first dual-transmission CVT, combined fuel consumption of 6.3L, and 3 driving modes.

[0151] question:

[0152] Introducing the 2.5L four-wheel drive – its "five killer features" + E-FOUR selling points

[0153] The above prompts are fed into the question-answering model to obtain... Figure 3 A reply that combines text and images.

[0154] The training method for the question-answering model in this application is described below:

[0155] like Figure 4 The diagram shown is a flowchart of a training method for a question-answering model provided in this application. Taking a server as the executing entity, the specific implementation process of this method includes the following steps S41-S44:

[0156] S41: The server performs the following operations for each sample data:

[0157] S411: Replace the non-text content in the sample data with the corresponding text content to obtain the sample text;

[0158] Specifically, the sample data contains relevant knowledge of the target domain. Taking a smart customer service system applied to a certain mobile phone A as an example, the sample data is the data of relevant knowledge of mobile phone A. The sample data may include data in the form of question-and-answer pairs. The question-and-answer pair data can be obtained by collecting data from the network or by inputting relevant knowledge data into an existing question-and-answer pair generation model and verifying the output question-and-answer pairs. No specific limitations are made here.

[0159] The sample data may contain both textual and non-textual content. Textual content refers to content expressed in written language, while non-textual content can include images, videos, audio, etc. Considering that non-textual content may not be understood by the initial model, this application proposes replacing non-textual content with corresponding textual content. Each piece of textual content includes at least a textual description of the corresponding non-textual content. The non-textual content is used as... Figure 5A For example, the text description would be "The picture shows a handle," while the non-text content would be... Figure 5B For example, the text description would be "The picture shows a car".

[0160] When characters are included in non-text content, those characters should also be included in the text description. Figure 5A The text description could also be "two gears up, one gear down, two gears down; there is a handle in the picture."

[0161] Furthermore, in order for non-text content to be displayed correctly to objects, the text content also includes link text to the corresponding non-text content, which links to the non-text content and indicates the storage address of the non-text content, for example... Figure 5A The text content is: [up two gears, up one gear, down two gears, the image shows a handle](https: / / qbot.qidian.mid&width=200.846&height=141.370&wRatio=0.370), where (https: / / qbot.qidian.mid&width=200.846&height=141.370&wRatio=0.370) is non-text content. When the answer output by the question-answering model contains text content, when rendering it to an object on the front end, the text content can be set to be rendered as the corresponding non-text content, and the object can view the non-text content normally.

[0162] like Figure 6 The diagram shown illustrates the sample text generation process in an embodiment of this application. The left side of the diagram shows the sample data, and the right side shows the sample text generated after replacing the images contained in the sample data with text content.

[0163] Optionally, the text description content (not text content) can be obtained in the following ways:

[0164] Method 1: Perform character recognition on non-text content, extract character information contained in the non-text content, and obtain text description content based on the character information;

[0165] Method 2: Extract features from non-text content to obtain content vectors; reconstruct features based on content vectors to obtain text description information of non-text content, and obtain text description content based on text description information;

[0166] Method 3: Perform character recognition on non-text content, extract the character information contained in the non-text content, extract features from the non-text content to obtain the content vector of the non-text content; perform feature reconstruction based on the content vector to obtain the text description information of the non-text content; and obtain the text description content based on the character information and the text description information.

[0167] Specifically, in method 1, OCR can be used to recognize characters in non-text content, extract the contained characters, and then directly use the extracted characters as text description content.

[0168] In Method 2, text description information of non-text content can be extracted through the image description model, and the text description information can be directly used as the text description content.

[0169] Furthermore, based on methods 1 and 2, the extracted characters and text description information can be combined as the text description content, i.e., method 3. For example... Figure 7 The diagram illustrates a process for obtaining text description content in an embodiment of this application. Taking an image as an example (non-text content), the image is subjected to OCR recognition to obtain the character information "up two gears, up one gear, down two gears, down two gears". The non-text content is then input into the image description model to obtain the text description information "There is a handle in the picture". Therefore, "up two gears, up one gear, down two gears, there is a handle in the picture" can be used as the text description content.

[0170] Based on the above approach, the model can understand the meaning of non-textual content through the textual content of non-textual content. In this way, when generating answers, it can judge the relevance between non-textual content and the contextual text. Therefore, it can also greatly improve the carry-out rate of non-textual content and the relevance between text and images.

[0171] Image description models typically employ an encoder-decoder framework, where the encoder can use convolutional neural networks (CNNs) to extract features, and the decoder can use long short-term memory networks (LSTMs) to reconstruct text based on the extracted features.

[0172] S412: Based on the obtained sample text and the preset prompt content template, construct the sample prompt content corresponding to the sample text;

[0173] The prompt content template is a template type that the model can recognize, used to help the model understand the semantic information of each part of the content. In this embodiment, the prompt content template pre-defines the position of each item and their order, so the corresponding sample prompt content can be generated according to the prompt content template.

[0174] The question-answering model trained in this embodiment can perform question-answer pair generation and reading comprehension tasks. Each task corresponds to a prompt content template, which is used to generate corresponding sample prompt content.

[0175] In an optional implementation, the prompt content template corresponding to the question-answer pair generation task (hereinafter referred to as the question-answer pair prompt content template) includes a first instruction content, which is used to instruct the initial model to output question-answer pairs based on text content. Step S412 can be implemented as follows:

[0176] Based on the sample text and question-and-answer pair prompt template, construct the first sample prompt content.

[0177] Since the sample text contains non-textual content and the question-and-answer pair prompt template contains the first instruction content, the first sample prompt content contains both the first instruction content and the text content.

[0178] Furthermore, the first sample prompt should also include known information to provide the initial model with reference knowledge for generating answers. This known information is extracted from the sample text. Taking sample prompt 1 as an example, it includes:

[0179] First instruction content:

[0180] Several question-and-answer pairs will be generated based on the following text.

[0181] Known content:

[0182] Lighting control handle

[0183] ! [Up two gears, up one gear, down two gears, the image shows a handle](https: / / qbot.qidian.qq.com / s / fzMv8hm4?size)

[0184] Lane change signal: Move the light control lever to the previous / next position to turn on the right / left lane change signal. The corresponding turn signal will flash 3 times and then turn off.

[0185] Turn signals: Moving the light control lever to the second position up / second position down activates the right / left turn signal, which will remain constantly illuminated. Moving the light control lever to the first position up / last position or returning the steering wheel to its original position deactivates the turn signals.

[0186] Automatic High Beam (FAB) system

[0187] ! [An exclamation mark is drawn inside a triangle in the image](https: / / qbot.qidian.qq.com / s / uRGeyzsO?size)

[0188] The automatic high beam system shares a camera with the lane assist system. When the system detects the taillights of the vehicle in front, the headlights of the oncoming vehicle, or high ambient light, it automatically suppresses the high beams (i.e., turns them off). When the ambient light dims and there are no vehicles in front, the suppression is lifted and the high beams are turned on again.

[0189] ! [An exclamation mark is drawn inside a triangle in the image](https: / / qbot.qidian.qq.com / s / uRGeyzsO?size)

[0190] In an optional implementation, the prompt content template corresponding to the reading comprehension task (hereinafter referred to as the reading comprehension prompt template) includes a second instruction content, used to indicate that: the initial model outputs the predicted answer to the sample question based on the text content, and step S412 can be implemented as follows:

[0191] Based on the sample text and reading comprehension prompt template, construct a second sample prompt content.

[0192] In this process, content is extracted from the sample text and placed into the reading comprehension prompt template to construct the second sample prompt content. Thus, the second sample prompt content includes the second instruction content, the sample question, and the text content.

[0193] In one alternative implementation, the second sample prompt content is constructed as follows:

[0194] Obtain sample questions; retrieve reference knowledge texts related to sample questions from sample texts and sample question-and-answer pairs; construct second sample prompt content based on reference knowledge texts, sample questions, and reading comprehension prompt templates.

[0195] Specifically, sample questions can be obtained from sample text and sample question-answer pairs, or they can be generated based on knowledge related to the target domain. There are no specific limitations here. Sample question-answer pairs can be generated based on sample text, for example, by inputting sample text into an existing question-answering model to obtain sample question-answer pairs. Sample text can also be question-answer pairs that contain knowledge related to the target domain collected from the network.

[0196] The second sample prompt should also include known content to provide reference knowledge for the initial model to generate answers. This known content can be extracted from reference knowledge text. For example, sample prompt 2 includes:

[0197] Second instruction content:

[0198] Based on the information provided below, answer the question concisely and professionally. If the answer cannot be found in the text, ignore the text and answer the question in Chinese.

[0199] Known content:

[0200] Lighting control handle

[0201] ! [Up two gears, up one gear, down two gears, the image shows a handle](https: / / qbot.qidian.qq.com / s / fzMv8hm4?size)

[0202] Lane change signal: Move the light control lever to the previous / next position to turn on the right / left lane change signal. The corresponding turn signal will flash 3 times and then turn off.

[0203] Turn signals: Moving the light control lever to the second position up / second position down activates the right / left turn signal, which will remain constantly illuminated. Moving the light control lever to the first position up / last position or returning the steering wheel to its original position deactivates the turn signals.

[0204] Automatic High Beam (FAB) system

[0205] ! [An exclamation mark is drawn inside a triangle in the image](https: / / qbot.qidian.qq.com / s / uRGeyzsO?size)

[0206] The automatic high beam system shares a camera with the lane assist system. When the system detects the taillights of the vehicle in front, the headlights of the oncoming vehicle, or high ambient light, it automatically suppresses the high beams (i.e., turns them off). When the ambient light dims and there are no vehicles in front, the suppression is lifted and the high beams are turned on again.

[0207] ! [An exclamation mark is drawn inside a triangle in the image](https: / / qbot.qidian.qq.com / s / uRGeyzsO?size)

[0208] Sample issue:

[0209] How do I use the light control handle to turn the turn signals on or off?

[0210] It should be noted that the above prompt template is only an example, and you can actually set it yourself as needed. This application does not impose any specific restrictions here.

[0211] S413: Based on the sample text, obtain the reference answer to the sample prompt content;

[0212] The server selects content from the sample text based on the prompt template to generate sample prompt content. Furthermore, it can construct reference answers for the sample prompt content based on the sample text. Taking sample prompt content 2 as an example, the corresponding reference answer is: Moving the light control lever to the second up / second down position will turn on the right / left turn signal, and the corresponding turn signal will remain constantly lit. Moving the light control lever to the first up / first down position or returning the steering wheel to its original position will turn off the turn signal. Operation illustration: ! [Second up, first down, second down; the image shows a lever](https: / / qbot.qidian.qq.com / s / fzMv8hm4?size).

[0213] Specifically, as mentioned earlier, the sample prompts are actually divided into two types. When a sample prompt is the second sample prompt, it is used for the reading comprehension task, and the model needs to output the answer to the question text. In this case, the reference answer of the sample prompt should also be the answer to the question text, which can be extracted from the sample question-and-answer pair or the sample text to guide the model in parameter adjustment. When a sample prompt is the first sample prompt, it is used for the question-and-answer pair generation task, and the model needs to output the question-and-answer pair. In this case, the reference answer of the sample prompt can be extracted from the sample question-and-answer pair.

[0214] S42: The server uses the constructed sample prompts to iteratively train the initial model and obtain the question-answering model.

[0215] In each iteration, the following steps are performed: the initial model predicts the answer based on a sample prompt, and the model parameters are adjusted based on the difference between the predicted answer and the reference answer of the sample prompt.

[0216] In this embodiment, the initial model can be a pre-trained large language model. Then, the server uses the constructed sample prompts to iteratively fine-tune the large language model to obtain the question-answering model.

[0217] Specifically, when fine-tuning the large language model, the sample prompts are input into the large language model, the large language model outputs the predicted answer, and then the parameters are adjusted.

[0218] Although large language models cannot understand the meaning of images, in this embodiment, non-text content is replaced with text content, enabling the large language model to understand non-text content through text content. This allows the large language model to fully integrate non-text content when generating answers, outputting answers that combine text and images, ensuring the image inclusion rate and image-text relevance of the answer. Furthermore, since only the large language model needs to process text content, the complexity of the model is effectively reduced, improving the computational efficiency and generalization ability of the question-answering model.

[0219] During training, the model iteratively adjusts its parameter values ​​until it can correctly predict answers based on sample prompts. To this end, the model uses self-learning techniques that teach it to adjust parameters to maximize the likelihood of correctly predicting answers in the training examples. After fine-tuning, LLM can easily adapt to perform multiple tasks using relatively small supervised datasets, and the resulting question-answering model can be directly used for question-answer pair generation and reading comprehension tasks. The LLM model tends to favor generating plausible image results when generating question-answer pairs or corresponding answers.

[0220] In this embodiment, firstly, the non-text content in the sample data is replaced with the corresponding text content to obtain sample text. Then, based on the obtained sample text and the preset prompt content template, the corresponding sample prompt content is constructed, and based on the sample text, the reference answer of the sample prompt content is obtained. Finally, the initial model is iteratively trained using each sample prompt content to obtain the question-answering model.

[0221] To help the initial model better understand images and output answers that combine text and images, this application replaces non-textual content with textual content when constructing sample text. This allows the initial model to understand non-textual content through textual content. In this way, when generating answers, non-textual content can be fully integrated to output answers that combine text and images, ensuring the image output rate and the relevance of the images and text. Consequently, when reading the answers, the user can quickly understand them, acquire the necessary knowledge, reduce the time spent solving problems, and improve information acquisition efficiency.

[0222] Furthermore, this application uses a preset prompt content template of the sample text to generate sample prompt content for iterative training. Training is carried out through prompt learning, which can achieve good training results without a large amount of training data, reducing the resource consumption of constructing training data and improving resource utilization.

[0223] In addition, this application modifies the sample data to iteratively train the initial model. The model only needs to process text information, thus effectively reducing the complexity of the model and improving its computational efficiency and generalization ability.

[0224] Optionally, before inputting a sample prompt into the large language model, the corresponding word segmentation sequence can be obtained as follows:

[0225] A sample prompt content is segmented into words, and each segment is semantically encoded to obtain a corresponding word semantic vector. Based on the position type of each segment, each segment is positionally encoded to obtain a corresponding position vector. Based on the word semantic vector and position vector of each segment, a comprehensive vector of the corresponding segment is obtained. The comprehensive vectors are combined according to the order of the corresponding segment in a sample prompt content to obtain a segmentation sequence.

[0226] The position type indicates whether the corresponding word segment originates from the text content of a sample text associated with a sample prompt. Specifically, the text content in the sample text refers to the non-text content within the sample material corresponding to the sample text. The sample material contains both text and non-text content. After replacing the non-text content with text content, the sample text can be considered as composed of text content and text content. The position type of a word segment originating from the text content is different from the position type of a word segment originating from the text content. For example, the position type of a word segment originating from the text content is type 1, while the position type of a word segment originating from the text content is type 2.

[0227] For example, such as Figure 8 The diagram shown is a schematic of a position vector in an embodiment of this application. For the sample text "light control handle! [up two gears, up one gear, down two gears, there is a handle in the picture](https: / / qidian.com)", the word segmentation from the text content is "! [up two gears, up one gear, down two gears, there is a handle in the picture](https: / / qidian.com)", the position vector of the word segmentation in the text content is set to 1, and the position vector of the word segmentation "light control handle" from the text content is set to 0.

[0228] Based on the above approach, positional encoding is performed simultaneously with semantic encoding of word segmentation. The positional vector and semantic vector are combined to form a composite vector, which is then fed into the large language model. This allows the model to focus on the contextual relationships between word segments and output the complete text content of the same non-textual content when generating the answer, ensuring the normal display of non-textual content.

[0229] LLM typically employs a Transformer-based architecture, which can be categorized into encoder-decoder structures, pure encoder structures (such as BERT), or pure decoder structures (such as GPT). This application primarily uses an encoder-decoder LLM as an example for illustration.

[0230] In an encoder-decoder architecture, the encoder is responsible for encoding the input text into a high-dimensional vector representation, and the decoder is responsible for decoding this high-dimensional vector representation into the target output text.

[0231] Optionally, the large language model includes an encoder and a decoder; the following steps are performed in step S42:

[0232] The word segmentation sequence corresponding to a sample prompt is input into the encoder for feature extraction to obtain the semantic features of the sample prompt; the semantic features are then input into the decoder for feature reconstruction to obtain the predicted answer.

[0233] Specifically, during the process of generating the answer, the decoder internally generates a series of vector representations. These vector representations represent the decoder's internal state at each step when generating the next word.

[0234] The decoder ultimately outputs a vocabulary of words. That is, when generating a word at each step, the decoder calculates a probability distribution corresponding to each word in the vocabulary. Then, by selecting the word with the highest probability (or through other sampling methods), the decoder directly outputs that word.

[0235] Therefore, the decoder outputs the text directly by calculating the probability distribution each time. This process is done step by step, generating one word at a time, until an end marker (such as a special end character) is generated.

[0236] During each generation step, the decoder outputs a probability distribution corresponding to each word in the vocabulary. Assuming the vocabulary has V words, the decoder outputs a vector of length V at each step, representing the probability of each word.

[0237] Optionally, in step S42, the model parameters are adjusted in the following way:

[0238] Based on the differences between the word segmentation of each target position type in the predicted answer and the word segmentation of each target position type in the reference answer, a first loss function is constructed. The target position type is represented as follows: the corresponding word segmentation comes from the text content of the sample text associated with the sample prompt content. Based on the differences between the word segmentation of each non-target position type in the predicted answer and the word segmentation of each non-target position type in the reference answer, a second loss function is constructed. Based on the first loss function, the second loss function, and the preset weight values, a target loss function is constructed, and the model parameters are adjusted based on the target loss function.

[0239] Specifically, as mentioned above, the position type is used to characterize whether the segmented words originate from the text content of the sample text. Target position type means that the segmented words originate from the text content of the sample text, and non-target position type means that the segmented words originate from the text content of the sample text. When generating the entire predicted answer sequence, the decoder generates each word step by step and calculates the corresponding loss at each step. The final total loss is the sum of the losses of all words, usually taken as the average of the losses of all words. However, in this embodiment, in order to enable the model to output the non-text content as a whole when outputting the predicted answer sequence, the losses of the segmented words of target position type and non-target position type are calculated separately, and the obtained first loss function and second loss function are weighted and summed to obtain the target loss function. This allows the model to focus more on outputting the non-text content as a whole, ensuring that the front end can correctly render the non-text content.

[0240] The loss function typically used is cross-entropy loss, which measures the difference between the probability distribution of each word generated by the model and the one-hot encoding of the actual target word. Specifically, assuming that in a certain generation step, the target word is (y_t), and the probability distribution generated by the model is (P(y_t|\text{context})), then the cross-entropy loss can be expressed as:

[0241] text{Loss}=-logP(y_t|text{context})

[0242] Here, text{context} represents the context information that the decoder relies on when generating (y_t), including previously generated words and the encoder's output.

[0243] After calculating the total loss (target loss function), the loss gradient is propagated back to the various parameters of the model (including the parameters of the encoder and decoder) through the backpropagation algorithm, and these parameters are updated using optimization algorithms (such as gradient descent) to minimize the loss.

[0244] like Figure 9The diagram shows the loss function calculation process in this embodiment. The reference answer is "light control handle! [up two gears, up one gear, down two gears, the diagram shows a handle](https: / / qidian.com)", and the predicted answer is "light box adjustment handle! [up one gear, up two gears, down two gears, the diagram shows a handle](https: / / qidian.com)". Each word in the reference answer has a corresponding one-hot encoded vector, and each word in the predicted answer has a corresponding probability distribution vector. The dimensions of the two types of vectors are the same. Each one-hot encoded vector and its corresponding probability distribution vector are used to calculate the loss s. When summing the calculated losses, the s corresponding to words with gray backgrounds in the reference answer (i.e., words of the target position type) are weighted to obtain the final total loss S.

[0245] Based on the above method, when calculating the loss function, the loss of segmentation of target position type and non-target position type is calculated separately, and the obtained first loss function and second loss function are weighted and summed to obtain the target loss function. This allows the model to focus more on the text content of non-text content in the overall output, so as to ensure the normal display of non-text content.

[0246] Optionally, the target loss function can be constructed in the following way:

[0247] The intermediate loss function is obtained by multiplying the first loss function by the weight value; the target loss function is constructed by summing the intermediate loss function and the second loss function.

[0248] Specifically, when the model generates the answer, it is expected that the text content of the non-text content in the predicted answer is exactly the same as the text content in the reference answer. This is necessary to correctly render the entire non-text content. Therefore, when calculating the loss function, the weight of the first loss function is increased to make the model pay more attention to the complete output text content. Thus, the preset weight value is greater than 1.

[0249] Based on the above approach, by assigning a weight greater than 1 to the first loss function, the model focuses on the loss corresponding to the text content that is not text content, and when non-text content is brought out in the answer, the text content is brought out completely.

[0250] The question-answering model training method in this application embodiment can serve the next generation of intelligent customer service. It aims to fully leverage the capabilities of large models, improving upon the shortcomings of existing technologies such as long cold start cycles, high operating costs, and poor transferability, thus facilitating a comprehensive upgrade of intelligent customer service scenarios. It utilizes only a large language model, rather than a multimodal model, to address the image output problem in customer service scenarios. In customer service scenarios, the key tasks are question-answer pair generation and reading comprehension. The former involves understanding the given material and extracting it into question-answer pairs, while the latter uses a retrieval system to recall material fragments and question-answer pairs to generate answers to the questions.

[0251] This application utilizes OCR technology and image description models to generate text information as annotations (text content) to assist in question-and-answer pair generation and reading comprehension tasks. Through prompt learning, the annotated question-and-answer pair generation and reading comprehension tasks are fine-tuned, effectively improving the quality of question-and-answer pair generation and segmentation encoding during the cold start phase of the next-generation intelligent customer service system. When the customer service system answers questions from users, it can also generate answers with images, ensuring the image output rate and image-text relevance of the answers.

[0252] like Figure 10 The diagram shown is an implementation flowchart of a question-answering model application method proposed in this application embodiment. Taking the server as the execution topic as an example, it includes the following steps:

[0253] S1001: The server retrieves the problem text;

[0254] S1002: The server constructs the prompt content based on the problem text and the preset prompt content template;

[0255] Fill in the question text and some relevant knowledge, such as question-and-answer pairs, document slices, and relevant information from search engines, in the prompt content template to construct the prompt content. The question-and-answer model can then generate the answer based on the prompt content.

[0256] S1003: The server inputs the prompt into the question-and-answer model to obtain the response answer output by the question-and-answer model.

[0257] The question-answering model is trained using the aforementioned training method, and specific implementation details are provided in the above embodiments and will not be repeated here. The response answer includes the answer text to the question text, as well as non-textual content related to the answer text. The textual content includes at least a textual description of the non-textual content. The method for obtaining the textual description content is also provided in the above embodiments and will not be repeated here.

[0258] In one optional implementation, before step S1001, the question-answering model first constructs first prompt content based on candidate text and question-answer pair prompt template, and then inputs the first prompt content into the question-answering model to obtain candidate question-answer pairs.

[0259] Specifically, the candidate text contains relevant knowledge of the target domain, and the question-answer pair prompt template includes the first instruction content. The first prompt content includes the first instruction content and the text content. The first instruction content is used to instruct the question-answering model to output the question-answer pair based on the text content. The construction method of the first prompt content is the same as the construction method of the first sample prompt content mentioned above, and will not be repeated here.

[0260] In one optional implementation, the prompt content template includes a reading comprehension prompt template, which includes second instruction content; step S1002 can be implemented as follows:

[0261] Based on the question text and reading comprehension prompt template, construct the second prompt content.

[0262] The second prompt includes the question text, the second instruction, and text content. The text content includes at least non-textual text descriptions. The second instruction indicates that the question-answering model outputs the answer to the question text based on the text content.

[0263] Specifically, from the candidate texts and candidate question-answer pairs, reference knowledge texts related to the question text are retrieved. The candidate texts contain relevant knowledge in the target domain, and the reference knowledge texts include text content. Based on the reference knowledge texts, the question text, and the reading comprehension prompt template, second prompt content is constructed. The construction method of the second prompt content is the same as that of the second sample prompt content described above, and will not be repeated here.

[0264] In an optional implementation, the text content further includes link text for linking to non-text content; the following steps may also be performed after step S1003:

[0265] Based on the linked text, retrieve non-text content; display the non-text content and the answer text.

[0266] Specifically, since the model's output response contains non-textual text content, in order for the object to view the non-textual content, it is necessary to obtain the non-textual content based on the link text and display the non-textual content and the answer text together to the object.

[0267] by Figure 3 For example, if the input question text is "Introduce the selling points of the 2.5L four-wheel drive - 'Five Killer Features' + E-FOUR", the question-and-answer model returns the answer, where the answer text is "The main selling points of the 2.5L four-wheel drive - 'Five Killer Features' + E-FOUR include the following points... The following is a schematic diagram of the 2.5L four-wheel drive - 'Five Killer Features' + E-FOUR:", and the car image is the non-text content related to the answer text.

[0268] This application proposes a basic framework for a text-based customer service question-and-answer scheme. First, data preprocessing is performed. Material data (source files) from the target domain are imported, parsed, and sliced. Non-text content such as images is converted into text descriptions and then stored as non-text text content. Second, a training dataset for question-and-answer pair generation and reading comprehension tasks is constructed. Each training data point consists of sample prompts and corresponding reference answers, including both text and non-text content. Then, a large language model is fine-tuned using the training dataset to enable the model to understand the meaning of image annotations, ensuring image output rate and text-image relevance. Finally, the fine-tuned model is applied to the question-and-answer system to achieve end-to-end question-and-answer.

[0269] like Figure 11 The diagram shown is a schematic of the framework of the question-answering system in this application embodiment. First, the relevant knowledge documents of product 1 are processed, and the non-text content is converted into text content to obtain candidate documents. The candidate documents are stored in the retrieval library and input into the trained large language model (i.e., the question-answering model) to generate question-answer pairs. The generated question-answer pairs are stored in the retrieval library. At this time, the preparation work is completed. When the question-answering system receives the question text of the object, it first retrieves the N most relevant knowledge from the retrieval library, then constructs prompt content based on the N knowledge, and inputs the prompt content into the trained large language model to obtain the output answer.

[0270] like Figure 12 The diagram shown illustrates the logic of a question-answering model training method in an embodiment of this application. First, sample data 1, sample data 2, sample data 3, sample data 4, and sample data 5, containing knowledge related to the target domain, are acquired. The sample data is then processed, replacing non-textual content with corresponding text content to obtain sample text 1, sample text 2, sample text 3, sample text 4, and sample text 5. At this point, each sample text is plain text and can be recognized by the large language model. Then, to incorporate prompt learning into model training, sample prompt content needs to be constructed so that the large language model can generate answers based on the sample prompt content. Taking the generation of one sample prompt content per sample text as an example, sample prompt content 1, sample prompt content 2, sample prompt content 3, sample prompt content 4, and sample prompt content 5 are input into the large language model. The large language model outputs a predicted answer, and parameters are adjusted based on the difference between the predicted answer and the reference answer to obtain the question-answering model.

[0271] Based on the same inventive concept, embodiments of this application also provide a training device 1300 for a question-answering model. For example... Figure 13 As shown, this is a schematic diagram of the structure of the training device 1300 for the question-answering model, which may include:

[0272] Processing unit 1301 is used to perform the following operations for each sample data:

[0273] The sample data is obtained by replacing the non-text content with the corresponding text content; wherein the text content includes at least the text description of the non-text content.

[0274] Based on the obtained sample text and the preset prompt content template, sample prompt content corresponding to the sample text is constructed. The sample prompt content includes text content and instruction content. The instruction content is used to instruct the initial model to output the answer based on the text content.

[0275] Based on the sample text, obtain the reference answer to the sample prompt content. The reference answer includes the text content and the sample answer related to the text content.

[0276] Training unit 1302 is used to iteratively train the initial model using the constructed sample prompts to obtain a question-answering model. In each iteration, the following steps are performed: the initial model predicts the answer based on a sample prompt, and the model parameters are adjusted based on the difference between the predicted answer and the reference answer of a sample prompt.

[0277] Optionally, the processing unit 1301 is specifically used to obtain the text description content through at least one of the following methods:

[0278] Character recognition is performed on non-text content to extract character information contained in the non-text content, and text description content is obtained based on the character information;

[0279] Feature extraction is performed on non-textual content to obtain content vectors; feature reconstruction is performed based on the content vectors to obtain textual description information of the non-textual content, and textual description content is obtained based on the textual description information.

[0280] Optionally, the prompt content template includes a question-and-answer pair prompt template, which includes the first instruction content;

[0281] Processing unit 1301 is specifically used for:

[0282] Based on the sample text and question-answer pair prompt template, the first sample prompt content is constructed. The first sample prompt content includes the first instruction content and the text content. The first instruction content is used to instruct the initial model to output the question-answer pair based on the text content.

[0283] Optionally, the prompt content template includes a reading comprehension prompt template, which includes the second instruction content;

[0284] Processing unit 1301 is specifically used for:

[0285] Based on the sample text and reading comprehension prompt template, a second sample prompt content is constructed. The second sample prompt content includes a second instruction content, a sample question, and text content. The second instruction content is used to instruct the initial model to output the predicted answer to the sample question based on the text content.

[0286] Optionally, based on the sample text and reading comprehension prompt template, a second sample prompt content is constructed. The processing unit 1301 is specifically used for:

[0287] The problem of obtaining samples;

[0288] Retrieve reference knowledge texts related to the sample questions from the sample text and sample question-answer pairs;

[0289] Based on the reference knowledge text, sample questions, and reading comprehension prompt templates, construct the second sample prompt content.

[0290] Optionally, the text content also includes link text to non-text content; the link text is used to link to the non-text content; the sample answer is related to both the link text and the non-text content.

[0291] Based on the same inventive concept, this application also provides an application device 1400 for a question-and-answer model. For example... Figure 14 As shown, this is a structural schematic diagram of the application device 1400 for the question-and-answer model, which may include:

[0292] Unit 1401 is used to retrieve the question text;

[0293] Construction unit 1402 is used to construct prompt content based on the question text and a preset prompt content template;

[0294] Extraction unit 1403 is used to input the prompt content into the question answering model to obtain the response answer output by the question answering model. The question answering model is obtained by the training method of the above question answering model. The response answer includes: the answer text for the question text, and the text content of non-textual content related to the answer text. The text content includes at least the text description content of the non-textual content.

[0295] Optionally, the text content may also include link text for linking to non-text content;

[0296] The device also includes a display unit 1404, for:

[0297] Retrieve non-text content based on linked text;

[0298] Display non-text content and answer text.

[0299] Optionally, the prompt content template includes a reading comprehension prompt template, which includes the second instruction content;

[0300] Building unit 1402 is specifically used for:

[0301] Based on the question text and reading comprehension prompt template, a second prompt content is constructed. The second prompt content includes the question text, the second instruction content, and the text content. The text content includes at least the non-text content text description content. The second instruction content is used to instruct the question-answering model to output the answer to the question text based on the text content.

[0302] Optionally, building unit 1402 is specifically used for:

[0303] From candidate texts and candidate question-answer pairs, retrieve reference knowledge texts related to the question text. Candidate texts contain relevant knowledge in the target domain, while reference knowledge texts include text content.

[0304] Based on the reference knowledge text, the question text, and the reading comprehension prompt template, construct the second prompt content.

[0305] Optionally, the prompt content template includes a question-and-answer pair prompt template, which includes the first instruction content;

[0306] Candidate question-answer pairs are obtained through the following method:

[0307] Based on candidate text and question-answer pair prompt templates, a first prompt content is constructed. The first prompt content includes a first instruction content and text content. The first instruction content is used to instruct the question-answering model to output a question-answer pair based on the text content.

[0308] Input the first prompt into the question-answering model to obtain candidate question-answer pairs.

[0309] Optionally, the text description content can be obtained through at least one of the following methods:

[0310] Character recognition is performed on non-text content to extract character information contained in the non-text content, and text description content is obtained based on the character information;

[0311] Feature extraction is performed on non-textual content to obtain content vectors; feature reconstruction is performed based on the content vectors to obtain textual description information of the non-textual content, and textual description content is obtained based on the textual description information.

[0312] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0313] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0314] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0315] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The server is shown. In this embodiment, the structure of the electronic device can be as follows. Figure 15 As shown, it includes a memory 1501, a communication module 1503, and one or more processors 1502.

[0316] The memory 1501 is used to store computer programs executed by the processor 1502. The memory 1501 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0317] Memory 1501 may be volatile memory, such as random-access memory (RAM); memory 1501 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1501 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1501 may be a combination of the above-described memories.

[0318] Processor 1502 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1502 is used to implement the above-described question-answering model training method when calling the computer program stored in memory 1501.

[0319] The communication module 1503 is used to communicate with terminal devices and other servers.

[0320] This application embodiment does not limit the specific connection medium between the memory 1501, communication module 1503, and processor 1502. This application embodiment... Figure 15 The memory 1501 and the processor 1502 are connected via a bus 1504, and the bus 1504 is in Figure 15 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1504 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 15 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0321] The memory 1501 stores a computer storage medium, which stores computer-executable instructions. These instructions are used to implement the question-answering model training method of this application embodiment. The processor 1502 is used to execute the above-described question-answering model training method, such as... Figure 4 As shown.

[0322] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 1 The terminal device is shown. In this embodiment, the electronic device can be structured as follows. Figure 16 As shown, it includes components such as: communication component 1610, memory 1620, display unit 1630, camera 1640, sensor 1650, audio circuit 1660, Bluetooth module 1670, processor 1680, etc.

[0323] The communication component 1610 is used to communicate with the server. In some embodiments, it may include a Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology, and the electronic device can send and receive information through the WiFi module.

[0324] The memory 1620 can be used to store software programs and data. The processor 1680 executes various functions of the terminal device and data processing by running the software programs or data stored in the memory 1620. The memory 1620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1620 stores an operating system that enables the terminal device to run. In this application, the memory 1620 may store the operating system and various application programs, and may also store a computer program that executes the training method of the question-answering model of the embodiments of this application.

[0325] The display unit 1630 can also be used to display information input by the object or information provided to the object, as well as a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 1630 may include a display screen 1632 disposed on the front of the terminal device. The display screen 1632 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 1630 can be used to display the training object interface of the question-and-answer model in the embodiments of this application, etc.

[0326] The display unit 1630 can also be used to receive input digital or character information and generate signal inputs related to object settings and function control of the terminal device. Specifically, the display unit 1630 may include a touch screen 1631 disposed on the front of the terminal device, which can collect touch operations on or near the object, such as clicking a button, dragging a scroll box, etc.

[0327] The touchscreen 1631 can be placed over the display screen 1632, or the touchscreen 1631 and the display screen 1632 can be integrated to realize the input and output functions of the terminal device. After integration, it can be referred to as a touch display screen. In this application, the display unit 1630 can display the application program and the corresponding operation steps.

[0328] Camera 1640 can be used to capture still images, and objects can post comments on images captured by camera 1640 through an application. There can be one or multiple cameras 1640. An optical image of an object is generated through a lens and projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to processor 1680 to be converted into a digital image signal.

[0329] The terminal device may also include at least one sensor 1650, such as an accelerometer 1651, a proximity sensor 1652, a fingerprint sensor 1653, and a temperature sensor 1654. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0330] Audio circuitry 1660, speaker 1661, and microphone 1662 provide an audio interface between the device and the terminal. Audio circuitry 1660 converts received audio data into electrical signals, which are then transmitted to speaker 1661, where they are converted into sound signals for output. The terminal device can also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 1662 converts collected sound signals into electrical signals, which are then received by audio circuitry 1660, converted back into audio data, and output to communication component 1610 for transmission to, for example, another terminal device, or to memory 1620 for further processing.

[0331] The Bluetooth module 1670 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 1670, thereby exchanging data.

[0332] The processor 1680 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 1620 and calling data stored in the memory 1620. In some embodiments, the processor 1680 may include one or more processing units; the processor 1680 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 1680. In this application, the processor 1680 can run the operating system, applications, user interface display and touch response, and the question-answering model training method of this embodiment. Furthermore, the processor 1680 is coupled to the display unit 1630.

[0333] In some possible implementations, various aspects of the question-answering model training method provided in this application can also be implemented as a program product, which includes a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the question-answering model training method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 4 The steps are shown in the figure.

[0334] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0335] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0336] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0337] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0338] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the target electronic device, partially on the target device, as a standalone software package, partially on the target electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the target electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0339] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0340] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0341] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0342] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0343] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0344] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0345] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0346] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for a question-answering model, characterized in that, The method comprises: The following operations are performed separately for each sample data: Replace the non-text content included in the sample data with corresponding text content to obtain sample text; The text content at least includes the text description content of the non-text content; Based on the obtained sample text and the preset prompt content template, the sample prompt content corresponding to the sample text is constructed, the sample prompt content includes the text content and instruction content, and the instruction content is used to instruct the initial model to output an answer based on the text content; Based on the sample text, a reference answer of the sample prompt content is obtained, and the reference answer includes the text content and a sample answer related to the text content; Each of the constructed sample prompt contents is used to iteratively train the initial model to obtain a question and answer model, wherein in each iteration, the following steps are performed: the initial model is used to predict an answer based on a sample prompt content, and the model parameters are adjusted based on the difference between the obtained predicted answer and the reference answer of the sample prompt content.

2. The method of claim 1, wherein, The text description content is obtained by at least one of the following ways: Character recognition is performed on the non-text content, character information contained in the non-text content is extracted, and the text description content is obtained based on the character information; Feature extraction is performed on the non-text content to obtain a content vector of the non-text content; feature reconstruction is performed based on the content vector to obtain text description information of the non-text content, and the text description content is obtained based on the text description information.

3. The method of claim 1, wherein, The prompt content template includes a question and answer pair prompt template, and the question and answer pair prompt template includes first instruction content; Based on the obtained sample text and the preset prompt content template, the sample prompt content corresponding to each of the sample texts is constructed, comprising: Based on the sample text and the question and answer pair prompt template, a first sample prompt content is constructed, the first sample prompt content includes the first instruction content and the text content, and the first instruction content is used to instruct the initial model to output a question and answer pair based on the text content.

4. The method of claim 1, wherein, The prompt content template includes a reading comprehension prompt template, and the reading comprehension prompt template includes second instruction content; Based on the obtained sample text and the preset prompt content template, the sample prompt content corresponding to each of the sample texts is constructed, comprising: Based on the sample text and the reading comprehension prompt template, a second sample prompt content is constructed, the second sample prompt content includes the second instruction content, a sample question, and the text content, and the second instruction content is used to instruct the initial model to output a predicted answer of the sample question based on the text content.

5. The method of claim 4, wherein, Based on the sample text and the reading comprehension prompt template, the second sample prompt content is constructed, comprising: The sample question is obtained; From the sample text and a sample question and answer pair, reference knowledge text related to the sample question is retrieved; Based on the reference knowledge text, the sample question, and the reading comprehension prompt template, the second sample prompt content is constructed.

6. The method according to any one of claims 1 to 5, wherein, The text content further includes link text of the non-text content, the link text being used for linking to the non-text content; the sample answer is related to the link text and the non-text content.

7. A method of applying a question and answer model, characterized by, The method comprises: obtaining question text; constructing prompt content based on the question text and a preset prompt content template; inputting the prompt content into the question and answer model to obtain a reply answer output by the question and answer model, the question and answer model being obtained based on the method in any one of claims 1 to 6, the reply answer including answer text for the question text and text content of non-text content related to the answer text, the text content at least including text description content of the non-text content.

8. The method of claim 7, wherein, The text content further includes link text for linking to the non-text content; The method further comprises: obtaining the non-text content based on the link text; displaying the non-text content and the answer text.

9. The method of claim 7, wherein, The prompt content template includes a reading comprehension prompt template, and the reading comprehension prompt template includes second instruction content; The constructing prompt content based on the question text and a preset prompt content template comprises: constructing second prompt content based on the question text and the reading comprehension prompt template, the second prompt content including the question text, the second instruction content, and the text content, the text content at least including text description content of the non-text content, and the second instruction content being used for instructing the question and answer model to output an answer to the question text based on the text content.

10. The method of claim 9, wherein, The constructing second prompt content based on the question text and the reading comprehension prompt template comprises: retrieving reference knowledge text related to the question text from candidate text and candidate question and answer pairs, the candidate text recording related knowledge of a target field, and the reference knowledge text including the text content; constructing the second prompt content based on the reference knowledge text, the question text, and the reading comprehension prompt template.

11. The method of claim 10, wherein, The prompt content template includes a question and answer pair prompt template, and the question and answer pair prompt template includes first instruction content; The candidate question and answer pairs are obtained in the following manner: constructing first prompt content based on the candidate text and the question and answer pair prompt template, the first prompt content including the first instruction content and the text content, and the first instruction content being used for instructing the question and answer model to output question and answer pairs based on the text content; inputting the first prompt content into the question and answer model to obtain the candidate question and answer pairs.

12. The method according to any one of claims 7 to 11, characterized in that, The text description content is obtained in at least one of the following manners: performing character recognition on the non-text content, extracting character information included in the non-text content, and obtaining the text description content based on the character information; performing feature extraction on the non-text content to obtain a content vector of the non-text content, performing feature reconstruction based on the content vector to obtain text description information of the non-text content, and obtaining the text description content based on the text description information.

13. An application apparatus of a question and answer model, characterized by, comprises: an obtaining unit configured to obtain question text; A constructing unit is configured to construct prompt content based on the question text and a preset prompt content template; An extracting unit is configured to input the prompt content into the question and answer model to obtain a reply answer output by the question and answer model, the question and answer model being obtained based on the method in any one of claims 1 to 6, and the reply answer including an answer text for the question text and a text content of non-text content related to the answer text, the text content at least including a text description content of the non-text content.

14. An electronic device, comprising: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the computer program causes the processor to execute the steps of the method in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, It comprises a computer program, and when the computer program runs on an electronic device, the computer program is used to make the electronic device execute the steps of the method in any one of claims 1-12.

Images