Large language model training method and device, information processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a large language model training method and device, an information processing method and device, electronic equipment, a storage medium and a program product, relates to the technical field of artificial intelligence, and reduces the illusion reply phenomenon of a large language model. The large language model training method comprises the steps of obtaining a target data set, wherein target data in the target data set comprises a preset question and a truth value reply of the preset question; auxiliary information of a preset problem in the target data is acquired; a first prompt, a preset question in the target data and auxiliary information of the preset question are input into a large language model, a reply of the preset question is generated, and the first prompt is used for instructing the large language model to output a negative reply under the condition that the preset question cannot be determined; and checking the reply of the large language model according to the truth value reply of the preset question, and finely adjusting the large language model according to a checking result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, information processing method, apparatus, electronic device, storage medium, and program product for a large language model. Background Technology

[0002] In recent years, with the exponential growth of the number of parameters and the scale of training data of large language models such as GPT-4, PaLM-E, and Gemini, the overall quality of question answering systems has been significantly improved. Users can obtain coherent and seemingly authoritative answers simply by asking questions in natural language.

[0003] However, these large language models commonly exhibit hallucinatory responses under the "genesis is rational" paradigm. Hallucinatory responses refer to the phenomenon where the content responded by a large language model does not match the actual data or deviates from the user's instructions.

[0004] Furthermore, as the instruction-following, context-learning, and logical reasoning capabilities inherent in large language models are transferred to the scenarios of multimodal question-answering systems, the hallucination response phenomenon inherent in large language models is not eliminated in multimodal question answering; instead, it presents new characteristics, such as visual hallucinations. Visual hallucinations refer to large language models giving affirmative descriptions of objects, attributes, or relationships that do not exist in an image, such as saying "red car" instead of "white truck."

[0005] Therefore, how to reduce the phenomenon of hallucinatory responses in large language models is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a training method, apparatus, information processing method, apparatus, electronic device, storage medium, and program product for large language models, so as to reduce the phenomenon of hallucination response in large language models.

[0007] In a first aspect, embodiments of the present invention provide a method for training a large language model, comprising:

[0008] Obtain the target dataset, wherein the target data in the target dataset includes a preset question and the truth response to the preset question;

[0009] Obtain auxiliary information about the preset problem from the target data;

[0010] The first prompt, the preset question in the target data, and auxiliary information of the preset question are input into the large language model to generate a response to the preset question. The first prompt instructs the large language model to output a negative response if it cannot determine the preset question.

[0011] The response of the large language model is verified based on the truth response of the preset question, and the large language model is fine-tuned based on the verification result.

[0012] Based on the above method, further improvements are made to the large language model using a low-rank adaptive fine-tuning method according to the verification results, including:

[0013] If the verification result of the response from the large language model is correct, at least one similar response that is similar to the truth response of the preset question is generated.

[0014] Based on the verification results and the at least one similar response, the large language model is fine-tuned using a low-rank adaptive fine-tuning method.

[0015] Based on further improvements to the above method, generating at least one similar response to the truth response of the preset question includes:

[0016] The second prompt, the preset question, and the truth value response of the preset question are input into the second large language model to generate at least one similar response that is similar to the truth value response of the preset question. The second prompt is used to prompt the second large language model to generate at least one similar response that is similar to the truth value response of the preset question based on the preset question and the truth value response of the preset question.

[0017] Based on a further improvement to the above method, after generating at least one similar response that is similar to the truth response to the preset problem, the method further includes:

[0018] Verify the at least one similar response based on the truth response to the preset question;

[0019] Furthermore, based on the verification results and the at least one similar response, the low-rank adaptive fine-tuning method is used to fine-tune the large language model, including:

[0020] Based on the verification results of at least one similar response and the verification results of the response of the large language model, the large language model is fine-tuned using a low-rank adaptive fine-tuning method.

[0021] Based on a further improvement of the above method, when the preset question is in the form of text and images, the auxiliary information for the preset question in the target data includes:

[0022] Retrieve at least one similar image to the image in the preset question using a search engine;

[0023] Extract text information from any one of the at least one similar images, and use the text information as auxiliary information for the preset question.

[0024] Based on a further improvement of the above method, when the preset question is in text form, the auxiliary information for obtaining the preset question from the target data includes:

[0025] Input a pre-defined question in text form into a search engine to obtain at least one Uniform Resource Locator (URL) related to the pre-defined question.

[0026] Extract resource information from any one of the at least one Uniform Resource Locators (URLs), and use the resource information as auxiliary information for the preset problem.

[0027] Based on a further improvement of the above method, the auxiliary information is used to provide the large language model with objective external references related to the preset question during the training phase, so as to help the large language model determine whether its internal knowledge is credible, thereby suppressing the generation of hallucinatory responses when the internal knowledge of the large language model is insufficient to answer the question.

[0028] Secondly, embodiments of the present invention provide an information processing method, including:

[0029] The problem of obtaining user input;

[0030] The large language model processes the problem and obtains a response, wherein the large language model is obtained using the training method of the large language model according to any one of the first aspects of the present invention;

[0031] The response will be returned to the user.

[0032] Further improvements to the above method also include:

[0033] The large language model obtains multiple responses based on the user's input question; and

[0034] The large language model is used to rank the multiple responses according to their relevance.

[0035] Thirdly, embodiments of the present invention provide a training apparatus for a large language model, comprising:

[0036] The first acquisition unit is used to acquire a target dataset, wherein the target data in the target dataset includes a preset question and the truth response to the preset question;

[0037] The second acquisition unit is used to acquire auxiliary information about a preset problem in the target data;

[0038] A first generation unit is configured to input a first prompt, a preset question from the target data, and auxiliary information of the preset question into a large language model, and generate a response to the preset question, wherein the first prompt is configured to instruct the large language model to output a negative response if it cannot determine the preset question; and

[0039] The verification unit is used to verify the response of the large language model based on the truth response of the preset question, and to fine-tune the large language model based on the verification result.

[0040] Fourthly, embodiments of the present invention provide an information processing apparatus, characterized in that it includes:

[0041] The third acquisition unit is used to acquire user input.

[0042] A processing unit is configured to process the question using a large language model to obtain a response, wherein the large language model is obtained using the training method of the large language model according to any one of the first aspects of the present invention;

[0043] The return unit is used to return the response to the user.

[0044] Fifthly, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, the device on which the storage medium is located executes the training method for a large language model as described in any one of the first aspects of the present invention, or the information processing method as described in any one of the second aspects of the present invention.

[0045] Sixthly, embodiments of the present invention provide an electronic device, comprising:

[0046] Memory, which stores executable programs;

[0047] A processor for running the program, wherein the program, when running, executes the training method for a large language model according to any one of the first aspects of the present invention, or the information processing method according to any one of the second aspects of the present invention.

[0048] In a seventh aspect, embodiments of the present invention provide a computer program product comprising a stored computer program that, when executed by a processor, implements the training method for a large language model as described in any one of the first aspects of the present invention, or the information processing method as described in any one of the second aspects of the present invention.

[0049] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0050] 1. The present invention effectively solves the problem of hallucination responses in large language models by introducing auxiliary information from preset questions and a label for responding with a negative answer when the large language model cannot provide an answer, and significantly reduces the probability of hallucination responses in large language models.

[0051] 2. This invention can handle preset questions in both image and text formats. Therefore, the solution can be applied to multimodal question answering systems, improving the performance of multimodal question answering systems and effectively solving the problem of visual illusions in the output of large language models.

[0052] 3. In fine-tuning a large language model, the present invention expands sample diversity and increases the generalization ability of the large language model by constructing similar responses that are similar to the correct responses, thereby preventing overfitting.

[0053] 4. The present invention introduces objective external references as a basis for judgment, and trains the large language model to actively output negative responses when the credibility of internal knowledge is insufficient. This effectively reduces the illusion phenomenon caused by insufficient credibility of internal knowledge. The ability to suppress illusion has been transformed from dependence on external tools to an intrinsic generative attribute of the model. The present invention directly improves the accuracy and reliability of the model's response.

[0054] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0055] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0056] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein may be implemented according to embodiments of the present invention is shown.

[0057] Figure 2 A flowchart illustrating a training method for a large language model according to an embodiment of the present invention is shown.

[0058] Figure 3 A flowchart illustrating a training method for a large language model according to another embodiment of the present invention is shown.

[0059] Figure 4 A flowchart illustrating an information processing method according to an embodiment of the present invention is shown.

[0060] Figure 5 A structural block diagram of a training apparatus for a large language model according to an embodiment of the present invention is shown.

[0061] Figure 6 A structural block diagram of an information processing apparatus according to an embodiment of the present invention is shown.

[0062] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention. Detailed Implementation

[0063] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0064] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of the present invention is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, and 104, one or more servers 120, and one or more communication networks 110 coupling the one or more client devices to the one or more servers 120. The client devices 101, 102, 103, and 104 can be configured to execute one or more applications.

[0065] In embodiments of the invention, one or more servers 120 may run one or more services or software applications that enable the training method of the large language model of the invention to be performed. In one exemplary embodiment, a large language model or a multimodal large language model may be deployed on one or more servers 120.

[0066] In some embodiments, one or more servers 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. In some embodiments, these services may be provided as web-based or cloud-based services, such as to users of client devices 101, 102, 103 and / or 104 under a Software as a Service (SaaS) model.

[0067] exist Figure 1In the configuration shown, one or more servers 120 may include one or more components that implement the functions performed by the one or more servers 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, and / or 104 can sequentially use one or more client applications to interact with one or more servers 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0068] Users can use client devices 101, 102, 103, and / or 104 to generate response data using a large language model or a multimodal large language model. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface, for example, outputting responses generated by the large language model or multimodal large language model in response to user input commands. Although Figure 1 Only four types of client devices are described, but those skilled in the art will understand that the invention can support any other type of client device, such as robotic devices.

[0069] Client devices 101, 102, 103, and / or 104 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google ChromeOS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Client devices are capable of executing various applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0070] One or more networks 110 may be any type of network well known to those skilled in the art, and may use any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0071] One or more servers 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. One or more servers 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, one or more servers 120 may run one or more services or software applications that provide the functionality described below.

[0072] The computing units in one or more servers 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. One or more servers 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0073] In some implementations, one or more servers 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103 and / or 104. One or more servers 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103 and / or 104.

[0074] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0075] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, a database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0076] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0077] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and apparatuses described in the present invention.

[0078] According to one aspect of the present invention, a method for training a large language model is provided. Figure 2 A flowchart illustrating a training method for a large language model according to an embodiment of the present invention is shown. Figure 2 As shown, the training method for this large language model includes:

[0079] Step 201: Obtain the target dataset.

[0080] In this embodiment, the target dataset can be a problem-based dataset, which may include multiple target datasets, each of which includes a preset problem and the truth response to the preset problem.

[0081] In this embodiment, the target dataset can be either a text classification dataset or a multimodal dialogue dataset. The data in the text classification dataset is text data, while the data in the multimodal dialogue dataset is image and text data.

[0082] In the example where the target dataset is a text classification dataset, the target dataset could be part of a Question-based Text Classification (QBQTC) dataset. The QBQTC dataset is part of CLUEBenchmark, which provides a text classification dataset containing various question types and complex contexts.

[0083] Table 1 below provides examples of five data points from the QBQTC dataset:

[0084]

[0085] Table 1

[0086] In Table 1, the title refers to the title in the corresponding Uniform Resource Locator (URL). The titles in Table 1 represent the responses to the input questions, and the label value of each data point represents the degree of similarity between the input question and the title.

[0087] In some embodiments, the target dataset can be a text classification dataset that meets a preset quality standard. For example, if the text classification dataset is the QBQTC dataset, multiple data points with a label value of 2 can be selected from the QBQTC dataset as the target dataset. In this example, titles with a label value of 2 represent the truth responses to the corresponding questions.

[0088] It should be noted that selecting a text classification dataset that meets the preset quality standards is beneficial, as it helps to further reduce the probability of hallucination responses in large language models and improve their accuracy.

[0089] In the example where the target dataset is a multimodal dialogue dataset, the target dataset can be a subset of the LLaVA-Instruct-150k dataset, the LLaVA-CN-150k dataset, the ShareGPT4V dataset, or the DocVQA dataset. The data in a multimodal dialogue dataset includes three fields: "id", "image", and "conversations". The "id" field represents the data identifier, the "image" field represents the image path to which the dialogue depends, and the "conversations" field represents a list of multi-turn dialogues, with each turn indicating the speaker and the text content of the question.

[0090] Below is an example of data from a multimodal dialogue dataset:

[0091]

[0092] In the example above, " The symbol "" indicates a placeholder, telling the model "look at the image here".

[0093] In some embodiments, the target dataset can be a multimodal dialogue dataset that meets a preset quality standard. For example, the preset quality standard may refer to the degree of matching between the "text response" and the "image content" exceeding a preset threshold. For instance, a Contrastive Language–Image Pretraining (CLIP) model can be used to calculate the similarity score s between the "text response" and the "image content." If the similarity score s is greater than 0.9, the preset quality standard is met; otherwise, it is not. In this example, a "text response" with a similarity score s greater than 0.9 is considered a ground truth response.

[0094] It should be noted that selecting a multimodal dialogue dataset that meets the preset quality standards is beneficial, as it helps to further reduce the probability of hallucination responses in multimodal large language models and improve their accuracy.

[0095] This invention can handle preset questions in both image and text formats, and therefore the solution can be applied to multimodal question answering systems, improving the performance of multimodal question answering systems and effectively solving the problem of visual illusions in the output of large language models.

[0096] Step 202: Obtain auxiliary information about the preset problem in the target data.

[0097] In this embodiment, the auxiliary information for the preset question refers to information related to the preset question in an external database. In this embodiment, the auxiliary information for the preset question in the target data can be obtained through a search engine. In the example where the target data is image-text data, the auxiliary information for the preset question may refer to the text information of images similar to the image in the preset question, retrieved by the search engine. In the example where the target data is text data, the auxiliary information for the preset question may refer to resource information in URLs related to the preset question, retrieved by the search engine.

[0098] In the example where the target data is text and image data, step 202 may include the following steps:

[0099] Step 2020: Use a search engine to retrieve at least one similar image to the image in the preset question.

[0100] Step 2022: Extract the text information of any one of the at least one similar images and use the text information as auxiliary information for the preset question.

[0101] The following explanation uses the target data as "text + image → reply answer" as an example to illustrate steps 2020 and 2022.

[0102] First, a search engine can be used to retrieve N images most similar to the image involved in the current problem (hereinafter referred to as the "current image"), where N is a positive integer greater than or equal to 1. In this embodiment, the search engine can vectorize the current image using a built-in image vector model, and then retrieve the N images most similar to the current image. N can take values ​​between 2 and 100, for example, N equals 10. After retrieving these N images, one image can be selected and its URL can be accessed. Finally, text information can be extracted from the URL of the image as auxiliary information for the current image.

[0103] In the example where the target data is text data, step 202 may include the following steps:

[0104] Step 2021: Input the preset question in text form into the search engine to obtain at least one Uniform Resource Locator (URL) related to the preset question.

[0105] Step 2023: Extract resource information from any one of the at least one Uniform Resource Locators (URLs) and use this resource information as auxiliary information for the preset question.

[0106] The following explanation uses "text question → reply answer" as an example to illustrate steps 2021 and 2023.

[0107] First, input the text question into a search engine to obtain the M most relevant URLs, where M is a positive integer greater than or equal to 1. Then, select one of these M URLs and click to enter the search results. M can be between 2 and 100, for example, M equals 10. Finally, extract the text information from the entered URL as supplementary information for the text question.

[0108] Step 203: Input the first prompt, the preset question in the target data, and the auxiliary information of the preset question into the large language model to generate a response to the preset question.

[0109] In this embodiment, the first prompt is used to guide the large language model to respond with a negative answer when it does not know the answer. In this embodiment, the first prompt can be freely designed; for example, the first prompt can be designed as follows:

[0110] You are a very helpful assistant, able to truthfully answer users' questions about the provided images. I am now providing the image and some text description; please provide any information and answers that may be related to the image. Please keep your answers concise and to the point. If you don't know the answer, simply reply "I don't know."

[0111] In this embodiment, a preset question (e.g., " The system takes the following information: "What color is the cabinet?", the auxiliary information of the preset question, and the first prompt as input to the large language model to generate the response R for the preset question.

[0112] In this embodiment, the large language model can be any multimodal large language model. For example, the large language model can be Llama-3.2-11B-Vision-Instruct.

[0113] In this embodiment, the auxiliary information is used to provide the large language model with objective external references related to the preset question during the training phase, so as to help the large language model determine whether its internal knowledge is credible, thereby suppressing the generation of hallucinatory responses when the internal knowledge of the large language model is insufficient to answer the question.

[0114] During the training phase, the input to the large language model includes: a pre-set question to be answered, relevant auxiliary information obtained from external sources, and a first prompt instructing it to output a negative response (such as "I don't know") when it cannot determine the answer.

[0115] Large language models need to synthesize responses to preset questions based on the knowledge encoded by their internal parameters (internal knowledge) and the provided external auxiliary information. This auxiliary information serves as a reference point when answering the preset questions. When the large language model's internal knowledge is clear and corroborates the external auxiliary information, it tends to generate a positive and specific response. Conversely, when the large language model's internal knowledge is vague, insufficient, or significantly inconsistent with or unable to support the information provided by the external auxiliary information, it is more likely to output a negative response based on the initial prompt.

[0116] Specifically, auxiliary information assists large language models in judging the credibility of knowledge; its essence is an implicit calibration achieved through training data. In each training iteration, the large language model compares its generated responses with preset ground truth responses. If the large language model forcibly generates a positive response (i.e., a hallucinatory response) due to insufficient internal knowledge, and this response does not match the ground truth, the large language model will receive a negative feedback signal through the loss function. Conversely, if the large language model outputs a negative response in a similar situation, and this behavior is verified as correct (i.e., the ground truth itself is "unknown" or the model should not have answered), the large language model will receive positive reinforcement.

[0117] The training process described above essentially teaches the model how to assess the sufficiency and credibility of its own parameterized knowledge (internal knowledge) in answering a question. External auxiliary information is used to generate a "credible / uncredible" discrimination signal for the model during training. After fine-tuning on large-scale data using methods such as low-rank adaptation (LoRA), the model gradually adjusts its internal computational paths and attention distribution, ultimately solidifying this discriminative ability into its weight parameters.

[0118] Therefore, once training is complete and deployed for inference, this large language model no longer needs to acquire external contextual information in real time. When faced with user questions, the model directly relies on its modified internal mechanism for computation. This mechanism simulates the discriminative process learned during training, analyzes the question, and assesses whether its existing knowledge can constitute a reliable and definite answer. If the assessment deems the knowledge sufficient and credible, the model will generate a positive response; if the assessment deems the knowledge ambiguous, conflicting, or insufficient, the model will proactively tend to generate a negative response based on training habits, rather than forcibly fabricating a potentially incorrect answer. The ability to suppress illusions has shifted from dependence on external tools to an intrinsic generative property of the model.

[0119] By introducing objective external references as a criterion, this approach trains a large language model to proactively output negative responses when the model's internal knowledge credibility is insufficient, thereby effectively reducing the illusion phenomenon caused by insufficient internal knowledge credibility. This method directly improves the accuracy and reliability of the model's responses.

[0120] Step 204: Verify the response of the large language model based on the truth response of the preset question, and fine-tune the large language model based on the verification results.

[0121] In this embodiment, the process of verifying the response of the large language model may include the following two aspects:

[0122] 1. If the response from the large language model is correct, the sample can be retained;

[0123] 2. If the response from the large language model is incorrect, it can be marked as a negative answer, which is the label of the sample.

[0124] In this embodiment, the correctness of the large language model's response can be determined based on the similarity between the truth response to a preset question and the response of the large language model. If the similarity between the two meets a preset condition, the response of the large language model is correct; otherwise, it is incorrect. For example, the preset condition could be that the similarity is greater than a similarity threshold.

[0125] In some embodiments, a second large language model can be used to validate the response of the large language model. For example, the truth response to a preset question, the response of the large language model, and validation hints can be input into the second large language model to obtain the validation result. For example, the second large language model can be a GPT-4o mini model.

[0126] In this embodiment, the large language model can be fine-tuned using the low-rank adaptation fine-tuning method based on the validated samples. Low-rank adaptation (LoRA) fine-tuning is an efficient parameter fine-tuning method based on low-rank decomposition. It significantly reduces computational cost and storage requirements while maintaining model performance by freezing the original model parameters and introducing a trainable low-rank matrix. LoRA fine-tuning is an existing technique for fine-tuning large models and will not be elaborated upon here.

[0127] The training method for large language models proposed in this invention effectively solves the problem of hallucination responses in large language models by using retrieval enhancement to generate data augmentation, and significantly reduces the probability of hallucination responses in large language models.

[0128] In some embodiments, step 204, "fine-tuning the large language model using a low-rank adaptive fine-tuning method based on the verification results," may include the following steps:

[0129] Step 2040: In response to the correct verification result of the response from the large language model, generate at least one similar response that is similar to the truth response of the preset question.

[0130] In this embodiment, if the response of the large language model is correct, at least one similar response that is similar to the truth response of the preset question can be generated.

[0131] In this embodiment, each similar response can be combined with the current preset question to form a training sample for training the large language model. Each similar response serves as the label for the current training sample.

[0132] This embodiment does not limit the method of creating similar responses; they can be created manually or generated with the help of tools (such as the second largest language model).

[0133] Step 2042: Based on the verification results and at least one similar response, fine-tune the large language model using the low-rank adaptive fine-tuning method.

[0134] In this embodiment, the LoRA fine-tuning method can be used to fine-tune the large language model by using samples obtained after verifying the responses of the large language model and samples composed of the current preset question and similar responses.

[0135] The training method for the large language model proposed in the above embodiments increases the generalization ability of the large language model by using multiple similar labels, thereby preventing overfitting.

[0136] In some embodiments, step 2040, "generating at least one similar response that is similar to the truth response to the preset question," may include:

[0137] The second prompt, the preset question, and the truth value response to the preset question are input into the second language model to generate at least one similar response that is similar to the truth value response to the preset question.

[0138] In this embodiment, the second prompt is used to prompt the second language model to generate at least one similar response that is similar to the truth response of the preset question, based on the preset question and the truth response of the preset question. In this embodiment, the second prompt can be freely designed; for example, the second prompt can be designed as follows:

[0139] Given a question and a standard answer, please help me generate 20 similar standard answers.

[0140] Output format:

[0141] 1.XX

[0142] 2.XX

[0143] 3.XX

[0144] ...

[0145] Question: {query}

[0146] Standard answer: {answer}

[0147] Figure 3 A flowchart illustrating a training method for a large language model according to another embodiment of the present invention is shown. Figure 3 As shown, the training method for this large language model includes:

[0148] Step 301: Obtain the target dataset.

[0149] Step 301 is basically the same as step 201, so it will not be repeated here.

[0150] Step 302: Obtain auxiliary information about the preset problem in the target data.

[0151] Step 302 is basically the same as step 202, so it will not be described again here.

[0152] Step 303: Input the first prompt, the preset question in the target data, and the auxiliary information of the preset question into the large language model to generate a response to the preset question.

[0153] Step 303 is basically the same as step 203, so it will not be repeated here.

[0154] Step 304: Verify the response of the large language model based on the truth response of the preset question.

[0155] Step 304 is basically the same as "verifying the response of the large language model based on the truth value response of the preset question" in step 204, so it will not be repeated here.

[0156] Step 305: In response to the verification result of the response from the large language model being correct, input the second prompt, the preset question, and the truth response to the preset question into the second large language model to generate at least one similar response that is similar to the truth response to the preset question.

[0157] In this embodiment, if the response of the large language model is correct, the second prompt, the preset question, and the truth value response of the preset question can be input into the second large language model to generate at least one similar response that is similar to the truth value response of the preset question.

[0158] In this embodiment, the second prompt is used to prompt the second language model to generate at least one similar response that is similar to the truth response of the preset question, based on the preset question and the truth response of the preset question. In this embodiment, the second prompt can be freely designed; for example, the second prompt can be designed as follows:

[0159] Given a question and a standard answer, please help me generate 20 similar standard answers.

[0160] Output format:

[0161] 1.XX

[0162] 2.XX

[0163] 3.XX

[0164] ...

[0165] Question: {query}

[0166] Standard answer: {answer}

[0167] Step 306: Verify the at least one similar response based on the truth response to the preset question.

[0168] In this embodiment, the verification process for similar responses may include the following two aspects:

[0169] 1. If the similar responses are correct, the sample can be retained;

[0170] 2. If a similar response is incorrect, the sample can be discarded.

[0171] In this embodiment, the correctness of a similar response can be determined based on the degree of similarity between the truth response to a preset question and the similar response itself. If the degree of similarity between the two meets a preset condition, the similar response is correct; otherwise, it is incorrect.

[0172] In some embodiments, a second large language model can be used to validate similar responses. For example, the truth responses, similar responses, and validation hints for a preset question can be input into the second large language model to obtain the validation results. For example, the second large language model can be a GPT-4o mini model.

[0173] Step 307: Based on the verification results of at least one similar response and the verification results of the response from the large language model, fine-tune the large language model using the low-rank adaptive fine-tuning method.

[0174] In this embodiment, the LoRA fine-tuning method can be used to fine-tune the large language model by using samples obtained after verifying the responses of the large language model and samples retained after verifying similar responses.

[0175] The present invention expands sample diversity, increases the generalization ability of the large language model, and prevents overfitting by constructing similar responses to the correct responses when fine-tuning a large language model.

[0176] According to another aspect of the present invention, an information processing method is provided. Figure 4 A flowchart illustrating an information processing method according to an embodiment of the present invention is shown. Figure 4 As shown, the information processing method includes:

[0177] Step 401: Obtain the user's input question.

[0178] Step 402: The large language model processes the question and obtains a response.

[0179] In this embodiment, the large language model can be obtained using any embodiment of the training method for the large language model provided by the present invention.

[0180] Step 403: Return the response to the user.

[0181] The information processing method proposed in this invention can reduce the probability of hallucination responses in large language models and improve user experience.

[0182] In some embodiments, Figure 4 The information processing methods in the text also include:

[0183] Step 406: The large language model obtains multiple responses based on the user's input question.

[0184] In this embodiment, a preset question and a first prompt can be input into the large language model to generate a response. This process can then be repeated to generate multiple responses.

[0185] Step 408: Use a large language model to rank the multiple responses according to their relevance.

[0186] In this embodiment, a preset question, multiple responses obtained in step 406, and reordering prompts can be input into a large language model for reasoning. The large language model outputs the most relevant response.

[0187] In this embodiment, the reordering suggestion is used to prompt the large language model to rank the multiple responses obtained in step 406 according to their relevance. In this embodiment, the reordering suggestion can be freely designed; for example, the reordering suggestion can be designed as follows:

[0188] You need to understand the question and the picture. I will then provide you with a set of answers and their numbers. Please sort them according to their relevance.

[0189] Output the list of most relevant labels, such as [x,xx,xx,……]. If there are no relevant items, output [].

[0190] Step 410: According to the set strategy, return one or more of the responses that are ranked first in relevance.

[0191] In this embodiment, the large language model can output the most relevant responses. The specific number of responses can be preset. For example, if the strategy is set to the top N, then the N most relevant responses can be output. Output prompts and the sorted responses can be input into the large language model, which will then output the most relevant responses. In this embodiment, the output prompts are used to guide the large language model to output the most relevant responses among the sorted responses.

[0192] According to another aspect of the present invention, a training apparatus for a large language model is provided. For example... Figure 5As shown, the device 500 includes: a first acquisition unit 510, configured to acquire a target dataset, wherein the target data in the target dataset includes a preset question and a truth value response to the preset question; a second acquisition unit 520, configured to acquire auxiliary information of the preset question in the target data; a first generation unit 530, configured to input a first prompt, the preset question in the target data, and the auxiliary information of the preset question into a large language model to generate a response to the preset question, wherein the first prompt is configured to instruct the large language model to output a negative response when it cannot determine the preset question; and a verification unit 540, configured to verify the response of the large language model based on the truth value response of the preset question, and to fine-tune the large language model based on the verification result.

[0193] It is understandable that the operation of units 510-540 in device 500 is related to... Figure 2 The operations of steps 201-204 are similar and will not be described in detail here.

[0194] According to another aspect of the present invention, an information processing apparatus is provided. For example... Figure 6 As shown, the device 600 includes: a third acquisition unit 610 for acquiring a question input by a user; a processing unit 620 for processing the question using a large language model to obtain a response, wherein the large language model is obtained using any embodiment of the training method for the large language model provided by the present invention; and a return unit 630 for returning the response to the user.

[0195] It is understandable that the operation of units 610-630 in device 600 is related to... Figure 4 The operations of steps 401-403 are similar and will not be described in detail here.

[0196] According to embodiments of the present invention, an electronic device, a computer-readable storage medium, and a computer program product are also provided.

[0197] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0198] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0199] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0200] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the training method for a large language model. For example, in some embodiments, the training method for a large language model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the training method for a large language model described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the training method for a large language model by any other suitable means (e.g., by means of firmware).

[0201] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0203] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0205] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0206] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0207] Compared with the prior art, the embodiments of the present invention can achieve at least one of the following beneficial effects:

[0208] 1. The present invention effectively solves the problem of hallucination responses in large language models by introducing pre-set question auxiliary information and a label for responding with a negative answer when the large language model cannot provide an answer, and significantly reduces the probability of hallucination responses in large language models.

[0209] 2. This invention can handle preset questions in both image and text formats. Therefore, the solution can be applied to multimodal question answering systems, improving the performance of multimodal question answering systems and effectively solving the problem of visual illusions in the output of large language models.

[0210] 3. In fine-tuning a large language model, the present invention expands sample diversity and increases the generalization ability of the large language model by constructing similar responses that are similar to the correct responses, thereby preventing overfitting.

[0211] 4. The present invention introduces objective external references as a basis for judgment, and trains the large language model to actively output negative responses when the credibility of internal knowledge is insufficient. This effectively reduces the illusion phenomenon caused by insufficient credibility of internal knowledge. The ability to suppress illusion has been transformed from dependence on external tools to an intrinsic generative attribute of the model. The present invention directly improves the accuracy and reliability of the model's response.

[0212] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a large language model, characterized in that, include: Obtain the target dataset, wherein the target data in the target dataset includes a preset question and the truth response to the preset question; Obtain auxiliary information about the preset problem from the target data; The first prompt, the preset question in the target data, and the auxiliary information of the preset question are input into the large language model to generate a response to the preset question. The first prompt is used to instruct the large language model to output a negative response when it cannot determine the answer to the preset question. as well as The response of the large language model is verified based on the truth response of the preset question, and the large language model is fine-tuned based on the verification result.

2. The method according to claim 1, characterized in that, Fine-tuning the large language model based on the verification results includes: If the verification result of the response from the large language model is correct, at least one similar response that is similar to the truth response of the preset question is generated. Based on the verification results and the at least one similar response, the large language model is fine-tuned.

3. The method according to claim 2, characterized in that, Generating at least one similar response to the truth response to the preset question includes: The second prompt, the preset question, and the truth value response of the preset question are input into the second large language model to generate at least one similar response that is similar to the truth value response of the preset question. The second prompt is used to prompt the second large language model to generate at least one similar response that is similar to the truth value response of the preset question based on the preset question and the truth value response of the preset question.

4. The method according to claim 3, characterized in that, After generating at least one similar response that is similar to the truth response to the preset question, the process further includes: Verify the at least one similar response based on the truth response to the preset question; Furthermore, based on the verification results and the at least one similar response, fine-tuning the large language model includes: Based on the verification results of at least one similar response and the verification results of the response of the large language model, the large language model is fine-tuned using a low-rank adaptive fine-tuning method.

5. The method according to claim 1, characterized in that, When the preset question is in the form of text and images, the auxiliary information for obtaining the preset question from the target data includes: Retrieve at least one similar image to the image in the preset question using a search engine; Extract text information from any one of the at least one similar images, and use the text information as auxiliary information for the preset question.

6. The method according to claim 1, characterized in that, When the preset question is in text form, the auxiliary information for obtaining the preset question from the target data includes: Input a pre-defined question in text form into a search engine to obtain at least one Uniform Resource Locator (URL) related to the pre-defined question. Extract resource information from any one of the at least one Uniform Resource Locators (URLs), and use the resource information as auxiliary information for the preset problem.

7. The method according to any one of claims 1-6, characterized in that, The auxiliary information is used to provide the large language model with objective external references related to the preset question during the training phase, so as to help the large language model determine whether its internal knowledge is credible, thereby suppressing the generation of illusory responses when the internal knowledge of the large language model is insufficient to answer the question.

8. An information processing method, characterized in that, include: The problem of obtaining user input; The large language model processes the problem and obtains a response, wherein the large language model is obtained using the training method of the large language model according to any one of claims 1 to 7; The response will be returned to the user.

9. The method according to claim 8, characterized in that, Also includes: The large language model obtains multiple responses based on the user's input question. as well as The large language model is used to rank the multiple responses according to their relevance; Based on the set strategy, return one or more of the responses ranked highest in relevance.

10. A training device for a large language model, characterized in that, include: The first acquisition unit is used to acquire a target dataset, wherein the target data in the target dataset includes a preset question and the truth response to the preset question; The second acquisition unit is used to acquire auxiliary information about a preset problem in the target data; The first generation unit is used to input the first prompt, the preset question in the target data and the auxiliary information of the preset question into the large language model, and generate a response to the preset question. The first prompt is used to instruct the large language model to output a negative response when it cannot determine the preset question. as well as The verification unit is used to verify the response of the large language model based on the truth response of the preset question, and to fine-tune the large language model based on the verification result.

11. An information processing device, characterized in that, include: The third acquisition unit is used to acquire user input. A processing unit is configured to process the question using a large language model to obtain a response, wherein the large language model is obtained using the training method for the large language model according to any one of claims 1 to 7; The return unit is used to return the response to the user.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the training method of the large language model according to any one of claims 1 to 7, or the information processing method according to any one of claims 8 to 9.

13. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the training method for a large language model according to any one of claims 1 to 7, or the information processing method according to any one of claims 8 to 9.

14. A computer program product, characterized in that, The computer program product includes a stored computer program that, when executed by a processor, implements the training method for a large language model as described in any one of claims 1 to 7, or the information processing method as described in any one of claims 8 to 9.

Images