A model training and information generation method and device, a storage medium and an apparatus

Abstract

The specification provides a model training and information generation method, device, storage medium and equipment. The model training method comprises: obtaining a plurality of reply information generated by a to-be-trained question and answer model based on a prompt word, and determining a token sequence corresponding to each reply information; for each token in the token sequence, determine the conditional distribution probability of the token output by the question and answer model under the condition that the previous tokens are given, and determine the relative accuracy score of the reply information corresponding to the token sequence according to the conditional distribution probability corresponding to each token contained in the token sequence; wherein the relative accuracy score is positively correlated with the conditional distribution probability corresponding to each token contained in the corresponding token sequence; determine the target reward value according to the relative accuracy score corresponding to each reply information, and update the model parameters of the question and answer model based on the target reward value to adjust the probability of the question and answer model outputting each reply information.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to a model training and information generation method, apparatus, storage medium and device. Background Technology

[0002] With the continuous development of Large Language Models (LLM) technology, it has been widely used in question-and-answer interaction scenarios such as intelligent customer service, code generation, risk assessment, and business decision-making. High-quality model-generated content relies on effective optimization training mechanisms, and how to improve the reliability and accuracy of model-generated content has become a key issue for the practical application of models.

[0003] However, existing training methods for question-answering models rely heavily on external validators, test cases, or standard answers for supervision, resulting in high sample construction costs. Furthermore, since content in question-answering scenarios typically lacks unified and clear judgment criteria, fixed labels can lead to the generation results being limited by a single standard, failing to meet the content generation needs of open question-answering scenarios and making it difficult to guarantee the accuracy and reliability of the generated content. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a model training method is proposed, comprising: Obtain multiple response messages generated by the question-answering model to be trained based on prompt words, and determine the token sequence corresponding to each response message; For each token in the token sequence, given its preceding tokens, the question-answering model outputs the conditional distribution probability of that token, and determines the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the token sequence; wherein, the relative accuracy score is positively correlated with the conditional distribution probabilities of each token in the corresponding token sequence. A target reward value is determined based on the relative accuracy score corresponding to each type of response information, and the model parameters of the question-answering model are updated based on the target reward value to adjust the probability of the question-answering model outputting each type of response information.

[0005] According to a second aspect of one or more embodiments of this specification, an information generation method is proposed, comprising: Obtain the user's input question statement and construct prompt words based on the question statement; The prompt words are input into a pre-trained question-and-answer model to generate response information, which is then returned to the user; wherein the question-and-answer model is trained using the method described above.

[0006] According to a third aspect of one or more embodiments of this specification, a model training apparatus is provided, comprising: The acquisition module is used to acquire multiple response information generated by the question-answering model to be trained based on prompt words, and to determine the token sequence corresponding to each response information; The determination module is used to determine, for each token in the token sequence, the conditional distribution probability of the question-answering model outputting that token, given its preceding tokens, and to determine the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the token sequence; wherein, the relative accuracy score is positively correlated with the conditional distribution probabilities of each token in the corresponding token sequence; The update module is used to determine the target reward value based on the relative accuracy score corresponding to each type of response information, and update the model parameters of the question-answering model based on the target reward value, so as to adjust the probability of the question-answering model outputting each type of response information.

[0007] According to a fourth aspect of one or more embodiments of this specification, an information generation apparatus is provided, comprising: The acquisition module is used to acquire the question input by the user and construct prompt words based on the question; An input module is used to input the prompt words into a pre-trained question-answering model, so as to generate response information through the question-answering model and return the response information to the user; wherein, the question-answering model is trained by the above method.

[0008] According to a fifth aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described above by executing the executable instructions.

[0009] According to a sixth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method described above.

[0010] According to a seventh aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0011] As can be seen from the above embodiments, this specification generates multiple response messages under the same prompt word and quantifies the relative accuracy score of different response content at the token generation level based on the probability predicted by at least one token generated before each token, thereby constructing an internal optimization constraint without external supervision. In this way, it can break free from the dependence on external validators, test cases, and standard answers, effectively avoiding the limitations of a single standard brought about by fixed labels, thus significantly improving the generation accuracy and reliability of the model in open-ended question-answering scenarios and meeting its diverse practical application needs. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the architecture of a model training service system provided in an exemplary embodiment; Figure 2 This is a schematic flowchart of a model training method provided in an exemplary embodiment; Figure 3 This is a flowchart illustrating the determination of a relative accuracy score, provided in an exemplary embodiment. Figure 4 This is an exemplary embodiment of a model training architecture diagram. Figure 5 This is a flowchart illustrating an information generation method provided in an exemplary embodiment; Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment; Figure 7 This is a block diagram of a model training apparatus provided in an exemplary embodiment; Figure 8 This is a block diagram of an information generation apparatus provided in an exemplary embodiment. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0014] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0015] While reinforcement learning with verifiable rewards (RVLR) can significantly improve model performance in tasks such as mathematical reasoning and code generation, these mechanisms typically rely on externally designed validators, test cases, or human-referenced answers for supervision. In open-ended tasks or situations where it is difficult to construct accurate verification mechanisms, the cost of obtaining effective rewards is high and the coverage is limited, hindering the large-scale promotion and application of this type of method.

[0016] To reduce reliance on external validators, self-supervised reward systems that operate "without validators" have emerged, constructing reward signals using statistical measures such as model confidence, self-consistent voting, entropy, or semantic entropy. However, confidence and entropy signals are often easily exploited by superficial strategies (such as repeating certain words or phrases, or increasing the length of generated sequences), causing the reward signal to gradually decouple from actual correctness in the later stages of training and resulting in insufficient generalization ability. Furthermore, self-consistent voting methods usually implicitly assume that the problem has a unique or at least decidable correct answer, which is often difficult to define or obtain a universally accepted reference standard for in open-ended tasks, thus making it difficult to generate reliable rewards stably. In relatively verifiable tasks such as code generation, external test cases are still usually required to determine correctness, thus not truly eliminating the external scoring mechanism.

[0017] Based on this, this specification provides a model training method that generates multiple response messages under the same prompt word and calculates the conditional distribution probability of each token in the token sequence corresponding to each response message. This quantifies the relative accuracy score of different response content at the token generation level, thereby eliminating the dependence on external validators, test cases, and standard answers and improving the generation accuracy and reliability of the model in open question answering scenarios.

[0018] The technical solutions described in the various embodiments of this application will be explained in detail below with reference to the accompanying drawings.

[0019] Figure 1 This is a schematic diagram of the architecture of a model training service system provided in an exemplary embodiment. For example... Figure 1As shown, the system may include a server 11, a network 12, and several electronic devices, such as a personal computer (PC) 13, a mobile phone 14, etc.

[0020] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted in a host cluster. During operation, server 11 can run server-side programs for a certain application to implement the relevant functions of that application. For example, when server 11 runs a model training service program, it can function as a corresponding model training service platform.

[0021] PC13 and mobile phone14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program of an application to achieve the relevant functions of that application. For example, when the electronic device runs a model training service program, it can act as a client for that model training service. The aforementioned model training service client application can be launched and run on the electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be achieved through a page displayed by a browser. This browser can be a standalone browser application or a browser module embedded in some applications.

[0022] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0023] Figure 2 This is a flowchart illustrating a model training method provided in an exemplary embodiment, including the following steps: S200: Obtain multiple response information generated by the question-answering model to be trained based on prompt words, and determine the token sequence corresponding to each response information; In this specification, the execution entity used to perform the model training method and the information generation method can be a designated device such as a server. Of course, it can also be a device such as a mobile phone or a computer, as well as a client installed on these devices. For ease of description, the following will use a server as the execution entity to explain the model training method and the information generation method provided in this specification.

[0024] During the training of the question-answering model, the server can first obtain the prompt words required for training the question-answering model.

[0025] The question-answering model can be a Large Language Model (LLM), and the prompt words can be prompt words collected in advance when users perform actual business, or prompt words that are automatically generated or manually constructed based on business scenarios.

[0026] In practical applications, the aforementioned prompts can be constructed based on information from multiple modalities of user input. That is, the information contained in the prompts can include one or more of text data, image data, and audio data. The question-answering model can perform different tasks based on the prompts in different business scenarios. For example: If the information contained in the prompt can include the corresponding code generation requirements (such as the required functions, syntax specification descriptions, etc.), then the question-answering model is used to perform code generation, and its output can include runnable program code, code comments, and syntax explanations. If the information contained in the prompt is risk control data to be evaluated (such as transaction data, identity information, performance records, etc.) and evaluation requirements, then the question-and-answer model is used to perform risk assessment, and the information it generates may include risk level, risk analysis report and response suggestions. If the prompt contains information such as operational data, decision-making conditions, and constraints, then the question-and-answer model is used to execute business decisions, and the information it generates may include decision-making schemes, feasibility analyses, and decision-making basis. If the prompt contains information such as user inquiry questions and business knowledge base information, then the question-answering model is used to perform intelligent question answering, and the information it generates may include question answers, relevant reference materials and guiding words; If the prompt contains information such as task instructions (e.g., data processing, information extraction, data recognition), target requirements, and scenario parameters, then the question-answering model is used to perform the corresponding task processing, and the information it generates may include processing results, execution logs, and status feedback.

[0027] The server can input prompt words into the question-answering model to be trained, and obtain multiple response messages generated by the model under the same prompt word.

[0028] The server can use prompt words to enable its question-answering model to generate multiple outputs. For example, the server can add task instructions to the basic prompt words to instruct the model to output multiple response results, such as "Please provide multiple answers with different approaches" or "Generate multiple possible results." Alternatively, different sampling parameters can be set to sample the model separately, thereby obtaining multiple response information under the same prompt word.

[0029] For each type of response information, it may include the final response result that the model needs to return to the user, as well as the reasoning logic information of that response result (such as Chain-of-Thought (CoT)).

[0030] In addition, the server can determine the token sequence corresponding to each type of response information.

[0031] In practical applications, the question-answering model generates and outputs individual tokens sequentially to gradually assemble complete response information. Each token in the token sequence is arranged in the order of generation by the model, and there are dependencies between them. After arranging and combining multiple tokens, the complete response information can be obtained.

[0032] Therefore, for the input prompt words This allows us to obtain a question-answering model. The output sequence of G tokens ( , ... ).

[0033] S202: For each token in the token sequence, given its preceding tokens, the question-answering model outputs the conditional distribution probability of that token, and determines the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the token sequence; wherein, the relative accuracy score is positively correlated with the conditional distribution probabilities of each token in the corresponding token sequence.

[0034] For token sequence A token generated at any time in the process ( The server can determine the conditional probability distribution of the token output by the question-answering model, given its preceding token.

[0035] The preceding token here may include Central At least one previous token ( ... In addition, prompt words can also be included. The corresponding token.

[0036] Given a preceding token, the question-answering model can predict multiple alternative tokens based on the preceding token. Each token corresponds to a conditional probability distribution, from which the server can then determine the correct answer. The corresponding conditional probability distribution, i.e. .

[0037] Since the above conditional probability distribution can reflect the credibility of the model in predicting and generating the current token directly, rather than randomly selecting or being externally forced to specify it, it can further reflect the accuracy and reliability of the response information, and can be used to quantitatively evaluate the quality of the generated response information.

[0038] The server can determine the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the sequence. This accuracy score is positively correlated with the conditional distribution probabilities of each token; that is, the higher the conditional distribution probability, the higher the confidence of the model in predicting that token, and therefore the higher the accuracy score of the response information. Conversely, the lower the conditional distribution probability, the lower the accuracy score of the response information.

[0039] It should be noted that the positive correlation here can refer to the conditional probability distribution corresponding to each token being positively correlated with the accuracy score of the response information, or it can refer to the conditional probability distribution corresponding to all tokens being positively correlated with the accuracy score of the response information as a whole (such as the mean or sum).

[0040] Furthermore, this specification provides a method for determining the relative accuracy score based on the conditional distribution probability, such as... Figure 3 As shown.

[0041] Figure 3 This is a flowchart of an exemplary embodiment for determining a relative accuracy score.

[0042] The server can determine the conditional probability distribution corresponding to each token based on the teacher-forced mechanism (i.e., forcing the real token from the previous time step to be used as the input for the current time step). For any token, the server can determine the negative log-likelihood loss value corresponding to the token based on the conditional probability distribution corresponding to the token, and the conditional probability distribution corresponding to the token is negatively correlated with the negative log-likelihood loss value.

[0043] For token sequence The server can determine the negative log-likelihood loss value corresponding to each token using the above method, and then take the mean of each loss value to obtain the sequence. The corresponding negative log-likelihood loss value can be expressed as: in, Corresponding sequence The total length.

[0044] After determining the negative log-likelihood loss value for each sequence, the server can further determine the relative accuracy score based on the negative log-natural loss.

[0045] Specifically, the server can calculate The mean of the negative log-likelihood loss values ​​corresponding to each token is used to determine... The corresponding negative log-likelihood loss value is then used to determine the gradient information of the question-answering model. This gradient information can be expressed as: in, These are the initial model parameters for the question-answering model.

[0046] Once the gradient information is determined, it is not directly used to update the model parameters. Instead, it is calculated using this gradient information. The norm (such as L2 norm) This data is then quantified to determine the relative accuracy score.

[0047] Optionally, in practical applications, for any parameter The following losses exist: In context change, The sign and magnitude of the average change depending on the token's position. When averaging over more tokens (i.e., when T is larger), the positive and negative terms tend to cancel each other out during summation, resulting in a smaller average magnitude. Therefore, as T increases, The amplitude is usually calculated according to Scaling. Intuitively, if per-token contributions are represented by random ±1 signs, this leads to... The value decreases as T increases, making gradient norm-based rewards susceptible to length hacking. Therefore, the server can be based on... The length T of the pair The correction was performed, and the result of the correction was... It can be represented as: The server can use this token sequence. The sequence length, with respect to norm Perform weighted processing and then correct the results. As The basic accuracy score. From this formula, we can see that the sequence length T and... They are negatively correlated, thus avoiding the length speculation problem where increased sequence length leads to artificially inflated reward values ​​(i.e., the model generates redundant, meaningless tokens to extend the sequence length and obtain higher reward values). Meanwhile, the norm... and The negative correlation results in sequences with higher corresponding conditional distribution probabilities (lower negative log-likelihood loss) receiving higher base accuracy scores.

[0048] Since the basic accuracy score of a response is calculated based on the tokens within its corresponding token sequence, in order to make it comparable between different sequences within the same group (i.e., a set of multiple response messages generated by the same prompt word), the server can normalize the basic accuracy score of each response message, thereby obtaining a relative accuracy score within the same evaluation system to characterize the quality of different response messages.

[0049] Specifically, the server can sort the responses in descending order of their corresponding baseline accuracy scores. The higher the ranking (the lower the position), the better the response. The larger the value, the higher the quality of the response information, and the more stable the training results. Therefore, the server can determine the relative accuracy score corresponding to each response information based on the ranking results, and the ranking of the response information is negatively correlated with the relative accuracy score.

[0050] In practical applications, for any response, the server can map the ranking position corresponding to that response to a specified data range to obtain the normalized accuracy score for that response. The normalized accuracy score can be expressed as: This formula maps sorting positions to the data range of [-1, 1].

[0051] Of course, in practical applications, it can also be mapped to other data ranges such as [-100, 100], and this manual does not make specific limitations on this.

[0052] The server can then further determine the mean and variance of the normalized accuracy score corresponding to each response, calculate the difference between the normalized accuracy score and the mean, and determine the ratio between the difference and the variance as the final relative accuracy score. This relative accuracy score can be expressed as: In this way, we can obtain information based on the prompt words. In a set of generated response information, each response information has a relative accuracy score, which is used to determine the quality of each response information in the same set of response information.

[0053] It should be noted that the above is a preferred example illustrating the method for determining the relative accuracy score. In practical applications, the specific calculation strategy for the relative accuracy score can be determined based on the actual needs of model training or business operations. For example: according to The corresponding conditional probability or negative log-likelihood loss value directly determines the relative accuracy score. (Conditional distribution probability and) The negative log-likelihood loss value is positively correlated with... (negative correlation) Will Corresponding basic accuracy score Directly determined as the relative accuracy score ( and (positively correlated) Will Corresponding normalized accuracy score Directly determined as the relative accuracy score ( and (Positive correlation)

[0054] S204: Determine the target reward value based on the relative accuracy score corresponding to each type of response information, and update the model parameters of the question-answering model based on the target reward value to adjust the probability of the question-answering model outputting each type of response information.

[0055] The server can determine the target reward value for the question-answering model based on the relative accuracy score corresponding to each type of response. It can then update the model parameters based on this target reward value, thereby adjusting the probability of the question-answering model outputting each type of response, strengthening responses with higher reward values, and suppressing responses with lower reward values. The probability of the updated question-answering model outputting each type of response is positively correlated with the relative accuracy score corresponding to each response.

[0056] The server can also summarize the relative accuracy scores for each type of response (e.g., calculate the average) and determine a target reward value, and then update the model parameters of the question-answering model based on the target reward value.

[0057] Of course, the server can determine multiple reward values ​​based on the relative accuracy score corresponding to each type of response (the target reward value at this time includes the branch reward value corresponding to each type of response), and use these reward values ​​to update the model parameters of the question-answering model.

[0058] Furthermore, to avoid training instability or distribution shift caused by excessively large model parameter updates, a parameter distribution constraint term can be set during the determination of the target reward value. The target reward value is determined based on the relative accuracy score corresponding to each response and the preset parameter distribution constraint term. The parameter distribution constraint term is used to characterize the distribution deviation of the model parameters before and after the update, and this deviation is positively correlated with the magnitude of the parameter distribution constraint term. The magnitude of the parameter distribution constraint term is negatively correlated with the target reward value.

[0059] The server can then update the model parameters of the question-answering model with the goal of maximizing the target reward value.

[0060] in, This represents the target reward value of the question-answering model. This represents the updated model parameters. This represents the model parameters before the update; This is the divergence regularization term (i.e., the parameter distribution constraint term), used to constrain the deviation of model parameters; , where is the probability ratio.

[0061] Once the preset number of training iterations has been reached, or the question-answering model has converged to a preset range, the question-answering model can be deployed to perform actual question-answering tasks (such as code generation, risk assessment, business decision-making, intelligent question answering, task processing, etc.) based on the model.

[0062] To facilitate understanding, this manual also provides an overall architecture diagram for model training, such as... Figure 4 As shown.

[0063] Among them, any prompt word After inputting the trained question-answering model, assume the model outputs two types of response information. and ,in, The logic is flawed; its corresponding conditional distribution probability is small, resulting in a large negative logarithmic loss and thus a decrease in the gradient norm. The higher the score, the lower the relative accuracy score and bonus value will be.

[0064] The logic holds true, and the corresponding conditional distribution probability is relatively large, resulting in a small negative logarithmic loss and thus a higher gradient norm. The smaller the score, the higher the relative accuracy score and bonus value.

[0065] Furthermore, this specification also provides an information generation method, such as... Figure 5 As shown.

[0066] Figure 5 This is a schematic flowchart of an information generation method provided in an exemplary embodiment, including the following steps: S500: Obtain the question input by the user and construct prompt words based on the question input; S502: Input the prompt words into a pre-trained question-answering model to generate response information through the question-answering model, and return the response information to the user; wherein, the question-answering model is trained by the above method.

[0067] In the process of performing actual question-and-answer business through the question-and-answer model, the server can obtain the question input by the user and construct prompt words based on the question and the user input data. The prompt words can contain one or more of the following: text information, image information, and audio information.

[0068] The server can then input the prompt into the question-and-answer model to generate response information (such as code data generated in the code generation task, risk assessment results generated in the risk assessment task, decision-making solutions generated in the business decision-making task, and question-and-answer results generated in the intelligent question-and-answer business, etc.) and return the response information to the user.

[0069] As can be seen from the above methods, this specification effectively overcomes the shortcomings of intrinsic rewards based on confidence or entropy, which are easily stolen by surface strategies, leading to training instability and reward degradation. When confidence, token entropy, and other output distribution statistics are used as rewards, they are easily manipulated by the model through repetition, templated representation, or lengthening of the output to "boost scores," resulting in reward length inflation and training instability. This solution uses teacher-forced conditional distribution probabilities as the intrinsic learning signal and introduces length correction and intra-group ranking normalization, ensuring that rewards do not depend on simple output distribution statistics and suppressing length bias, thus reducing the risk of exploitation by surface strategies. This approach effectively overcomes the shortcomings of majority voting-based self-supervised rewards in open-ended tasks, which struggle to generate stable pseudo-labels and are prone to self-consistent amplification bias. Majority voting typically requires aggregating multiple samples under the same cue word into a single "pseudo-label." However, in open-ended or multi-solution tasks, candidate answers are scattered and may be valid simultaneously, making it difficult to obtain stable and consistent pseudo-labels, thus rendering the training signal unreliable. Furthermore, pseudo-labels originate from the model's own predictions, easily leading to confirmation bias and self-reinforcing early errors, thereby impairing generalization ability and later accuracy. This scheme does not rely on pseudo-label aggregation but instead calculates the intrinsic signal for each output sequence and performs relative ranking within the group to form a reward, thus fundamentally reducing the risk of self-consistent voting leading to pseudo-label self-reinforcement.

[0070] This solution effectively overcomes the drawbacks of semantic entropy rewards relying on external semantic detection modules, which leads to longer processing paths and increased maintenance costs. Semantic entropy typically requires semantic equivalence clustering before calculating entropy on semantic clusters, thus usually necessitating the introduction of independent semantic detection modules, resulting in increased components, longer processing paths, and higher maintenance costs. The reward calculation process in this solution only involves the model itself, requiring no external components, thereby reducing system maintenance costs.

[0071] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 6 As shown, device 600 mainly consists of a communication interface 602, a user interface 604, a processor 606, and a data storage 608. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 610. The communication interface 602 enables device 600 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 602 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 602 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 602 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 602 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0072] User interface 604 includes receiving user input and providing output to the user. Therefore, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 604 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 604 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 600 may support remote access from other devices via communication interface 602 or another physical interface (not shown). User interface 604 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 604 may also be configured as a display device for rendering or displaying text fragments.

[0073] Processor 606 may contain one or more general-purpose processors and / or special-purpose processors.

[0074] Data storage 608 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 606. Data storage 608 may include removable and non-removable components.

[0075] Processor 606 is capable of executing program instructions 618 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 608 to perform the various functions described herein. Data storage 608 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 600, enable device 600 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 618 by processor 606 may result in processor 606 using data 612.

[0076] For example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 600 and one or more applications 620 (e.g., a browser, social application, or game application). Similarly, data 612 may include operating system data 616 and application data 614. Operating system data 616 is primarily accessible to the operating system 622, while application data 614 is primarily accessible to one or more applications 620. Application data 614 may reside in a file system visible or hidden from the user of device 600.

[0077] Application 620 can communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs help application 620 read and / or write application data 614, transmit or receive information via communication interface 602, receive or display information on user interface 604, etc.

[0078] In some terminology, application 620 may be simply referred to as "app". Furthermore, application 620 can be downloaded to device 600 through one or more online app stores or app markets. However, applications can also be installed on device 600 in other ways, such as through a web browser or a physical interface on device 600 (e.g., a USB port).

[0079] Please refer to Figure 7 and Figure 8 Model training devices and information generation devices can be applied to, for example... Figure 4 The device shown is used to implement the technical solution described in this specification.

[0080] Figure 7 This is a block diagram of a model training apparatus provided in an exemplary embodiment, comprising: The acquisition module 700 is used to acquire multiple response information generated by the question-answering model to be trained based on prompt words, and to determine the token sequence corresponding to each response information; The determining module 702 is used to determine, for each token in the token sequence, the conditional distribution probability of the question-answering model outputting that token, given its preceding tokens, and to determine the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the token sequence; wherein, the relative accuracy score is positively correlated with the conditional distribution probabilities of each token in the corresponding token sequence; The update module 704 is used to determine the target reward value based on the relative accuracy score corresponding to each type of response information, and update the model parameters of the question-and-answer model based on the target reward value, so as to adjust the probability of the question-and-answer model outputting each type of response information.

[0081] Optionally, the determining module 702 is specifically used to: determine the negative log-likelihood loss value corresponding to the token based on the conditional distribution probability corresponding to the token, wherein the conditional distribution probability corresponding to the token is negatively correlated with the negative log-likelihood loss value; and determine the relative accuracy score based on the negative log-likelihood loss value corresponding to each token in the token sequence.

[0082] Optionally, the determining module 702 is specifically used to: determine the gradient information of the question-answering model based on the negative log-likelihood loss value corresponding to each token in the token sequence, and calculate the norm of the gradient information; determine the basic accuracy score of the response information corresponding to the token sequence based on the norm; sort the response information in descending order of the basic accuracy score corresponding to each type of response information, and determine the relative accuracy score corresponding to each response information based on the sorting result; wherein, the norm and the basic accuracy score are negatively correlated, and the sorting position of the response information is negatively correlated with the relative accuracy score.

[0083] Optionally, the determining module 702 is specifically used to: for each response information, map the ranking position corresponding to the response information to a specified data interval to obtain the normalized accuracy score corresponding to the response information; determine the mean and variance of the normalized accuracy scores corresponding to each response information; calculate the difference between the normalized accuracy score and the mean, and determine the ratio between the difference and the variance as the relative accuracy score.

[0084] Optionally, the determining module 702 is specifically used to perform weighted processing on the norm according to the sequence length of the token sequence, and determine the basic accuracy score according to the weighted processing result; wherein the sequence length is negatively correlated with the basic accuracy score.

[0085] Optionally, the update module 704 is specifically used to determine the target reward value based on the relative accuracy score corresponding to each type of response information and a preset parameter distribution constraint term; wherein, the parameter distribution constraint term is used to characterize the distribution deviation of the model parameters before and after the update; the deviation is positively correlated with the magnitude of the parameter distribution constraint term; the magnitude of the parameter distribution constraint term is negatively correlated with the target reward value; updating the model parameters of the question-answering model based on the target reward value specifically includes: updating the model parameters of the question-answering model with the goal of maximizing the target reward value.

[0086] Optionally, the question-answering model includes a large language model (LLM), and the probability of each type of response output by the updated question-answering model is positively correlated with the relative accuracy score corresponding to each response.

[0087] Figure 8 This is a block diagram of an information generation apparatus provided in an exemplary embodiment, comprising: The acquisition module 800 is used to acquire the question input by the user and construct prompt words based on the question; The input module 802 is used to input the prompt words into a pre-trained question-answering model, so as to generate response information through the question-answering model and return the response information to the user; wherein, the question-answering model is trained by the above method.

[0088] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0089] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0090] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0091] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0092] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0093] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0094] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0095] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0096] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0097] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0098] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A model training method, comprising: Obtain multiple response messages generated by the question-answering model to be trained based on prompt words, and determine the token sequence corresponding to each response message; For each token in the token sequence, given its preceding tokens, the question-answering model outputs the conditional distribution probability of that token, and determines the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the token sequence; wherein, the relative accuracy score is positively correlated with the conditional distribution probabilities of each token in the corresponding token sequence. A target reward value is determined based on the relative accuracy score corresponding to each type of response information, and the model parameters of the question-answering model are updated based on the target reward value to adjust the probability of the question-answering model outputting each type of response information.

2. The method as described in claim 1, determining the relative accuracy score of the response information corresponding to the token sequence, specifically includes: Based on the conditional distribution probability corresponding to the token, determine the negative log-likelihood loss value corresponding to the token, wherein the conditional distribution probability corresponding to the token is negatively correlated with the negative log-likelihood loss value; The relative accuracy score is determined based on the negative log-likelihood loss value corresponding to each token in the token sequence.

3. The method as described in claim 2, wherein the relative accuracy score is determined based on the negative log-likelihood loss value corresponding to each token in the token sequence, specifically includes: Based on the negative log-likelihood loss value corresponding to each token in the token sequence, the gradient information of the question answering model is determined, and the norm of the gradient information is calculated. Based on the norm, determine the basic accuracy score of the response information corresponding to the token sequence; The responses are sorted in descending order of their base accuracy scores, and the relative accuracy scores for each response are determined based on the sorting results. The norm and the basic accuracy score are negatively correlated, and the ranking of the response information is negatively correlated with the relative accuracy score.

4. The method as described in claim 3, wherein determining the relative accuracy score corresponding to each response based on the ranking result specifically includes: For each response, the ranking position corresponding to the response is mapped to a specified data range to obtain the normalized accuracy score corresponding to the response. Determine the mean and variance of the normalized accuracy scores for each response; Calculate the difference between the normalized accuracy score and the mean, and determine the ratio between the difference and the variance as the relative accuracy score.

5. The method as described in claim 3, wherein determining the basic accuracy score of the response information corresponding to the token sequence based on the norm specifically includes: The norm is weighted according to the sequence length of the token sequence, and the basic accuracy score is determined based on the weighting result. The sequence length is negatively correlated with the baseline accuracy score.

6. The method as described in claim 1, wherein the target reward value is determined based on the relative accuracy score corresponding to each type of response information, specifically includes: The target reward value is determined based on the relative accuracy score corresponding to each response and the preset parameter distribution constraint term; wherein, the parameter distribution constraint term is used to characterize the distribution deviation of the model parameters before and after the update; the deviation is positively correlated with the magnitude of the parameter distribution constraint term; the magnitude of the parameter distribution constraint term is negatively correlated with the target reward value; The model parameters of the question-answering model are updated based on the target reward value, specifically including: The model parameters of the question-answering model are updated with the goal of maximizing the target reward value.

7. The method as described in claim 1, wherein the question-answering model includes a Large Language Model (LLM), and the probability of the updated question-answering model outputting each type of response information is positively correlated with the relative accuracy score corresponding to each response information.

8. An information generation method, comprising: Obtain the user's input question statement and construct prompt words based on the question statement; The prompt words are input into a pre-trained question-and-answer model to generate response information, and the response information is returned to the user; wherein the question-and-answer model is trained by the method described in any one of claims 1 to 7.

9. A model training device, comprising: The acquisition module is used to acquire multiple response information generated by the question-answering model to be trained based on prompt words, and to determine the token sequence corresponding to each response information; The determination module is used to determine, for each token in the token sequence, the conditional distribution probability of the question-answering model outputting that token, given its preceding tokens, and to determine the relative accuracy score of the response information corresponding to the token sequence based on the conditional distribution probabilities of each token in the token sequence; wherein, the relative accuracy score is positively correlated with the conditional distribution probabilities of each token in the corresponding token sequence; The update module is used to determine the target reward value based on the relative accuracy score corresponding to each type of response information, and update the model parameters of the question-answering model based on the target reward value, so as to adjust the probability of the question-answering model outputting each type of response information.

10. An information generation device, comprising: The acquisition module is used to acquire the question input by the user and construct prompt words based on the question; An input module is used to input the prompt words into a pre-trained question-answering model, so as to generate response information through the question-answering model and return the response information to the user; wherein the question-answering model is trained by the method described in any one of claims 1 to 7.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-8 by executing the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

Images