Reinforcement learning training method and system based on military document and answer similarity
By adopting similarity-based reinforcement learning training methods in the military field, automatically extracting questions and evaluating the loyalty of model answers, the problems of high cost and low efficiency in traditional training methods are solved, and high-quality model training is achieved, satisfying the military Domain requirements for model reliability.
Patent Information
- Application Number
- CN202411106016.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
AI Technical Summary
In the military field, traditional model training methods require a lot of manual annotation data and high training costs, and in the absence of standard answers, training a reliable and reference-compliant model is a huge challenge.
Using reinforcement learning training methods based on military documents and answer similarity, through automatic extraction of questions and generation of model answers, calculate the similarity between answers and reference materials, evaluate the loyalty of model answers and rewards, update strategy and value network parameters, to Reduce training costs and improve model efficiency and quality.
It effectively reduces training costs, improves model training efficiency and quality, meets the military field's demand for model reliability, reduces manual annotation work, and ensures loyalty in model answers.
Smart Images

Figure CN119005290A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a reinforcement learning training method and system based on military documents and answer similarity. Background Art
[0002] In the field of modern military research and decision-making, large models are increasingly used because they can process huge data sets and conduct deeper analysis and answer generation. However, the training cost behind these models is very high. In order to obtain accurate and reliable models, a large amount of training data and a precise annotation process are required. In the military field, the need for model reliability is particularly evident, because wrong or misleading answers may lead to serious consequences. In addition, even with a large amount of corpus data, it is still a huge challenge to train a reliable model that is highly consistent with the reference material in the absence of standard answers.
[0003] Traditional model training methods face difficulties due to the particularity of the military field, so it is necessary to design a training reward mechanism to reduce the demand for manual labeling and improve the efficiency of training. Summary of the invention
[0004] In view of this, the purpose of this application is to propose a reinforcement learning training method and system based on military documents and answer similarities, aiming to effectively reduce training costs and improve model training efficiency and quality.
[0005] In order to achieve one of the above disclosed purposes, the present application provides a reinforcement learning training method based on military documents and answer similarity, the method comprising:
[0006] Obtain reference materials, automatically extract questions from the reference materials using a reinforcement learning model, and form sample pairs of the questions and the reference materials as training data;
[0007] splicing the question and the reference material as inputs of the reinforcement learning model to generate a model answer;
[0008] Calculating the similarity between the model answer and the reference material;
[0009] Based on the similarity, evaluating the fidelity of the model answer to the reference material, and obtaining a loyalty reward;
[0010] Calculating the policy loss of the policy network and the value loss of the value network in the reinforcement learning model;
[0011] The loss function is used to iterate the gradients of the network parameters of the policy network and the value network to update the network parameters.
[0012] As a further improvement of an implementation mode of the present application, the calculating the similarity between the model answer and the reference material includes:
[0013] Processing the reference material and the model answer separately as sentences;
[0014] Calculate the similarity between the sentence answered by the model and the reference material using a text embedding model to generate a two-dimensional matrix, wherein each row of the matrix represents the similarity between the model answer and different sentences of the reference material, and each column of the matrix represents the similarity between different sentences answered by the model;
[0015] Use cosine distance to evaluate the similarity between embedding vectors;
[0016] The calculation formula of the similarity is:
[0017] Similarity=Normalize(E a )·Normalize(E r );
[0018] Among them, E a The embedding vector representing the sentence answered by the model; E r The embedding vector representing the sentence of the reference material; Normalize means normalizing the vector.
[0019] As a further improvement of an implementation mode of the present application, the step of evaluating the loyalty of the model answer to the reference material based on the similarity to obtain a loyalty reward includes:
[0020] For each sentence answered by the model, recall the top N similarity values among the similarities of all sentences in the reference material, where N is a hyperparameter;
[0021] If the actual number of similarities is less than N, recall is performed based on the actual number;
[0022] Using the normalization factor as a weight, a loyalty reward is calculated for each sentence answered by the model.
[0023] As a further improvement of an embodiment of the present application, for the first N similarity values x i , the calculation formula of the smoothed maximum value is:
[0024]
[0025] Among them, α represents the adjustment factor, and the value of i ranges from 1 to N. When α is a positive value with a large absolute value, the smoothed maximum value is close to the maximum value function; when α is a negative value with a large absolute value, the smoothed maximum value is close to the minimum value function.
[0026] The calculation formula of the normalization factor is:
[0027] f=m γ ;
[0028] Where m represents the number of sentences answered by the model, and γ is a value between [-1, 1];
[0029] The calculation formula for the loyalty reward is:
[0030] reward = f × maximum.
[0031] As a further improvement of an implementation method of the present application, the calculation formula of the strategy loss is:
[0032]
[0033] Among them, A t represents the advantage value, π represents the current policy network, π old represents the policy network that updates the previous step, a t represents the action at time t, s t Represents the state at time t;
[0034] The calculation formula for the value loss is:
[0035] L V =MSE(Return, Reward);
[0036] Among them, Return represents the model answer score predicted by the value network, Reward represents the loyalty reward, and MSE is the mean square error.
[0037] As a further improvement of an implementation mode of the present application, the obtaining of reference materials includes:
[0038] Acquire corpus data, perform text cleaning and document segmentation on the corpus data, and generate the reference material.
[0039] As a further improvement of an implementation mode of the present application, the step of splicing the question and the reference material as inputs of the reinforcement learning model to generate a model answer includes:
[0040] Importance sampling is used to sample the output of the same input multiple times to generate multiple model answers.
[0041] Based on the same inventive concept, the present application also provides a reinforcement learning training system based on military documents and answer similarity, including:
[0042] An extraction module, used to obtain reference materials, automatically extract questions from the reference materials using a reinforcement learning model, and form sample pairs of the questions and the reference materials as training data;
[0043] A generation module, used to combine the question and the reference material as inputs of the reinforcement learning model to generate a model answer;
[0044] A first calculation module, used for calculating the similarity between the model answer and the reference material;
[0045] An evaluation module, used for evaluating the loyalty of the model answer to the reference material based on the similarity, and obtaining a loyalty reward;
[0046] A second calculation module is used to calculate the policy loss of the policy network and the value loss of the value network in the reinforcement learning model;
[0047] The optimization module is used to iterate the gradients of the network parameters of the policy network and the value network using a loss function to update the network parameters.
[0048] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned reinforcement learning training methods based on military documents and answer similarities.
[0049] Based on the same inventive concept, the present application also provides an electronic device, comprising: a processor and a memory; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any of the steps of the reinforcement learning training method based on military documents and answer similarity as described above.
[0050] Compared with the prior art, the technical effect of the present invention is that: the present application can eliminate manual labeling and effectively reduce training costs by automatically extracting questions and giving rewards through the use of a reinforcement learning model. In addition, the present application is based on similarity rewards, so that the model answers can better follow the reference materials, thereby improving the efficiency and quality of model training, and can meet the requirements of low tolerance for model hallucinations in specific fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the implementation methods or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0052] Figure 1 A flowchart of a reinforcement learning training method based on military documents and answer similarity provided in one embodiment of the present application;
[0053] Figure 2 A flowchart of evaluating similarity provided for another embodiment of the present application;
[0054] Figure 3 A flowchart of a normalization process provided for another embodiment of the present application;
[0055] Figure 4 A schematic diagram of a reinforcement learning training system based on military documents and answer similarity provided for an implementation method of the present application;
[0056] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0057] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.
[0058] Implementation methods It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the implementation methods of this application should be understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the implementation methods of this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0059] Related technologies propose a scalable method called "instruction back-translation" for fine-tuning large language models to follow instructions. This method uses a large amount of unlabeled data to increase and edit high-quality training examples through an iterative self-training algorithm to improve model performance. The method includes two core steps: self-enhancement and self-screening. Self-enhancement refers to augmenting unlabeled data by generating instructions to generate candidate training data. Self-screening refers to self-screening high-quality examples by predicting the quality of instructions to train a model that can follow instructions. This method improves the performance of the model through continuous iterative training, and each iteration uses a better intermediate model to select data for training the next iteration.
[0060] However, this training method still requires some manually labeled seed data for training in order to solve the problem of model hallucination, which increases the operating cost.
[0061] The present application embodiment provides a reinforcement learning training method based on military documents and answer similarity, such as Figure 1 As shown, the reinforcement learning training method based on military documents and answer similarity includes the following steps:
[0062] Step S100, obtain reference materials, use the reinforcement learning model to automatically extract questions from the reference materials, and form sample pairs of questions and reference materials as training data.
[0063] In one possible implementation of the present application, step S100 further includes:
[0064] Step S110, obtaining corpus data, performing text cleaning and document segmentation on the corpus data, and generating reference materials.
[0065] Specifically, in this application, a large model is used as a policy model in reinforcement learning. In order for the model to understand the text, some preprocessing work needs to be done on the corpus data, such as text cleaning, document segmentation, etc. The organized corpus is a segment of text that can be used by the model, and each segment of text is a reference material. Then, the large model is used to extract questions from the reference material, and the model will look for possible questions from the material based on the context. For example, in an article about strategic deployment, the model may extract questions such as "What is an effective strategic deployment" or "How to implement strategic deployment". Finally, organize the questions and reference materials into sample pairs to construct training data.
[0066] Step S100 automatically extracts questions from the corpus to construct training data for the model, thereby avoiding a large amount of manual labeling work and reducing operating costs.
[0067] Step S200, concatenating the question and reference material as input to the reinforcement learning model to generate a model answer.
[0068] In one possible implementation of the present application, step S200 further includes:
[0069] Step S210, using importance sampling to sample the same input multiple times and generate multiple model answers.
[0070] Specifically, in the forward propagation process, the question and reference materials are concatenated as input data, and the model makes an answer based on the input, that is, generates a model answer. In this process, in order to increase the sampled data, it is necessary to do importance sampling on the same input, sample the same input multiple times at higher temperature values and top_p values, output different model answers, and increase the number and diversity of samples. Among them, the same input refers to multiple identical inputs, and here, the input refers to the text content of the input.
[0071] Among them, importance sampling is a method of extracting samples from a probability distribution so that the sampling results can reflect the importance of each event in the original distribution. The top_p (top probability) value and temperature value are commonly used parameters when generating text. The top_p value controls the cutting of the words with the highest probability when generating text until the cumulative probability exceeds the set top_p value. A smaller top_p value will result in more conservative and predictable generated text because only the words with the highest probability are selected, while a larger top_p value may lead to more diversity and randomness. The temperature value adjusts the selectivity of the model for the distribution of words when generating text. A higher temperature value will increase the diversity of the generated text because it makes the model more inclined to select words with lower probabilities, while a lower temperature value will result in more deterministic text. In other words, when the value is higher, the generated text may be more creative and varied, but it may also lead to semantic inaccuracy or incoherence; a lower temperature helps to generate more accurate and predictable text, but it may appear monotonous or lack novelty.
[0072] Step S300, calculating the similarity between the model answer and the reference material.
[0073] In one possible implementation of the present application, Figure 2 As shown, for step S300, it also includes:
[0074] Step S310, processing the reference material and the model answer separately into sentences;
[0075] Step S320, using the text embedding model to calculate the similarity between the sentence of the model answer and the reference material, and generate a two-dimensional matrix, wherein each row of the matrix represents the similarity between the model answer and different sentences of the reference material, and each column of the matrix represents the similarity between different sentences of the reference material and the model answer;
[0076] Step S330, using cosine distance to evaluate the similarity between embedded vectors;
[0077] The calculation formula for similarity is:
[0078] Similarity=Normalize(E a)·Normalize(E r );
[0079] Among them, E a is the embedding vector of the sentence answered by the model; E r is the embedding vector of the sentence of the reference material; Normalize means normalizing the vector.
[0080] Specifically, first, the reference material and the model answer are divided into sentences. Then, the text embedding model is used to calculate the similarity between each sentence in the model answer and each sentence in the reference material. These similarity values are organized into a two-dimensional matrix. Each row of the matrix represents the similarity between a sentence in the model answer and different sentences in the reference material; each column represents the similarity between a sentence in the reference material and different sentences in the model answer. When calculating the similarity, the cosine distance is used to evaluate the similarity between the embedded vectors. Step S300 can quantify the similarity between the model answer and the reference material at the sentence level, so as to facilitate the subsequent optimization of loyalty evaluation and model training.
[0081] Step S400, based on the similarity, evaluate the loyalty of the model answer to the reference material and obtain a loyalty reward.
[0082] In one possible implementation of the present application, Figure 3 As shown, for step S400, it also includes:
[0083] Step S410, for each sentence answered by the model, recall the top N similarity values among the similarities of all sentences in the reference material, where N is a hyperparameter;
[0084] Step S420, if the actual number of similarities is less than N, recall is performed according to the actual number;
[0085] Step S430, using the normalization factor as a weight, calculate the loyalty reward for each sentence answered by the model.
[0086] In one possible implementation of the present application, step S400 further includes:
[0087] For the first N similarity values x i , the calculation formula of the smoothing maximum value is:
[0088]
[0089] Among them, α is the adjustment factor, and the value of i ranges from 1 to N. When α is a positive value with a large absolute value, the smoothed maximum value is close to the maximum value function; when α is a negative value with a large absolute value, the smoothed maximum value is close to the minimum value function.
[0090] Specifically, in order to evaluate the fidelity of the answers generated by the model to the reference material, it is necessary to make a judgment based on the similarity results. The specific method is to find the top N similarity values among the similarities between each sentence in the model's answer and all the sentences in the reference material. Among them, N is a pre-set hyperparameter. If the actual number of similarities is less than N, recall according to the actual number. For example, if N is set to 5, but there are only 3 similarity values between a sentence in the model's answer and all the sentences in the reference material, then recall these 3 similarity values. In this way, the most important similarity information related to each sentence can be screened out, so as to more accurately evaluate fidelity.
[0091] The calculation formula of the normalization factor f is:
[0092] f=m γ ;
[0093] Where m represents the number of sentences answered by the model, and γ is a value between [-1, 1];
[0094] The calculation formula for loyalty reward is:
[0095] reward = f × maximum.
[0096] Specifically, when using the smoothed maximum as the loyalty of each sentence, the impact of the number of sentences on the reward value must also be considered. Different answers may contain different numbers of sentences, so the reward value of each model answer sentence needs to be normalized to ensure that the reward value can reflect the impact of the number of sentences. The final loyalty reward is the product of the smoothed maximum and the normalization factor, forming a method to quantify the consistency between the model-generated text and the reference material. By introducing adjustment factors and normalization factors, the evaluation results are made more detailed and fair, and the loyalty of the model answer to the reference material can be more accurately evaluated.
[0097] Step S500, calculating the policy loss of the policy network and the value loss of the value network in the reinforcement learning model.
[0098] Specifically, in the AC (Actor-Critic) framework of reinforcement learning, the policy network (Actor) and the value network (Critic) need to be updated simultaneously. The policy network is responsible for learning and executing actions, and its update goal is to maximize rewards, that is, to select actions that can bring higher expected rewards. The value network is responsible for evaluating the actions taken by the policy, that is, predicting the long-term value of these actions, and its update goal is to fit the reward, that is, to accurately predict the return of the action.
[0099] In one possible implementation of the present application, step S500 further includes:
[0100] Step S510, the calculation formula of strategy loss is:
[0101]
[0102] Among them, A t is the advantage value, π represents the current policy network, π old represents the policy network that updates the previous step, a t represents the action at time t, s t Represents the state at time t. This loss function adjusts the strategy by comparing the probability of the new and old policy networks for the same action, so that actions with higher advantage values are selected more frequently, rather than simply following the current best strategy.
[0103] Specifically, the strategy loss maximizes the reward by maximizing both the probability of the sampled action and its advantage value, and converts the gradient ascent into gradient descent by negation. Since the strategy loss is the maximum value of the current strategy divided by the previous strategy multiplied by the advantage value, and the previous strategy is a fixed value, the maximum reward can be obtained.
[0104] Step S520, value loss L V The calculation formula is:
[0105] L V =MSE(Return, Reward);
[0106] Among them, Return represents the model answer score predicted by the value network, that is, the estimation of the long-term return of the current action sequence; Reward is the reward, specifically refers to the loyalty reward obtained in step S400; MSE is the mean square error, which is a common indicator for evaluating the difference between the predicted value and the actual value.
[0107] Specifically, in reinforcement learning, MSE (Mean Squared Error) is usually used to measure the accuracy of the value network's prediction. The role of the value network in reinforcement learning is to evaluate the long-term return or expected return under the current state to help the decision-making agent choose the appropriate action.
[0108] Specifically, the predicted value of the value network is usually expressed as the value of a state or state-action pair, which represents the expected or estimated value of the future cumulative reward that the agent can obtain starting from this state. This value can be estimated by the reward during the training process, which is the accumulation of the immediate rewards obtained by the agent after performing actions in the environment.
[0109] In the process of training the value network, the agent interacts with the environment and collects experience data including status, actions performed, rewards obtained, etc. Through these data, the difference between the actual reward value and the value predicted by the value network can be calculated, that is, the mean square error. The smaller the mean square error value, the closer the value predicted by the value network is to the actual reward value, indicating that the prediction ability of the value network is better.
[0110] Through step S500, the reinforcement learning model can learn effective strategies in a complex environment and evaluate the long-term effects of these strategies.
[0111] Step S600, using the loss function to iterate the gradients of the network parameters of the policy network and the value network to update the network parameters.
[0112] In the back propagation process of reinforcement learning training, the gradient of the loss function to the network parameters is used to iteratively update the network parameters. The main goal of this process is to minimize the loss and optimize the model performance. The loss function refers to the calculation formula of the policy loss and value loss, and minimizing the loss means minimizing the policy loss and value loss.
[0113] Specifically, after the model performs forward propagation and obtains the predicted result, the value of the loss function is calculated by comparing it with the actual result. Then, using the backpropagation algorithm, starting from the output layer, the gradient of the loss function for each network parameter is calculated layer by layer. These gradients represent the degree of influence of the change in the parameter on the loss function. The direction of updating the network parameters is to reduce the loss function, that is, to adjust the parameters in the opposite direction of the gradient. By repeating this process continuously, the model can gradually learn the optimal parameter configuration, thereby improving the performance of the model. In the process of reinforcement learning, the loss function is related to the reward signal, and the model obtains experience and feedback by interacting with the environment.
[0114] In this application, first, the model automatically extracts relevant questions from the corpus and constructs training data; in the training stage, the reference materials provided by the user are used to generate answers. Then the similarity between the model answer and the reference material is calculated to evaluate its fidelity. Finally, the model is trained for reinforcement learning using similarity as a reward function, and the model performance is prioritized by maximizing rewards and minimizing losses. By automatically constructing a reinforcement learning training data method, including automatically extracting questions and automatically giving rewards, manual labeling is eliminated; then reinforcement learning training is performed based on similarity rewards, so that the model answers can better follow the reference materials. This method effectively reduces the training cost, improves the training efficiency and quality of the model, and makes the answers follow the reference data to a higher degree, which can meet the low tolerance requirements for model hallucinations in specific fields (such as the military field).
[0115] like Figure 4As shown, another embodiment of the present application discloses a reinforcement learning training system based on military documents and answer similarity, including:
[0116] The extraction module is used to obtain reference materials, automatically extract questions from the reference materials using a reinforcement learning model, and form sample pairs of questions and reference materials as training data;
[0117] The generation module is used to concatenate questions and reference materials as input to the reinforcement learning model and generate model answers;
[0118] The first calculation module is used to calculate the similarity between the model answer and the reference material;
[0119] An evaluation module is used to evaluate the fidelity of the model answer to the reference material based on similarity and obtain a loyalty reward;
[0120] The second calculation module is used to calculate the policy loss of the policy network and the value loss of the value network in the reinforcement learning model;
[0121] The optimization module is used to iterate the gradients of the network parameters of the policy network and the value network using the loss function to update the network parameters.
[0122] This system uses an automatic method to build reinforcement learning training data, including automatic question extraction and automatic reward giving, eliminating the need for manual labeling; then reinforcement learning training is performed based on similarity rewards, so that the model answers can better follow the reference material. This method effectively reduces the training cost, improves the training efficiency and quality of the model, and makes the answers follow the reference material data to a higher degree, which can meet the low tolerance requirements for model hallucinations in the military field.
[0123] In one possible implementation of the present application, the extraction module further includes:
[0124] The preprocessing module is used to obtain corpus data, perform text cleaning and document segmentation on the corpus data, and generate reference materials.
[0125] In one possible implementation of the present application, the first computing module further includes:
[0126] The sentence processing module is used to process the reference materials and model answers separately;
[0127] The matrix module is used to calculate the similarity between the sentence answered by the model and the reference material using the text embedding model to generate a two-dimensional matrix;
[0128] Quantization module for evaluating the similarity between embedding vectors using cosine distance.
[0129] Figure 5A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0130] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the implementation methods of this specification.
[0131] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0132] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0133] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0134] The bus 1050 includes pathways that transmit information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).
[0135] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the implementation scheme of this specification, and does not necessarily include all the components shown in the figure.
[0136] The electronic device of the above-mentioned embodiment is used to implement the corresponding reinforcement learning training method based on military documents and answer similarity in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method implementation method, which will not be repeated here.
[0137] Based on the same inventive concept, corresponding to any of the above-mentioned implementation methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the reinforcement learning training method based on military documents and answer similarity as described in any of the above implementation methods.
[0138] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0139] The computer instructions stored in the storage medium of the above-mentioned embodiment are used to enable the computer to execute the reinforcement learning training method based on military documents and answer similarity as described in any of the above embodiments, and have the beneficial effects of the corresponding method implementation method, which will not be repeated here.
[0140] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. This narrative style of the specification is merely for the sake of clarity, and those skilled in the art should take the specification as a whole. Under the concept of the present application, the above embodiments or the technical features in different embodiments may be appropriately combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0141] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0142] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0143] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. A reinforcement learning training method based on military documents and answer similarity, characterized in that: include: Obtain reference materials, automatically extract questions from the reference materials using a reinforcement learning model, and form sample pairs of the questions and the reference materials as training data; splicing the question and the reference material as inputs of the reinforcement learning model to generate a model answer; Calculating the similarity between the model answer and the reference material; Based on the similarity, evaluating the fidelity of the model answer to the reference material, and obtaining a loyalty reward; Calculating the policy loss of the policy network and the value loss of the value network in the reinforcement learning model; The loss function is used to iterate the gradients of the network parameters of the policy network and the value network to update the network parameters.
2. The reinforcement learning training method based on military documents and answer similarity according to claim 1 is characterized in that: The calculating the similarity between the model answer and the reference material comprises: Processing the reference material and the model answer separately as sentences; Calculate the similarity between the sentence answered by the model and the reference material using a text embedding model to generate a two-dimensional matrix, wherein each row of the matrix represents the similarity between the model answer and different sentences of the reference material, and each column of the matrix represents the similarity between different sentences answered by the model; Use cosine distance to evaluate the similarity between embedding vectors; The calculation formula of the similarity is: Similarity=Normalize(E a )·Normalize(E r ); Among them, Similarity represents similarity; E a The embedding vector representing the sentence answered by the model; E r The embedding vector representing the sentence of the reference material; Normalize means normalizing the vector.
3. The reinforcement learning training method based on military documents and answer similarity according to claim 1 is characterized in that: The step of evaluating the loyalty of the model answer to the reference material based on the similarity and obtaining a loyalty reward comprises: For each sentence answered by the model, recall the top N similarity values among the similarities of all sentences in the reference material, where N is a hyperparameter; If the actual number of similarities is less than N, recall is performed based on the actual number; Using the normalization factor as a weight, a loyalty reward is calculated for each sentence answered by the model.
4. The reinforcement learning training method based on military documents and answer similarity according to claim 3 is characterized in that: For the first N similarity values x i , the calculation formula of the smoothed maximum value is: Among them, maximum represents the smoothing maximum value, α represents the adjustment factor, and the value of i ranges from 1 to N; The calculation formula of the normalization factor is: f=m γ ; Where f represents the normalization factor, m represents the number of sentences answered by the model, and the value of γ is [-1, 1]; The calculation formula for the loyalty reward is: reward = f × maximum; Among them, reward means loyalty reward.
5. The reinforcement learning training method based on military documents and answer similarity according to claim 4 is characterized in that: The calculation formula of the strategy loss is: Among them, L P represents the strategy loss, A t represents the advantage value, π represents the current policy network, π old represents the policy network that updates the previous step, a t represents the action at time t, S t Represents the state at time t; The calculation formula for the value loss is: L V =MSE(Reward,Reward); Among them, L V Represents value loss, Reward represents the model answer score predicted by the value network, Reward represents the loyalty reward, and MSE is the mean square error.
6. The reinforcement learning training method based on military documents and answer similarity according to claim 1 is characterized in that: The obtaining of reference materials includes: Acquire corpus data, perform text cleaning and document segmentation on the corpus data, and generate the reference material.
7. The reinforcement learning training method based on military documents and answer similarity according to claim 1 is characterized in that: The step of combining the question and the reference material as inputs of the reinforcement learning model to generate a model answer includes: Importance sampling is used to sample the output of the same input multiple times to generate multiple model answers.
8. A reinforcement learning training system based on military documents and answer similarity, characterized in that: The system comprises: An extraction module, used to obtain reference materials, automatically extract questions from the reference materials using a reinforcement learning model, and form sample pairs of the questions and the reference materials as training data; A generation module, used to combine the question and the reference material as inputs of the reinforcement learning model to generate a model answer; A first calculation module, used for calculating the similarity between the model answer and the reference material; An evaluation module, used for evaluating the loyalty of the model answer to the reference material based on the similarity, and obtaining a loyalty reward; A second calculation module is used to calculate the policy loss of the policy network and the value loss of the value network in the reinforcement learning model; The optimization module is used to iterate the gradients of the network parameters of the policy network and the value network using a loss function to update the network parameters.
9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the reinforcement learning training method based on military document and answer similarity as described in any one of claims 1 to 7.
10. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the reinforcement learning training method based on military documents and answer similarities as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Efficient network retrieval enhanced answering method and system oriented to human preferences
CN116501843A
Intelligent question answering method based on large language model
CN117033597A
Long document question and answer processing method and system in special field
CN117196002A
Industrial question and answer model training method based on reinforcement learning and knowledge base matching
CN117763127A
Question and answer model training method, text processing method and reward model training method
CN118350463A
Cited By
Keyword guidance and large language model near-end strategy optimization combined key sentence extraction method
CN120509402A
