Method and apparatus for training information acquisition capability of artificial intelligence model

Abstract

This application relates to a training method and apparatus for improving the information acquisition capability of an artificial intelligence model. The method includes: acquiring a training sample set for the i-th training round, wherein the training sample set includes a first training sample pair with a first dynamic ratio and a second training sample pair with a second dynamic ratio, the second training sample pair being obtained by performing semantic reconstruction processing on a third training sample pair, and both the first and third training sample pairs being training sample pairs from the original training sample set; wherein i is a positive integer greater than or equal to 1; training the artificial intelligence model based on the training sample set; determining the reward function corresponding to the training sample set based on the running data of the i-th training round; and iteratively updating the parameters of the artificial intelligence model based on the result of the reward function. This method enables large speech models to be applied to complex search application scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method and apparatus for the information acquisition capability of an artificial intelligence model. Background Technology

[0002] With the development of deep learning technology, large language models can be used to answer user questions and interact with users.

[0003] However, because the training samples used specifically for training information acquisition capabilities are of a single type, large language models in related technologies still suffer from the problem of not being able to adaptively select the most suitable search strategy when dealing with search problems of different complexities. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method and apparatus for improving the information acquisition capabilities of an artificial intelligence model that enables a large speech model to adaptively select the most suitable search strategy, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for training the information acquisition capability of an artificial intelligence model, including:

[0006] Obtain the training sample set for the i-th training round. The training sample set includes a first training sample pair with a first dynamic ratio and a second training sample pair with a second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. Both the first training sample pair and the third training sample pair are training sample pairs in the original training sample set. Where i is a positive integer greater than or equal to 1.

[0007] The artificial intelligence model is trained based on the training sample set. The reward function corresponding to the training sample set is determined according to the running data of the i-th training round, and the parameters of the artificial intelligence model are iteratively updated based on the result of the reward function.

[0008] In one embodiment, the method further includes:

[0009] Based on the result of the reward function, the first dynamic ratio is adjusted to obtain the third dynamic ratio, and the second dynamic ratio is adjusted to obtain the fourth dynamic ratio;

[0010] The fourth training sample pair with the third dynamic ratio is determined from the original training sample set, and the fifth training sample pair with the fourth dynamic ratio is determined from the original training sample set;

[0011] Perform semantic reconstruction processing on the fifth training sample pair to obtain the sixth training sample pair;

[0012] The set consisting of the fifth training sample pair and the sixth sample pair is determined as the training sample set for the (i+1)th training round.

[0013] In one embodiment, the step of training the artificial intelligence model based on the training sample set, and determining the reward function corresponding to the training sample set based on the running data of the i-th training round, includes:

[0014] Based on the operational data, the result correctness score, thought chain behavior score, and information acquisition strategy score corresponding to the training sample set of the i-th training round are determined respectively.

[0015] The reward function is determined based on the correctness score of the results, the thought chain behavior score, and the information acquisition strategy score.

[0016] In one embodiment, adjusting the first dynamic ratio to obtain a third dynamic ratio based on the result of the reward function, and adjusting the second dynamic ratio to obtain a fourth dynamic ratio, includes:

[0017] Obtain the first reward function result corresponding to the first training sample pair and the second reward function result corresponding to the second training sample pair in the i-th training round;

[0018] Based on the mathematical statistics of the first reward function result, the first dynamic ratio is adjusted to obtain the third dynamic ratio;

[0019] Based on the mathematical statistics of the second reward function result, the second dynamic ratio is adjusted to obtain the fourth dynamic ratio.

[0020] In one embodiment, determining the information acquisition strategy score for the i-th training round includes:

[0021] Based on the running data of the i-th training round, the semantic correlation between the search result vector of the artificial intelligence model when processing any training sample pair and any reference vector corresponding to the training sample pair is obtained; the reference vector is the standard search result vector corresponding to the processing of the training sample pair.

[0022] Based on the operational data, the similarity between the search results of the artificial intelligence model when processing the training sample pairs and the historical search results is obtained;

[0023] Based on the artificial intelligence model, the semantic relevance and result similarity of each training sample pair are processed to determine the information acquisition score corresponding to the training sample set.

[0024] In one embodiment, determining the information acquisition score corresponding to the training sample set based on the semantic relevance and result similarity of each training sample pair using the artificial intelligence model includes:

[0025] For each training sample pair, the unit information acquisition score of each training sample pair is determined based on the difference between the information gain value of the training sample pair and the information acquisition cost corresponding to the training sample pair; the information gain value is determined at least based on the semantic relevance and result similarity corresponding to the training sample pair.

[0026] Based on the unit information acquisition score of each training sample pair, the information acquisition score corresponding to the training sample set is determined.

[0027] In one embodiment, the information acquisition cost includes a first information acquisition cost and a second information acquisition cost; the process of acquiring the information acquisition cost includes:

[0028] For the first training sample pair, the first information acquisition cost is determined based on the first initial information acquisition cost, the first base, and the number of information acquisition cycles corresponding to the first training sample pair.

[0029] For the second training sample pair, the second information acquisition cost is determined based on the second initial information acquisition cost, the second base, and the number of information acquisition cycles corresponding to the second training sample pair.

[0030] In one embodiment, the cost of acquiring the first initial information is greater than the cost of acquiring the second initial information; the first base is greater than the second base.

[0031] In one embodiment, in each training round, the second dynamic ratio is greater than the first dynamic ratio.

[0032] Secondly, this application also provides a training device for the information acquisition capability of an artificial intelligence model, comprising:

[0033] The acquisition module is used to acquire the training sample set of the i-th training round. The training sample set includes a first training sample pair with a first dynamic ratio and a second training sample pair with a second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. Both the first training sample pair and the third training sample pair are training sample pairs in the original training sample set. Wherein, i is a positive integer greater than or equal to 1.

[0034] The training module is used to train the artificial intelligence model based on the training sample set, determine the reward function corresponding to the training sample set according to the running data of the i-th training round, and iteratively update the parameters of the artificial intelligence model based on the result of the reward function.

[0035] The training method and apparatus for the information acquisition capability of the aforementioned artificial intelligence model, for the i-th training round, determines a first training sample pair with a first dynamic ratio and a third training sample pair with a second dynamic ratio from the original training sample set, and performs semantic reconstruction processing on the third training sample pair to obtain a second training sample pair. The first training sample pair with the first dynamic ratio and the second training sample pair with the second dynamic ratio are constructed as the training sample set for the i-th training round. This makes the composition of the training samples in the constructed training sample set more dynamic and rich. Moreover, since the second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair, the problems of the training sample pairs in the constructed training sample set are also more complex and diverse. Thus, the artificial intelligence model can be trained to respond to different problems based on different training sample sets, enabling the artificial intelligence model to have a more comprehensive information acquisition adaptability and better suited to application scenarios involving complex information processing or mixed processing of multiple different types of information. Setting the first dynamic ratio and the second dynamic ratio can ensure that the structure of the training samples is dynamically adjusted according to the results of the previous training round. Compared with the existing technology where the composition ratio of each type of training sample is fixed in each training round, configuring the dynamic ratio can better improve the multi-round training effect. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating a method for training the information acquisition capability of an artificial intelligence model in one embodiment.

[0038] Figure 2 This is a flowchart illustrating a method for training the information acquisition capability of an artificial intelligence model in another embodiment.

[0039] Figure 3 This is a flowchart illustrating a method for training the information acquisition capability of an artificial intelligence model in another embodiment.

[0040] Figure 4 This is a flowchart illustrating a method for training the information acquisition capability of an artificial intelligence model in another embodiment.

[0041] Figure 5 This is a flowchart illustrating a method for training the information acquisition capability of an artificial intelligence model in another embodiment.

[0042] Figure 6 This is a structural block diagram of a training device for the information acquisition capability of an artificial intelligence model in one embodiment;

[0043] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0046] In one embodiment, such as Figure 1 As shown, a training method for improving the information acquisition capability of an artificial intelligence model is provided. This embodiment illustrates the method by applying it to a computer device. It is understood that this method can also be applied to servers, mobile devices, and systems including both computer devices and servers, and is implemented through the interaction between the computer device and the server. In this embodiment, the method includes the following steps:

[0047] S201, obtain the training sample set for the i-th training round. The training sample set includes the first training sample pair with the first dynamic ratio and the second training sample pair with the second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. The first training sample pair and the third training sample pair are both training sample pairs in the original training sample set. Where i is a positive integer greater than or equal to 1.

[0048] Here, the i-th training round can be any round in the training process of the artificial intelligence model. It is understood that there is a certain upper limit to the number of training rounds of the artificial intelligence model. In this embodiment, i is a positive integer greater than or equal to 1. At the same time, the value of i must be less than the upper limit of the number of training rounds of the artificial intelligence model. For example, the upper limit of rounds can be 20, 30, 22, 25, etc. In this embodiment, the upper limit of the number of training rounds is not limited, and the value can be determined in combination with the actual training process.

[0049] The original training sample set in this embodiment may include multiple pairs of training sample pairs composed of <question-answer>, that is, QA pairs. Exemplarily, the original training sample set may include different types of QA pairs, including but not limited to factual retrieval QA pairs, multi-round exploratory QA pairs, counterfactual QA pairs, etc. As an example, a factual retrieval QA pair may be like: <What is the number of newborns in XX District in XX year? - XX people>; a multi-round exploratory QA pair may be like: <What are the models of new energy vehicles with the highest global sales volume and the standard cruising range in the XXth quarter of XX year? - The model of new energy vehicle with the highest global sales volume in the XXth quarter of XX year is XXX → Obtain the mileage data of all versions of this model of new energy vehicle → Its standard cruising range is XX>; a counterfactual QA pair may be like: <What is the working principle of a perpetual motion machine? - There is no perpetual motion machine, and this concept violates the law of conservation of energy>.

[0050] Exemplarily, for the i-th training round, a first training sample pair with a first dynamic ratio and a third training sample pair with a second dynamic ratio can be selected from the original training sample set. Here, it should be noted that for different training rounds, the values of the first dynamic ratio and the second dynamic ratio can be the same as those in the previous training round, or different. Preferably, for each training round, the values of the first dynamic ratio and the second dynamic ratio can be different from those in the previous training round, so as to enrich the training sample set corresponding to each training round, enabling the artificial intelligence model to make information acquisition responses to different questions in different training rounds.

[0051] It can be understood that in order to enrich the richness of the training sample set in each training round, semantic reconstruction processing can be performed on a relatively large proportion of the third training sample pairs, so that the proportion of the second training sample pairs in the training sample set is relatively large. As an optional implementation manner, in order to enable the model to perform better when dealing with complex questions, the above-mentioned second dynamic ratio is greater than the first dynamic ratio.

[0052] Furthermore, semantic reconstruction processing can be performed on the selected third training sample pair to obtain the second training sample pair in the training sample set of the i-th training round. For example, the semantic reconstruction processing performed on the third training sample pair refers to rewriting the question portion of the third training sample pair, reconstructing it into content with richer semantic information, while not changing the answer. For instance, the question portion in the third training sample is "The famous B of the A century died in the XX battle. What was the name of the warhorse B rode in that battle? In which museum is the specimen of this horse now preserved?" This question portion can be semantically reconstructed as "Please find the leader of this XX dynasty, also known as 'XX'. His daughter later became a famous queen and invited C to be her tutor. This leader died in a battle that took place in XX year. Please find the name of the warhorse the king rode in that fatal battle, and the current location of the specimen of this warhorse." It can be seen that the semantically reconstructed question portion has richer semantic information and a more detailed description of the problem.

[0053] Optionally, the semantic reconstruction processing performed on the third training sample pair can be either semantic reconstruction processing of at least one core constraint in the question part of the third training sample pair, or semantic reconstruction processing of at least one auxiliary constraint and at least one core constraint in the question part of the third training sample pair. It should be noted that the core constraint in the question part of the third training sample pair refers to the constraint whose deletion or modification would affect the answer to the question, while the auxiliary constraint refers to the constraint whose deletion or modification would not affect the answer to the question. Assuming that the question part of a training sample consists of constraint A + constraint B + constraint C, if the response information can be uniquely determined based solely on constraints A and B, then constraints A and B are core constraints, and constraint C is an auxiliary constraint; if the response information can be uniquely determined solely on constraints B and C, then constraints B and C are core constraints, and constraint A is an auxiliary constraint. Furthermore, performing semantic reconstruction processing on the question part will not change the corresponding answer.

[0054] Understandably, performing semantic reconstruction on the third training sample enriches the semantic information of the resulting second training sample, making it more complex. This inevitably increases the number of thought rounds the AI ​​model needs to process the semantically reconstructed second training sample. Furthermore, the number of thought rounds corresponds to several different semantically reconstructed second training samples in the training sample set. Each training round may contain a large number of training samples corresponding to different thought rounds, improving the AI ​​model's search ability when handling various problems. Additionally, each time samples requiring semantic reconstruction are selected, several pairs of first and third training samples are randomly chosen from all training samples. This means that in different training rounds, a small portion of the third training sample pairs may be the same, while most are different, for semantic reconstruction. The resulting second training sample pairs are also random, increasing the diversity of the second training samples in each training round. This improves the generalization ability of the AI ​​model during training. Simultaneously, the small number of repetitive training sample pairs allows the model to quickly and fully learn the feature patterns in the training data, reducing the risk of underfitting and overfitting.

[0055] S202, the artificial intelligence model is trained based on the training sample set. The reward function corresponding to the training sample set is determined according to the running data of the i-th training round, and the parameters of the artificial intelligence model are iteratively updated based on the result of the reward function.

[0056] In this embodiment, the artificial intelligence model can be trained based on the training sample set of the i-th training round to obtain the running data of the training sample set. Based on the running data of the i-th training round, the reward function corresponding to the training sample set is determined. Taking the score of the reward function as the core, the score result is transformed into a loss that the artificial intelligence model can optimize through the loss function. Then, based on the running data of the artificial intelligence model running the training sample set, the gradient from parameter change to loss change is calculated for all parameters of the artificial intelligence model to construct gradient information. Based on the gradient information, the thought chain decision logic of the artificial intelligence model is iteratively updated to update the parameters of the artificial intelligence model.

[0057] The following will explain the process of training the artificial intelligence model based on the training sample set of the i-th training epoch, and determining the reward function corresponding to the training sample set of the i-th training epoch based on the running data of the i-th training epoch. In one embodiment, such as Figure 2 As shown, the above S202 includes:

[0058] S301, Based on the running data, determine the result correctness score, thought chain behavior score, and information acquisition strategy score corresponding to the training sample set of the i-th training round.

[0059] In this embodiment, the result correctness score refers to whether the answer output by the artificial intelligence model running the training sample set is correct, that is, whether the answer output by the artificial intelligence model is the same as the A in the QA pair in the training sample set. Matching detection can be achieved by string matching method. If it matches, the result correctness score is 1; if it does not match, the result correctness score is 0.

[0060] In this embodiment, the thought chain behavior score refers to whether the processing of the artificial intelligence model conforms to preset rules. The thought chain behavior score is 1 only when the processing of the artificial intelligence model fully conforms to the preset rules; otherwise, it is 0. For example, taking interleaved thinking as an example, the preset behavior rule is that in the 0th to T-1th search cycle, <thinking x-behavior y-response z> is ​​executed. In the last search cycle T, only thinking x is executed. The thinking content is mainly used to integrate the search content and generate the answer. It is no longer necessary to perform search operations or obtain search feedback. Furthermore, each behavior of the artificial intelligence model needs to be represented using a corresponding format. For example, the thinking content is represented using... <think> and< / think> Enclosing, the content of the behavior is used <search> and< / search> Enclose the response content using <observation> and< / observation> The thought process for a short thought chain question with a T=1 interval includes thinking -> execution -> response -> thinking. Optionally, the preset behavior rules can be other rules, as long as the model outputs according to the preset behavior rules. As an example, taking the behavior rules in the example above as the preset behavior rules, we can examine the keywords in the code generated by the AI ​​model after receiving the question. <think> 、< / think> , <search> 、< / search> , <observation> 、< / observation> The order in which the rules are observed determines whether the behavior meets the requirements of the preset behavioral rules. If it does, the behavior score of the thought chain is 1. If any part does not meet the preset rules, the behavior score of the thought chain is 0.

[0061] In this embodiment, the information acquisition strategy score is used to characterize whether each step of the artificial intelligence model's search is beneficial to the solution. The detailed process of determining the information acquisition strategy score for the i-th training round will be explained below. In one embodiment, such as... Figure 3 As shown, determining the information acquisition strategy score for the i-th training round in S301 above includes:

[0062] S401, Based on the running data of the i-th training round, obtain the semantic correlation between the search result vector of the artificial intelligence model when processing any training sample pair and any reference vector corresponding to the training sample pair; the reference vector is the standard search result vector corresponding to each search behavior when processing the training sample pair.

[0063] The reference vector is the standard search result vector corresponding to the training sample pair processed by the artificial intelligence model. When the training sample pair is the first training sample pair, the reference vector can be the standard search result vector corresponding to the first training sample pair. When the training sample pair is the third training sample pair, the reference vector can be the standard search result vector corresponding to the third training sample pair.

[0064] Understandably, by leveraging the semantic correlation between the search result vector and any corresponding reference vector of the training sample pair when the AI ​​model processes the training sample pair, it can be ensured that the search content of the AI ​​model closely matches the preset topic. As an example, for the search result corresponding to a search behavior, the search result of the AI ​​model can first be converted into a first latent space vector based on the embedding model. Then, the semantic correlation between the first latent space vector and all reference vectors is calculated, and the maximum semantic correlation with the first latent space vector is taken as the semantic correlation between the search result vector and the reference vectors for that search period. Here, the latent space vectors obtained from all search results have the same dimension. Taking the set of all reference vectors corresponding to the first training sample pair as G1 and the set of all reference vectors corresponding to the third training sample pair as G2 as an example, when the artificial intelligence model processes the first training sample pair, it can calculate the semantic correlation between the search result vector of the artificial intelligence model processing any first training sample pair and all reference vectors in G1. Then, the maximum semantic correlation is taken as the semantic correlation between the search result vector of the first training sample pair and any reference vector corresponding to the first training pair. When the artificial intelligence model processes the second training sample pair, it can calculate the semantic correlation between the search result vector of the artificial intelligence model processing any second training sample pair and all reference vectors in G2. Then, the maximum semantic correlation is taken as the semantic correlation between the search result vector of the second training sample pair and any reference vector corresponding to the second training pair.

[0065] Additionally, it should be noted that the vectors mentioned in the embodiments of this application include, but are not limited to, dense vectors or sparse vectors. As an optional implementation, the semantic relevance calculation method includes: i. angle-based measures, such as cosine similarity; ii. spatial distance-based measures, such as Euclidean distance and Manhattan distance; iii. projection-based measures, such as dot product; and any mathematical operation that can characterize the closeness of two multidimensional vectors in a high-dimensional space. It should also be noted that before performing the semantic relevance calculation, the search result vector and any reference vector need to be normalized so that the final semantic relevance falls within the interval [0, 1]. Preferably, L2 normalization can be used when normalizing the search result vector and any reference vector.

[0066] S402, based on the running data, obtain the similarity between the search results of the artificial intelligence model when processing training sample pairs and the historical search results.

[0067] For example, with The similarity between the latest search result and existing historical search results when the artificial intelligence model processes training sample pairs is represented. It can be used to control the novelty of new search results from artificial intelligence models compared to historical search results, and to prevent the model from executing overly similar search strategies and / or providing overly similar search results in different search cycles through novelty rewards.

[0068] As an optional implementation, the first latent space vector can be invoked again to calculate the semantic similarity between the first latent space vector and all historical search result vectors. The highest semantic similarity is taken as the result similarity between the search result and the historical search result when the AI ​​model processes training sample pairs. It should be noted that the calculation method for result similarity is based on the same principle as the calculation method for semantic relevance, and will not be repeated here. Furthermore, for the 0th cycle when the AI ​​model processes training sample pairs, since there is no historical data, , .

[0069] S403, based on the artificial intelligence model, processes the semantic correlation and result similarity of each training sample pair to determine the information acquisition score corresponding to the training sample set.

[0070] As an optional implementation method, such as Figure 4 As shown, the above S403 includes:

[0071] S501, for each training sample pair, the unit information acquisition score of each training sample is determined based on the difference between the information gain value of the training sample pair and the information acquisition cost corresponding to the training sample pair; the information gain value is determined at least based on the semantic relevance and result similarity of the training sample pair.

[0072] For example, with This indicates the semantic relevance between training samples. Indicates the similarity of results. The information acquisition cost for a training sample pair is represented by the unit information acquisition score for that training sample pair. It can be represented as In the formula, This represents the information gain value of the training sample pair. It should be noted that the information gain value of the training sample pair is... and Product rather than and The weighted sum is used because the product better reflects the synergistic relationship between the two parameters. The multiplication relationship can better highlight scenarios where both parameters score highly. If one parameter is large and the other is small, the overall result will be lowered. The multiplication method is suitable for scenarios where both important parameters need to be optimized simultaneously, so that the artificial intelligence model tends to search for content in each round that is neither off-topic nor lacks new reference information. In contrast, if the weighted sum of the relevance score and the novelty score is calculated, there is no such synergistic effect. If one of them is large, the smaller of the other will not have a significant impact on the final result.

[0073] In this embodiment, for each training sample pair, the unit information acquisition score of the training sample pair can be determined based on the difference between the information gain value of the training sample pair and the information acquisition cost corresponding to the training sample pair. For example, the information acquisition cost may include a first information acquisition cost and a second information acquisition cost. For the first training sample pair, the first information acquisition cost can be determined based on the first initial information acquisition cost, the first base, and the number of information acquisition cycles corresponding to the first training sample pair; for the second training sample pair, the second information acquisition cost can be determined based on the second initial information acquisition cost, the second base, and the number of information acquisition cycles corresponding to the second training sample pair. Wherein, the first initial information acquisition cost is greater than the second initial information acquisition cost, and the first base is greater than the second base. For example, with... Indicates the cost of acquiring the first piece of information. In the formula, This represents the initial cost of acquiring the first piece of information. Indicates the first base. This indicates the number of information acquisition cycles. Then, using... Indicates the cost of acquiring the second piece of information mentioned above. In the formula, This represents the cost of acquiring the second initial information. Indicates the second base. This indicates the number of information acquisition cycles. Among them, , The preferred value range is [1.5, 2]. The preferred value range is [1.02, 1.15], which ensures that the higher the number of rounds, the better. The higher the value, the better it aligns with the token consumption rules when a context exists.

[0074] It should be noted that the above , This is a preset value, representing the step cost of one cycle of computation. If the current training sample pair is the original QA pair, then the corresponding... If the current training sample is a rewritten Q'A pair, then the corresponding , We generally believe that the information gain from each round of searching decreases. If no information acquisition cost is set for the training sample pair, and only the information gain value of the training sample pair is considered, a positive reward value will be obtained as long as the newly searched content is slightly different from the historical content. This may lead the AI ​​model to choose a lengthy search strategy in order to achieve a higher score, or even the search strategy may enter an infinite loop. Taking the first training sample pair as an example, in other words, if the information gain of the newly searched information does not reach the required level... This will cause the unit information acquisition score of the training sample pair to become negative. Negative reward values ​​during training can prompt the model to stop quickly. For the original QA pair that should undergo fewer cycles of reflection, a larger reward is assigned... This allows the AI ​​model to stop thinking as soon as it has found enough relevant content; for rewritten Q'A pairs that should undergo more cycles of reflection, a smaller [percentage] is assigned. This prompts the model to think through as many rounds as possible until the value of the new content it finds is less than [a certain value]. Up to that point. Preferred, , Additionally, it should be noted that the preferred range of values ​​mentioned above is merely an example and is not limited to restricting the range of values; the range of values ​​can also include other values.

[0075] S502, based on the unit information acquisition score of each training sample pair, determine the information acquisition score corresponding to the training sample set.

[0076] For example, in this embodiment, the sum of the unit information acquisition scores of each training sample pair can be determined as the information acquisition score corresponding to the training sample set of the i-th training round of the artificial intelligence model.

[0077] S302. Determine the reward function based on the result correctness score, the thought chain behavior score, and the information acquisition strategy score.

[0078] For example, suppose the correctness score of the result is The score for the thought chain behavior is The information acquisition strategy score is Then the reward function It can be represented as In the formula, The values ​​of all values ​​are between (0, 1). The most important factor is that the AI ​​model can answer the question correctly; therefore, other terms should not outweigh the consideration of whether the result is correct. As an optional implementation method, it is recommended to... The value is greater than The value of .

[0079] Based on the above embodiments, as an optional implementation, the dynamic ratio of the first training sample pair and the dynamic ratio of the second training sample pair in the (i+1)th training round can be updated and determined based on the reward function corresponding to the training sample set in the i-th training round, thereby determining the training sample set in the (i+1)th training round. For ease of distinction from the i-th training round, the first training sample pair in the (i+1)th training round will be referred to as the fourth training sample pair, and the second training sample pair in the (i+1)th training round will be referred to as the sixth training sample pair. Figure 5 As shown, the above method also includes:

[0080] S601, based on the result of the reward function, adjust the first dynamic ratio to obtain the third dynamic ratio, and adjust the second dynamic ratio to obtain the fourth dynamic ratio.

[0081] For example, the reward function for the i-th training epoch may include a first reward function corresponding to the first training sample pair in the i-th training epoch, and a second reward function corresponding to the second training sample pair. In this embodiment, as an optional implementation, the reward function calculation formula described in the above embodiments can be used: We obtain the first reward function result corresponding to the first training sample pair and the second reward function result corresponding to the second training sample pair in the i-th training round. Then, based on the mathematical statistics of the first reward function result, we adjust the first dynamic ratio to obtain the third dynamic ratio. Based on the mathematical statistics of the second reward function result, we adjust the second dynamic ratio to obtain the fourth dynamic ratio. Assume that the average reward score corresponding to the first training sample pair is... The average reward score for the second training sample pair is Then the fourth dynamic ratio can be based on the formula Determine, in the formula, This represents the fourth dynamic scale; the third dynamic scale can be based on the formula. Determine, in the formula, This indicates the third dynamic ratio. It is understandable that when... A lower score indicates that the model performs poorly on rewriting samples. If the value approaches 1, it means that the proportion of samples rewritten in the next round will also approach 1; conversely, if... A high score indicates that the model's performance on rewritten samples in that round of training has met expectations. It will decrease.

[0082] S602, determine the fourth training sample pair with the third dynamic ratio from the original training sample set, and determine the fifth training sample pair with the fourth dynamic ratio from the original training sample set.

[0083] Understandably, the dynamic ratios of the original training sample pairs and the semantically reconstructed training sample pairs in the training sample set are different in each training epoch. Therefore, after determining the third dynamic ratio of the original training sample pairs and the fourth dynamic ratio of the semantically reconstructed training sample pairs in the (i+1)th training epoch, the fourth training sample pair with the third dynamic ratio and the fifth training sample pair with the fourth dynamic ratio can be determined from the original training sample set. Furthermore, the sample partitioning or extraction methods can differ in different training epochs to ensure data diversity and improve the model's generalization ability.

[0084] S603, perform semantic reconstruction processing on the fifth training sample pair to obtain the sixth training sample pair.

[0085] In this embodiment, the semantic reconstruction processing performed on the fifth training sample pair can refer to the semantic reconstruction processing performed on the third training sample pair described in the above embodiments. The principles of the two are similar, and this embodiment will not repeat them here.

[0086] S604, the set consisting of the fifth training sample pair and the sixth sample pair is determined as the training sample set for the (i+1)th training round.

[0087] In this embodiment, after performing semantic reconstruction processing on the fifth training sample pair of the fourth dynamic ratio, the set of the obtained sixth training sample pair and the fourth training sample pair of the third dynamic ratio can be determined as the training sample set of the (i+1)th training round.

[0088] In the training method for the information acquisition capability of the aforementioned artificial intelligence model, for the i-th training round, a first training sample pair with a first dynamic ratio and a third training sample pair with a second dynamic ratio are determined from the original training sample set. Semantic reconstruction processing is then performed on the third training sample pair to obtain a second training sample pair. The first training sample pair with the first dynamic ratio and the second training sample pair with the second dynamic ratio are constructed as the training sample set for the i-th training round. This makes the composition of the training samples in the constructed training sample set more dynamic and richer. Furthermore, since the second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair, the problems of the training sample pairs in the constructed training sample set are also more complex and diverse. Therefore, the artificial intelligence model can be trained to respond to different problems based on different training sample sets, enabling the artificial intelligence model to have a more comprehensive information acquisition adaptability and better suited for application scenarios involving complex information processing or mixed processing of various types of information. Setting the first and second dynamic ratios ensures that the structure of the training samples is dynamically adjusted according to the results of the previous training round. Compared with the existing technology where the proportion of each type of training sample is fixed in each training round, configuring dynamic ratios can better improve the multi-round training effect.

[0089] To facilitate understanding by those skilled in the art, the following provides a detailed description of the training method for the information acquisition capability of the artificial intelligence model disclosed herein. This method may include:

[0090] S1, obtain the training sample set for the i-th training round. The training sample set includes the first training sample pair with the first dynamic ratio and the second training sample pair with the second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. The first training sample pair and the third training sample pair are both training sample pairs in the original training sample set. Where i is a positive integer greater than or equal to 1. The second dynamic ratio is greater than the first dynamic ratio.

[0091] S2, train the artificial intelligence model based on the training sample set, and determine the result correctness score and thought chain behavior score corresponding to the training sample set of the i-th training round according to the running data of the i-th training round.

[0092] S3, based on the running data of the i-th training round, obtain the semantic correlation between the search result vector of the artificial intelligence model when processing any training sample pair and any reference vector corresponding to the training sample pair; the reference vector is the standard search result vector corresponding to the training sample pair.

[0093] S4, based on the running data, obtains the similarity between the search results of the artificial intelligence model when processing training sample pairs and the historical search results.

[0094] S5, for each training sample pair, the unit information acquisition score of each training sample pair is determined based on the difference between the information gain value of the training sample pair and the information acquisition cost corresponding to the training sample pair; the information gain value is determined at least based on the semantic relevance and result similarity corresponding to the training sample pair. Specifically, for the first training sample pair, the first information acquisition cost is determined based on the first initial information acquisition cost, the first base, and the number of information acquisition cycles corresponding to the first training sample pair; for the second training sample pair, the second information acquisition cost is determined based on the second initial information acquisition cost, the second base, and the number of information acquisition cycles corresponding to the second training sample pair; the first initial information acquisition cost is greater than the second initial information acquisition cost; the first base is greater than the second base.

[0095] S6. Based on the unit information acquisition score of each training sample pair, determine the information acquisition score corresponding to the training sample set.

[0096] S7. Based on the result correctness score, the thought chain behavior score, and the information acquisition strategy score, determine the reward function corresponding to the training sample set in the i-th training round.

[0097] S8 updates the parameters of the artificial intelligence model iteratively based on the results of the reward function.

[0098] S9, obtain the first reward function result corresponding to the first training sample pair in the i-th training round and the second reward function result corresponding to the second training sample pair.

[0099] S10, based on the mathematical statistics of the first reward function result, adjust the first dynamic ratio to obtain the third dynamic ratio.

[0100] S11. Based on the mathematical statistics of the second reward function result, adjust the second dynamic ratio to obtain the fourth dynamic ratio.

[0101] S12, determine the fourth training sample pair with the third dynamic ratio from the original training sample set, and determine the fifth training sample pair with the fourth dynamic ratio from the original training sample set.

[0102] S13, perform semantic reconstruction processing on the fifth training sample pair to obtain the sixth training sample pair.

[0103] S14, determine the set consisting of the fifth training sample pair and the sixth sample pair as the training sample set for the (i+1)th training round.

[0104] S15, train the artificial intelligence model based on the training sample set of the (i+1)th training round.

[0105] It should be noted that the descriptions of the above steps can be found in the relevant descriptions in the above embodiments, and their effects are similar, so they will not be repeated here.

[0106] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0107] Based on the same inventive concept, this application also provides a training device for the information acquisition capability of an artificial intelligence model, which is used to implement the training method for the information acquisition capability of the artificial intelligence model described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in the one or more embodiments of the training device for the information acquisition capability of an artificial intelligence model provided below can be found in the limitations of the training method for the information acquisition capability of an artificial intelligence model described above, and will not be repeated here.

[0108] In one exemplary embodiment, such as Figure 6 As shown, a training device for the information acquisition capability of an artificial intelligence model is provided, comprising: an acquisition module 10 and a training module 11, wherein:

[0109] The acquisition module 10 is used to acquire the training sample set of the i-th training round. The training sample set includes a first training sample pair with a first dynamic ratio and a second training sample pair with a second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. Both the first training sample pair and the third training sample pair are training sample pairs in the original training sample set. Where i is a positive integer greater than or equal to 1.

[0110] Training module 11 is used to train the artificial intelligence model based on the training sample set. It determines the reward function corresponding to the training sample set based on the running data of the i-th training round, and iteratively updates the artificial intelligence model parameters based on the result of the reward function.

[0111] Optionally, the second dynamic ratio is greater than the first dynamic ratio.

[0112] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0113] Based on the above embodiments, optionally, the above device further includes: an adjustment module, a first determining module, a processing module, and a second determining module, wherein:

[0114] The adjustment module is used to adjust the first dynamic ratio to obtain the third dynamic ratio based on the result of the reward function, and to adjust the second dynamic ratio to obtain the fourth dynamic ratio.

[0115] The first determining module is used to determine the fourth training sample pair with the third dynamic ratio from the original training sample set, and the fifth training sample pair with the fourth dynamic ratio from the original training sample set.

[0116] The processing module is used to perform semantic reconstruction processing on the fifth training sample pair to obtain the sixth training sample pair.

[0117] The second determining module is used to determine the set consisting of the fifth training sample pair and the sixth sample pair as the training sample set for the (i+1)th training round.

[0118] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0119] Based on the above embodiments, optionally, the training module 11 includes: a first determining unit and a second determining unit, wherein:

[0120] The first determining unit is used to determine the result correctness score, thought chain behavior score, and information acquisition strategy score corresponding to the training sample set of the i-th training round, based on the running data.

[0121] The second determining unit is used to determine the reward function based on the result correctness score, the thought chain behavior score, and the information acquisition strategy score.

[0122] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0123] Based on the above embodiments, optionally, the adjustment module includes: an acquisition unit, a first adjustment unit, and a second adjustment unit, wherein:

[0124] The acquisition unit is used to acquire the first reward function result corresponding to the first training sample pair and the second reward function result corresponding to the second training sample pair in the i-th training round.

[0125] The first adjustment unit is used to adjust the first dynamic ratio based on the mathematical statistics of the first reward function result to obtain the third dynamic ratio.

[0126] The second adjustment unit is used to adjust the second dynamic ratio based on the mathematical statistics of the second reward function result to obtain the fourth dynamic ratio.

[0127] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0128] Based on the above embodiments, optionally, the first determining unit is specifically used to obtain the semantic correlation between the search result vector of the artificial intelligence model when processing any training sample pair and any reference vector corresponding to the training sample pair, based on the running data of the i-th training round; the reference vector is the standard search result vector corresponding to the training sample pair; based on the running data, obtain the result similarity between the search result of the artificial intelligence model when processing the training sample pair and the historical search result; and based on the semantic correlation and result similarity of each training sample pair processed by the artificial intelligence model, determine the information acquisition score corresponding to the training sample set.

[0129] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0130] Based on the above embodiments, optionally, the first determining unit is specifically used to determine the unit information acquisition score of each training sample pair based on the difference between the information gain value of the training sample pair and the information acquisition cost corresponding to the training sample pair; the information gain value is determined at least based on the semantic relevance and result similarity corresponding to the training sample pair; and the information acquisition score corresponding to the training sample set is determined based on the unit information acquisition score of each training sample pair.

[0131] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0132] Based on the above embodiments, optionally, the information acquisition cost includes a first information acquisition cost and a second information acquisition cost; the first determining unit is specifically used to determine the first information acquisition cost for the first training sample pair based on the first initial information acquisition cost, the first base, and the number of information acquisition cycles corresponding to the first training sample pair; and to determine the second information acquisition cost for the second training sample pair based on the second initial information acquisition cost, the second base, and the number of information acquisition cycles corresponding to the second training sample pair.

[0133] Optionally, the cost of acquiring the first initial information is greater than the cost of acquiring the second initial information; the first base is greater than the second base.

[0134] The training device for the information acquisition capability of the artificial intelligence model provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0135] The modules in the training device for the information acquisition capability of the aforementioned artificial intelligence model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0136] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores processing parameters corresponding to the artificial intelligence model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for the information acquisition capabilities of an artificial intelligence model.

[0137] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0140] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training the information acquisition ability of an artificial intelligence model, characterized in that, The method includes: Obtain the training sample set for the i-th training round. The training sample set includes a first training sample pair with a first dynamic ratio and a second training sample pair with a second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. Both the first training sample pair and the third training sample pair are training sample pairs in the original training sample set. Where i is a positive integer greater than or equal to 1. The artificial intelligence model is trained based on the training sample set. The reward function corresponding to the training sample set is determined according to the running data of the i-th training round, and the parameters of the artificial intelligence model are iteratively updated based on the result of the reward function.

2. The method according to claim 1, characterized in that, The method further includes: Based on the result of the reward function, the first dynamic ratio is adjusted to obtain the third dynamic ratio, and the second dynamic ratio is adjusted to obtain the fourth dynamic ratio; The fourth training sample pair with the third dynamic ratio is determined from the original training sample set, and the fifth training sample pair with the fourth dynamic ratio is determined from the original training sample set; The fifth training sample pair is subjected to semantic reconstruction processing to obtain the sixth training sample pair; The set consisting of the fifth training sample pair and the sixth sample pair is determined as the training sample set for the (i+1)th training round.

3. The method according to claim 2, characterized in that, The step of training the artificial intelligence model based on the training sample set, and determining the reward function corresponding to the training sample set based on the running data of the i-th training round, includes: Based on the operational data, the result correctness score, thought chain behavior score, and information acquisition strategy score corresponding to the training sample set of the i-th training round are determined respectively. The reward function is determined based on the correctness score of the results, the thought chain behavior score, and the information acquisition strategy score.

4. The method according to claim 3, characterized in that, The step of adjusting the first dynamic ratio to obtain the third dynamic ratio based on the result of the reward function, and adjusting the second dynamic ratio to obtain the fourth dynamic ratio, includes: Obtain the first reward function result corresponding to the first training sample pair and the second reward function result corresponding to the second training sample pair in the i-th training round; Based on the mathematical statistics of the first reward function result, the first dynamic ratio is adjusted to obtain the third dynamic ratio; Based on the mathematical statistics of the second reward function result, the second dynamic ratio is adjusted to obtain the fourth dynamic ratio.

5. The method according to claim 3, characterized in that, Determine the information acquisition strategy score for the i-th training round, including: Based on the running data of the i-th training round, the semantic correlation between the search result vector of the artificial intelligence model when processing any training sample pair and any reference vector corresponding to the training sample pair is obtained; the reference vector is the standard search result vector corresponding to the processing of the training sample pair. Based on the operational data, the similarity between the search results of the artificial intelligence model when processing the training sample pairs and the historical search results is obtained; Based on the artificial intelligence model, the semantic relevance and result similarity of each training sample pair are processed to determine the information acquisition score corresponding to the training sample set.

6. The method according to claim 5, characterized in that, The process of determining the information acquisition score corresponding to the training sample set based on the semantic relevance and result similarity of each training sample pair using the artificial intelligence model includes: For each training sample pair, the unit information acquisition score of each training sample pair is determined based on the difference between the information gain value of the training sample pair and the information acquisition cost corresponding to the training sample pair; the information gain value is determined at least based on the semantic relevance and result similarity corresponding to the training sample pair. Based on the unit information acquisition score of each training sample pair, the information acquisition score corresponding to the training sample set is determined.

7. The method according to claim 6, characterized in that, The information acquisition cost includes a first information acquisition cost and a second information acquisition cost; the process of acquiring the information acquisition cost includes: For the first training sample pair, the first information acquisition cost is determined based on the first initial information acquisition cost, the first base, and the number of information acquisition cycles corresponding to the first training sample pair. For the second training sample pair, the second information acquisition cost is determined based on the second initial information acquisition cost, the second base, and the number of information acquisition cycles corresponding to the second training sample pair.

8. The method according to claim 7, characterized in that, The cost of acquiring the first initial information is greater than the cost of acquiring the second initial information; the first base number is greater than the second base number.

9. The method according to claim 1, characterized in that, The second dynamic ratio is greater than the first dynamic ratio.

10. A training device for the information acquisition capability of an artificial intelligence model, characterized in that, The device includes: The acquisition module is used to acquire the training sample set of the i-th training round. The training sample set includes a first training sample pair with a first dynamic ratio and a second training sample pair with a second dynamic ratio. The second training sample pair is obtained by performing semantic reconstruction processing on the third training sample pair. Both the first training sample pair and the third training sample pair are training sample pairs in the original training sample set. Wherein, i is a positive integer greater than or equal to 1. The training module is used to train the artificial intelligence model based on the training sample set, determine the reward function corresponding to the training sample set according to the running data of the i-th training round, and iteratively update the parameters of the artificial intelligence model based on the result of the reward function.

Images