A no-label inference enhancement method based on generator- verifier co-evolution

Abstract

The application discloses a kind of label-free inference enhancement methods based on generator-verification collaborative evolution, which utilizes large language model as generator and verifier simultaneously: first, generator generates result set multiple times on inference query inference, and drafts pseudo-label to divide positive and negative sample groups through majority vote;Subsequently, the verifier verifies the positive sample to form a positive verification set, and generates a verification result for the negative sample to form a negative verification set;Based on the positive and negative sets, contrast training samples are constructed to correct errors and optimize the inference and verification capabilities of the model. The optimized model can be directly used for inference and verification of target queries. Experiments show that on mathematical reasoning and cross-domain question answering datasets, this method significantly improves inference and verification accuracy, while training stability is superior and has outstanding cross-distribution generalization capability.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and in particular relates to a label-free reasoning enhancement method based on generator-verifier co-evolution, which is applicable to complex reasoning tasks such as mathematical reasoning and cross-domain question answering. Background Technology

[0002] In recent years, large language models have developed rapidly, demonstrating powerful processing capabilities in various intelligent tasks such as complex reasoning and question answering, becoming a core technological support in the field of artificial intelligence. With the continuous expansion of application scenarios, higher demands are placed on the reasoning accuracy and generalization ability of large language models. However, traditional model optimization methods often rely on large amounts of high-quality labeled data. Obtaining such data not only requires significant human and material resources but also faces difficulties in many specialized fields, limiting the further application of the models.

[0003] Label-free learning, as a training paradigm that does not rely on external labeled data, is gradually becoming a key direction for improving the capabilities of large language models. Its core idea is to utilize the model's own output or the inherent features of the data to construct effective supervision signals, enabling the model to self-optimize and iterate. This learning method can overcome the limitations of labeled data, reduce training costs, and is particularly suitable for complex reasoning scenarios lacking standard labels, providing a feasible path for the unsupervised evolution of large language models.

[0004] Among the many implementations of unlabeled learning, majority voting is one of the mainstream technical approaches. The core logic of this method is that, for a single reasoning task, the model generates multiple independent reasoning paths and corresponding results. The answer that appears most frequently is selected as a "pseudo-label" based on statistical frequency, and this pseudo-label is used as the criterion for judging the correctness of the reasoning result, which is then used in the reward calculation for reinforcement learning. The underlying assumption is that, among diverse reasoning paths, consistently occurring answers are more likely to be close to the true solution. This method does not require external labels and can quickly provide training signals to the model, thus it is widely used in the unlabeled reasoning optimization of large language models.

[0005] While Test-Time Reinforcement Learning (TTRL), a representative method based on majority voting, achieves label-free training to some extent, it still has significant technical limitations. Because its training objective is to maximize the consistency between the model's output and the pseudo-labels from the majority vote, as training progresses, the model's inference output gradually converges towards a single answer, leading to a continuous decrease in the diversity of inference paths. When the pseudo-labels generated by the majority vote are themselves incorrect, the model continuously reinforces this erroneous perception, forming a vicious cycle of "false consensus," making it difficult to detect and correct systematic biases. In this situation, the model's inference ability not only fails to continuously improve but may also degrade due to the accumulation of errors, failing to meet the accuracy and stability requirements of complex inference tasks. Summary of the Invention

[0006] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies: Firstly, existing unlabeled reinforcement learning relies on majority voting to generate pseudo-labels, easily falling into a "consensus trap," leading to a collapse in reasoning diversity, accumulation of systematic errors, and a continuous decline in reward and label accuracy. Secondly, the self-verification mechanism relies on external label training, creating a circular dependency with the unlabeled training scenario, making it unable to independently achieve error filtering and capability improvement. Thirdly, existing methods have weak self-correction capabilities, poor cross-domain reasoning generalization, and difficulty adapting to the dynamic needs of complex reasoning tasks. To address these shortcomings, this invention proposes an unlabeled reasoning enhancement method based on generator-verifier co-evolution.

[0007] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0008] In a first aspect, the present invention provides a label-free reasoning enhancement method based on generator-verifier co-evolution, comprising the following steps:

[0009] S1. For each reasoning query in the reasoning task dataset, the large language model to be enhanced for reasoning is first used as a generator to perform multiple reasoning operations on the reasoning query. After each reasoning operation, a reasoning result containing the answer and reasoning path is generated. Thus, the reasoning query and each reasoning result are used as a reasoning sample to form a reasoning set.

[0010] S2. By majority voting, draft the pseudo-labels corresponding to the inference query from the inference results, and divide the inference set into positive and negative sample groups according to the pseudo-labels;

[0011] S3. Using the large language model as a validator, the positive sample group is validated, and the positive sample group that passes the validation and the pseudo-label of the inference query are retained to form a positive validation set;

[0012] S4. The answer with the lowest frequency in the negative sample group is taken as the verification answer. The verifier generates multiple verification results corresponding to the verification answer and a verification label for each verification result. The inference query, verification result and verification label are used as a negative verification sample to form a negative verification set. The number of negative verification samples is the same as the number of positive verification samples.

[0013] S5. Construct comparative training samples based on positive and negative validation sets, and correct the inference results of all samples that fail validation in the two validation sets, in order to jointly optimize the error recognition ability of the large language model as a validator and the inference ability of the large language model as a generator.

[0014] S6. Input the target query into the optimized large language model. The optimized large language model acts as a generator to reason about the target query and generate an output containing the answer and reasoning path. When it is necessary to determine whether the output is correct, the optimized large language model acts as a validator to verify the output and obtain a judgment result indicating whether the output is correct.

[0015] Based on the above scheme, each step can be implemented in the following preferred manner.

[0016] As a preferred embodiment of the first aspect above, in step S2, the answer with the highest frequency in the reasoning set is used as the pseudo-label; for each reasoning result in the reasoning set, when the answer in the reasoning result is consistent with the pseudo-label, the reasoning sample corresponding to the reasoning result is added to the positive sample group, and when the answer in the reasoning result is inconsistent with the pseudo-label, the reasoning sample corresponding to the reasoning result is added to the negative sample group.

[0017] As a preferred embodiment of the first aspect mentioned above, the specific process of step S3 is as follows:

[0018] S31. When the number of inference samples in the positive sample group is greater than or equal to the preset verification budget threshold. At that time, random sampling was performed from the positive sample group. The positive sample group consists of a set of inference samples to be verified. When the number of inference samples in the positive sample group is less than the preset verification budget threshold, the set of inference samples in the positive sample group consists of all the inference samples in the positive sample group.

[0019] S32. After validating each inference sample in the validation set, the validator outputs a binary validation label; where a validation label of 1 indicates that the inference of the sample is correct and passes the validation; a validation label of 0 indicates that the inference of the sample is incorrect and fails the validation.

[0020] S33. When the number of inference samples with verification label 1 in the set to be verified exceeds half of its own number of inference samples, retain the pseudo label and positive sample group corresponding to the inference query, and use the inference sample and corresponding verification label in the positive sample group as a positive verification sample to form a positive verification set.

[0021] As a preferred embodiment of the first aspect above, in step S5, for all samples with a verification label of 0 in both the positive and negative verification sets, the generator generates a new sample based on the inference query, the inference result, and the verification label. Correcting the reasoning result, To correct the number of inference results.

[0022] As a preferred embodiment of the first aspect mentioned above, in step S5, the large language model is optimized using the answer-anchored group relative strategy optimization algorithm, and the total reward is obtained by adding the format reward and the accuracy reward; for the inference results of positive verification samples in the positive verification set, the pseudo-labels are grouped and the intra-group dominance value is calculated; for the corrected inference results and the verification results of negative verification samples in the negative verification set, standard prefix grouping is used and the intra-group dominance value is calculated; based on the intra-group dominance value maximizing the objective function of the group relative strategy optimization algorithm, the parameters of the generator and the verifier are jointly updated.

[0023] Furthermore, the accuracy reward for the generator is calculated as follows: for each sample, if the answer generated by the generator is the same as the pseudo-label, the accuracy reward for that sample is 1; otherwise, the accuracy reward is 0.

[0024] Furthermore, the accuracy reward for the validator is calculated as follows: For each sample, when the answer generated by the generator is the same as the pseudo-label and the validator outputs a verification label of 1, the accuracy reward for that sample is 1; when the answer generated by the generator is the same as the pseudo-label and the validator outputs a verification label of 0, the accuracy reward for that sample is 0; when the answer generated by the generator is different from the pseudo-label and the validator outputs a verification label of 1, the accuracy reward for that sample is 0; when the answer generated by the generator is different from the pseudo-label and the validator outputs a verification label of 0, the accuracy reward for that sample is 1.

[0025] In a second aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the label-free reasoning enhancement method based on generator-verifier co-evolution as described in any of the solutions of the first aspect above.

[0026] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the label-free reasoning enhancement method based on generator-verifier co-evolution as described in any of the solutions of the first aspect above.

[0027] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0028] The memory is used to store computer programs;

[0029] The processor is configured to, when executing the computer program, implement the label-free reasoning enhancement method based on generator-verifier co-evolution as described in any of the schemes of the first aspect above.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. Significantly improved reasoning accuracy: On the mathematical reasoning (MATH500, AMC, AIME24) and cross-domain question answering (GPQA-Diamond) datasets, compared with the TTRL baseline, the method of this invention improves the final reasoning accuracy by an average of 4.7%-5.9% on three large language models of different sizes, with Qwen3-1.7B-Base showing a 7.7% improvement on the AMC task;

[0032] 2. Significantly enhanced verification capability: The self-verification accuracy of the method of this invention has increased from approximately 55% initially to 81.2%-86.5%, effectively identifying "high consensus errors." The True Negative Rate has increased from 22% to 50%, while the False Positive Rate has remained stable below 5%.

[0033] 3. Excellent training stability: The method of this invention maintains a reward accuracy of over 90% throughout the entire process, avoiding the reward accuracy degradation problem of TTRL, and the label accuracy gradually increases from 75% to 88%;

[0034] 4. Excellent generalization ability: The method of this invention also achieves stable improvement on the cross-domain question answering test set GPQA-Diamond, realizing reliable inference across distributed scenarios. Attached Figure Description

[0035] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0036] Figure 2 A flowchart for constructing the positive validation set;

[0037] Figure 3 This is a schematic diagram of the components of a computer electronic device according to the present invention. Detailed Implementation

[0038] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0039] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0040] like Figure 1 As shown, in a preferred embodiment of the present invention, to solve the above-mentioned technical problems, the present invention proposes a label-free reasoning enhancement method based on generator-verifier co-evolution. This method achieves bidirectional supervision and mutual promotion of capabilities by having a single model alternately assume the roles of generator and verifier. The specific technical solution is as follows:

[0041] S1. For each reasoning query in the reasoning task dataset, the large language model to be enhanced by reasoning is first used as a generator to perform multiple reasoning operations on the reasoning query. After each reasoning operation, a reasoning result containing the answer and reasoning path is generated. Thus, the reasoning query and each reasoning result are used as a reasoning sample to form a reasoning set.

[0042] It should be noted that in step S1 of this invention, a large language model serves as the generator. The role takes a reasoning task dataset as input. For each reasoning query... Generator sampling Each of the four independent reasoning paths, along with its corresponding answer, forms a reasoning set. ;in, Indicates the index of the reasoning result; The number of reasoning results corresponding to one reasoning query; For the first inference set One reasoning result; It is a reasoning sample in the reasoning set.

[0043] S2. Draft pseudo-labels corresponding to the inference query from the inference results through majority voting, and divide the inference set into positive and negative sample groups based on the pseudo-labels.

[0044] It should be noted that in step S2 of this invention, the answer that appears most frequently in the reasoning set is used as the pseudo-label. For each inference result in the inference set, if the answer in that inference result matches the pseudo-label, then the inference sample corresponding to that inference result is added to the positive sample group. If the answer in the inference result is inconsistent with the pseudo-label, then the inference sample corresponding to the inference result is added to the negative sample group. .

[0045] S3. Using the large language model as a validator, the positive sample group is validated, and the positive sample group that passes the validation and the pseudo-label of the inference query are retained to form a positive validation set.

[0046] It should be noted that, as Figure 2 As shown, the specific process of step S3 of the present invention is as follows:

[0047] S31. When the positive sample group Number of inference samples in Greater than or equal to the preset verification budget threshold At that time, random sampling was performed from the positive sample group. The positive sample group consists of a set of inference samples to be verified. When the number of inference samples in the positive sample group is less than the preset verification budget threshold, the positive sample group consists of all inference samples to be verified. .

[0048] S32. Verifier After validating each inference sample in the validation set, output a binary validation label; where a validation label of 1 indicates that the inference of the inference sample is correct and passes the validation; and a validation label of 0 indicates that the inference of the inference sample is incorrect and fails the validation.

[0049] S33. When the number of inference samples with a verification label of 1 in the set to be verified exceeds half of the number of its own inference samples, retain the pseudo-label corresponding to that inference query. The positive sample group is used as a positive verification sample, and the inference sample and its corresponding verification label in the positive sample group are used as a positive verification sample to form a positive verification set. .

[0050] In this embodiment S3, the validator and the generator use the same large language model. If the set to be verified... middle The number of samples exceeded Then retain the pseudo-label of the inference query. And the corresponding positive sample group; otherwise, discard, thus avoiding training contamination by erroneous pseudo-labels. The resulting positive validation set is represented as follows: , This is a positive validation sample in the set. To validate the first positive sample group The result of the inference, To verify the label.

[0051] S4. The answer with the lowest frequency in the negative sample group is taken as the verification answer. The validator simultaneously generates multiple verification results corresponding to the verification answer and a verification label for each verification result. The inference query, verification result, and verification label are used as a negative verification sample to form a negative verification set. The number of negative validation samples is the same as the number of positive validation samples.

[0052] In this embodiment S4, from the negative sample group The answer with the lowest frequency is selected from the options, and the validator generates the result. The validation results are then used to form a negative validation set. , This is a negative validation sample in the set. For the generated first Verification results. Because... Therefore, a set of contrast samples was formed to prevent the validator from overfitting to the features of positive or negative samples.

[0053] S5. Construct comparative training samples based on positive and negative validation sets, and correct the inference results of all samples that fail validation in both validation sets. This is used to jointly optimize the error recognition capability of the large language model as a validator and the inference capability of the large language model as a generator.

[0054] It should be noted that in step S5 of this invention, for all samples with a verification label of 0 in both the positive and negative verification sets, the generator... Generate based on reasoning query, reasoning result, and verification label. There are 1 corrective inference result, where each corrective inference result is represented as: , Indicates the index of the corrected reasoning result. To correct the number of inference results.

[0055] It should be noted that in step S5 of this invention, the Answer-Anchored GRPO algorithm is used to optimize the large language model, and the total reward is obtained by adding the format reward and the accuracy reward. For the inference results of positive verification samples in the positive verification set, the pseudo-labels are grouped and the intra-group dominance value is calculated. For the corrected inference results and the verification results of negative verification samples in the negative verification set, standard prefix grouping is used and the intra-group dominance value is calculated. Based on the objective function of the Answer-Anchored GRPO algorithm that maximizes the intra-group dominance value, the parameters of the generator and the verifier are jointly updated to ensure collaborative optimization between the two roles.

[0056] Furthermore, the accuracy reward of the generator is calculated as follows: for each sample, if the answer generated by the generator is the same as the pseudo-label, the accuracy reward for that sample is 1; otherwise, the accuracy reward is 0.

[0057] Furthermore, the accuracy reward of the above validator is calculated as follows: For each sample, when the answer generated by the generator is the same as the pseudo-label and the verification label output by the validator is 1, the accuracy reward of the sample is 1; when the answer generated by the generator is the same as the pseudo-label and the verification label output by the validator is 0, the accuracy reward of the sample is 0; when the answer generated by the generator is different from the pseudo-label and the verification label output by the validator is 1, the accuracy reward of the sample is 0; when the answer generated by the generator is different from the pseudo-label and the verification label output by the validator is 0, the accuracy reward of the sample is 1.

[0058] In this embodiment, the total reward is represented as ,in The format reward is 1 when the output format of the sample conforms to the preset structural constraints, and 0 otherwise. The reward is for accuracy. The total reward for each sample. Subtract the average of the total rewards within the group Divide by the standard deviation of the total reward within the group This allows us to obtain the advantage value for each sample:

[0059]

[0060] By maximizing the objective function corresponding to this advantage value, the large language model The parameters are updated to improve the probability of the model generating high-quality reasoning paths and correct answers.

[0061] It should also be noted that, in this invention, the number of inference paths sampled by the generator... Validate budget threshold The number of corrective reasoning results in the self-correction phase While maintaining balance, certain adjustments can be made. Furthermore, the validator... The threshold for consensus filtering when validating inference samples ( The value can be adjusted to 0.6-0.7 depending on the task type to improve the purity of the pseudo-label. When constructing negative validation samples, in addition to the method provided in this invention, "lowest frequency negative samples" can be changed to "random negative samples" to achieve similar results.

[0062] S6. Input the target query into the optimized large language model. The optimized large language model acts as a generator to reason about the target query and generate an output containing the answer and reasoning path. When it is necessary to determine whether the output is correct, the optimized large language model acts as a validator to verify the output and obtain a judgment result indicating whether the output is correct.

[0063] It should be noted that in this invention, reasoning queries, answers, and target queries are all in text format.

[0064] To better demonstrate the specific implementation and technical effects of the present invention, the label-free reasoning enhancement method based on generator-verifier co-evolution shown in steps S1 to S6 of the above preferred implementation is applied to a specific example.

[0065] Example

[0066] The implementation process of the label-free reasoning enhancement method based on generator-verifier co-evolution used in this embodiment is as described above and will not be repeated here.

[0067] This invention is validated on several base models, including Qwen3-1.7B-Base, Llama-3.2-3B-Instruct, and Qwen2.5-7B. The training set uses the MATH dataset, which contains a large number of mathematical reasoning tasks. The test set uses three mathematical reasoning test sets: MATH500, AMC, and AIME24, and one cross-domain question answering test set: GPQA-Diamond. The baseline is compared to the base model and the current mainstream unlabeled reinforcement learning method TTRL. The baseline samples 64 inference paths for majority voting and 32 for training, according to the original configuration. The method of this invention samples 32 inference paths (N=32), has a validation budget of M=8, a self-correction sample size of K=6, and undergoes two rounds of iterative training.

[0068] The method of this invention significantly outperforms the baseline TTRL in both initial and final accuracy across multiple test sets, with an average initial accuracy improvement of approximately 2.7%-3.8% and an average final accuracy improvement of approximately 4.7%-5.7%, reaching a difference of 8.9% on the AMC task. Furthermore, the method of this invention exhibits cross-domain generalization. On the GPQA-Diamond test set, the final accuracy of all baseline models after self-correction was lower than their initial accuracy, while the final accuracy of the method of this invention (33.6%-37.2%) was higher than the initial accuracy (32.9%-36.2%). Moreover, as shown in Table 2, the self-validation accuracy of the method of this invention achieved an overall improvement of approximately 20%-40% across multiple test sets, demonstrating that the joint optimization process greatly enhances the model's ability as a validator.

[0069] Table 1. Initial accuracy / Final accuracy of the present invention on multiple test sets

[0070]

[0071] Table 2. Self-validated accuracy of the present invention on multiple test sets

[0072]

[0073] It is understood that the label-free reasoning enhancement method based on generator-verifier co-evolution described in S1-S6 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the label-free reasoning enhancement method based on generator-verifier co-evolution provided in the above embodiments. This product includes a computer program / instruction that, when executed by a processor, can implement the label-free reasoning enhancement method based on generator-verifier co-evolution as described in the above embodiments.

[0074] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the label-free reasoning enhancement method based on generator-verifier co-evolution provided in the above embodiments, such as... Figure 3 As shown, it includes a memory and a processor;

[0075] The memory is used to store computer programs;

[0076] The processor is configured to implement the label-free reasoning enhancement method based on generator-verifier co-evolution in the above embodiments when executing the computer program.

[0077] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0078] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the generator-verifier co-evolution-based label-free reasoning enhancement method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the generator-verifier co-evolution-based label-free reasoning enhancement method in the above embodiments.

[0079] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S6.

[0080] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0081] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0082] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A label-free reasoning enhancement method based on generator-verifier co-evolution, characterized in that, Includes the following steps: S1. For each reasoning query in the reasoning task dataset, the large language model to be enhanced for reasoning is first used as a generator to perform multiple reasoning operations on the reasoning query. After each reasoning operation, a reasoning result containing the answer and reasoning path is generated. Thus, the reasoning query and each reasoning result are used as a reasoning sample to form a reasoning set. S2. By majority voting, draft the pseudo-labels corresponding to the inference query from the inference results, and divide the inference set into positive and negative sample groups according to the pseudo-labels; S3. Using the large language model as a validator, the positive sample group is validated, and the positive sample group that passes the validation and the pseudo-label of the inference query are retained to form a positive validation set; S4. The answer with the lowest frequency in the negative sample group is taken as the verification answer. The verifier generates multiple verification results corresponding to the verification answer and a verification label for each verification result. The inference query, verification result and verification label are used as a negative verification sample to form a negative verification set. The number of negative verification samples is the same as the number of positive verification samples. S5. Construct comparative training samples based on positive and negative validation sets, and correct the inference results of all samples that fail validation in the two validation sets, in order to jointly optimize the error recognition ability of the large language model as a validator and the inference ability of the large language model as a generator. S6. Input the target query into the optimized large language model. The optimized large language model acts as a generator to reason about the target query and generate an output containing the answer and reasoning path. When it is necessary to determine whether the output is correct, the optimized large language model acts as a validator to verify the output and obtain a judgment result indicating whether the output is correct.

2. The label-free reasoning enhancement method based on generator-verifier co-evolution as described in claim 1, characterized in that, In step S2, the answer with the highest frequency in the reasoning set is used as the pseudo-label. For each reasoning result in the reasoning set, if the answer in the reasoning result is consistent with the pseudo-label, the reasoning sample corresponding to the reasoning result is added to the positive sample group. If the answer in the reasoning result is inconsistent with the pseudo-label, the reasoning sample corresponding to the reasoning result is added to the negative sample group.

3. The label-free reasoning enhancement method based on generator-verifier co-evolution as described in claim 1, characterized in that, The specific process of step S3 is as follows: S31. When the number of inference samples in the positive sample group is greater than or equal to the preset verification budget threshold. At that time, random sampling was performed from the positive sample group. The positive sample group consists of a set of inference samples to be verified. When the number of inference samples in the positive sample group is less than the preset verification budget threshold, the set of inference samples in the positive sample group consists of all the inference samples in the positive sample group. S32. After validating each inference sample in the validation set, the validator outputs a binary validation label; where a validation label of 1 indicates that the inference of the sample is correct and passes the validation; a validation label of 0 indicates that the inference of the sample is incorrect and fails the validation. S33. When the number of inference samples with verification label 1 in the set to be verified exceeds half of its own number of inference samples, retain the pseudo label and positive sample group corresponding to the inference query, and use the inference sample and corresponding verification label in the positive sample group as a positive verification sample to form a positive verification set.

4. The label-free reasoning enhancement method based on generator-verifier co-evolution as described in claim 1, characterized in that, In step S5, for all samples with a verification label of 0 in both the positive and negative verification sets, the generator generates a new sample based on the inference query, the inference result, and the verification label. Correcting the reasoning result, To correct the number of inference results.

5. The label-free reasoning enhancement method based on generator-verifier co-evolution as described in claim 1, characterized in that, In step S5, the large language model is optimized using the answer-anchored group relative strategy optimization algorithm, and the total reward is obtained by adding the format reward and the accuracy reward. For the inference results of positive verification samples in the positive verification set, the pseudo-labels are grouped and the within-group dominance value is calculated. For the corrected inference results and the verification results of negative verification samples in the negative verification set, standard prefix grouping is used and the within-group dominance value is calculated. Based on the objective function of the group relative strategy optimization algorithm that maximizes the within-group dominance value, the parameters of the generator and the validator are jointly updated.

6. The label-free reasoning enhancement method based on generator-verifier co-evolution as described in claim 5, characterized in that, The accuracy reward for the generator is calculated as follows: for each sample, if the answer generated by the generator is the same as the pseudo-label, the accuracy reward for that sample is 1; otherwise, the accuracy reward is 0.

7. The label-free reasoning enhancement method based on generator-verifier co-evolution as described in claim 5, characterized in that, The accuracy reward for the validator is calculated as follows: For each sample, when the answer generated by the generator is the same as the pseudo-label and the validator outputs a verification label of 1, the accuracy reward for that sample is 1; when the answer generated by the generator is the same as the pseudo-label and the validator outputs a verification label of 0, the accuracy reward for that sample is 0; when the answer generated by the generator is different from the pseudo-label and the validator outputs a verification label of 1, the accuracy reward for that sample is 0; when the answer generated by the generator is different from the pseudo-label and the validator outputs a verification label of 0, the accuracy reward for that sample is 1.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the label-free reasoning enhancement method based on generator-verifier co-evolution as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the label-free reasoning enhancement method based on generator-verifier co-evolution as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the label-free reasoning enhancement method based on generator-verifier co-evolution as described in any one of claims 1 to 7.

Images