GRPO-like self-optimization prompt generation method and system
By using a GRPO-like self-optimizing suggestion generation method, and leveraging semantic advantages to describe and update external experience bases, the high accuracy requirements of large language models in specific professional fields are addressed, achieving low-cost, efficient model optimization and cross-domain applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ZHONGKE JINDEZHU INTELLIGENT TECH CO LTD
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-10
AI Technical Summary
Existing reinforcement learning methods suffer from high computational costs, overfitting risks and limited generalization in large language models, as well as strong data dependence, making it difficult to achieve high accuracy requirements in specific professional fields.
A GRPO-like self-optimizing prompt generation method is adopted. Candidate outputs are generated through a large language model, and a reward model is used to evaluate and generate semantic advantage descriptions. These descriptions are then transformed into modification operation instructions to update the external experience base, serving as contextual prompts to guide the model in optimizing its output.
It significantly reduces training and implementation costs, improves adaptability and generalization ability, avoids the risk of overfitting, and enables flexible transfer across domains and tasks.
Smart Images
Figure CN121835807A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of artificial intelligence and natural language processing, and particularly relates to a GRPO-like self-optimization prompt generation method and system. BACKGROUND
[0002] In recent years, large language models (LLMs) have shown excellent performance in various tasks. However, in specific professional fields (such as medical diagnosis, complex mathematical reasoning, and code generation), directly using general large models often cannot meet the high-precision requirements. Existing solutions mainly rely on supervised fine-tuning (SFT) or reinforcement learning (RL), such as PPO (Proximal Policy Optimization) or GRPO (Group Relative Policy Optimization).
[0003] Although existing reinforcement learning methods are effective, they have the following significant drawbacks: 1. High computational cost: For models with a large number of parameters (such as 70B or more), reinforcement learning training requires tens of thousands of GPU hours and has extremely high requirements for hardware memory.
[0004] 2. Risk of overfitting and limited generalization: Reinforcement learning based on parameter updates is prone to overfitting to specific training data, leading to degradation of the model's ability on other general tasks.
[0005] 3. Data dependence: Traditional reinforcement learning requires a large amount of high-quality labeled data or reward models, and the maintenance cost of fine-tuned models is high, making it difficult to be compatible with general API-type models (which cannot modify parameters).
[0006] Therefore, there is an urgent need for a self-optimization prompt generation scheme that can achieve similar GRPO-like strategy self-evolution effects without expensive parameter gradient updates, and can be flexibly adapted to various general large models. SUMMARY
[0007] The purpose of the present application is to provide a GRPO-like self-optimization prompt generation method and system.
[0008] According to a first aspect of the present application, a GRPO-like self-optimization prompt generation method is provided, which comprises: Step S1: processing an input query sentence using a large language model to generate a group of candidate outputs to form an output group; Step S2: evaluating each of the candidate outputs through a reward model to obtain a reward score corresponding to each of the candidate outputs; Step S3: the large language model performs semantic comparison on the output group based on the reward score corresponding to each of the candidate outputs, and generates a semantic advantage description in natural language form, wherein the semantic advantage description can be used to replace the numerical advantage function in the reinforcement learning model; Step S4: converting the semantic advantage description into a modification operation instruction, and updating an external experience library based on the modification operation instruction, wherein the external experience library is used to independently store policy experience; Step S5: injecting the policy experience in the updated external experience library as a context prompt into the input of the large language model, guiding the large language model to output a new output group that is biased towards a high reward score when processing a new query sentence.
[0009] Optionally, in the step S1, the large language model ensures that a group of different candidate outputs are generated by setting the sampling temperature to a preset temperature value or using a Top-P sampling strategy.
[0010] Optionally, in the step S1, the large language model is a parameter-frozen general model or a closed-source model called through an API.
[0011] Optionally, in the step S2, the reward model is a score record based on artificial feedback or a pre-trained neural network reward model.
[0012] Optionally, the step S3 specifically includes: identifying a winning candidate output with a high reward score and a disadvantaged candidate output with a low reward score from the output group based on the reward score corresponding to each of the candidate outputs; the large language model performs semantic comparison on the winning candidate output with a high reward score and the disadvantaged candidate output with a low reward score, and obtains reasons for the advantages and disadvantages of the outputs and improvement experience; generating a text containing specific error attribution and correction suggestions based on the reasons for the advantages and disadvantages of the outputs and the improvement experience, to obtain the semantic advantage description.
[0013] Optionally, in the step S4, the modification operation instruction is an addition instruction, a modification instruction, a deletion instruction, or a retention instruction.
[0014] Optionally, in the step S4, the external experience library stores the policy experience in the form of structured text or knowledge graph in units of natural language rules.
[0015] According to a second aspect of the present application, a self-optimizing prompt generation system of a GRPO-like system is provided, the system comprising: A first processing module configured to process an input query statement using a large language model to generate a group of candidate outputs to form an output population; A second processing module configured to evaluate each of the candidate outputs by a reward model to obtain a reward score corresponding to each of the candidate outputs; A third processing module configured to perform semantic comparison on the output population based on the reward score corresponding to each of the candidate outputs by the large language model to generate a semantic advantage description in natural language form, wherein the semantic advantage description can be used to replace a numerical advantage function in a reinforcement learning model; A fourth processing module configured to convert the semantic advantage description into a modification operation instruction and update an external experience library based on the modification operation instruction, wherein the external experience library is used to store policy experience independently; A fifth processing module configured to inject the policy experience in the updated external experience library as a context prompt into the input of the large language model to guide the large language model to output a new output population that is biased towards a high reward score when processing a new input query statement.
[0016] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the self-optimizing prompt generation method of a GRPO-like system according to any one of the first aspect of the present application when executing the computer program.
[0017] According to a fourth aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores a computer program, and the computer program implements the steps of the self-optimizing prompt generation method of a GRPO-like system according to any one of the first aspect of the present application when executed by a processor.
[0018] The present application has the following beneficial effects: As can be seen from the above solution, the self-optimizing prompt generation method and system of a GRPO-like system provided by the embodiments of the present application have the following beneficial effects: The GRPO-like self-optimization prompt generation method provided by the application is first beneficial to significantly reduce the training and implementation cost and improve the landing feasibility: by abandoning the gradient update mechanism that must be relied on in traditional reinforcement learning, all strategy optimization effects are realized through the context operation in the reasoning stage, without the need to invest a large amount of computing power resources for model parameter training iteration, thereby greatly reducing the computing power consumption, shortening the optimization period, and reducing the dependence on high-configuration hardware devices, effectively controlling the cost threshold of technology landing, and being more easily applied in various resource conditions; secondly, it has wide adaptability and can be compatible with mainstream general large models: using the model parameter freezing design idea, without the need for targeted fine-tuning or modification of the large language model itself, the method can be directly applied to DeepSeek, Qwen and other mainstream general large language models, thereby significantly improving the universality and adaptation range of the technical solution, reducing the adaptation cost of cross-model application, and facilitating the rapid docking of existing model resources in different scenarios; thirdly, it is beneficial to avoid overfitting risk and enhance the generalization ability and cross-domain migration: an external experience library independent of the model parameter space is constructed, and the strategy experience is stored in natural language rules and can be flexibly combined, thereby fundamentally avoiding the decline in generalization ability caused by excessive parameter fitting in the model training process, and the independence of the external experience library enables it to be independent of the original model and task scenario, realizing flexible migration and reuse across domains and tasks, and further improving the application range and practical value of the technical solution. BRIEF DESCRIPTION OF DRAWINGS
[0019] Figure 1 A flowchart of a GRPO-like self-optimization prompt generation method according to an embodiment is provided. Figure 2 A specific implementation process diagram of a GRPO-like self-optimization prompt generation method according to an embodiment is provided. Figure 3 A structure diagram of a GRPO-like self-optimization prompt generation system according to an embodiment is provided. Figure 4 An electronic device according to an embodiment is provided. DETAILED DESCRIPTION
[0020] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.
[0021] According to a first aspect of the present application, a GRPO-like self-optimization prompt generation method is provided, as shown inFigure 1 As shown, the method comprises, Step S1: processing the input query sentence by using a large language model to generate a group of candidate outputs to form an output group; Step S2: evaluating each candidate output by using a reward model to obtain the reward score corresponding to each candidate output; Step S3: the large language model performs semantic comparison on the output group based on the reward score corresponding to each candidate output to generate a semantic advantage description in natural language form, wherein the semantic advantage description can be used to replace the numerical advantage function in the reinforcement learning model; Step S4: converting the semantic advantage description into a modification operation instruction, and updating the external experience library based on the modification operation instruction, wherein the external experience library is used to independently store policy experience; Step S5: injecting the policy experience in the updated external experience library into the input of the large language model as a context prompt to guide the large language model to output a new output group that is biased towards a high reward score when processing a new query sentence.
[0022] Optionally, in the GRPO-like self-optimizing prompt generation method of the embodiment of the application, the large language model ensures that a group of different candidate outputs are generated by setting the sampling temperature to a preset temperature value or using a Top-P sampling strategy.
[0023] In this embodiment, the candidate outputs are a set of natural language responses generated by the large language model through a specific sampling strategy and are not identical. Specifically, the model sets the sampling temperature to a preset value (such as 0.7-0.9, balancing randomness and semantic rationality), or uses a Top-P sampling strategy (selecting a subset of candidate words with a cumulative probability reaching a preset threshold for sampling) to avoid generating results with repeated semantics or single expression, and ensure the diversity of the output group to support subsequent semantic comparison and advantage mining.
[0024] Specifically, as shown in the formula, Figure 2 In this embodiment, for the input query sentence q, the large language model πθ generates G candidate outputs {O1, O2, …, OG} to form an output group. G} to form an output group.
[0025] Optionally, in the GRPO-like self-optimizing prompt generation method of the embodiment of the application, the large language model is a parameter-frozen general model or a closed-source model called through an API.
[0026] The general model with parameter freezing in this embodiment is a general large language model whose model weight is fixed after completing pre-training or supervised fine-tuning. In the execution process of the method of this embodiment, only the text generation and semantic understanding capabilities of the model are called, and the network parameters of the model itself are not updated and iterated. The closed-source model called through the API is a commercial large language model called through a third-party interface. The complete network architecture and weight file of the model do not need to be deployed locally. The input query sentence can be processed and the candidate output can be generated through a standardized interface. By using the general model with parameter freezing or the closed-source model called through the API, the power consumption and parameter iteration risk of local fine-tuning of the large language model are avoided, and the deployment and landing cost of the method is greatly reduced. A large amount of computing resources is not needed to perform secondary training on the large language model. Only the original text processing capability of the model can be used to generate the candidate output.
[0027] Optionally, in the GRPO-like self-optimizing prompt generation method, the reward model is a pre-trained neural network reward model or a score record based on artificial feedback.
[0028] The score record based on artificial feedback in this embodiment refers to structured score data formed after professional personnel evaluate the quality of the candidate output generated by the large language model. The score record needs to clearly associate the semantic accuracy, logical integrity, and expression fluency of the candidate output, and can be directly used as a basis for determining the reward score. The pre-trained neural network reward model is a neural network model with autonomous evaluation capability trained by large-scale labeled data, which can automatically analyze the semantics of the candidate output and output a quantitative reward score. By using the two types of reward models in this embodiment, different application scenarios can be flexibly adapted. The artificial feedback score record is suitable for scenarios with extremely high evaluation accuracy requirements, and the pre-trained neural network reward model is suitable for scenarios that require efficient and automated evaluation, both of which can provide reliable reward signal support for subsequent semantic comparison and strategy optimization.
[0029] Specifically, as shown in Figure 2 , the reward score r i of each candidate output O i is scored by using an external reward model R in this embodiment.
[0030] Optionally, in the GRPO-like self-optimizing prompt generation method, the specific process in which the large language model performs semantic comparison on the output group based on the reward scores corresponding to each candidate output is as follows: Identify the high-reward candidate output and the low-reward candidate output from the output group based on the reward scores corresponding to each candidate output. The large language model performs semantic comparison on the winning candidate output with high reward score and the inferior candidate output with low reward score, to obtain reasons and improvement experience of the winning and inferior outputs. Based on the reasons and improvement experience of the winning and inferior outputs, a text containing specific error attribution and correction suggestions is generated to obtain a semantic advantage description.
[0031] In this embodiment, the process of semantic comparison of the large language model on the output group based on the reward score is divided into three steps, namely, winning and inferior output screening, semantic difference analysis and advantage description generation, so as to realize the conversion from quantitative score to qualitative semantic advantage, as follows: In the first step, the reward score of each candidate output is used as the screening basis, and a preset score threshold is set (for example, the candidate outputs with the top 30% score are determined as winning candidate outputs, and the candidate outputs with the bottom 30% score are determined as inferior candidate outputs), so as to accurately separate the two types of candidate outputs with significant score difference from the output group; In the second step, the large language model performs deep semantic comparison on the winning candidate output and the inferior candidate output, analyzes the differences between them in core dimensions such as semantic accuracy, logical integrity and demand fit, and then locates the problem roots (such as semantic deviation, logical discontinuity and information missing) of the inferior output and the advantages of the winning output, and extracts reusable improvement experience; In the third step, based on the above analysis results, a structured text containing error attribution and correction suggestions, i.e. semantic advantage description, is generated, wherein the error attribution clearly indicates the specific problem types and causes of the inferior output, and the correction suggestions correspond to the advantage characteristics of the winning output, providing clear semantic basis for subsequent conversion into modification operation instructions.
[0032] Specifically, referring to Figure 2 In this embodiment, the large language model compares the candidate outputs in the group at the semantic level, generates a semantic advantage description A_text in natural language form, and the description can replace the traditional numerical advantage i , that is, the traditional GRPO uses numerical advantage.
[0033] (1) When the reward scores of the candidate outputs in the group are obviously superior and inferior, the large language model generates reason summary and improvement experience through self-reflection, wherein A_text represents the language experience corresponding to the high-quality strategy.
[0034] Optionally, in the GRPO-like self-optimization prompt generation method of the embodiment of the application, the modification operation instruction is an addition instruction, a modification instruction, a deletion instruction or a retention instruction.
[0035] In this embodiment, the four types of instructions jointly constitute an instruction system covering all scene optimization requirements, and can accurately respond to different types of optimization instructions in the semantic advantage description, and are specifically defined as follows: the adding instruction is used to supplement the missing core information in the existing strategy experience; the modifying instruction is used to correct the deviation or non-standard content in the existing strategy experience; the deleting instruction is used to eliminate the redundant and invalid information in the existing strategy experience; and the keeping instruction is used to retain the effective content in the existing strategy experience that meets the high reward output characteristics and does not need to be optimized. The four types of instructions can be used alone or in combination to ensure that the external experience library can be accurately and efficiently updated according to the semantic advantage description, and to provide reliable strategy experience support for subsequent guiding the large language model output to shift to a high reward score.
[0036] Optionally, in the GRPO-like self-optimization prompt generation method, the external experience library stores the strategy experience in the form of a structured text or a knowledge graph in units of natural language rules.
[0037] In this embodiment, the smallest storage unit of the external experience library is a natural language rule expression with complete semantics, and each unit corresponds to an independent strategy experience, ensuring that the strategy experience is understandable and reusable; the structured text storage form can use a preset template to organize the experience, making the experience content clear and easy to quickly search; and the knowledge graph storage form constructs a semantic network of the strategy experience through the association relationship between nodes and edges, and intuitively presents the logical association between different experiences. The two storage forms can be flexibly selected.
[0038] Specifically, referring to FIG. 2, Figure 3 In this embodiment, the semantic advantage description A_text is converted into a modifying operation instruction (an adding instruction Add, a modifying instruction Modify, a deleting instruction Delete, or a keeping instruction Keep), so as to update the external experience library E, wherein the external experience library E is used to store refined domain knowledge and strategy experience.
[0039] In subsequent reasoning processing (that is, when processing a new query sentence), the large language model πθ performs conditional generation through πθ(y|q, E), so that the output new output group distribution naturally shifts to a high reward strategy, thereby realizing no-training optimization.
[0040] According to a second aspect of the present application, a GRPO-like self-optimization prompt generation system is provided, referring to FIG. 3, Figure 4 As shown in FIG. 3, the system 300 includes: A first processing module 301 configured to process an input query sentence using a large language model to generate a group of candidate outputs to form an output group. The second processing module 302 is configured to evaluate each candidate output by a reward model to obtain a reward score corresponding to each candidate output. The third processing module 303 is configured to perform semantic comparison on the output group based on the reward score corresponding to each candidate output by the large language model, and generate a semantic advantage description in a natural language form, wherein the semantic advantage description can be used to replace the numerical advantage function in the reinforcement learning model. The fourth processing module 304 is configured to convert the semantic advantage description into a modification operation instruction, and update the external experience library based on the modification operation instruction, wherein the external experience library is used to independently store the policy experience. The fifth processing module 305 is configured to inject the policy experience in the updated external experience library into the input of the large language model as a context prompt, so as to guide the large language model to output a new output group that is biased to a high reward score when processing a new query sentence.
[0041] In summary, compared with the traditional technology, the self-optimization prompt generation method of the GRPO provided by the embodiment of the present application has the following remarkable beneficial effects: 1. Break through the shackles of traditional technology and significantly reduce the training and implementation cost: the method of the embodiment of the present application innovatively converts the "policy gradient optimization" of GRPO into "context space optimization", without updating the model parameters, and discards the gradient update mechanism that must be relied on in the traditional reinforcement learning. By reconfiguring the process of "numerical advantage→policy update" in the traditional reinforcement learning into a new path of "semantic advantage→experience evolution→context injection", all policy optimization effects are realized through the context operation of the external experience library E in the inference stage, without the need to invest a large amount of computing resources for model parameter training iteration, which not only helps to greatly reduce the computing consumption and shorten the optimization period, but also reduces the dependence on high-configuration hardware devices, effectively controls the cost threshold of technology landing, and is easier to popularize and apply under various resource conditions.
[0042] 2. Wide adaptability and more accurate evaluation, compatible with mainstream general-purpose large models: the method of the embodiment of the present application adopts the design idea of model parameter freezing, without the need to fine-tune or modify the large language model itself, and can be directly applied to DeepSeek, Qwen and other mainstream general-purpose large language models, breaking through the dependence limitation of traditional technology on specific model architecture, improving the universality and adaptation range of the technical solution, and reducing the adaptation cost of cross-model application; in addition, the method of the embodiment of the present application innovatively proposes a semantic group advantage mechanism, which replaces the numerical advantage of the traditional GRPO with the natural language evaluation generated by the model itself, and the advantage evaluation is realized relying on the semantic comparison ability of the model on the outputs in the group, which is more adaptive to the natural language processing core characteristics of the general-purpose large model, makes the optimization evaluation more accurate and the explainability stronger.
[0043] 3. Avoid overfitting risk, enhance generalization ability and cross-domain migration: The external experience library E constructed by the method of the embodiment of the application is independent of the model parameter space, the strategy experience is stored in natural language rules and can be flexibly combined, and the generalization ability decline problem caused by parameter overfitting in the model training process can be fundamentally avoided; meanwhile, the independence of the external experience library enables it to be separated from the original model and task scene, to realize flexible migration and reuse across domains and tasks, and to further improve the application breadth and practical value of the technical solution.
[0044] According to a third aspect of the application, an electronic device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the self-optimization prompt generation method of a GRPO-like according to any one of the first aspect of the application when executing the computer program.
[0045] Figure 4 As shown in the structural diagram of an electronic device according to an embodiment of the application, Figure 4 the electronic device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.
[0046] Those skilled in the art can understand, Figure 4 the structure shown in the above-mentioned structure, only the structure of part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.
[0047] According to a fourth aspect of the application, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, and the computer program being executed by a processor to implement the steps of the self-optimization prompt generation method of a GRPO-like according to any one of the first aspect of the application.
[0048] The above is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can also be made several improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.
Claims
1. A self-optimizing suggestion generation method similar to GRPO, characterized in that, The method includes: Step S1: Process the input query statement using a large language model to generate a set of candidate outputs to form an output group; Step S2: Evaluate each candidate output using a reward model to obtain a reward score corresponding to each candidate output; Step S3: The large language model performs semantic comparison on the output group based on the reward scores corresponding to each candidate output, and generates a semantic advantage description in natural language form, wherein the semantic advantage description can be used to replace the numerical advantage function in the reinforcement learning model; Step S4: Transform the semantic advantage description into modification operation instructions, and update the external experience base based on the modification operation instructions, wherein the external experience base is used to independently store policy experience; Step S5: Inject the updated policy experience from the external experience base as contextual cues into the input of the large language model, guiding the large language model to output a new group of outputs that are shifted towards higher reward scores when processing new query statements.
2. The self-optimization suggestion generation method for a GRPO-like system according to claim 1, characterized in that, In step S1, the large language model ensures that a set of distinct candidate outputs are generated by setting the sampling temperature to a preset temperature value or by adopting a Top-P sampling strategy.
3. The self-optimizing suggestion generation method for a GRPO-like system according to claim 1, characterized in that, In step S1, the large language model is either a general model with frozen parameters or a closed-source model called via API.
4. The self-optimizing suggestion generation method for a GRPO-like system according to claim 1, characterized in that, In step S2, the reward model is a rating record based on human feedback or a pre-trained neural network reward model.
5. The self-optimization suggestion generation method for a GRPO-like system according to claim 1, characterized in that, Step S3 specifically includes: Based on the reward scores corresponding to each candidate output, identify the winning candidate output with high reward scores and the inferior candidate output with low reward scores from the output group. The large language model performs semantic comparison between the winning candidate output with high reward scores and the inferior candidate output with low reward scores to obtain the reasons for the superior and inferior outputs and the experience for improvement. Based on the reasons for the output of advantages and disadvantages and the experience of improvement, a text containing specific error attributions and correction suggestions is generated to obtain the semantic advantage description.
6. The self-optimization suggestion generation method for a GRPO-like system according to claim 1, characterized in that, In step S4, the modification operation instruction is an add instruction, a modify instruction, a delete instruction, or a keep instruction.
7. The self-optimizing suggestion generation method for a GRPO-like system according to claim 1, characterized in that, In step S4, the external experience base stores the strategy experience in units of natural language rules and in the form of structured text or knowledge graph.
8. A self-optimizing suggestion generation system similar to GRPO, characterized in that, The system includes: The first processing module is configured to process the input query statement using a large language model to generate a set of candidate outputs to form an output group. The second processing module is configured to evaluate each of the candidate outputs using a reward model to obtain a reward score corresponding to each of the candidate outputs. The third processing module is configured to perform semantic comparison of the output group based on the reward scores corresponding to each candidate output by the large language model, and generate a semantic advantage description in natural language form, wherein the semantic advantage description can be used to replace the numerical advantage function in the reinforcement learning model. The fourth processing module is configured to convert the semantic advantage description into modification operation instructions and update the external experience base based on the modification operation instructions, wherein the external experience base is used to independently store strategy experience. The fifth processing module is configured to inject the updated strategy experience from the external experience base as contextual hints into the input of the large language model, guiding the large language model to output a new output group that is shifted towards higher reward scores when processing new query statements.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the self-optimizing hint generation method for a GRPO-like device as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a self-optimizing suggestion generation method for a GRPO-like system as described in any one of claims 1 to 7.