Method for training difficulty assessment model, storage medium and computing device

Abstract

A method for training a difficulty assessment model, a storage medium and a computing device, the difficulty assessment model being a large language model, the method comprising: obtaining a training sample, the training sample comprising a reasoning instruction and a difficulty tag corresponding to the reasoning instruction; evaluating the difficulty of the reasoning instruction by using a difficulty evaluation model to determine an output result, wherein the output result at least comprises the comprehensive prediction difficulty; a first reward is determined according to the difference between the comprehensive prediction difficulty and the difficulty label, a second reward is determined according to a preset content constraint and an output result, the content constraint shows a plurality of specified content items required to be contained in the output result, and the specified content items comprise comprehensive prediction difficulty items and sub-prediction difficulty items corresponding to the evaluation dimensions; and parameter adjustment is performed on the difficulty assessment model according to the first reward and the second reward, and rule-based reinforcement learning is introduced, so that the output result of the difficulty assessment model for the reasoning instruction not only comprises the comprehensive prediction difficulty, but also comprises an analysis process for obtaining the comprehensive prediction difficulty.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of data processing technology, and in particular relate to a method, storage medium, and computing device for training a difficulty assessment model. Background Technology

[0002] Instruction pairs (typically, an instruction pair consists of an inference instruction and the correct result corresponding to that inference instruction, and the inference instruction includes at least the question to be processed) are commonly used corpora in the training process of large language models, and are widely used in stages such as model parameter tuning and model inference ability evaluation.

[0003] On the one hand, when adjusting the parameters of a large language model, technicians can select instruction pairs that are compatible with the current reasoning ability of the large language model based on the difficulty level of the instruction pairs, and make targeted parameter adjustments to the large language model.

[0004] On the other hand, when evaluating the reasoning ability of large language models, technicians often use reasoning instructions of varying difficulty to test the large language models in order to accurately assess the capability boundaries of the large language models' reasoning ability.

[0005] The prerequisite for all of these processes is a relatively standardized and objective difficulty assessment of the reasoning instructions beforehand. Currently, the difficulty assessment of reasoning instructions often relies on expert scoring. However, when faced with a large number of reasoning instructions to be assessed, this human-based difficulty assessment is limited by human efficiency, making it difficult to simultaneously achieve high efficiency and high completion (e.g., recording the assessment process in written form while simultaneously determining the predicted difficulty). This consequently affects the reference value of the difficulty levels obtained from the assessment.

[0006] Therefore, there is an urgent need for a solution to train a difficulty assessment model, in order to at least partially solve the above problems. Summary of the Invention

[0007] This specification provides an embodiment of a method, storage medium, and computing device for training a difficulty assessment model, including:

[0008] This specification provides, in a first aspect, a method for training a difficulty assessment model, wherein the difficulty assessment model is a large language model, and the method includes:

[0009] Obtain training samples, which include inference instructions and difficulty labels corresponding to the inference instructions;

[0010] The difficulty of the inference instruction is evaluated using the difficulty assessment model, and the output result is determined. The output result includes at least the comprehensive prediction difficulty.

[0011] The first reward is determined based on the difference between the comprehensive prediction difficulty and the difficulty label, and the second reward is determined based on the preset content constraints and the output results. The content constraints indicate several specified content items that the output results need to include, and the specified content items include the comprehensive prediction difficulty item and the sub-prediction difficulty items corresponding to each evaluation dimension.

[0012] Based on the first reward and the second reward, the parameters of the difficulty assessment model are adjusted.

[0013] A second aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0014] A third aspect of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in the first aspect.

[0015] The technical solution for training a difficulty assessment model provided in this embodiment introduces rule-based reinforcement learning, which enables the difficulty assessment model to output not only the comprehensive predicted difficulty but also the analysis process for obtaining the comprehensive predicted difficulty, thereby completing the difficulty assessment of the inference instruction with high efficiency and high quality. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating the technical concept of a training difficulty assessment model in one embodiment of this specification;

[0018] Figure 2 This is a flowchart illustrating a method for training a difficulty assessment model in one embodiment of this specification.

[0019] Figure 3 This is a flowchart illustrating a sample preparation stage in one embodiment of this specification. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0021] To address the various problems associated with relying on human intervention for difficulty assessment of inference instructions, this embodiment employs rule-based reinforcement learning to train a large language model. During training, the large language model simultaneously learns the ability to assess the difficulty of inference instructions and to explain the basis for such assessments. Consequently, the trained large language model can perform efficient and high-quality difficulty assessments of inference instructions.

[0022] Figure 1 This document illustrates a schematic diagram of the technical concept of a training difficulty assessment model according to one embodiment. First, it should be noted that in this embodiment, the difficulty assessment model can be a pre-trained large language model, thus possessing a certain level of text processing capability before training in this embodiment. The training process of the difficulty assessment model is described below using one iteration of the training process as an example (each iteration can utilize multiple training samples to train the difficulty assessment model, or it can use only one training sample; this embodiment only uses one training sample as an example).

[0023] During the training of the difficulty assessment model, firstly, inference instructions and their corresponding difficulty labels can be obtained as training samples. The difficulty labels of these instructions can be determined manually or by referencing expert model predictions for those instructions; this specification does not impose any restrictions. After obtaining the inference instructions, they and preset prompts can be input into the difficulty assessment model to obtain the output. These prompts may include content constraints on the output of the difficulty assessment model, enabling the model to perform the corresponding difficulty assessment task based on the prompts.

[0024] Generally, as a fundamental training objective, the output of this difficulty assessment model should include the comprehensive predicted difficulty of the reasoning instruction. However, the comprehensive predicted difficulty, as a predicted value for the difficulty level of the reasoning instruction, is often represented as a scalar (e.g., the value range of the comprehensive predicted difficulty can be 0-100, with higher values ​​indicating higher difficulty) or a categorical data point (e.g., the comprehensive predicted difficulty can be intuitively represented as "high difficulty," "medium difficulty," or "low difficulty"). In fact, the reasoning difficulty of a reasoning instruction is often difficult to measure simply using a "difficulty level" on a uniform scale. Under different assessment dimensions such as "the number and cross-complexity of the disciplines involved in the reasoning instruction," "the depth and cutting-edge nature of the professional knowledge required to assess the reasoning instruction," and "the complexity and abstraction of the reasoning process," even two reasoning instructions rated at the same difficulty level can exhibit significant differences. Simply assessing the difficulty level of the reasoning instruction would undoubtedly result in an overly simplistic assessment, lacking guidance for subsequent applications. To enhance the interpretability of the overall prediction difficulty and provide more diverse and refined application directions for inference instructions, the content constraints of this embodiment may also indicate the sub-prediction difficulty items required to be included in the output results for each evaluation dimension. Therefore, under ideal conditions (or after training), the output results of this difficulty evaluation model may also include the sub-prediction difficulty corresponding to each evaluation dimension.

[0025] Of course, on the one hand, as the most basic task objective, the comprehensive prediction difficulty needs to comprehensively and accurately represent the difficulty level of each inference instruction; on the other hand, in the early stages of training (or when the number of training iterations is small), the output of the difficulty assessment model may not conform to the content constraints of the prompt word. Therefore, based on these two requirements, this embodiment establishes a rule-based reward function. The first reward is determined according to the difference between the comprehensive prediction difficulty and the difficulty label, and the second reward is determined according to the degree of matching between the output and the content constraints. Combining the first and second rewards, reinforcement learning methods are used to adjust the parameters of the difficulty assessment model. Thus, during the iterative training process, the difficulty assessment model gradually learns the ability to predict the difficulty level of inference instructions and simultaneously provide analysis processes corresponding to multiple assessment dimensions (i.e., providing sub-predicted difficulty corresponding to each assessment dimension).

[0026] Figure 2 This document illustrates a flowchart of a method for training a difficulty assessment model according to one embodiment of this specification. This method can be executed by a computing device or server with computing capabilities, and this specification does not impose any limitations on it. Figure 2 The method shown illustrates the steps required for one round of parameter tuning in a difficulty assessment model. The complete training process can be achieved by iteratively executing this method for several rounds, which includes:

[0027] Step S201: Obtain training samples, which include inference instructions and difficulty labels corresponding to the inference instructions.

[0028] Specifically, the reasoning instructions are in text form, and each instruction includes at least one problem to be reasoned about. This problem can be a mathematical problem, a physical problem, a coding problem, a logical reasoning problem, or any other verifiable problem with an objective answer. Of course, the reasoning instruction may also include background information corresponding to the problem to be reasoned about, but this specification does not impose any restrictions on this.

[0029] In addition, to ensure the availability of the first reward in subsequent steps, the inference instruction must also have a corresponding difficulty label. The difficulty label of this inference instruction can be determined manually or by referencing the prediction results of an expert model for that inference instruction. The method of obtaining the difficulty label by referencing an expert model will be discussed later and will not be elaborated upon here.

[0030] In practice, a training sample set consisting of several different training samples (and the difficulty label corresponding to each training sample) can be established in advance, and the training samples obtained in step S201 can be randomly drawn from this training sample set.

[0031] Step S203: Using the difficulty assessment model, assess the difficulty of the reasoning instruction and determine the output result, which includes at least the comprehensive prediction difficulty.

[0032] After obtaining the reasoning instruction, the reasoning instruction can be input into the difficulty assessment model so that the difficulty assessment model can evaluate the difficulty of the reasoning instruction and obtain the corresponding output result.

[0033] In some implementations, preset prompts (referred to as evaluation method prompts in this embodiment) can be used. These prompts, along with inference instructions, are then input into the difficulty assessment model to control the content of the output. In this embodiment, content constraints are indicated by these evaluation method prompts. Specifically, these constraints may include several specified content items required for the output, including a comprehensive prediction difficulty item and sub-prediction difficulty items corresponding to each assessment dimension. Correspondingly, the evaluation method prompts may include the scoring criteria for each assessment dimension.

[0034] As mentioned above, simply predicting the difficulty level is insufficient to fully reflect the characteristics of the reasoning instructions. Therefore, this embodiment introduces content constraints into the difficulty assessment model by introducing assessment method prompts—specifically, the scoring criteria corresponding to each assessment dimension. This aims to ensure that the output of the difficulty assessment model includes sub-predicted difficulties in addition to the overall predicted difficulty, thereby further interpreting the overall predicted difficulty from each assessment dimension.

[0035] In some implementations, the evaluation dimensions specifically include at least one of the following: domain span, professional depth, logical complexity, computational requirements, and information completeness.

[0036] By standardizing the reasoning process, this embodiment establishes an evaluation system based on five dimensions: domain span, professional depth, logical complexity, computational requirements, and information completeness. Specifically, the domain span dimension evaluates the "number and cross-disciplinary complexity of the reasoning instructions"; professional depth evaluates the "depth and cutting-edge nature of the professional knowledge required by the reasoning instructions"; logical complexity evaluates the "complexity and abstraction of the reasoning process"; computational requirements evaluate the "complexity of the numerical calculations and algorithms required by the reasoning instructions"; and information completeness evaluates the "completeness and clarity of the information within the reasoning instructions themselves."

[0037] Furthermore, for any evaluation dimension, the evaluation criterion can be the correspondence between the feature performance of the inference instruction under that evaluation dimension and the sub-difficulty level. Thus, the sub-prediction difficulty corresponding to that evaluation dimension in the output result is the predicted value of the sub-difficulty level of that evaluation dimension.

[0038] To enable the difficulty assessment model to further understand the relationship between each assessment dimension and the final comprehensive prediction difficulty, some implementations may include the weight corresponding to each assessment dimension in the prompt words. Specifically, for any assessment dimension, the weight corresponding to that assessment dimension is the weight of the sub-prediction difficulty corresponding to that assessment dimension when the comprehensive prediction difficulty is obtained by weighted fusion of the sub-prediction difficulties.

[0039] To more clearly illustrate the specific content of the evaluation method prompts, the following example uses evaluation dimensions including domain breadth, professional depth, logical complexity, computational requirements, and information completeness to provide a scoring standard for each evaluation dimension in one embodiment. First, it should be noted that in this embodiment, the difficulty level ranges from 1 to 5, where 1 represents the lowest difficulty and 5 represents the highest difficulty. Correspondingly, the sub-difficulty levels for each evaluation dimension also range from 1 to 5, with 1 representing the lowest difficulty and 5 representing the highest difficulty. Regarding the breadth of the field, the scoring criteria could be: "Involving a single discipline concept – 1; Crossing two disciplines – 2; Integrating multidisciplinary knowledge – 3+"; Regarding the depth of expertise, the scoring criteria could be: "Common sense knowledge – 1; Understanding of required professional terminology – 2; Involving cutting-edge theories or complex principles – 3; Requires specialized knowledge in the field – 4+"; Regarding logical complexity, the scoring criteria could be: "Involving only linear reasoning with 3 or fewer steps – 1; Involving branch condition judgments – 2; Involving multi-layered nested reasoning – 3; Requires proof by contradiction or abstract reasoning – 4+"; Regarding computational requirements, the scoring criteria could be: "No computation or only simple arithmetic required – 1; Requires multi-step algebraic operations – 2; Requires special algorithms – 3; Involves complex numerical analysis – 4+"; Regarding information completeness, the scoring criteria could be: "The question stem contains all conditions – 1; Requires additional reasonable assumptions based on the question stem – 2; The question stem contains unstated implicit conditions – 3+". Therefore, by arranging each assessment dimension, its corresponding scoring criteria, and weights into assessment method prompts, a clear and standardized multi-dimensional difficulty quantification method can be provided for the difficulty assessment model, thereby making the output of the model more interpretable. In the scoring criteria, "+" indicates a score above the corresponding number.

[0040] Step S205: Determine the first reward based on the difference between the comprehensive prediction difficulty and the difficulty label, and determine the second reward based on the preset content constraints and the output result. The content constraints indicate several specified content items that the output result needs to include, and the specified content items include the comprehensive prediction difficulty item and the sub-prediction difficulty items corresponding to each evaluation dimension.

[0041] As mentioned above, the main drawback of existing technologies lies in their inability to balance speed and interpretability in difficulty assessment. In this embodiment, a difficulty assessment model trained on a large language model is used to assess the difficulty of inference instructions, naturally satisfying the speed improvement requirement compared to manual scoring. To meet the interpretability improvement requirement, this embodiment employs rule-based reinforcement learning to adjust the parameters of the difficulty assessment model, determining the comprehensive reward score for each output result by comprehensively considering both the accuracy of the predicted difficulty and the content contained in the output results.

[0042] Specifically, a rule-based reward function is preset, which includes a first reward term and a second reward term. The first reward term is similar to the loss in supervised learning—indicating the difference between the overall predicted difficulty and the difficulty label. The second reward term evaluates the content of the output—indicating the difference between the actual content contained in the output and the preset content constraints. In some implementations, the first reward can be set to a larger value corresponding to a smaller difference between the overall predicted difficulty and the difficulty label; the second reward can be set to a larger value corresponding to the more specified content items contained in the output. This specification does not impose restrictions on the specific formulas for determining the first and second rewards.

[0043] Furthermore, to enhance the interpretability of the output, the specified content items can also include a confidence score item representing the credibility of the overall prediction difficulty. Thus, users can determine the credibility of the overall prediction difficulty of the inference instruction based on the confidence score in the output, and then further process the inference instruction according to the credibility (e.g., filtering based on credibility, manually evaluating inference instructions with low credibility, etc.).

[0044] On the other hand, to facilitate the application of this inference instruction in subsequent processes, the content constraint can also include format constraints for the output results. Thus, in the output results of the trained difficulty assessment model, the instruction content items are arranged in a form that conforms to these format constraints. By processing each inference instruction using this difficulty assessment model, a uniformly formatted output result can be obtained, facilitating subsequent storage and application of the output results.

[0045] It should be noted that step S203 above describes the implementation method of presetting evaluation method prompts to control the content of the output result. In other implementation methods, evaluation method prompts may not be set, only content constraints may be preset, and the content of the output result may be controlled by using the first reward and second reward determined according to the content constraints in steps S205 and S207. Of course, in this implementation method, due to the lack of clear scoring criteria as guidance, the parameter convergence speed of the difficulty assessment model is slower, which will not be elaborated on here.

[0046] Step S207: Adjust the parameters of the difficulty assessment model based on the first reward and the second reward.

[0047] After determining the first and second rewards, they are fused using preset weights (these weights are independent of the weights used in the weighted fusion of the sub-prediction difficulties mentioned earlier) to determine the comprehensive reward corresponding to the inference instruction. Based on this comprehensive reward, the parameters of the difficulty assessment model can be adjusted using reinforcement learning methods.

[0048] To facilitate understanding of the specific process of parameter adjustment, the following is a brief introduction to reinforcement learning methods.

[0049] As mentioned above, reinforcement learning methods typically involve multiple rounds of iterative parameter tuning, such as... Figure 2 The method shown illustrates the steps required for one round. It should be noted that the difficulty assessment model in this embodiment is a large language model. Since an output of a large language model consists of several tokens, each token is generated iteratively based on the preceding tokens in the current round's output. Therefore, the generation process of each token in an output constitutes a (state-action) trajectory required for reinforcement learning, and the comprehensive reward is the reward for the entire trajectory in the current round.

[0050] Furthermore, since the comprehensive reward in this embodiment is determined according to preset rules, unlike the loss in supervised regression, this comprehensive reward is not differentiable relative to the model parameters. Therefore, in step S207, after determining the comprehensive reward for the current round, the policy gradient method can be used to determine the adjustment targets of each parameter in the difficulty assessment model, and the parameters of the difficulty assessment model can be adjusted accordingly. This embodiment does not improve the specific implementation of the policy gradient method; therefore, only a brief introduction to the implementation idea of ​​the policy gradient method is given here—the policy gradient method first establishes an expected reward function, which represents the expected value of the comprehensive reward obtained under different model parameters. Then, the actual reward corresponding to the current round is substituted into the expected reward function, and the gradient of the expected reward function is estimated using Monte Carlo simulation sampling. Based on the determined gradient, the parameters of the difficulty assessment model can be adjusted, thereby avoiding the back-derivative calculation in the training process of supervised regression, which is suitable for the parameter adjustment using comprehensive rewards in this embodiment.

[0051] This specification does not restrict the specific implementation of the policy gradient method.

[0052] like Figure 2 The method shown is to train a difficulty assessment model. By introducing rule-based reinforcement learning, the output of the difficulty assessment model for reasoning instructions can include not only the comprehensive predicted difficulty but also the analysis process for obtaining the comprehensive predicted difficulty. This can complete the difficulty assessment of reasoning instructions with high efficiency and high quality.

[0053] In addition to the above, the process of training the difficulty assessment model faces another challenge—how to define an "ideal" difficulty level to obtain the labels corresponding to the training samples. If expert ratings are used as labels, the label determination process is still influenced by subjective factors, and consequently, the credibility and universality of the labels will be questioned. Therefore, this embodiment provides a more objective method for defining difficulty levels—determining the corresponding difficulty level based on the reasoning accuracy of the expert model for reasoning instructions. The expert model can be a state-of-the-art (SOTA) large language model. For example, the expert model can be a large language model that ranks highly in various benchmark tests.

[0054] Figure 3 A schematic flowchart of a sample preparation stage in one embodiment of this specification is shown. Figure 3 As shown, the following section uses a training sample (inference instruction) as an example to briefly introduce the sample preparation stage. Figure 3 As shown, in the sample preparation stage, several expert models (one or more) are first used to reason about the inference instruction. Each expert model can perform independent reasoning multiple times for the instruction. Assuming there are n expert models, and each model performs m independent reasoning operations for the instruction, a total of n*m reasoning operations can be performed for the instruction, resulting in n*m reasoning results. Then, based on the correct results corresponding to the instruction, the percentage of correct reasoning results out of the n*m ​​results can be determined. Finally, based on the number of correct reasoning results (c) and the total number of reasoning results (n*m), the reasoning accuracy (c / (n*m)) for the instruction can be determined.

[0055] Therefore, by pre-setting a training sample set consisting of several training samples and performing the above process on each training sample in the training sample set, the inference accuracy corresponding to each training sample can be determined. Furthermore, based on the inference accuracy corresponding to each training sample, the difficulty level corresponding to each training sample can be determined, serving as the label for each training sample.

[0056] In some implementations, the training samples in the training sample set can be sorted according to their corresponding inference accuracy, and the difficulty level of each training sample can be determined based on its ranking. To illustrate the specific method for determining the difficulty level, continuing from the previous section on difficulty level division—"the difficulty level range is set to 1-5, where 1 represents the lowest difficulty and 5 represents the highest difficulty"—for example, the difficulty level corresponding to the 20% of training samples with the lowest inference accuracy can be set to 5, and so on… while the difficulty level corresponding to the 20% of training samples with the highest inference accuracy can be set to 1. Of course, the correspondence between ranking and difficulty level can be set by the implementer of this embodiment, and this specification does not impose any restrictions here.

[0057] In some implementations, a pre-defined correspondence between inference accuracy and difficulty level can be used to determine the difficulty level for each training sample (for example, in some implementations, the correspondence between inference accuracy and difficulty level can be shown in Table 1).

[0058] Reasoning accuracy: Difficulty level: 0-20％ 5 (Highest Difficulty) 20％-40％ 4 (Higher difficulty) 40％-60％ 3 (Medium Difficulty) 60％-80％ 2 (Lower Difficulty) 80％-100％ 1 (Lowest Difficulty)

[0059] Table 1

[0060] To improve the data quality of the training sample set after the difficulty levels are determined, some implementations involve data cleaning after determining the prediction accuracy for each training sample. Specific methods for data cleaning include: removing training samples with excessively low prediction accuracy; removing training samples with unclear descriptions; and optimizing the proportion of training samples at each difficulty level (to ensure a more even distribution of training samples at each difficulty level after data cleaning, allowing the difficulty assessment model to fully learn its difficulty assessment capabilities for each level of training sample). This specification does not impose any limitations on these methods.

[0061] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them outstanding generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0062] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0063] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0064] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0065] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0066] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0067] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0068] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0069] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0073] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0074] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0075] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0078] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for training a difficulty assessment model, wherein the difficulty assessment model is a large language model, the method comprising: Obtain training samples, which include inference instructions and difficulty labels corresponding to the inference instructions; Using the aforementioned difficulty assessment model, the reasoning difficulty of the reasoning instruction is assessed, and the output result is determined. The output result includes at least the comprehensive prediction difficulty. The first reward is determined based on the difference between the comprehensive prediction difficulty and the difficulty label, and the second reward is determined based on the preset content constraints and the output results. The content constraints indicate several specified content items that the output results need to include, and the specified content items include the comprehensive prediction difficulty item and several sub-prediction difficulty items corresponding to the evaluation dimensions. Based on the first reward and the second reward, the parameters of the difficulty assessment model are adjusted through reinforcement learning.

2. The method as described in claim 1, wherein, The difficulty label corresponding to the reasoning instruction is determined according to the following steps: Based on the preset number of repetitions k, the expert model performs k independent inferences according to the inference instructions to determine k inference results; Based on the k inference results and the correct results corresponding to the inference instruction, the inference accuracy rate corresponding to the inference instruction is determined; Based on the reasoning accuracy rate, determine the difficulty label corresponding to the reasoning instruction.

3. The method as described in claim 2, wherein, Based on a preset number of repetitions k, an expert model performs k independent inferences according to the inference instructions to determine k inference results, specifically including: Based on a preset number of repetitions k, multiple expert models with different parameters are used to perform k independent inferences according to the inference instructions to determine k inference results.

4. The method as described in claim 1, wherein the difficulty assessment model is used to assess the difficulty of the reasoning instruction and determine the output result, specifically includes: The reasoning instructions and preset evaluation method prompts are input into the difficulty evaluation model to obtain the output of the difficulty evaluation model. The evaluation method prompts indicate the content constraints and the scoring criteria corresponding to each evaluation dimension.

5. The method of claim 4, wherein, The content constraints also include: format constraints for the output results, which indicate how each specified content item is arranged in the output results.

6. The method of claim 4, wherein, The evaluation method prompt also indicates the weight of each evaluation dimension's sub-prediction difficulty in the overall prediction difficulty.

7. The method of claim 1, wherein, The aforementioned evaluation dimensions specifically include at least one of the following: domain span, professional depth, logical complexity, computational requirements, and information completeness.

8. The method of claim 1, wherein, The specified content item also includes a confidence score item that represents the credibility of the overall prediction difficulty.

9. The method of claim 1, wherein, Based on the first reward and the second reward, the parameters of the difficulty assessment model are adjusted using reinforcement learning, including: The first reward and the second reward are weighted and combined to obtain the comprehensive reward; The comprehensive reward is substituted into the pre-constructed expected reward function as the actual reward for the current round, and the gradient of the expected reward function is estimated using Monte Carlo simulation sampling; where the expected reward function represents the expected value of the comprehensive reward obtained under different model parameters; The parameters of the difficulty assessment model are adjusted based on the determined gradient.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.

11. A computing device comprising a memory and a processor, the memory storing executable code, wherein the processor, when executing the executable code, implements the method as claimed in any one of claims 1-9.

Images