Question query method, apparatus and device based on adaptive series group, and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of financial science and technology and the field of medical treatment, and discloses a question query method, device and equipment based on a self-adaptive series group and a medium, which are applied to a high-frequency quantitative transaction decision scene or an electronic medical record reasoning scene. Generating a complete thinking chain sequence and an initial answer based on the query question; according to a preset truncation position, truncating the complete thinking chain sequence to generate a truncation subsequence and a basic answer, and constructing a series group and an answer set; calculating a correctness index of the answer set and an average correctness rate of the series group; performing dynamic reward switching according to the average accuracy and a preset threshold value to generate a reward set, and calculating a dominant set; updating the initial strategy model according to the advantage set to obtain a target strategy model; and reasoning based on the to-be-reasoned query question through the target strategy model to generate a target answer. According to the method, the query accuracy and efficiency of the multi-step problem query task are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech and healthcare, and in particular to a question query method, apparatus, device and medium based on adaptive serial groups. Background Technology

[0002] With the rapid development of artificial intelligence technology, pre-trained language models have demonstrated powerful capabilities in natural language processing tasks, particularly excelling in complex multi-step question query and question-answering systems. In multi-step question query tasks in natural language processing, Chain-of-Thought (CoT) can be utilized for question reasoning and querying. As an emerging reasoning paradigm, CoT decomposes complex problems into multiple intermediate steps by simulating the step-by-step reasoning process of humans, significantly improving the model's performance in tasks such as mathematical reasoning and common-sense question answering. For example, applying CoT reasoning and querying methods to high-frequency quantitative trading decision-making scenarios in the fintech field can quickly generate answers to questions related to trading scenarios, reducing trading decision response latency and improving the accuracy of question queries. This CoT reasoning and querying method may also be applied to electronic medical record reasoning scenarios in the medical field, accurately retrieving answers to questions related to inaccurate medical records and improving the efficiency of outpatient medical record processing.

[0003] Existing chain-like reasoning generation methods based on reinforcement learning (such as GRPO) mainly rely on the 0 / 1 reward of the final result, a coarse-grained evaluation method with significant drawbacks. During training, the model generates lengthy reasoning chains containing many irrelevant steps or repetitive content. This redundancy not only wastes computational resources but, more seriously, distracts the model from key reasoning steps, thus reducing the accuracy of the final answer. To address the reasoning efficiency problem, existing techniques attempt to introduce length penalty mechanisms (such as the length penalty strategy of GRPO-λ), but this approach presents a new technical contradiction: applying efficiency-first pressure too early before the model's reasoning ability is fully developed can lead to drastic fluctuations or even crashes in the training process, making it difficult to achieve a balance between training robustness and reasoning efficiency. Furthermore, existing methods lack a dynamic evaluation mechanism for the reasoning quality at different stages of the reasoning chain, failing to intelligently adjust the training strategy based on the model's actual performance, which is also a significant reason for the unsatisfactory training stability. Summary of the Invention

[0004] The purpose of this application is to propose a question query method, apparatus, device and medium based on adaptive serial groups, so as to improve the query accuracy and efficiency of multi-step question query tasks.

[0005] To address the aforementioned technical problems, embodiments of this application provide a problem query method based on adaptive cascade groups, including:

[0006] Obtain a pre-trained language model and a query question, use the pre-trained language model as an initial strategy model, and generate a complete thought chain sequence and an initial answer based on the query question using the initial strategy model;

[0007] The complete thought chain sequence is truncated according to a preset truncation position to generate truncated sub-sequences and corresponding basic answers. Then, a series group and a set of answers corresponding to the series group are generated based on the complete thought chain sequence, the truncated sub-sequences, the initial answers, and the basic answers.

[0008] Calculate the correctness index for each answer in the answer set, and calculate the average correctness rate of the concatenated group based on the correctness index;

[0009] Rewards are dynamically switched based on the average accuracy and preset threshold to generate reward sets for each thought chain, and an advantage function is calculated based on the reward sets to generate an advantage set.

[0010] The model parameters of the initial policy model are updated using gradients based on the advantage set and the concatenation group to obtain the target policy model;

[0011] Obtain the query question to be reasoned, and generate the target answer by reasoning based on the query question through the target strategy model.

[0012] To address the aforementioned technical problems, embodiments of this application provide a problem query device based on an adaptive cascade group, comprising:

[0013] The initial strategy model loading module is used to obtain a pre-trained language model and a query question, use the pre-trained language model as the initial strategy model, and generate a complete thought chain sequence and an initial answer based on the query question using the initial strategy model.

[0014] The thought chain truncation module is used to truncate the complete thought chain sequence according to a preset truncation position, generate truncated sub-sequences and the basic answers corresponding to the truncated sub-sequences, and generate concatenation groups and answer sets corresponding to the concatenation groups based on the complete thought chain sequence, the truncated sub-sequences, the initial answers and the basic answers;

[0015] The correctness index calculation module is used to calculate the correctness index of each answer in the answer set, and calculate the average correctness rate of the series group based on the correctness index.

[0016] The reward dynamic switching module is used to dynamically switch rewards based on the average accuracy and a preset threshold to generate reward sets for each thought chain, and to calculate the advantage function based on the reward sets to generate an advantage set.

[0017] The strategy model training module is used to perform gradient updates on the model parameters of the initial strategy model based on the advantage set and the concatenation group to obtain the target strategy model.

[0018] The model reasoning module is used to obtain the query question to be reasoned and to generate the target answer by reasoning based on the query question through the target strategy model.

[0019] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, such that the one or more processors implement the problem query method based on adaptive cascade groups as described above.

[0020] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the problem query method based on adaptive cascade groups as described above.

[0021] This invention provides a question query method, apparatus, device, and medium based on adaptive concatenation groups. The method includes: acquiring a pre-trained language model and a query question; using the pre-trained language model as an initial strategy model, and generating a complete thought chain sequence and an initial answer based on the query question using the initial strategy model; truncating the complete thought chain sequence according to a preset truncation position to generate truncated sub-sequences and corresponding basic answers; generating concatenation groups and corresponding answer sets based on the complete thought chain sequence, the truncated sub-sequences, the initial answers, and the basic answers; calculating the correctness index of each answer in the answer set, and calculating the average correctness rate of the concatenation group based on the correctness index; dynamically switching rewards based on the average correctness rate and a preset threshold to generate reward sets for each thought chain, and calculating an advantage function based on the reward sets to generate an advantage set; updating the model parameters of the initial strategy model using gradients based on the advantage set and the concatenation group to obtain a target strategy model; acquiring a query question to be reasoned about, and generating the target answer by reasoning based on the query question to be reasoned about using the target strategy model. This invention, through the construction of a concatenated group structure comprising complete thought chains and truncated subsequences, combined with a dynamically switching reward mechanism and position decay weight calculation, achieves refined evaluation and control of the multi-step question query process, thereby improving query efficiency. During training, the reward strategy is intelligently adjusted based on the model's actual performance, avoiding training instability caused by prematurely imposing efficiency constraints and effectively suppressing redundant inference steps, thus improving query accuracy. Attached Figure Description

[0022] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an application environment for a problem query method based on adaptive serial groups in one embodiment of the present invention;

[0024] Figure 2 This is a flowchart illustrating the implementation of the problem query method based on adaptive serial groups provided in this application embodiment;

[0025] Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S1;

[0026] Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S2;

[0027] Figure 5 yes Figure 2 A flowchart illustrating a specific implementation of step S4;

[0028] Figure 6 yes Figure 2 A flowchart illustrating a specific implementation of step S5;

[0029] Figure 7 yes Figure 2 A flowchart illustrating a specific implementation of step S6;

[0030] Figure 8 This is a schematic diagram of a problem query device based on an adaptive cascade group provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0035] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the problem query method based on adaptive serial groups provided in this application embodiment is generally executed by the server, and correspondingly, the problem query device based on adaptive serial groups is generally configured in the server.

[0037] The problem query method based on adaptive cascaded groups provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server receives the query question to be reasoned from the client and generates the target answer based on the query question. In this invention, the server sends the target answer to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0038] The problem query method based on adaptive serial groups provided in this application can be applied to high-frequency quantitative trading decision-making scenarios in the financial technology field, or to electronic medical record reasoning scenarios in the medical field.

[0039] In existing technologies, multi-step reasoning tasks in the field of natural language processing often employ chain-like thought generation methods based on reinforcement learning. These methods rely on the binary reward signal of the final result for model training, making it impossible to quantitatively evaluate the effectiveness of intermediate reasoning steps. When the model generates lengthy thought chains, redundant steps divert computational resources from the attention mechanism, weakening key reasoning components. Existing improvements attempt to compress the inference step length using a fixed-length penalty factor; however, when the model's reasoning ability is not fully established, such rigid constraints can disrupt the stability of the training process, causing gradient oscillations or even model collapse.

[0040] To address these issues, researchers discovered that quality monitoring of intermediate inference steps and dynamic adjustment of training strategies are key to overcoming existing bottlenecks. Analysis revealed that overthinking stems from the model's lack of ability to evaluate inference paths at different stages, while training instability arises from a mismatch between efficiency optimization and capability development. Based on this, a proposal was made to construct multi-granularity inference units during training and establish a phased reward mechanism, enabling the model to automatically adapt to differentiated optimization objectives at different training stages. Therefore, this application proposes a question query method based on adaptive concatenation groups, applied to multi-step reasoning tasks in natural language processing, including: acquiring a pre-trained language model and a query question; using the pre-trained language model as an initial policy model, and generating a complete thought chain sequence and an initial answer based on the query question using the initial policy model; truncating the complete thought chain sequence according to a preset truncation position to generate truncated sub-sequences and corresponding basic answers, and generating concatenation groups and corresponding answer sets based on the complete thought chain sequence, truncated sub-sequences, initial answers, and basic answers; calculating the correctness index of each answer in the answer set, and calculating the average accuracy of the concatenation group based on the correctness index; dynamically switching rewards based on the average accuracy and a preset threshold to generate reward sets for each thought chain, and calculating the advantage function based on the reward sets to generate an advantage set; updating the model parameters of the initial policy model using gradients based on the advantage set and the concatenation group to obtain a target policy model; acquiring the query question to be reasoned about, and generating the target answer by reasoning based on the query question to be reasoned about using the target policy model.

[0041] Specifically, the pre-trained language model first generates a thought chain containing a complete logical deduction process and its corresponding answer, forming a baseline reasoning path. The original thought chain is then structurally segmented using multiple preset cutoff points, generating sub-sequences with varying reasoning depths. Each sub-sequence undergoes secondary reasoning by the model to generate intermediate answers, which, together with the complete chain answer, constitute a multi-dimensional evaluation sample set. The system calculates the average accuracy of answers at each stage in real time. When this value is below a set threshold, an all-or-nothing reward mechanism is used to strengthen basic reasoning ability; when it is above the threshold, a position decay factor is introduced to progressively optimize long-chain reasoning. By calculating the relative advantage value of each reasoning path, the model parameters are updated in the direction of improving key reasoning links. During the reasoning phase, the model autonomously determines the termination condition based on the quality of the real-time generated thought chain, shortening redundant reasoning steps while ensuring the correctness of the answer.

[0042] Please see Figure 2 , Figure 2 A specific implementation of the problem query method based on adaptive cascade groups is shown.

[0043] It should be noted that if substantially the same result is obtained, the method of this invention does not necessarily require further clarification. Figure 2 Limited to the sequence shown, this method includes the following steps:

[0044] S1: Obtain a pre-trained language model and a query question, use the pre-trained language model as an initial strategy model, and generate a complete thought chain sequence and an initial answer based on the query question using the initial strategy model.

[0045] Specifically, for query question q, based on the pre-policy π of the initial policy model... θ Generate a complete thought chain sequence T = (t1, t2, ..., t n ) and the initial answer a0.

[0046] Please see Figure 3 , Figure 3 A specific implementation of step S1 is shown below:

[0047] S11: Obtain the pre-trained language model and the query question, and obtain optimization parameters, wherein the optimization parameters include a decay factor, a preset truncation position, and a preset threshold.

[0048] S12: Using the pre-trained language model as the initial policy model, the complete thought chain sequence and the initial answer are generated based on the query question using the current policy of the initial policy model.

[0049] Specifically, during model initialization, a ternary optimization parameter set is loaded synchronously. The generated complete thought chains are physically segmented at preset truncation positions, providing basic units for subsequent construction of concatenated groups. When the accuracy is below a preset threshold in the early training phase, a binary reward strategy without decay is used to maintain training stability. Once the accuracy reaches the target, a gradually increasing reward decay is applied to long thought chains through a decay factor, guiding the model to shorten inference steps while maintaining accuracy. This parameterized control mechanism prioritizes inference accuracy in the early training phase and gradually optimizes inference efficiency in the later stages, achieving a dynamic balance between robustness and efficiency. This application dynamically judges the training phase through preset thresholds, activating the decay mechanism only after the accuracy reaches the target, avoiding premature interference with the formation of the model's inference ability.

[0050] The decay factor is a dynamic coefficient used to adjust the degree of reward decay. It can be implemented using a hyperparameter with a value between 0.8 and 0.95, applying a gradual penalty to long thought chains in the later stages of training. The preset truncation position is the physical dividing point that controls the length of the basic thought chain. It can be implemented as a percentage of the sequence length or a fixed number of tokens, such as 30% of the total sequence length or the first 50 tokens. The preset threshold is the critical accuracy indicator for dividing the training phase. It can be implemented using an empirical value between 0.7 and 0.85, used to trigger the switching of different reward strategies.

[0051] S2: Truncate the complete thought chain sequence according to the preset truncation position, generate truncated sub-sequences and the basic answers corresponding to the truncated sub-sequences, and generate concatenation groups and answer sets corresponding to the concatenation groups based on the complete thought chain sequence, the truncated sub-sequences, the initial answers and the basic answers.

[0052] Specifically, at multiple preset cutoff positions By truncating the complete thought chain sequence, we obtain the truncated subsequence. And generate the basic answer a k Combine the complete thought chain sequence and all truncated subsequences into a concatenated group. Where m is the number of truncated subsequences.

[0053] Please see Figure 4 , Figure 4 A specific implementation of step S2 is shown below:

[0054] S21: The complete thought chain sequence is truncated at the preset truncation position to generate the truncated subsequence.

[0055] S22: Input the truncated subsequence into the initial strategy model to generate the basic answer.

[0056] S23: Combine the complete thought chain sequence and the truncated sub-sequence to generate the concatenation group, and generate the answer set based on the initial answer and the basic answer.

[0057] Specifically, after the complete thought chain sequence is divided into truncated subsequences containing different reasoning stages, each subsequence independently generates a basic answer to verify its logical consistency. The combination of the complete chain and the truncated chain forms a concatenation group, forcing the model to simultaneously learn the ability to construct a complete reasoning chain and the validity judgment of key nodes during training. The answer set establishes a quantifiable evaluation benchmark for different reasoning stages by comparing the consistency difference between the initial answer and the basic answer. For example, when the preset truncation position is set to 50% of the sequence length, the truncated subsequence contains the first half of the reasoning steps, and its generated basic answer needs to be compared with the initial answer of the complete chain. If the difference between the two exceeds a threshold, gradient adjustment is triggered. This application realizes dynamic granular control of the reasoning path during training, solving the training instability problem caused by fixed-length penalty. Through the independent answer generation mechanism of the truncated subsequence, validity supervision is provided for intermediate reasoning steps, avoiding the generation of redundant or invalid thought chains. The combined design of the concatenation group and the answer set enables the model to simultaneously optimize the completeness of reasoning and the conciseness of steps, improving reasoning efficiency while ensuring answer accuracy.

[0058] Here, a truncated subsequence refers to a stage-specific reasoning segment extracted from the complete reasoning chain. This can be achieved by retaining all steps from the starting position to the truncated position, ensuring the subsequence possesses the semantic integrity to independently generate an answer. A basic answer refers to a stage-specific conclusion generated based on the truncated subsequence. This can be achieved by inputting the subsequence into the initial policy model for forward reasoning, providing a validation benchmark for the effectiveness of intermediate steps. A concatenation group refers to a comparative training unit containing both the complete and truncated chains. This can be achieved by arranging and combining reasoning paths of different granularities to form multi-level supervision signals. The answer set refers to the evaluation set consisting of the initial answer and the basic answer. This can be achieved by aggregating the output results of different reasoning paths, providing multi-dimensional feedback to the model.

[0059] S3: Calculate the correctness index of each answer in the answer set, and calculate the average accuracy rate of the cascaded group based on the correctness index.

[0060] Specifically, the formula for calculating the accuracy index is as follows:

[0061]

[0062] Wherein, δ(a) k ) is the accuracy indicator, a k This is the k-th answer in the answer set.

[0063] The formula for calculating the average accuracy rate is as follows:

[0064]

[0065] Where R is the average accuracy rate.

[0066] S4: Dynamically switch rewards based on the average accuracy and preset threshold to generate reward sets for each thought chain, and calculate the advantage function based on the reward sets to generate an advantage set.

[0067] Please see Figure 5 , Figure 5 A specific implementation of step S4 is shown below:

[0068] S41: If the average accuracy is less than the preset threshold, then the initial strategy model is trained using a binary reward strategy.

[0069] S42: If the average accuracy is greater than or equal to the preset threshold, then calculate the positional decay weight based on the decay factor and the complete thought chain sequence.

[0070] Specifically, the formula for calculating the location attenuation weight is as follows: Where β∈(0,1) is the decay factor; when p0=n, it corresponds to the complete chain sequence, with no decay.

[0071] S43: Calculate the decay reward based on the position decay weight and the average accuracy, and generate the reward set for each thought chain based on the decay reward.

[0072] S44: Calculate the advantage function based on the reward set to generate the advantage set.

[0073] Specifically, when the model's average accuracy is below a preset threshold, it indicates that its reasoning ability is not yet stable. At this point, a binary reward strategy is adopted, allocating rewards only based on the correctness of the final answer, avoiding the introduction of complex penalties that could cause gradient oscillations in the early stages of training. When the average accuracy exceeds the threshold, the model has acquired basic reasoning ability. At this point, positional decay weights are calculated for each step of the complete thought chain based on a decay factor, reducing the reward weights for subsequent redundant steps. The advantage function calculation compares the decayed rewards of each step with the average reward benchmark to generate a relative advantage value to optimize the policy gradient update direction.

[0074] The average accuracy rate refers to the average of the correctness metrics of all answers in the answer set. Specifically, it can be calculated by comparing the number of correct answers to the total number of answers. This metric is used to evaluate the overall reasoning ability of the model at the current stage. The binary reward strategy involves assigning a binary reward of 0 or 1 to the correctness of an answer. This is achieved by comparing the degree of matching between the answer and the standard answer, providing a stable reward signal in the early stages of model training. Position decay weights are weight values ​​calculated based on the position of the reasoning step in the complete thought chain. Decaying reward is a dynamic reward value that combines the position decay weights and the average accuracy rate.

[0075] Further, step S44 includes: calculating an average reward benchmark based on the reward set using a preset formula; and calculating the relative advantage of each thought chain according to the reward set and the average reward benchmark to generate the advantage set.

[0076] Specifically, during model training, the rewards generated by all truncated subsequences are first averaged using a pre-defined formula to generate a dynamic baseline value that matches the current inference complexity. For example, when the number of truncated subsequences is 5, the baseline value is obtained by summing the correctness indices corresponding to the 5 basic answers and dividing by 5. Then, for each thought chain, the difference between its reward and the baseline value is calculated as the relative advantage. For example, if a thought chain has a reward of 0.8 and a baseline value of 0.7, its relative advantage is 0.1. This dual-calculation mechanism allows the model to distinguish between high-value inference steps and redundant steps. For instance, when a step significantly increases the reward compared to the baseline, its advantage value guides the model to strengthen that inference path. By using the number of truncated subsequences as a formula parameter, the evaluation baseline is dynamically adjusted during the training phase. For example, a larger number of truncations is set in the early stages of training to retain more inference steps, and then gradually reduced in the later stages to improve efficiency. This application solves the problem that existing reinforcement learning methods cannot dynamically adjust the evaluation weights of intermediate inference steps, achieving a balance between training stability and inference efficiency. Specifically, this is reflected in the early stages of training by using a dynamic benchmark to retain effective inference steps and prevent premature collapse, and in the later stages by using relative advantages to select high-value steps and shorten the inference path. At the same time, it avoids the excessive suppression of immature models by fixed-length penalties, ensuring robust convergence during the training process.

[0077] The average reward benchmark refers to the arithmetic mean of the rewards corresponding to all truncated subsequences. This can be achieved by summing the rewards and dividing by the number of truncated subsequences, and is used to establish a dynamic evaluation benchmark to reflect the overall performance level in the current training phase. The relative advantage refers to the difference between the reward of a single thought chain and the average benchmark. This can be achieved by calculating the algebraic difference between the reward and the benchmark for each chain, and is used to quantify the differences in the contribution of different inference paths to model optimization. The number of truncated subsequences refers to the number of inference step segments divided by a preset truncation position. This can be achieved by dynamically adjusting the truncation position parameter, and is used to control the balance between evaluation granularity and computational complexity.

[0078] The default formula for calculating the average reward benchmark is:

[0079]

[0080] Where b is the average reward benchmark, r j The reward for the j-th thought chain is m, where m is the number of the truncated subsequences.

[0081] The formula for calculating relative advantage is:

[0082] A k =r k -b;

[0083] Among them, A k For the k-th relative advantage, generate an advantage set from all sets of relative advantages.

[0084] S5: Update the model parameters of the initial policy model using gradients based on the advantage set and the concatenation group to obtain the target policy model.

[0085] Specifically, the formula for calculating the policy gradient is:

[0086]

[0087] in, The expectation for the series group; Given a query question q, the model generates a thought chain. The probability (strategy); This is a gradient strategy.

[0088] Please see Figure 6 , Figure 6 A specific implementation of step S5 is shown below:

[0089] S51: Calculate the policy gradient based on the advantage set, the cascade group, and the current parameters of the initial policy model.

[0090] S52: Update the parameters of the initial policy model according to the policy gradient, and iteratively train the updated initial policy model to generate the target policy model.

[0091] Specifically, during the model training phase, the policy gradient is first calculated based on the advantage set and the current model parameters. The relative value of the thought chain is dynamically integrated into the gradient update process, avoiding the gradient oscillation problem caused by relying on a single final reward in traditional methods. Then, the model parameters are updated according to the policy gradient, replacing discrete policy adjustments with continuous optimization of the parameter space, thus preserving the exploratory characteristics of reinforcement learning. Finally, the updated model undergoes multiple rounds of iterative training. In each iteration, the thought chain is regenerated and the advantage set is calculated, gradually converging to a balance that maintains inference ability while improving efficiency. This application effectively solves the model instability problem caused by a single reward mechanism in reinforcement learning training. Through dynamic gradient updates and iterative verification mechanisms, inference efficiency is gradually optimized while maintaining inference accuracy, avoiding the risk of collapse caused by premature efficiency pressure during training, and achieving a synergistic improvement in model robustness and inference efficiency.

[0092] The advantage set refers to the set of relative value assessments of each thought chain obtained through the advantage function calculation. Specifically, it can be calculated using the difference between the reward set and the average reward benchmark, reflecting the differences in contribution of different thought chains during training. The policy gradient refers to the gradient direction calculated based on the advantage function and the current model parameters. Specifically, it can be implemented using the gradient estimation formula in the policy gradient algorithm, guiding the direction of model parameter updates. Iterative training refers to the process of performing multiple rounds of validation and parameter adjustment on the updated model. Specifically, it can be implemented using a fixed number of training rounds or convergence condition judgments, ensuring the stability of model optimization.

[0093] S6: Obtain the query question to be reasoned, and generate the target answer by reasoning based on the query question through the target strategy model.

[0094] Please see Figure 7 , Figure 7 A specific implementation of step S6 is shown below:

[0095] S61: Obtain the query question to be reasoned.

[0096] S62: The target strategy model is used to gradually generate a target thinking chain based on the query question to be reasoned, and the preset cutoff position is monitored in real time on the target thinking chain.

[0097] S63: If the preset truncation position is detected, stop generating the target thought chain and generate the current thought chain answer based on the target thought chain.

[0098] S64: If the correctness index of the current thought chain answer is 1, then the current thought chain is taken as the target answer.

[0099] Specifically, the target policy model executes a step-by-step generation mechanism during the inference phase, triggering the position monitoring module each time an inference step is generated. When the number of generated steps reaches a preset truncation position, the generation of subsequent steps is immediately terminated. At this point, the generated thought chain fragment is input into the verification module for immediate answer verification. The setting of this truncation position comprehensively considers the model's performance stability at different stages; for example, it can be set to a smaller value in the later stages of training to improve efficiency. The verification module generates a correctness index by comparing the semantic consistency between the current answer and the standard answer. The final answer is only output when this index is 1, avoiding the continuation of invalid inference chains. This dual verification mechanism, through the synergy of real-time monitoring and immediate feedback, achieves dynamic optimization of the inference path length while ensuring answer accuracy. If the correctness index of the current thought chain answer is 0, the thought chain continues to be generated until the next preset truncation position is reached. At this point, it is determined whether the correctness of the current thought chain answer is 1. If yes, the current thought chain is used as the target answer; otherwise, the thought chain continues to be generated until the correctness of the current thought chain answer is 1.

[0100] The target policy model refers to the inference model that has been updated through gradients. Specifically, it can be implemented using a language model architecture based on policy gradient optimization, and its parameter update direction is driven by the calculation results of the advantage function.

[0101] This application can be applied to high-frequency quantitative trading decision-making scenarios in the fintech field. The problem encountered in this scenario is that real-time market data analysis requires rapid chain-like reasoning (event identification → impact assessment → trading signal generation), while traditional models suffer from high latency due to "overthinking." The embodiments of this application can implement an early exit mechanism and dynamic rewards. The early exit mechanism outputs trading signals in advance at preset nodes (such as volatility mutation points); the dynamic reward strengthens the efficient decision-making path through decaying rewards during the training phase. This application can reduce response latency and improve signal accuracy. This application can also be applied to electronic medical record reasoning scenarios in the medical field. The problem encountered in this scenario is that there are redundant steps in medical record text reasoning (symptom extraction → differential diagnosis → treatment plan). This application can implement concatenation group construction in this scenario. The concatenation group construction can set different cutoff points, such as cutoff point 1 (p1): outputting the top 3 suspected diseases based on the main symptoms; cutoff point 2 (p2): outputting the final diagnosis based on the test results. This application can improve the efficiency of outpatient medical record processing.

[0102] In this embodiment, a pre-trained language model and a query question are obtained. The pre-trained language model is used as an initial strategy model, and a complete thought chain sequence and an initial answer are generated based on the query question using the initial strategy model. The complete thought chain sequence is truncated according to a preset truncation position to generate truncated sub-sequences and corresponding basic answers. A concatenation group and an answer set corresponding to the concatenation group are generated based on the complete thought chain sequence, the truncated sub-sequences, the initial answer, and the basic answer. The correctness index of each answer in the answer set is calculated, and the average accuracy rate of the concatenation group is calculated based on the correctness index. Rewards are dynamically switched based on the average accuracy rate and a preset threshold to generate reward sets for each thought chain. An advantage function is calculated based on the reward sets to generate an advantage set. The model parameters of the initial strategy model are updated using gradients based on the advantage set and the concatenation group to obtain a target strategy model. A query question to be reasoned is obtained, and the target answer is generated by reasoning based on the query question to be reasoned using the target strategy model. This invention, through the construction of a concatenated group structure comprising complete thought chains and truncated subsequences, combined with a dynamically switching reward mechanism and position decay weight calculation, achieves refined evaluation and control of the multi-step question query process, thereby improving query efficiency. During training, the reward strategy is intelligently adjusted based on the model's actual performance, avoiding training instability caused by prematurely imposing efficiency constraints and effectively suppressing redundant inference steps, thus improving query accuracy.

[0103] This application addresses the overthinking problem caused by the lack of quality monitoring in intermediate inference steps by accurately identifying redundant inference steps through a phased evaluation mechanism. A dynamic reward switching strategy avoids model instability caused by inappropriate constraints applied in the early stages of training, creating a positive synergy between inference efficiency optimization and model capability improvement. A comparative learning mechanism for multi-granularity inference paths enhances the model's ability to focus on key inference steps, significantly reducing computational resource consumption while maintaining answer accuracy.

[0104] Please refer to Figure 8 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a problem query device based on adaptive serial groups. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0105] like Figure 8 As shown, the problem query device based on adaptive concatenation groups in this embodiment includes: an initial strategy model loading module 71, a thought chain truncation module 72, a correctness index calculation module 73, a reward dynamic switching module 74, a strategy model training module 75, and a model inference module 76, wherein:

[0106] The initial strategy model loading module 71 is used to acquire a pre-trained language model and a query question, use the pre-trained language model as the initial strategy model, and generate a complete thought chain sequence and an initial answer based on the query question using the initial strategy model.

[0107] The thought chain truncation module 72 is used to truncate the complete thought chain sequence according to a preset truncation position, generate truncated sub-sequences and the basic answers corresponding to the truncated sub-sequences, and generate a series group and the answer set corresponding to the series group based on the complete thought chain sequence, the truncated sub-sequences, the initial answers and the basic answers;

[0108] The correctness index calculation module 73 is used to calculate the correctness index of each answer in the answer set, and calculate the average correctness rate of the series group based on the correctness index.

[0109] The reward dynamic switching module 74 is used to dynamically switch rewards according to the average accuracy and preset threshold to generate reward sets for each thought chain, and to calculate the advantage function based on the reward sets to generate an advantage set.

[0110] The strategy model training module 75 is used to perform gradient updates on the model parameters of the initial strategy model based on the advantage set and the concatenation group to obtain the target strategy model.

[0111] The model reasoning module 76 is used to obtain the query question to be reasoned and to generate the target answer by reasoning based on the query question through the target strategy model.

[0112] Furthermore, the initial strategy model loading module 71 includes:

[0113] The model acquisition unit is used to acquire the pre-trained language model and the query question, as well as to acquire optimization parameters, wherein the optimization parameters include a decay factor, a preset truncation position, and a preset threshold.

[0114] An initial answer generation unit is used to take the pre-trained language model as the initial policy model and generate the complete thought chain sequence and the initial answer based on the query question using the current policy of the initial policy model.

[0115] Furthermore, the thought chain truncation module 72 includes:

[0116] A truncation unit is used to truncate the complete thought chain sequence according to the preset truncation position to generate the truncated subsequence;

[0117] A basic answer generation unit is used to generate the basic answer by inputting the truncated subsequence into the initial strategy model;

[0118] The concatenation group generation unit is used to combine the complete thought chain sequence and the truncated sub-sequence to generate the concatenation group, and to generate the answer set based on the initial answer and the basic answer.

[0119] Furthermore, the reward dynamic switching module 74 includes:

[0120] The first case unit is used to train the initial strategy model using a binary reward strategy if the average accuracy is less than the preset threshold.

[0121] The second case unit is used to calculate the position decay weight based on the decay factor and the complete thought chain sequence if the average accuracy is greater than or equal to the preset threshold.

[0122] The reward set calculation unit is used to calculate the decay reward based on the position decay weight and the average accuracy, and generate the reward set for each thought chain based on the decay reward;

[0123] The advantage set generation unit is used to calculate the advantage function based on the reward set and generate the advantage set.

[0124] Furthermore, the advantage set generation unit includes:

[0125] An average reward benchmark calculation unit is used to calculate the average reward benchmark based on the reward set using a preset formula.

[0126] The relative advantage calculation unit is used to calculate the relative advantage of each thought chain based on the reward set and the average reward benchmark, so as to generate the advantage set;

[0127] The preset formula is:

[0128]

[0129] Where b is the average reward benchmark, r j The reward for the j-th thought chain is m, where m is the number of the truncated subsequences.

[0130] Furthermore, the policy model training module 75 includes:

[0131] The policy gradient calculation unit is used to calculate the policy gradient based on the advantage set, the cascade group, and the current parameters of the initial policy model.

[0132] The gradient update unit is used to update the parameters of the initial policy model according to the policy gradient, and to iteratively train the updated initial policy model to generate the target policy model.

[0133] Furthermore, the model inference module 76 includes:

[0134] A query question acquisition unit is used to acquire the query question to be inferred.

[0135] The truncation position monitoring unit is used to generate a target thought chain step by step based on the query question to be reasoned through the target strategy model, and to monitor the preset truncation position on the target thought chain in real time.

[0136] The current thought chain answer generation unit is used to stop generating the target thought chain if the preset truncation position is detected, and to generate the current thought chain answer based on the target thought chain;

[0137] The target answer generation unit is used to take the current thinking chain as the target answer if the correctness index of the current thinking chain answer is 1.

[0138] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0139] Computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that... Figure 9 Only a computer device 8 with three components—memory 81, processor 82, and network interface 83—is shown. It should be understood that implementing all shown components is not required; more or fewer components may be implemented alternatively. Those skilled in the art will understand that this computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices.

[0140] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0141] The memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the memory 81 may include both internal storage units and external storage devices of the computer device 8. In this embodiment, the memory 81 is typically used to store the operating system and various application software installed on the computer device 8, such as program code for a problem query method based on adaptive cascade groups. In addition, the memory 81 may also be used to temporarily store various types of data that have been output or will be output.

[0142] In some embodiments, processor 82 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, processor 82 is used to run program code stored in memory 81 or process data, for example, to run the program code of the above-described problem query method based on adaptive cascade groups, to implement various embodiments of the problem query method based on adaptive cascade groups.

[0143] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0144] This application also provides another embodiment, namely, a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the problem query method based on the adaptive cascade group described above.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0146] Obviously, the embodiments described above are merely some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of protection of this application.

Claims

1. A question query method based on adaptive concatenation groups, characterized in that, include: Obtain a pre-trained language model and a query question, use the pre-trained language model as an initial strategy model, and generate a complete thought chain sequence and an initial answer based on the query question using the initial strategy model; The complete thought chain sequence is truncated according to a preset truncation position to generate truncated sub-sequences and corresponding basic answers. Then, a series group and a set of answers corresponding to the series group are generated based on the complete thought chain sequence, the truncated sub-sequences, the initial answers, and the basic answers. Calculate the correctness index for each answer in the answer set, and calculate the average correctness rate of the concatenated group based on the correctness index; Rewards are dynamically switched based on the average accuracy and preset threshold to generate reward sets for each thought chain, and an advantage function is calculated based on the reward sets to generate an advantage set. The model parameters of the initial policy model are updated using gradients based on the advantage set and the concatenation group to obtain the target policy model; Obtain the query question to be reasoned, and generate the target answer by reasoning based on the query question through the target strategy model.

2. The problem query method based on adaptive cascaded groups according to claim 1, characterized in that, The process of acquiring a pre-trained language model and a query question, using the pre-trained language model as an initial policy model, and generating a complete thought chain sequence and an initial answer based on the query question using the initial policy model includes: Obtain the pre-trained language model and the query question, and obtain optimization parameters, wherein the optimization parameters include a decay factor, a preset truncation position, and a preset threshold; The pre-trained language model is used as the initial policy model, and the complete thought chain sequence and the initial answer are generated based on the query question using the current policy of the initial policy model.

3. The problem query method based on adaptive cascaded groups according to claim 1, characterized in that, The step of truncating the complete thought chain sequence according to a preset truncation position to generate truncated sub-sequences and corresponding basic answers, and generating concatenation groups and corresponding answer sets based on the complete thought chain sequence, the truncated sub-sequences, the initial answers, and the basic answers, includes: The complete thought chain sequence is truncated at the preset truncation position to generate the truncated subsequence; The truncated subsequence is input into the initial strategy model to generate the basic answer; The complete thought chain sequence and the truncated sub-sequence are combined to generate the concatenated group, and the answer set is generated based on the initial answer and the basic answer.

4. The problem query method based on adaptive cascaded groups according to claim 2, characterized in that, The step of dynamically switching rewards based on the average accuracy and a preset threshold to generate reward sets for each thought chain, and calculating an advantage function based on the reward sets to generate an advantage set, includes: If the average accuracy is less than the preset threshold, then the initial strategy model is trained using a binary reward strategy. If the average accuracy is greater than or equal to the preset threshold, then the positional decay weight is calculated based on the decay factor and the complete thought chain sequence. The decay reward is calculated based on the position decay weight and the average accuracy, and the reward set for each thought chain is generated based on the decay reward. The advantage set is generated by calculating the advantage function based on the reward set.

5. The problem query method based on adaptive serial groups according to claim 4, characterized in that, The step of calculating the advantage function based on the reward set to generate the advantage set includes: The average reward benchmark is calculated based on the reward set using a preset formula; The relative advantage of each thought chain is calculated based on the reward set and the average reward benchmark to generate the advantage set.

6. The problem query method based on adaptive concatenation groups according to any one of claims 1 to 5, characterized in that, The step of updating the model parameters of the initial policy model according to the advantage set and the concatenation group to obtain the target policy model includes: The policy gradient is calculated based on the advantage set, the cascade group, and the current parameters of the initial policy model; The parameters of the initial policy model are updated according to the policy gradient, and the updated initial policy model is iteratively trained to generate the target policy model.

7. The problem query method based on adaptive concatenation groups according to any one of claims 1 to 5, characterized in that, The process of obtaining the query question to be reasoned about and generating the target answer by reasoning based on the query question using the target strategy model includes: Obtain the query question to be reasoned; The target strategy model generates a target thought chain step by step based on the query question to be reasoned, and the preset cutoff position is monitored in real time on the target thought chain; If the preset truncation position is detected, stop generating the target thought chain, and generate the current thought chain answer based on the target thought chain; If the correctness index of the current thought chain answer is 1, then the current thought chain is taken as the target answer.

8. A problem query device based on adaptive cascaded groups, characterized in that, include: The initial strategy model loading module is used to obtain a pre-trained language model and a query question, use the pre-trained language model as the initial strategy model, and generate a complete thought chain sequence and an initial answer based on the query question using the initial strategy model. The thought chain truncation module is used to truncate the complete thought chain sequence according to a preset truncation position, generate truncated sub-sequences and the basic answers corresponding to the truncated sub-sequences, and generate concatenation groups and answer sets corresponding to the concatenation groups based on the complete thought chain sequence, the truncated sub-sequences, the initial answers and the basic answers; The correctness index calculation module is used to calculate the correctness index of each answer in the answer set, and calculate the average correctness rate of the series group based on the correctness index. The reward dynamic switching module is used to dynamically switch rewards based on the average accuracy and a preset threshold to generate reward sets for each thought chain, and to calculate the advantage function based on the reward sets to generate an advantage set. The strategy model training module is used to perform gradient updates on the model parameters of the initial strategy model based on the advantage set and the concatenation group to obtain the target strategy model. The model reasoning module is used to obtain the query question to be reasoned and to generate the target answer by reasoning based on the query question through the target strategy model.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the problem query method based on adaptive cascade groups as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the problem query method based on adaptive cascade groups as described in any one of claims 1 to 7.

Images