Model training method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model training method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining preset question and answer training data, wherein the question and answer training data comprises input question data and a model output result corresponding to the input question data; reasoning trajectory generation is carried out based on each piece of input question data and the corresponding model output result, and a plurality of candidate reasoning trajectories are obtained; performing confusion degree evaluation based on the plurality of candidate reasoning trajectories to obtain a confusion degree score corresponding to each candidate reasoning trajectory; a target reasoning trajectory is determined from the plurality of candidate reasoning trajectories based on the confusion score to guide model training. According to the embodiment of the invention, stable model training can be carried out on a large language model used for business scenes such as intelligent investment advisers or insurance claims and the like in a low-cost and high-efficiency manner, the logic stability of the large language model during business processing is improved, and business services with high professionality and depth are provided.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, electronic device and storage medium. Background Technology

[0002] The Deep Reasoning paradigm has driven significant advancements in large-scale language models across specialized fields. This paradigm emphasizes investing more computational resources in model reasoning to achieve advanced cognitive capabilities such as multi-step logical deduction and complex risk decomposition, achieving great success in financial subfields with clearly defined verification standards, such as actuarial mathematics and quantitative trading code generation. However, migrating deep reasoning capabilities to business areas such as open financial consulting and complex insurance claims processing still faces significant challenges.

[0003] In verifiable fields such as high-frequency trading or financial auditing, reinforcement learning can effectively guide model exploration through explicit reward signals, such as the profit and loss results of strategy backtesting or the balance of financial statement data. However, in open fields such as investment research report writing and non-standard insurance claims, the quality of the output depends on subjective or composite standards such as the rigor of the argument, the comprehensiveness of risk disclosure, and the compliance of the interpretation of terms, lacking a single, objective standard answer. This makes it difficult to build accurate reward models to evaluate the quality of generated financial solutions, causing reinforcement learning processes to often face problems such as low sample efficiency, high training costs, and poor convergence stability.

[0004] As an alternative, instruction distillation relies on extracting reasoning trajectories from a more robust closed-source teacher model. This not only incurs high costs for acquiring specialized data and calling APIs, but also fundamentally limits the capabilities of the student model to the teacher model, making it difficult to generate novel investment perspectives or claims settlement approaches that surpass existing expert systems. Furthermore, in complex financial and insurance transactions, high-quality case queries and their corresponding complete logical reasoning chains are extremely scarce, further restricting the effectiveness of this method.

[0005] In summary, existing technologies struggle to balance low cost, high efficiency, and scalability when building deep reasoning capabilities for open-ended generative tasks. Therefore, how to achieve stable training results for large language models using new paradigms without relying on expensive external teacher models or unstable reinforcement learning processes has become a critical technical problem urgently needing to be solved in the field of artificial intelligence. Summary of the Invention

[0006] The main objective of this application is to provide a model training method, apparatus, electronic device, and storage medium, which aims to perform stable model training on large language models in a low-cost and high-efficiency manner.

[0007] To achieve the above objectives, a first aspect of this application proposes a model training method, the method comprising: Obtain preset question-and-answer training data, wherein the question-and-answer training data includes input question data and model output results corresponding to the input question data; Based on each input question data and the corresponding model output result, a reasoning trajectory is generated to obtain multiple candidate reasoning trajectories; The perplexity is evaluated based on multiple candidate reasoning trajectories to obtain a perplexity score for each candidate reasoning trajectory. The target inference trajectory is determined from multiple candidate inference trajectories based on the perplexity score to guide model training.

[0008] In some embodiments, the generation of inference trajectories based on each input question data and the corresponding model output result yields multiple candidate inference trajectories, including: An initial inference trajectory is generated by each input question data and the corresponding model output result; The initial inference trajectory is segmented to obtain a trajectory sequence containing multiple logical segments; For each of the aforementioned logical segments, an optimization and replacement process is performed to generate multiple candidate replacement segments; Based on each candidate replacement fragment, a corresponding candidate inference trajectory is generated, and multiple candidate inference trajectories are constructed.

[0009] In some embodiments, generating an initial inference trajectory using each input question data and the corresponding model output includes: Obtain multiple preset differentiated prompt strategies; Based on the aforementioned differentiated prompting strategies, the input question data and the model output results are subjected to inference processing to obtain multiple initial candidate trajectories; Each of the initial candidate trajectories is evaluated for quality, and the initial candidate trajectory that meets the preset quality standard is determined as the initial inference trajectory.

[0010] In some embodiments, the optimization and replacement process for each of the logical segments, generating multiple candidate replacement segments, includes: Each logical segment in the trajectory sequence is masked, and the model output is predicted and evaluated based on the masked trajectory sequence to obtain the sensitivity index of the corresponding logical segment. In response to the sensitivity index being higher than a preset threshold, the corresponding logic segment is identified as a logic segment to be optimized; For each of the aforementioned logic segments to be optimized, an optimization and replacement process is performed.

[0011] In some embodiments, determining the target reasoning trajectory from a plurality of candidate reasoning trajectories based on the perplexity score includes: From the multiple candidate reasoning trajectories, a preliminary trajectories whose perplexity scores meet the preferred criteria are selected; Based on the initially selected trajectory, a verification is generated to obtain the verification output result; The semantic consistency of the verification output and the model output is compared to obtain a consistency score. In response to the consistency score meeting the preset safety standard, the initial trajectory is determined as the target inference trajectory.

[0012] In some embodiments, the method further includes: In response to determining the target inference trajectory from the plurality of candidate inference trajectories, the initial inference trajectory is updated using the target inference trajectory; Based on the updated initial inference trajectory, the segmentation process, the optimization and replacement process, and the candidate inference trajectory construction steps are repeated; In response to the confusion score satisfying the preset convergence condition, the repeated execution of the steps is stopped, and the final target inference trajectory is output.

[0013] In some embodiments, the perplexity evaluation based on multiple candidate reasoning trajectories to obtain a perplexity score for each candidate reasoning trajectory includes: For each candidate inference trajectory, it is used as a preconditional context, and the model output is used as the target prediction sequence. The generation probability distribution of the target prediction sequence under the given preconditions is calculated using a pre-defined evaluation model. The perplexity is calculated based on the generated probability distribution to obtain a numerical perplexity score.

[0014] To achieve the above objectives, a second aspect of this application provides a model training apparatus, the apparatus comprising: The training data acquisition module is used to acquire preset question-and-answer training data, which includes input question data and model output results corresponding to the input question data; The reasoning trajectory generation module is used to generate a reasoning trajectory based on each input question data and the corresponding model output result, so as to obtain multiple candidate reasoning trajectories. The reasoning trajectory evaluation module is used to evaluate the perplexity based on multiple candidate reasoning trajectories and obtain a perplexity score for each candidate reasoning trajectory. A training module is used to determine a target reasoning trajectory from multiple candidate reasoning trajectories based on the perplexity score, in order to guide model training.

[0015] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the model training method described in the first aspect.

[0016] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method described in the first aspect.

[0017] The model training method, apparatus, electronic device, and storage medium proposed in this application acquire pre-set question-and-answer training data, which includes input question data and the corresponding model output results. Based on each input question data and the corresponding model output result, inference trajectories are generated to obtain multiple candidate inference trajectories. Perplexity is evaluated based on these candidate inference trajectories to obtain a perplexity score for each candidate inference trajectory. Based on the perplexity score, a target inference trajectory is determined from the multiple candidate inference trajectories to guide model training. Therefore, this application generates multiple candidate inference trajectories based on input samples and output results, then uses the perplexity score to measure the degree to which the candidate inference trajectories explain the known high-quality model output results. Through a reverse reasoning approach, the candidate inference trajectory that minimizes the perplexity of the output result, i.e., has the highest generation probability, is selected as the target inference trajectory for model training, achieving stable and efficient training guidance and avoiding the convergence problem of reinforcement learning. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the model training method provided in the embodiments of this application; Figure 2 yes Figure 1 A flowchart illustrating step S102 in the process; Figure 3 yes Figure 2 A flowchart illustrating step S201 in the process; Figure 4 yes Figure 2 A flowchart illustrating step S203 in the process; Figure 5 yes Figure 1A flowchart illustrating step S104 in the process; Figure 6 This is another schematic flowchart of the model training method provided in the embodiments of this application; Figure 7 yes Figure 1 A flowchart illustrating step S103 in the process; Figure 8 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0022] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0023] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0024] Based on this, embodiments of this application provide a model training method, apparatus, electronic device, and storage medium, aiming to perform stable model training on large language models in a low-cost and high-efficiency manner.

[0025] The model training method, apparatus, electronic device, and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the model training method in the embodiments of this application is described.

[0026] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0027] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0028] The model training method provided in this application relates to the field of artificial intelligence technology. The model training method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model training method, but is not limited to the above forms.

[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0030] Figure 1 This is a schematic diagram of an optional process for the model training method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0031] Step S101: Obtain preset question-and-answer training data, which includes input question data and the model output results corresponding to the input question data; Step S102: Generate inference trajectories based on each input question data and the corresponding model output results to obtain multiple candidate inference trajectories; Step S103: Perform perplexity evaluation based on multiple candidate reasoning trajectories to obtain a perplexity score for each candidate reasoning trajectory; Step S104: Determine the target reasoning trajectory from multiple candidate reasoning trajectories based on the perplexity score to guide model training; Steps S101 to S104 of this embodiment involve acquiring preset question-answering training data, which includes input question data and the corresponding model output results; generating multiple candidate inference trajectories based on each input question data and the corresponding model output result; evaluating the perplexity of the multiple candidate inference trajectories to obtain a perplexity score for each candidate inference trajectory; and determining the target inference trajectory from the multiple candidate inference trajectories based on the perplexity score to guide model training. Thus, this application generates multiple candidate inference trajectories based on input samples and output results, then uses the perplexity score to measure the degree to which the candidate inference trajectories explain the known high-quality model output results. Through reverse reasoning, the candidate inference trajectory that minimizes the perplexity of the output result, i.e., has the highest generation probability, is selected as the target inference trajectory for model training, achieving stable and efficient training guidance and avoiding the convergence problem of reinforcement learning.

[0032] In step S101 of some embodiments, preset question-answering training data is obtained. This training data mainly includes paired input questions and corresponding model outputs. The input questions can be various open-ended business questions, complex logical inquiries, or prompts for creative writing, while the model outputs are high-standard answers or solutions to these questions. This data can originate from selected public datasets, historical business records, or high-quality corpora that have undergone manual review.

[0033] In specific financial investment research scenarios, the input questions can be requests for investment value analysis of specific listed companies or inquiries about the market impact of macroeconomic policies. The corresponding model outputs are professional research reports, investment ratings, or risk warnings written by senior analysts. In this case, the pre-set question-and-answer training data covers multi-dimensional financial corpus information, ranging from macroeconomic indicators and industry development trends to company financial statements. In insurance business scenarios, the input questions can be user reports of incidents and on-site photos, and the model outputs standard claims determinations or reasons for claim rejection.

[0034] In practical applications of smart healthcare or assisted diagnosis, the input query data can be a patient's chief complaint, physical examination report data, or a textual description of medical images. The corresponding model output is a diagnostic conclusion, treatment plan suggestion, or medication guidance given by an expert from a top-tier hospital. In this case, the pre-set question-and-answer training data can come from anonymized electronic medical records, authoritative medical guidelines, or real doctor-patient dialogue records.

[0035] In step S102 of some embodiments, reasoning trajectories are generated based on each input question data and the corresponding model output, resulting in multiple candidate reasoning trajectories. Specifically, a language model with basic generative capabilities can be used, with the input question data and model output as contextual constraints, or specific prompting engineering can guide the model to infer the intermediate thought process. To explore multiple possible logical paths, a single path is not generated; instead, parallel sampling or multiple generation are performed to obtain multiple candidate reasoning trajectories with different content. These candidate reasoning trajectories represent multiple possible thought chains that lead from the same question to the same answer.

[0036] In step S103 of some embodiments, perplexity evaluation is performed based on multiple candidate reasoning trajectories to obtain a perplexity score for each candidate reasoning trajectory. This process verifies the rationality of the logic. Each candidate reasoning trajectory can be used as a precondition, combined with the input question data, and input into an evaluation model to obtain the corresponding perplexity score. The perplexity score is used to quantify the explanatory power of a candidate reasoning trajectory in generating the known model output, and can measure the quality of model reasoning. If a candidate reasoning trajectory is logically rigorous and its derivation is natural, then given the candidate reasoning trajectory, the probability of generating the corresponding model output should be high, and the corresponding perplexity score will be low; conversely, if the candidate reasoning trajectory is logically chaotic or irrelevant to the model output, the perplexity score will be high.

[0037] In step S104 of some embodiments, a target inference trajectory is determined from multiple candidate inference trajectories based on the perplexity score to guide model training. The calculated perplexity scores can be sorted or filtered, and the trajectory(s) with the best perplexity scores can be selected as the target inference trajectory. This target inference trajectory is considered the best logical path to explain the model's output. Subsequently, the input question data, the target inference trajectory, and the model output are concatenated into a complete training sample, and these synthesized high-quality samples are used for supervised fine-tuning of the large language model. In this way, the model no longer merely learns to imitate the final answer, but learns the deep reasoning logic hidden behind the answer.

[0038] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S204: Step S201: Generate an initial inference trajectory using each input question data and the corresponding model output result; Step S202: Perform segmentation processing based on the initial inference trajectory to obtain a trajectory sequence containing multiple logical segments; Step S203: Perform optimization and replacement processing on each logical segment to generate multiple candidate replacement segments; Step S204: Generate a corresponding candidate inference trajectory based on each candidate replacement fragment, and construct multiple candidate inference trajectories.

[0039] In step S201 of some embodiments, an initial reasoning trajectory is generated using each input question data and the corresponding model output result. In specific implementations, the input question data and model output results can be simultaneously used as context input to the large language model, along with specific prompts, requiring the model to reconstruct or reverse-reason the thought process that may occur from the input to the output. The model can output a continuous logical explanation text based on the input question data and the corresponding model output results, which is the initial reasoning trajectory. Although this initial trajectory may not be perfect in terms of logical coherence or depth, it contains the basic logical framework connecting the question and the answer.

[0040] Taking insurance claims settlement as an example, the generated initial reasoning trajectory can be concretely reflected in the thought process of a claims settlement expert: first, confirming whether the policy is still valid; second, analyzing whether the cause of the accident falls within the scope of the exclusion clauses; third, calculating the claim amount based on the loss assessment list; and finally, drawing a comprehensive claim settlement conclusion. In this way, the initial reasoning trajectory generated by the model transforms the originally black-box claims settlement decision-making process into an explicit and visible logical chain, providing a foundation for subsequent review and evaluation.

[0041] In step S202 of some embodiments, the initial reasoning trajectory is segmented to obtain a trajectory sequence containing multiple logical segments. To enable fine-grained correction of the reasoning process, the continuous initial reasoning trajectory needs to be broken down into smaller, operable units. Natural language processing techniques can be used to identify paragraph breaks, logical connectors, or punctuation marks in the trajectory to determine the segmentation points. Logical connectors include, for example, "firstly," "secondly," and "finally." Through segmentation, the originally lengthy thought process is transformed into a trajectory sequence composed of several logical segments arranged sequentially. Each logical segment typically represents an independent thinking step, the planning of a sub-task, or a specific argumentation stage. This structured processing transforms the complex global optimization problem into a series of relatively simple local optimization problems.

[0042] In step S203 of some embodiments, an optimization and replacement process is performed on each logical segment to generate multiple candidate replacement segments. A local search strategy can be adopted, focusing on a specific logical segment in the trajectory sequence each time. By keeping the preceding context of a logical segment, the input question data, and the model output unchanged, the generating model is required to regenerate multiple possible content variations for that specific position, which are the candidate replacement segments. These candidate replacement segments may contain more detailed explanations, different perspectives, or self-corrections from the previous step. Through parallel sampling or diversified prompts, a rich variety of possibilities can be explored at this local node, thereby avoiding the limitation of falling into a single thinking path.

[0043] In step S204 of some embodiments, a corresponding candidate inference trajectory is generated based on each candidate replacement fragment, thus constructing multiple candidate inference trajectories. After generating several candidate replacement fragments for a certain logical position, these logical fragments can be filled back into the corresponding positions of the original trajectory sequence, replacing the original logical fragments, while keeping the logical fragments of other parts unchanged. Thus, a completely new and complete inference path is derived through each small local modification. By repeating the above operation on different logical positions, multiple candidate inference trajectories can be constructed. Each trajectory represents a unique logical deduction method, providing a broad search space for subsequent perplexity-based optimization.

[0044] Steps S201 to S204 effectively achieve deep search and optimization of the inference trajectory. By decomposing long-chain inference tasks into local segments for targeted optimization, the computational complexity of generating long text logic chains is significantly reduced, and the diversity and robustness of the inference process are greatly enhanced. This allows the model to repeatedly refine and polish local details, thereby uncovering deeper thought processes that are easily overlooked in one-time generation, which are logically more rigorous and better explain high-quality output results, ultimately greatly improving the quality of the synthetic training data.

[0045] Please see Figure 3 In some embodiments, step S201 may include, but is not limited to, steps S301 to S303: Step S301: Obtain a variety of preset differentiated prompting strategies; Step S302: Based on each differentiated prompting strategy, perform inference processing on the input question data and model output results to obtain multiple initial candidate trajectories; Step S303: Perform quality assessment on each initial candidate trajectory, and determine the initial candidate trajectory that meets the preset quality standard as the initial inference trajectory.

[0046] In step S301 of some embodiments, multiple preset differentiated prompting strategies are obtained. To avoid a single initialization path causing subsequent optimization to fall into local optima, multiple sets of prompt word templates with different styles and focuses are pre-configured. These differentiated prompting strategies may include, but are not limited to, step-by-step decomposition strategies, reverse deduction strategies, critical reflection strategies, and outline filling strategies. Among them, the step-by-step decomposition strategy focuses on linear planning according to logical steps, the reverse deduction strategy focuses on tracing back to key turning points from the result, the critical reflection strategy focuses on continuous self-questioning and correction during the reasoning process, and the outline filling strategy focuses on building a macro framework first and then filling in the details. The purpose of obtaining these strategies is to provide diverse thinking perspectives for subsequent reasoning generation.

[0047] In step S302 of some embodiments, based on each differentiated prompting strategy, inference processing is performed on the input question data and model output results to obtain multiple initial candidate trajectories. Each of the differentiated prompting strategies obtained above can be applied to the current input question data and model output results to drive the large language model to perform parallel inference generation tasks. This means that for the same question and the same standard answer, multiple different ways of thinking can be used to explain the logical relationship between the two.

[0048] For example, applying a step-by-step decomposition strategy might generate a trajectory that focuses on execution details, while applying an outline-filling strategy might generate a trajectory with a stronger sense of structure. In this way, multiple initial candidate trajectories with different content, structures, and logical flows can be generated at once.

[0049] In step S303 of some embodiments, each initial candidate trajectory is evaluated for quality, and the initial candidate trajectories that meet the preset quality standards are determined as the initial inference trajectories. Since the quality of trajectories generated by different prompting strategies may vary, a filtering mechanism is needed. This can be achieved by calculating the perplexity score of each initial candidate trajectory relative to the model output, or by filtering using preset heuristic rules, such as trajectory length, logical connector density, and format compliance. By comparing the evaluation results of each initial candidate trajectory, those trajectories with logical breaks or obvious irrationality are eliminated, and the trajectory(s) with the highest evaluation scores are selected and formally determined as the initial inference trajectories.

[0050] Steps S301 to S303 effectively address the path dependency problem that may arise from a single initialization strategy. By introducing diverse thinking patterns and selecting the best ones during the algorithm startup phase, this technical solution significantly improves the quality baseline of the initial inference trajectory, thereby preventing subsequent local search algorithms from prematurely falling into poor-quality local optima. This, in turn, enhances the convergence speed of the overall reverse engineering inference process and the logical rigor of the final synthesized data.

[0051] Please see Figure 4 In some embodiments, step S203 may include, but is not limited to, steps S401 to S403: Step S401: Mask each logical segment in the trajectory sequence, and predict and evaluate the model output based on the masked trajectory sequence to obtain the sensitivity index of the corresponding logical segment. Step S402: In response to the sensitivity index being higher than a preset threshold, the corresponding logic segment is identified as a logic segment to be optimized. Step S403: Perform optimization and replacement processing for each logic segment to be optimized.

[0052] In step S401 of some embodiments, each logical segment in the trajectory sequence is masked, and the model output is predicted and evaluated based on the masked trajectory sequence to obtain the sensitivity index of the corresponding logical segment. The core of this step is to quantify the importance of each logical step to the final result. By adopting a similar masking strategy, a logical segment in the trajectory sequence is temporarily removed or replaced with a special mask symbol in turn, while other contextual information is preserved. Subsequently, this incomplete, masked trajectory sequence is input into the evaluation model to observe how the model's prediction probability or perplexity of the previous model output changes in this state. The sensitivity index is usually defined as the difference in perplexity or the magnitude of probability change before and after masking. If masking a logical segment causes a sharp increase in the model's perplexity of the model output, it indicates that the logical segment carries key derivation logic, and the sensitivity index is high. Conversely, if the change in perplexity is negligible, it indicates that the segment may be redundant information or invalid reasoning, and the sensitivity index is low.

[0053] In a specific exemplary embodiment, in applications such as drug interaction analysis or prescription review, logical segments in the trajectory sequence may correspond to analytical steps regarding contraindications to specific drug components. If masking the logical segment "detecting a patient's penicillin allergy history" does not significantly increase the perplexity of the model's prediction of "prescribing amoxicillin," meaning the model considers not prescribing the medication reasonable, or the model ignores the allergy risk, this may indicate a security vulnerability in the original logical chain. Conversely, if masking this segment makes the prediction of safe medication use highly uncertain, it indicates that the allergy history screening step is a highly sensitive critical risk control point. In this way, safety-critical steps in medical reasoning can be accurately captured.

[0054] In step S402 of some embodiments, in response to a sensitivity index exceeding a preset threshold, the corresponding logical segment is identified as a logical segment to be optimized. To improve the algorithm's operating efficiency, not all logical segments are optimized indiscriminately; instead, a filtering threshold is set for selection. The calculated sensitivity index is compared with a preset numerical threshold. Only those highly sensitive segments that contribute significantly to the explanatory power of the results and whose absence would cause the logical chain to break are marked as logical segments to be optimized. Through a strategy for allocating computing resources, it is ensured that subsequent generation computing power is concentrated on the core logical links that most need refinement, while ignoring minor details that have little impact on the overall quality.

[0055] In step S403 of some embodiments, optimization and replacement processing is performed for each logical segment to be optimized. After determining the key optimization targets, the generation operation is focused on these logical segments to be optimized. By calling the generation model, multiple new candidate versions are generated specifically for these highly sensitive positions while keeping the context unchanged. Since these positions have been identified as key logical points, optimizing and replacing them often leads to a significant improvement in the overall reasoning quality. By focusing on the key nodes in the thought chain, an attempt is made to find alternatives that are more logically coherent and have stronger arguments.

[0056] By introducing a dynamic pruning and attention mechanism based on sensitivity analysis in steps S401 to S403, the computational efficiency and targeting of the reverse inference process are significantly improved. This not only greatly reduces the computational overhead in large-scale trajectory search, but also avoids the model from performing ineffective iterative iterations on irrelevant details, thereby converging to a high-quality inference trajectory more quickly and achieving a dual optimization of model training efficiency and effectiveness.

[0057] Please see Figure 5 In some embodiments, step S104 may include, but is not limited to, steps S501 to S504: Step S501: Select the initial trajectory from multiple candidate reasoning trajectories that meets the preferred criteria for confusion score; Step S502: Generate and verify the initial trajectory to obtain the verification output result; Step S503: Compare the semantic consistency between the verification output and the model output to obtain a consistency score; Step S504: In response to the consistency score meeting the preset safety standard, the initial trajectory is determined as the target inference trajectory.

[0058] In step S501 of some embodiments, preliminary inference trajectories that meet the preferred criteria are selected from multiple candidate inference trajectories. By setting a preferred criterion, such as selecting the top few candidate inference trajectories with the lowest perplexity scores, or by setting a threshold for perplexity scores, all candidate inference trajectories with perplexity scores below the threshold are retained. Through this operation, low-quality candidate inference trajectories with illogical reasoning that clearly cannot reasonably explain the model's output are eliminated, and the remaining preliminary trajectories are probabilistically considered to be the logical paths most likely to lead to the correct result.

[0059] In step S502 of some embodiments, a verification is performed based on the initially selected trajectory to obtain a verification output result. To verify whether the initially selected trajectory truly and logically guides the model's output result, the selected trajectory can be used as an input prompt or pre-context and input again into the generation model. The model is then required to generate a new output answer based solely on this reasoning trajectory, without considering the original model's output result; this is the verification output result. By simulating the model's behavior during actual reasoning, if the reasoning logic is correct, then following this logical chain should lead to a result highly similar to the original answer.

[0060] In step S503 of some embodiments, a semantic consistency comparison is performed between the verification output and the model output to obtain a consistency score. Natural language processing (NLP) techniques are used to compare the semantic similarity between the newly generated verification output and the original model output. This can be achieved by calculating the cosine similarity of the text embedding vectors, using traditional NLP metrics, or directly using another large model for semantic discrimination. The consistency score quantifies the degree of consistency between the verification output and the original model output in terms of core meaning, key information points, and conclusions. A higher consistency score indicates that the initial selection trajectory not only probabilistically fits the original answer but also effectively reproduces the core content of the original answer in terms of actual logical guidance.

[0061] In step S504 of some embodiments, in response to the consistency score meeting a preset safety standard, the initial selected trajectory is determined as the target inference trajectory. By setting a safety standard, such as a similarity score greater than 0.8, only when the consistency score exceeds this standard will the corresponding initial selected trajectory be finally confirmed as a true and valid target inference trajectory. This can exclude initial selected trajectories that, although they have low perplexity, are actually logically constructed from illusions and will cause the generated results to deviate significantly from the original meaning, thus obtaining the target inference trajectory. The finally determined target inference trajectory will be used to construct training data.

[0062] Steps S501 to S504 construct a safety firewall to prevent logical illusions and overfitting. This effectively addresses the risk of logical consistency but erroneous conclusions that may result from relying solely on perplexity metrics, ensuring that the synthesized inference trajectory is not only statistically sound but also highly accurate and reproducible in terms of actual semantic guidance. This screening strategy enhances the reliability of the target inference trajectory, thereby guaranteeing that the model, after fine-tuning, can truly grasp the correct inference logic, rather than merely memorizing superficial textual patterns.

[0063] Please see Figure 6 In some embodiments, the embodiments of this application may also include, but are not limited to, steps S601 to S603: Step S601: In response to determining the target inference trajectory from multiple candidate inference trajectories, the initial inference trajectory is updated using the target inference trajectory; Step S602: Based on the updated initial inference trajectory, repeat the segmentation processing, optimization and replacement processing, and candidate inference trajectory construction steps. In step S603, in response to the confusion score satisfying the preset convergence condition, the repeated execution of the step is stopped, and the final target inference trajectory is output.

[0064] In step S601 of some embodiments, in response to determining a target inference trajectory from multiple candidate inference trajectories, the initial inference trajectory is updated using the target inference trajectory. Once a target inference trajectory that is better than the previous one or can significantly reduce perplexity is selected in the current round, the task does not stop immediately. Instead, this refined target inference trajectory is used as a new benchmark. The initial inference trajectory used in the previous iteration is overwritten or replaced by this target inference trajectory, thereby solidifying the results of this round of optimization and preparing for a deeper search and refinement in the next round.

[0065] In step S602 of some embodiments, based on the updated initial inference trajectory, the segmentation processing, optimization and replacement processing, and candidate inference trajectory construction steps are repeatedly executed. After establishing a new starting point, the core generation engine is restarted. The updated initial inference trajectory is logically re-segmented, and weak links or potential optimization space are scanned again. Subsequently, a new round of replacement generation and candidate construction is performed for the new logical segments. As the initial inference trajectory is continuously optimized, the candidate segments generated by the model in the next round tend to be more accurate and profound, thereby achieving a spiral improvement in inference quality.

[0066] In step S603 of some embodiments, in response to the perplexity score meeting a preset convergence condition, the repeated execution of the step is stopped, and the final target inference trajectory is output. After each iteration, the trend of the perplexity score is monitored in real time. The convergence condition can be set as the difference between the scores of two adjacent rounds being less than a preset minimum value, or the score no longer decreasing or even starting to rebound, or the preset maximum number of iterations being reached. Once any of these conditions is met, it indicates that the current target inference trajectory has reached the limit of the model's capabilities or a local optimum, and the marginal benefit of continuing iteration is extremely low. At this point, the loop can be terminated, and the retained target inference trajectory can be output as the final result for subsequent model training.

[0067] Through steps S601 to S603, a closed-loop iterative optimization feedback mechanism is constructed, transforming the one-time generation task into a progressive search and refinement process. This overcomes the shortcomings of single generation, which easily gets trapped in local optima or produces shallow logic, by continuously optimizing through repeated refinement and self-correction. As the iteration deepens, the logical fit between the inference trajectory and the model output gradually approaches the extreme value, ultimately synthesizing a high-quality target inference trajectory that is logically rigorous, rich in detail, and has extremely high explanatory power, greatly improving the quality of the training samples.

[0068] Please see Figure 7 In some embodiments, step S103 may include, but is not limited to, steps S701 to S703: Step S701: For each candidate inference trajectory, use it as the precondition context and use the model output as the target prediction sequence. Step S702: Calculate the generation probability distribution of the target prediction sequence under the context of the preconditions using a preset evaluation model; Step S703: Calculate the perplexity based on the generation probability distribution to obtain a numerical perplexity score.

[0069] In step S701 of some embodiments, for each candidate inference trajectory, it is used as a preconditional context, and the model output is used as the target prediction sequence. In this step, the input data is serialized and reassembled. Specifically, the currently generated candidate inference trajectory is concatenated after the input question data to form a combined text sequence containing input information and intermediate inference processes. This combined text sequence is set as the context input of the language model. At the same time, the preset model output is set as the label sequence to be predicted. Thus, a standard conditional probability calculation task is constructed, that is, given the input sample and candidate inference trajectory, calculating the probability that the model generates a specific model output result.

[0070] In step S702 of some embodiments, the generation probability distribution of the target prediction sequence under the preconditional context is calculated using a preset evaluation model. The constructed combined text sequence is input into a pre-trained basic large language model. The basic large language model is in inference mode or evaluation mode, without parameter updates. Then, based on the autoregressive generation principle, for each word in the target prediction sequence, its conditional probability is calculated under the context conditions consisting of the combined text sequence and all preceding words. By traversing the entire length of the target prediction sequence, the probability value corresponding to each position in the target prediction sequence is obtained, forming a complete generation probability distribution sequence.

[0071] In step S703 of some embodiments, perplexity is calculated based on the generation probability distribution to obtain a numerical perplexity score. Mathematical operations are performed on the obtained generation probability distribution sequence. This can be done by first calculating the logarithm of each conditional probability, then obtaining the average negative log-likelihood of these logarithmic values, and finally performing an exponential operation on this average to obtain the perplexity score. This numerical perplexity score directly characterizes the statistical correlation between the target prediction sequence and the preconditional context. A lower score indicates a higher generation probability of the target prediction sequence under the constraint of the current candidate inference trajectory, and a stronger causal relationship between the two; a higher score indicates a weaker statistical correlation.

[0072] Through steps S701 to S703, an automated evaluation process based on probability theory was established through sequence concatenation, conditional probability calculation, and statistical index transformation. By leveraging the statistical modeling capabilities of a basic large language model for text sequences, the explanatory power of the inference trajectory on the model's output is quantified into specific mathematical indicators. By calculating and outputting a numerical perplexity score, the effectiveness of different candidate inference trajectories in deriving specific output results can be objectively measured. This provides a quantitative basis for accurately selecting high-quality inference data from massive candidate data, ensuring the objectivity of the selection process and the consistency of the calculation.

[0073] This application's embodiments acquire preset question-answering training data, which includes input question data and the corresponding model output results. Based on each input question data and its corresponding model output result, inference trajectories are generated, resulting in multiple candidate inference trajectories. Perplexity is evaluated based on these candidate inference trajectories, yielding a perplexity score for each. Finally, a target inference trajectory is determined from the multiple candidate inference trajectories based on the perplexity score to guide model training. Therefore, this application generates multiple candidate inference trajectories based on input samples and output results, then uses the perplexity score to measure the interpretability of the candidate inference trajectories to the known high-quality model output results. Through a reverse reasoning approach, the candidate inference trajectory that minimizes the perplexity of the output result (i.e., has the highest generation probability) is selected as the target inference trajectory for model training, achieving stable and efficient training guidance and avoiding the convergence difficulty problem of reinforcement learning.

[0074] Please see Figure 8 This application also provides a model training apparatus that can implement the above-described model training method. The apparatus includes: The training data acquisition module is used to acquire preset question-and-answer training data, which includes input question data and the model output results corresponding to the input question data. The reasoning trajectory generation module is used to generate reasoning trajectories based on each input question data and the corresponding model output results, resulting in multiple candidate reasoning trajectories. The reasoning trajectory evaluation module is used to evaluate the perplexity based on multiple candidate reasoning trajectories and obtain a perplexity score for each candidate reasoning trajectory. The training module is used to determine the target inference trajectory from multiple candidate inference trajectories based on perplexity scores to guide model training.

[0075] The specific implementation of this model training device is basically the same as the specific implementation of the model training method described above, and will not be repeated here.

[0076] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described model training method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0077] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 using the model training method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0078] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described model training method.

[0079] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] The model training method, apparatus, electronic device, and storage medium provided in this application acquire preset question-and-answer training data, which includes input question data and the corresponding model output results. Based on each input question data and the corresponding model output result, inference trajectories are generated to obtain multiple candidate inference trajectories. Perplexity is evaluated based on these candidate inference trajectories to obtain a perplexity score for each candidate inference trajectories. Based on the perplexity score, a target inference trajectories are determined from the multiple candidate inference trajectories to guide model training. Therefore, this application generates multiple candidate inference trajectories based on input samples and output results, and then uses the perplexity score to measure the degree to which the candidate inference trajectories explain the known high-quality model output results. Through a reverse reasoning method, the candidate inference trajectory that minimizes the perplexity of the output result, i.e., has the highest generation probability, is selected as the target inference trajectory for model training, achieving stable and efficient training guidance and avoiding the convergence problem of reinforcement learning.

[0081] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0082] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0085] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0086] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that, The method includes: Obtain preset question-and-answer training data, wherein the question-and-answer training data includes input question data and model output results corresponding to the input question data; Based on each input question data and the corresponding model output result, a reasoning trajectory is generated to obtain multiple candidate reasoning trajectories; The perplexity is evaluated based on multiple candidate reasoning trajectories to obtain a perplexity score for each candidate reasoning trajectory. The target inference trajectory is determined from multiple candidate inference trajectories based on the perplexity score to guide model training.

2. The method according to claim 1, characterized in that, The inference trajectory is generated based on each input question data and the corresponding model output result, resulting in multiple candidate inference trajectories, including: An initial inference trajectory is generated by each input question data and the corresponding model output result; The initial inference trajectory is segmented to obtain a trajectory sequence containing multiple logical segments; For each of the aforementioned logical segments, an optimization and replacement process is performed to generate multiple candidate replacement segments; Based on each candidate replacement fragment, a corresponding candidate inference trajectory is generated, and multiple candidate inference trajectories are constructed.

3. The method according to claim 2, characterized in that, The step of generating an initial inference trajectory by using each input question data and the corresponding model output includes: Obtain multiple preset differentiated prompt strategies; Based on the aforementioned differentiated prompting strategies, the input question data and the model output results are subjected to inference processing to obtain multiple initial candidate trajectories; Each of the initial candidate trajectories is evaluated for quality, and the initial candidate trajectory that meets the preset quality standard is determined as the initial inference trajectory.

4. The method according to claim 2, characterized in that, The optimization and replacement process for each logical segment generates multiple candidate replacement segments, including: Each logical segment in the trajectory sequence is masked, and the model output is predicted and evaluated based on the masked trajectory sequence to obtain the sensitivity index of the corresponding logical segment. In response to the sensitivity index being higher than a preset threshold, the corresponding logic segment is identified as a logic segment to be optimized; For each of the aforementioned logic segments to be optimized, an optimization and replacement process is performed.

5. The method according to claim 1, characterized in that, The step of determining the target reasoning trajectory from multiple candidate reasoning trajectories based on the perplexity score includes: From the multiple candidate reasoning trajectories, a preliminary trajectories whose perplexity scores meet the preferred criteria are selected; Based on the initially selected trajectory, a verification is generated to obtain the verification output result; The semantic consistency of the verification output and the model output is compared to obtain a consistency score. In response to the consistency score meeting the preset safety standard, the initial trajectory is determined as the target inference trajectory.

6. The method according to claim 1, characterized in that, The method further includes: In response to determining the target inference trajectory from the plurality of candidate inference trajectories, the initial inference trajectory is updated using the target inference trajectory; Based on the updated initial inference trajectory, the segmentation process, the optimization and replacement process, and the candidate inference trajectory construction steps are repeated; In response to the confusion score satisfying the preset convergence condition, the repeated execution of the steps is stopped, and the final target inference trajectory is output.

7. The method according to claim 1, characterized in that, The perplexity evaluation based on multiple candidate reasoning trajectories, to obtain a perplexity score for each candidate reasoning trajectory, includes: For each candidate inference trajectory, it is used as a preconditional context, and the model output is used as the target prediction sequence. The generation probability distribution of the target prediction sequence under the given preconditions is calculated using a pre-defined evaluation model. The perplexity is calculated based on the generated probability distribution to obtain a numerical perplexity score.

8. A model training device, characterized in that, The device includes: The training data acquisition module is used to acquire preset question-and-answer training data, which includes input question data and model output results corresponding to the input question data; The reasoning trajectory generation module is used to generate a reasoning trajectory based on each input question data and the corresponding model output result, so as to obtain multiple candidate reasoning trajectories. The reasoning trajectory evaluation module is used to evaluate the perplexity based on multiple candidate reasoning trajectories and obtain a perplexity score for each candidate reasoning trajectory. A training module is used to determine a target reasoning trajectory from multiple candidate reasoning trajectories based on the perplexity score, in order to guide model training.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1 to 7.

Images