Training method of reasoning model, question and answer method, electronic device and storage medium

Abstract

Embodiments of the present specification provide a reasoning model training method, a question and answer method, an electronic device and a storage medium. The reasoning model training method comprises: inputting a sample question into a reasoning model, and guiding the reasoning model to generate corresponding prediction results based on multiple reasoning tracks. Each reasoning track comprises multiple reasoning steps executed in sequence. The prediction result corresponding to each reasoning track is determined based on the reasoning results of the multiple reasoning steps. A mixed reasoning step in the multiple reasoning steps comprises: determining an anchor word element with a matching degree higher than a target threshold and a neighborhood word element of the anchor word element in a semantic dimension based on the matching degree of the current context information and each candidate word element corresponding to the reasoning model; performing mixed processing on the anchor word element and the neighborhood word element to obtain a reasoning result of the mixed reasoning step; and adjusting the reasoning strategy of the reasoning model based on the prediction results corresponding to the multiple reasoning tracks respectively.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a training method for a reasoning model, a question-answering method, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence technology, the application of large language models is becoming more and more widespread, and the requirements for the reasoning ability of large language models are also getting higher and higher.

[0003] Large language models can typically be further trained using reinforcement learning algorithms. For large language models designed to handle complex reasoning tasks, the reinforcement learning training process requires them to generate diverse reasoning trajectories to explore better reasoning strategies and improve their reasoning capabilities. However, current large language models have significant limitations in exploring different reasoning trajectories, lack deep logical diversity, and perform poorly in exploring entirely new reasoning trajectories, making it difficult to effectively improve their reasoning abilities.

[0004] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it can be considered prior art in this disclosure. Summary of the Invention

[0005] This specification provides a training method, question-answering method, electronic device, and storage medium for a reasoning model, which can improve the reasoning ability of the reasoning model.

[0006] Firstly, this specification provides a training method for a reasoning model, comprising: obtaining a sample question; inputting the sample question into a reasoning model and guiding the reasoning model to generate corresponding prediction results based on multiple reasoning trajectories, each reasoning trajectory including multiple reasoning steps executed sequentially, the prediction result corresponding to each reasoning trajectory being determined based on the reasoning results of the multiple reasoning steps, wherein a hybrid reasoning step among the multiple reasoning steps includes: determining anchor words with a matching degree higher than a target threshold and the neighboring words of the anchor words in the semantic dimension based on the matching degree between the current context information and each candidate word corresponding to the reasoning model, wherein in the first reasoning step, the current context information includes the sample question, and in subsequent reasoning steps, the current context information includes the sample question and the reasoning results determined in the previous reasoning steps; performing hybrid processing on the anchor words and the neighboring words to obtain the reasoning result of the hybrid reasoning step; and adjusting the reasoning strategy of the reasoning model based on the prediction results corresponding to the multiple reasoning trajectories.

[0007] Secondly, this specification also provides a question-and-answer method, comprising: receiving a target question sent by a terminal device; inputting the target question into an inference model to guide the inference model to answer the target question and obtain answer information, wherein the inference model is trained based on the method described in the first aspect; and sending the answer information to the terminal device.

[0008] Thirdly, this specification also provides an electronic device, comprising: at least one storage medium storing at least one instruction set; and at least one processor communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set and executes the method described in the first or second aspect according to the instructions of the at least one instruction set.

[0009] Fourthly, this specification also provides a computer-readable non-volatile storage medium, wherein the computer-readable non-volatile storage medium stores at least one instruction set, which, when executed by at least one processor, implements the method described in the first or second aspect above.

[0010] The training method, question-answering method, electronic device, and storage medium for the inference model provided in this specification guide the inference model to generate corresponding prediction results based on multiple inference trajectories during the training process. The inference strategy of the inference model is then adjusted based on these prediction results. Each inference trajectory includes multiple inference steps, among which a hybrid inference step is performed. In this hybrid inference step, anchor words and their neighboring words with a matching degree higher than a target threshold are identified from the candidate words corresponding to the inference model. The anchor words and neighboring words are then mixed to obtain the corresponding inference result. The inference result of this hybrid inference step can cover more information and can embed perturbations within the effective semantic range in the inference trajectory. While keeping the exploration center of the inference model aligned with a better inference trajectory, it expands the trajectory exploration range of the inference model. Therefore, it can encourage the inference model to explore a better inference trajectory with minimal impact on the logical coherence of the inference, thereby improving the inference model's inference ability.

[0011] The training methods, question-answering methods, electronic devices, and storage media for the inference model provided in this specification are partially listed in the following description. The inventive aspects of the training methods, question-answering methods, electronic devices, and storage media for the inference model provided in this specification can be fully explained through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of a question-and-answer scenario provided according to an embodiment of this specification is shown; Figure 2 A hardware structure diagram of an electronic device provided according to an embodiment of this specification is shown; Figure 3 A flowchart is shown of a training method for an inference model provided according to an embodiment of this specification; Figure 4 A simplified flowchart illustrating an implementation of a hybrid inference step according to an embodiment of this specification is shown. Figure 5 A simplified flowchart illustrating a training method for an inference model according to embodiments of this specification is shown; and Figure 6 A flowchart of a question-and-answer method provided according to an embodiment of this specification is shown. Detailed Implementation

[0014] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0015] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. The term “a plurality” refers to two or more, and “at least one” refers to one or more. The terms “first,” “second,” etc., may be used in this specification to describe various information, but such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, “first” may also be referred to as “second” without departing from the scope of embodiments of this specification, and similarly, “second” may also be referred to as “first.”

[0016] The term "at least one of A, B, or C" includes seven options: A only, B only, C only, both A and B, both A and C, both B and C, and both A, B, and C. Similarly, the statement "at least one of multiple items" refers to all possible combinations based on these items. The term "and / or" refers to any or all possible combinations of one or more associated listed items. For example, "A and / or B" includes three options: A only, B only, and both A and B. "A, B, and / or C" is equivalent to "at least one of A, B, or C".

[0017] The term "comprising" is an open-ended description and should be understood as "including but not limited to," potentially including other content beyond what has been described. When used in this specification, the terms "comprising," "including," and / or "containing" mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0018] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0019] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0020] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them excellent generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0021] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0022] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0023] With the development of artificial intelligence technology, large language models are increasingly being applied to people's work and daily lives, and the requirements for their effectiveness are becoming increasingly stringent. Large language models typically provide services to users in a question-and-answer format. They receive user input questions, reason based on those questions, generate corresponding responses, and then provide feedback to the user. The reasoning ability of large language models is crucial to the accuracy and reliability of the generated responses. This is especially true in scenarios involving scientific questions (such as those in mathematics, biology, physics, or chemistry), where the accuracy and reliability of the responses are paramount, thus demanding even higher levels of reasoning ability from large language models.

[0024] Typically, large language models with basic capabilities are further trained using reinforcement learning algorithms to improve their reasoning abilities. Among these, the Group Relative Policy Optimization (GRPO) algorithm is a highly efficient training method for large language models, making it a mainstream training algorithm. The core mechanism of GRPO is that, during training, for each input question, the large language model generates a series of diverse reasoning trajectories. By calculating the relative advantages among these trajectories, the policy network of the large language model is optimized, giving it stronger reasoning capabilities. In this specification, a reasoning trajectory refers to the entire process from the input question to the final answer, and may include a complete sequence of reasoning results. When training a large language model using the GRPO algorithm, there is no need to train a value network outside the policy network, simplifying the training process. If the large language model can generate more diverse and effective reasoning trajectories, its reasoning ability will be even stronger; therefore, it is necessary to improve the efficiency of the large language model's exploration of reasoning trajectories.

[0025] A large language model can correspond to a vocabulary containing tens of thousands or even more candidate tokens. Each candidate token in the vocabulary can be obtained based on samples used during the training of the large language model's basic capabilities. The response information generated by the large language model can be composed of combinations of tokens from this vocabulary. For example, if the vocabulary contains all numbers and operators, a response information obtained using this vocabulary would be "1+2=3", where "1", "+", "2", "=", and "3" are different tokens. In training a large language model based on the GRPO algorithm, each inference trajectory consists of multiple sequentially executed inference steps, also known as generation steps. Each inference step determines one token, and the final inference result corresponding to a inference trajectory (referred to as the prediction result in this specification) can be obtained by sequentially combining the tokens determined by each inference step.

[0026] In one related technique, a large language model generates different inference trajectories for each inference step of each inference trajectory for the same problem by randomly sampling from multiple possible lexical units in the vocabulary. This approach is limited to shallow exploration at the discrete lexical unit level, and the generated inference trajectories are usually just repetitions of surface wording (such as simply changing "1+2" ​​to "2+1"), making it difficult to touch on deep logical changes, resulting in a large amount of redundancy in the generated inference trajectories.

[0027] Another related technique attempts to explore inference trajectories within a continuous embedding space. This involves injecting noise (such as Gaussian noise) into the embedding vectors of the words determined in each inference step to alter the inference results and generate different inference trajectories. However, this approach suffers from semantic drift due to the anisotropy of the embedding space—meaning the density of word embeddings in the model is uneven across different directions. Simple noise injection can cause semantic drift (i.e., even small perturbations can cause the representation vectors to detach from the valid semantic region), leading to chaotic inference logic or even the generation of irrelevant gibberish.

[0028] Therefore, a mechanism is urgently needed for large language models to achieve both in-depth exploration of inference trajectories and ensure semantic consistency and logical coherence during their training process based on reinforcement learning algorithms. This mechanism aims to improve the training efficiency and final inference performance of large language models in inference tasks. The training based on reinforcement learning algorithms will be referred to as reinforcement training below.

[0029] This specification provides a training method, question-answering method, electronic device, and storage medium for a reasoning model. The reasoning model described in this specification is a large language model. During the reinforcement training process of the reasoning model, each reasoning trajectory includes multiple reasoning steps executed sequentially, including a hybrid reasoning step. In this hybrid reasoning step, anchor words and their neighboring words with a matching degree higher than a target threshold are identified from the candidate words corresponding to the reasoning model. The anchor words and neighboring words are then mixed to obtain the corresponding reasoning result. The reasoning result of this hybrid reasoning step can cover more information and can embed perturbations within the effective semantic range in the reasoning trajectory. While ensuring that the exploration center of the reasoning model still adheres to a better reasoning trajectory, the trajectory exploration range of the reasoning model is expanded. Therefore, it can encourage the reasoning model to explore a better reasoning trajectory with minimal impact on the logical coherence of reasoning, thereby improving the reasoning ability of the reasoning model.

[0030] The reasoning model provided in this manual can be applied to a variety of scenarios. For example, it can be applied to reasoning scenarios in certain scientific fields, such as mathematical reasoning, physical reasoning, and chemical reasoning. It can also be applied to general scenarios such as textual reasoning, social sentiment reasoning, or economic reasoning.

[0031] Figure 1 This diagram illustrates a question-answering scenario according to an embodiment of this specification, specifically an application scenario 100 of the question-answering method and the training method for the inference model provided in this specification. Figure 1 As shown, the application scenario 100 may include a target user 110, a terminal device 120, a server 130, and a network 140.

[0032] The target user 110 can be a user who engages in dialogue with the inference model, such as a user who asks a question. The target user 110 can operate the terminal device 120 to engage in dialogue with the inference model, such as inputting a target question into the dialogue model through the terminal device 120 and obtaining the response information output by the inference model.

[0033] Terminal device 120 can be a device that interacts with the inference model in response to a dialogue triggering action by target user 110. For example, the dialogue triggering action may include target user 110 clicking an icon of a dialogue application based on the inference model on terminal device 120, or opening a dialogue webpage based on the inference model, causing terminal device 120 to display the dialogue page. The dialogue triggering action may also include the user entering a target question on the dialogue page.

[0034] In some embodiments, the question-and-answer method can be executed on a terminal device 120. In this case, the terminal device 120 may store data or instructions for executing the question-and-answer method described herein, and may execute or be used to execute the data or instructions. In some embodiments, the terminal device 120 may include hardware devices with data processing capabilities and the necessary programs required to drive the hardware devices.

[0035] In some embodiments, terminal device 120 may include mobile devices, tablets, laptops, desktop computers, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, the smart home device may include smart TVs, or any combination thereof. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, the virtual reality device or the augmented reality device may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in device in the motor vehicle may include an in-vehicle computer, an in-vehicle television, etc.

[0036] Server 130 may be a server providing various services, such as a backend server supporting a chat application installed on terminal device 120 or a chat webpage opened on terminal device 120. Server 130 may be a single server or a server cluster consisting of multiple servers. In some embodiments, the question-and-answer method may be executed on server 130. In this case, server 130 may store data or instructions for executing the question-and-answer method described herein, and may execute or be used to execute said data or instructions. In some embodiments, server 130 may include hardware devices with data information processing capabilities and the necessary programs required to drive the hardware devices to work.

[0037] In some embodiments, the inference model can be deployed in server 130. Terminal device 120 can send a target question input by target user 110 to server 130, whereby the inference model infers the target question and generates corresponding response information. Then, server 130 sends the response information back to terminal device 120 for display to target user 110. This realizes the interaction process between target user 110 and the inference model. Server 130 can communicate with multiple terminal devices 120 and receive target questions sent by terminal devices 120. In some embodiments, terminal device 120 and server 130 can cooperate to implement the question-and-answer method provided in this specification.

[0038] In some embodiments, the training method for the inference model can be executed on server 130. In this case, server 130 may store data or instructions for executing the training method for the inference model described herein, and may execute or be used to execute said data or instructions. In some embodiments, server 130 may include hardware devices with data processing capabilities and the necessary programs required to drive the hardware devices.

[0039] Network 140 is a medium used to provide a communication connection between terminal device 120 and server 130. Network 140 can facilitate the exchange of information or data. Figure 1 As shown, terminal device 120 and server 130 can connect to network 140 and transmit information or data to each other through network 140. In some embodiments, network 140 can be any type of wired or wireless network, or a combination thereof. For example, network 140 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks. In some embodiments, network 140 may include one or more network access points. For example, network 140 may include wired or wireless network access points, such as base stations or Internet switching points, through which one or more components of terminal device 120 and server 130 can connect to network 140 to exchange data or information.

[0040] It should be understood that Figure 1 The number of terminal devices 120, servers 130, and networks 140 shown is merely illustrative. Any number of terminal devices 120, servers 130, and networks 140 can be included depending on implementation needs.

[0041] It should be noted that all user data obtained in this manual has been authorized by the user and does not involve user privacy.

[0042] Figure 2 A hardware structure diagram of an electronic device provided according to an embodiment of this specification is shown. Figure 2 Electronic device 200 in the middle can be used as Figure 1 Server 130 in the specification executes the training method or question-and-answer method of the inference model described in this specification.

[0043] like Figure 2 As shown, the electronic device 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, the electronic device 200 may also include a communication port 250 and an internal communication bus 210. The electronic device 200 may also include an input / output component (I / O component) 260.

[0044] The internal communication bus 210 can connect to different system components. For example, the internal communication bus 210 can connect to storage medium 230, processor 220, communication port 250, and I / O component 260, etc.

[0045] I / O component 260 supports input / output between electronic device 200 and other components.

[0046] Communication port 250 is used for data communication between electronic device 200 and the outside world. For example, communication port 250 can be used for data communication between electronic device 200 and a network. Communication port 250 can be a wired communication port or a wireless communication port.

[0047] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set may include computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc.

[0048] At least one processor 220 may be communicatively connected to at least one storage medium 230. When the electronic device 200 is running, at least one processor 220 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the training method or question-answering method of the inference model provided in this specification. The processor 220 may execute the steps included in the training method of the inference model or the steps included in the question-answering method. The processor 220 may be in the form of one or more processors. In some embodiments, the processor 220 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0049] For illustrative purposes only, the electronic device 200 in the accompanying drawings shows only one processor 220. However, it should be noted that the electronic device 200 in this specification may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor or by multiple processors in combination. For example, if the processor 220 of the electronic device 200 described in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0050] Figure 3 A flowchart illustrating a training method for an inference model according to an embodiment of this specification is shown. As previously described, the electronic device 200 can serve as... Figure 1 The server 130 in the electronic device 200 executes the training method P300 of the inference model described in this specification. Specifically, the processor 220 in the electronic device 200 can read the instruction set stored in its local storage medium, and then execute the training method P300 of the inference model described in this specification according to the provisions of the instruction set. For ease of description, the training method of the inference model will be referred to as the model training method below, and the device executing the method will be referred to as the model training device. For example, the model training device can be... Figure 1 Server 130 in the context of this model. The model training method can be based on the aforementioned GRPO algorithm. For example... Figure 3 As shown, method P300 may include the following steps 310 to 330.

[0051] Step 310: Obtain the sample problem.

[0052] Sample questions can be defined by staff or collected from user-submitted questions in historical conversations. The inference model requires reinforcement training based on multiple sample questions, and each sample question requires multiple iterative reinforcement training iterations. This manual only describes the process of one reinforcement training iteration for a single sample question; all other training iterations can be referenced in this manual.

[0053] The questions input into the reasoning model in this specification can be explicit textual statements, such as "Is principle X xx?". In some embodiments, the question can also be a statement or other textual statements, or even just a few phrases. Regardless of the form of expression, the input information can be considered as a question that the reasoning model needs to determine the answer to. For example, if the text input into the reasoning model is "principle X", it can be considered as the corresponding question "What is principle X?".

[0054] In some embodiments, the model training device may obtain sample questions from a sample set. Steps 320 and 330 may be performed for each obtained sample question.

[0055] Step 320: Input the sample problem into the inference model and guide the inference model to generate corresponding prediction results based on multiple inference trajectories. Each inference trajectory includes multiple inference steps executed sequentially. The prediction result corresponding to each inference trajectory is determined based on the inference results of the multiple inference steps. Among the multiple inference steps, there is at least one hybrid inference step.

[0056] In some embodiments, the model training device can obtain prompts corresponding to the inference model, and guide the large language model to infer the sample question based on these prompts to obtain the corresponding prediction results. The inference model is a type of large language model, such as a large language model based on the Transformer architecture. A large language model is a prompt-based generative model whose output is highly dependent on the guidance and control of the prompts. Specific prompts need to be constructed for each large language model to instruct it to perform the corresponding task. The prompts are essentially structured instructions or query information that can transform task intent into contextual information that the model can understand and execute, used to guide, constrain, and customize the output behavior of the large language model.

[0057] For example, the prompt information may include one or more components such as role definition, task instructions, contextual information, input / output format constraints, and examples. The role definition section is used to assign a specific role to the model (e.g., "You are an expert in xx"), enabling it to generate information from the perspective and expertise of that role. The task instructions section describes the specific task the model needs to perform (e.g., "Based on your knowledge, answer question xx using k problem-solving approaches"). The contextual information section provides background knowledge, reference information, or specific data to be processed related to the task. The input / output format constraints section explicitly specifies the format requirements for the model's output (e.g., "Output in JSON format"), ensuring the parsability of the output and its match with actual needs. The examples section provides one or more example input-output pairs to demonstrate the desired task execution, enabling the model to output the required results faster and in a more standardized manner.

[0058] For example, the model training device can concatenate the obtained sample question with the prompt information and input it into the inference model. This prompt information guides the inference model to reason based on multiple inference trajectories, thereby generating prediction results for each inference trajectory. The prompt information also guides the inference model to sequentially execute multiple inference steps for each inference trajectory, and specifies the steps that should be performed for each inference step. Thus, guided by this prompt information, the inference model can perform reinforcement learning according to the GRPO algorithm.

[0059] In some embodiments, the model training device can specify the number 's' of inference trajectories to be generated for the inference model. For example, this number 's' can be added to the prompt information input into the inference model. Based on this number 's', the inference model performs 's' independent inference processes, resulting in 's' inference trajectories after completing the inference. In some embodiments, the execution of these 's' inference trajectories can be parallel.

[0060] In this specification, each inference trajectory may include at least one hybrid inference step among its multiple inference steps. This hybrid inference step includes steps 322 and 324 as described below. The large language model can execute steps 322 and 324 as guided by prompts to implement the hybrid inference step and obtain the inference result corresponding to the hybrid inference step.

[0061] Step 322: Based on the matching degree between the current context information and each candidate word corresponding to the inference model, determine the anchor word with a matching degree higher than the target threshold, as well as the neighbor word of the anchor word in the semantic dimension. In the first inference step, the current context information includes the sample question. In subsequent inference steps, the current context information includes the sample question and the inference results determined in previous inference steps.

[0062] In each inference step of an inference trajectory, the inference model needs to determine the corresponding inference result based on the current context information. This context information is determined based on the sample question that the inference trajectory addresses. For example, in the first inference step, the current context information may only include the sample question.

[0063] In a reasoning trajectory, the reasoning steps are executed sequentially and are logically related. Subsequent reasoning steps rely on previous ones for their reasoning; therefore, the context information for a subsequent reasoning step can be determined based on the reasoning results established in previous steps. For example, in reasoning steps following the first inference step in a reasoning trajectory, the corresponding context information can include the sample question and the reasoning results established in previous steps.

[0064] As mentioned earlier, the inference model corresponds to a vocabulary, which includes multiple candidate lexical units. In each inference step of each inference trajectory, the inference model can determine the matching degree between each candidate lexical unit and the current context information.

[0065] In some embodiments, the inference model can determine the logits distribution of each candidate lexical unit, where each logit represents the degree of matching between the candidate lexical unit and the current context information. The inference model can calculate the logits for each candidate lexical unit at once to determine an overall logit distribution vector, which includes the logits of each candidate lexical unit, with each position in the vector corresponding to a candidate lexical unit in the vocabulary. For example, the current context information is input into a linear layer, which can yield the logit distribution of each candidate lexical unit.

[0066] The above logical values ​​are unnormalized raw scores. In some embodiments, the logical values ​​of each candidate word can be normalized, and the resulting probability can be used as the matching degree of the candidate word. For example, the SoftMax function can be used to process the distribution of these logical values ​​to obtain the probability distribution of each candidate word.

[0067] In some embodiments, temperature scaling can be applied to the logical values ​​of each candidate word to adjust the smoothness between the logical values ​​corresponding to each candidate word. For example, the temperature-scaled values ​​of each candidate word can be used to characterize the matching degree between the candidate word and the current context information. In some embodiments, the logical values ​​of each candidate word can be temperature-scaled and then normalized to obtain the probability distribution of each candidate word, and then the probability can be used as the matching degree of the candidate word.

[0068] The target threshold in step 322 can be a fixed value set manually, or it can be a threshold that can be dynamically adjusted. In some embodiments, the inference model can use the candidate word with the highest matching degree with the current context information as the anchor word. For example, the inference model can use the argmax function to determine the anchor word based on the matching degree of each candidate word. In some embodiments, the inference model can randomly select one word as the anchor word from multiple (e.g., the top n highest) candidate words with a matching degree higher than the target threshold.

[0069] After determining the anchor word, the inference model uses it as a benchmark to determine the semantically similar neighboring words, that is, words that are semantically similar to the anchor word. In some embodiments, there can be multiple neighboring words, which can be distributed around the anchor word. For example, the number of neighboring words k can be set manually, and the inference model can retrieve k neighboring words with high semantic similarity to the anchor word from among the candidate words.

[0070] In some embodiments, the inference model can obtain the embedding vectors of each candidate word. Based on the embedding vectors of each candidate word, the inference model can determine neighboring words that are semantically similar to the anchor word. For example, for the vocabulary corresponding to the inference model, a corresponding embedding matrix can be pre-generated, which includes the embedding vectors of each candidate word, and each embedding vector can include 238 elements.

[0071] In some embodiments, the inference model can calculate the cosine similarity between lexical units, using this cosine similarity to characterize semantic similarity, and thus determine the neighboring lexical units of the anchor lexical unit. Specifically, the inference model can determine the cosine similarity between the embedding vectors of other lexical units besides the anchor lexical unit and the embedding vector of the anchor lexical unit. Then, in descending order of similarity, a target number (e.g., k) of candidate lexical units with the highest cosine similarity are determined as the semantic neighbors of the anchor lexical unit. Cosine similarity effectively measures semantic similarity and can utilize the clustering properties of the embedding space to determine neighboring lexical units that are close to the anchor lexical unit in the embedding space. This ensures that the determination of neighboring lexical units conforms to the geometric characteristics of the embedding space (i.e., the direction in the embedding space determines semantics), effectively adapting to the anisotropy of the embedding space and facilitating subsequent effective mixing of lexical units.

[0072] For example, the inference model can determine the cosine similarity between each other word and the anchor word, and based on the determined cosine similarity, sort these other words in descending order of similarity, and then select the first k words in the resulting sequence as neighboring words. Alternatively, the inference model can also sort these other words in ascending order of similarity, and then select the last k words in the resulting sequence as neighboring words.

[0073] In the above embodiments, the inference model, after determining the anchor word, excludes the anchor word from the vocabulary and calculates the semantic similarity between other words and the anchor word, thus determining k neighboring words. In some embodiments, after determining the anchor word, the inference model may not exclude the anchor word from the vocabulary, but directly calculate the semantic similarity between all words in the vocabulary and the anchor word. Then, the k+1 words with the highest similarity are selected from the vocabulary, and these k+1 words include the anchor word and the k neighboring words. These k+1 words are then directly used for candidate processing.

[0074] In some embodiments, before step 322, such as before reinforcement training of the inference model, k neighboring words corresponding to each candidate word in the vocabulary can be pre-determined. The method for determining these neighboring words can be the same as in the embodiments described above. Then, the correspondence between each candidate word and its semantically neighboring words is stored in the target storage space. In other words, this correspondence is pre-stored in the target storage space before reinforcement training of the inference model. Accordingly, in step 322, after determining the anchor word, the inference model can determine the neighboring words of the anchor word based on this correspondence.

[0075] In this way, during the reinforcement training process, the inference model no longer needs to determine the semantic similarity between the anchor word and other words in each inference step. Instead, it can directly find the neighboring words corresponding to the anchor word based on the correspondence. This reduces the computation steps, improves the efficiency of determining neighboring words, and correspondingly improves the model training efficiency.

[0076] Step 324: Perform mixed processing on the anchor word and the neighbor word to obtain the reasoning result of the mixed reasoning step.

[0077] In the hybrid reasoning step, the reasoning model can determine the corresponding reasoning result based on anchor words and their neighboring words. For example, by mixing anchor words and their neighboring words, the reasoning result of the hybrid reasoning step can be obtained. This reasoning result can be used as the input for the next reasoning step. In this way, the reasoning model does not simply sample discrete words from the vocabulary as the reasoning result during the reasoning process, but also uses certain perturbations embedded in neighboring words to expand the exploration space of the reasoning model and help it find a better reasoning method.

[0078] Since the neighboring lexicon is located in the local semantic neighborhood of the anchor lexicon, the embedded perturbation is constrained by the anchor lexicon as the exploration center. This ensures that the inference result generated after the perturbation is embedded remains within the effective semantic flow. Therefore, it reduces the probability of semantic drift, that is, it reduces the probability of the intermediate state generated in the inference step deviating from the original logical inference path, and reduces the probability of generating irrelevant or erroneous inference steps. Furthermore, since the exploration center is an anchor lexicon with a high degree of matching with the context information, it can closely follow the original optimal inference path of the inference model, preventing the instability of the inference process caused by using a lexicon with a low degree of matching as the exploration center.

[0079] In some embodiments, the inference model can perform mixed processing on anchor terms and neighboring terms in the embedding space. For example, the embedding vectors of anchor terms and neighboring terms can be mixed to obtain the inference result. For instance, the inference model can linearly combine the embedding vectors of anchor terms and neighboring terms based on weights. This allows for the addition of perturbations to the inference process within a continuous embedding space, i.e., semantically guided embedding-level perturbations. This increases the information content of the inference result, enabling the inference to reach deeper logical changes and enhancing the efficiency of the inference model in effectively exploring different inference trajectories.

[0080] Since the embedding space of inference models typically exhibits a clear directionality, embedding-level exploration during the deployment of the inference model must adapt to local semantics. In the above embodiment, by constructing a perturbation mechanism at the embedding layer, the embedding vectors of the determined anchor lexical units are mixed with the embedding vectors of neighboring lexical units. Because these neighboring lexical units are obtained based on the similarity of the embedding vectors, they naturally cluster along the same semantic direction as the anchor lexical units. Therefore, the mixed embedding vectors are still confined to the effective semantic region, while providing the necessary randomness for effective expansion. This mechanism can bridge the gap between discrete lexical-level inference and continuous embedding-level exploration, enabling the inference model to explore a wider space that standard lexical-level sampling cannot reach.

[0081] In some embodiments, before step 324, the model training device may first obtain the embedding vectors of each candidate word. In step 324, the model training device may assign weights to the anchor word and neighboring words based on the matching degree of the anchor word and the matching degree of the neighboring words. Then, based on the weights of the anchor word and the neighboring words, the embedding vectors of the anchor word and the neighboring words are weighted to obtain the inference result of the hybrid inference step.

[0082] For example, higher weights can be assigned to words with higher matching degrees, and lower weights to words with lower matching degrees. The sum of the weights of the anchor word and the weights of each neighboring word is 1. Taking the identification of two neighboring words for the anchor word as an example, the resulting inference is... Where D represents the embedding vector of the anchor word, a represents the weight of the anchor word, L1 represents the embedding vector of a neighboring word, b represents the weight of that neighboring word, L2 represents the embedding vector of another neighboring word, and c represents the weight of that neighboring word.

[0083] In some embodiments, the model training device can normalize the matching degree of anchor words and the matching degree of neighboring words to obtain the weights of anchor words and neighboring words. Assuming that the matching degree of each candidate word is represented by the probability obtained after normalizing the logistic value, and the probability of the anchor word determined from each candidate word is 0.6, the probability of one neighboring word is 0.1, and the probability of another neighboring word is 0.1, then the weight assigned to the anchor word can be 0.75, and the weights assigned to both neighboring words can be 0.125.

[0084] In the above embodiments, the reasoning result of the hybrid reasoning step is a hybrid vector constructed in a continuous space. This hybrid vector contains both the main semantics of the anchor lexicon and introduces controlled continuous perturbations. Compared to reasoning methods that select discrete lexicons from a finite vocabulary, this hybrid vector can express subtle semantic changes that discrete lexicons cannot cover, thus significantly expanding the exploration space and helping the reasoning model discover better reasoning paths.

[0085] In the above embodiments, the degree of influence of neighboring words on the reasoning result is determined based on the semantic similarity between neighboring words and anchor words. Thus, the direction and intensity of the perturbation embedded during the reasoning process can be adaptively adjusted based on this semantic similarity to constrain the degree of perturbation. This expands the exploration range while maintaining consistency with the model's current semantic preferences as much as possible, preventing the reasoning result from being pushed into meaningless regions and preventing the reasoning trajectory from deviating from the normal trajectory, thereby maintaining the coherence of the reasoning logic.

[0086] In some embodiments, for each candidate word in the vocabulary, in addition to pre-storing its correspondence with its corresponding neighboring words, the weights of each candidate word and its corresponding neighboring words can also be pre-stored. During reinforcement training, after determining the anchor word in the hybrid inference step, the inference model can directly determine the corresponding neighboring words based on the pre-stored correspondences, and obtain the weights of the anchor word and each neighboring word based on the pre-stored weights, without needing to assign weights to each word in real time. This ensures the repeatability of the information used in the reinforcement training phase, simplifies the inference process of the inference model, and improves training efficiency.

[0087] Figure 4 A simplified flowchart illustrating an implementation of a hybrid inference step according to an embodiment of this specification is shown. Figure 4 As shown, in the hybrid inference step, the inference model can obtain the matching degree between each candidate word in the vocabulary and the current context information, and select anchor words from the candidate words based on the matching degree (corresponding to the aforementioned step 322). This matching degree can be determined by the inference model before this hybrid inference step. Figure 4 Each small rectangle within the first dashed box represents the matching degree between a candidate word (such as C1, C2, C3, C4, etc.) and the current context information. The length of the rectangle represents the magnitude of the matching degree. The inference model can select the candidate word with the highest matching degree as the anchor word, such as selecting C2 as the anchor word.

[0088] Subsequently, the inference model determines semantically similar neighboring lexical units based on the anchor lexical unit (corresponding to step 322 above). The inference model can determine neighboring lexical units by calculating the cosine similarity between the embedding vectors of lexical units in the embedding space. For example, refer to... Figure 4 The diagram within the second dashed box illustrates that lexical units within a range centered on anchor lexical unit C2 in the embedding space can be considered as neighboring lexical units of anchor lexical unit C2. For example... Figure 4 If the cosine similarity between the embedding vector of a word within the dashed circular region and the embedding vector of the anchor word C2 is greater than the target threshold, then the words within this region can be considered as neighboring words. Assume that the words within this region include C1, C2, C3, C4, and C5.

[0089] The inference model also assigns weights to anchor words and their neighbors based on the matching degree between the anchor words and their neighbors (corresponding to step 324 above). For example... Figure 4 Each small rectangle in the third dashed box corresponds to a word, and the length of the rectangle represents the magnitude of the corresponding weight. Then, based on the assigned weights, the inference model linearly combines the embedding vectors of the anchor word and the embedding vectors of each neighboring word, and uses the resulting mixed vector as the inference result of the mixed inference step.

[0090] Steps 322 and 324 described above only cover the steps executed within a single hybrid inference step. Other inference steps may precede and follow this hybrid inference step, and the inference result of the hybrid inference step can be used as input for the next inference step, such as as part of the context information of the next inference step. For any inference trajectory, if one of its inference steps is a hybrid inference step, its implementation can refer to the relevant descriptions in steps 322 and 324 above.

[0091] In some embodiments, multiple inference steps within each inference trajectory can all be hybrid inference steps. In some embodiments, in addition to hybrid inference steps, multiple inference steps within each inference trajectory also include standard inference steps; that is, some inference steps are hybrid inference steps, and some are standard inference steps. The standard inference step can be a method of directly sampling discrete lexical units in the vocabulary to obtain the inference result. This standard inference step includes sampling lexical units from each candidate lexical unit based on the matching degree between the current context information and the candidate lexical units corresponding to the inference model, and using the sampled target lexical unit as the inference result of the standard inference step. The sampling probability of each candidate lexical unit is positively correlated with its matching degree. In some embodiments, the inference model can randomly sample lexical units based on this matching degree; candidate lexical units with high matching degrees have a higher probability of being selected, but candidate lexical units with low matching degrees may also be selected.

[0092] The method for determining the matching degree is described in the previous section on step 322, and will not be repeated here. In some embodiments, the inference model can first determine the multiple words with a matching degree greater than a preset threshold or the top-ranking words based on the matching degree of each candidate word, and then sample the target word from these multiple words. In this way, the determined target word can be a word that is more consistent with the inference logic, preventing the sampling of words with too low relevance from causing errors in the inference logic.

[0093] In some embodiments, during the training of the inference model, for each inference step in each inference trajectory, the inference model determines the inference method used in that inference step based on a gating sampling mechanism, i.e., whether the inference step is a hybrid inference step or a standard inference step. When the gating mechanism is triggered, the inference method corresponding to the hybrid inference step is used, and the mixed result of multiple lexical units (such as the aforementioned mixed vector) is used as the input to the next inference step. When the gating mechanism is not triggered, the inference method corresponding to the standard inference step is used, and the target lexical unit sampled from the vocabulary is used as the input to the next inference step, such as using the embedding vector corresponding to the target lexical unit as the input to the next inference step.

[0094] For example, the inference model can randomly determine whether an inference step is a hybrid inference step or a standard inference step based on a preset mixing rate, where the preset mixing rate represents the probability of selecting a hybrid inference step. This preset mixing rate is a hyperparameter that can be specified by the developer or determined empirically. The preset mixing rate can be less than 0.5 to encourage the use of standard inference steps in the inference trajectory, ensuring inference stability. For example, the preset mixing rate can be 0.2, 0.15, or other probabilities.

[0095] In some embodiments, the model training device can obtain a Bernoulli distribution mask based on a preset mixing rate, and then determine whether the current inference step is a mixed inference step or a standard inference step based on the mask information. The Bernoulli distribution mask information *m* is a random variable with a probability of *r* of being 1 and a probability of *1-r* of being 0. Random trials are performed based on this probability *r*, and the resulting trial is a Bernoulli distribution mask, with the result being either *m=1* or *m=0*. One trial can be called one sampling. For example, each trial may include: generating a random number between 0 and 1, and determining whether the random number is less than *r*. If it is less, the trial result is *m=1*; otherwise, *m=0*. In the embodiments of this specification, the preset mixing rate can be used as the probability *r* that *m* is 1. The model training device performs random trials based on this preset mixing rate to obtain a Bernoulli distribution mask. When the trial result is *m=1*, the current inference step uses the mixed inference step method; when the trial result is *m=0*, the current inference step uses the standard inference step method.

[0096] In the above embodiments, in each inference trajectory of the inference model, the hybrid inference step and the standard inference step are used intermittently. This prevents the inference model from over-relying on the continuous perturbations added in the hybrid inference step, which could lead to the final inference result being inconsistent with conventional linguistic logic. Consequently, it can improve the stability of the inference model in long-chain inference and maintain the semantic stability of the generated data. Furthermore, by setting a smaller mixing rate, the number of hybrid inference steps in the inference trajectory is relatively small compared to the standard inference steps, ensuring that most inference steps still yield discrete lexical units, which can further mitigate the potential risks of training instability and semantic drift.

[0097] Each inference trajectory of the inference model includes at least one hybrid inference step and at least one standard inference step. The inference result generated in each inference step is an intermediate state of the inference trajectory. Based on the inference results of each inference step in an inference trajectory, the inference model generates the prediction result (i.e., the final inference result) corresponding to that inference trajectory. In some embodiments, the model training device can combine the anchor words determined in the hybrid inference step and the target words sampled in the standard inference step in the order of the inference steps to obtain the prediction result corresponding to the inference trajectory. That is, the inference results determined in the hybrid inference step are only used to assist the inference model in exploring inference trajectories, improving the difference between different inference trajectories while conforming to the inference logic, and are not directly used as fields in the prediction result.

[0098] Since the predicted result corresponding to the inference trajectory is the answer information to the sample question, it needs to meet the requirements of normal text expression. Therefore, the predicted result can be obtained by combining word units from the vocabulary to ensure that the inference model outputs normal answer information. The inference result in the hybrid inference step is the result of mixing multiple word units (such as a hybrid vector), and this result may not correspond to a single actual word unit. Based on this, for the hybrid inference step, its corresponding anchor word units can be used to compose the predicted result of the inference trajectory to ensure the normal implementation of the model function.

[0099] In addition, determining the prediction results in this way can help the subsequent model training equipment to verify the response information and determine the reward for each inference trajectory, so that there is no need to redesign the optimization method when optimizing the model parameters, and the inference model trajectory exploration and standard training optimization methods are compatible.

[0100] In some embodiments, the data generated in each inference step can be recorded using a data table. For example, for each hybrid inference step, the anchor terms determined in that inference step can be recorded; for each standard inference step, the target terms determined in that inference step can be recorded. After an inference trajectory is completed, the anchor terms corresponding to each hybrid inference step and the target terms corresponding to each standard inference step can be found through table lookup. These terms are then combined according to the execution order of the inference steps to obtain the prediction result corresponding to the inference trajectory.

[0101] In some embodiments, during the training of the inference model, previously generated data may be reused multiple times, such as data generated in certain inference steps. Recording the data generated in the inference steps facilitates the accurate reproduction of previous inference states and subsequent processing.

[0102] In this specification, the execution of each inference step in each inference trajectory by the inference model is random. For example, the inference method used for the inference step at the same position in different inference trajectories may be different, and even if the inference method is the same, the inference results may be different. Therefore, the multiple inference trajectories generated by the inference model can be different, and the prediction results corresponding to different inference trajectories will also be different.

[0103] Step 330: Based on the prediction results corresponding to the multiple inference trajectories, adjust the inference strategy of the inference model.

[0104] During each training iteration of the inference model for a given sample problem, corresponding prediction results can be obtained based on multiple inference trajectories. The model training device can then adjust the inference model based on these prediction results. The inference model may include a policy network, which determines the inference strategy employed during the inference process. Adjusting the inference model can involve modifying the parameters of this policy network, also known as adjusting the inference strategy of the inference model.

[0105] In this specification, the inference model is reinforced and trained based on the GRPO algorithm. Accordingly, based on this algorithm, the model training device can perform intra-group comparisons of the prediction results corresponding to the multiple inference trajectories to determine how to adjust the inference strategy of the inference model. In some embodiments, step 330 may include steps s21 to s24 below.

[0106] Step s21: Based on the prediction results corresponding to the multiple inference trajectories, obtain the reward information for each inference trajectory.

[0107] In some embodiments, the model training device also obtains standard response information corresponding to the sample question, and then compares this standard response information with the prediction result of each inference trajectory to determine the reward information of that inference trajectory. In some embodiments, the model training device also analyzes each inference step in each inference trajectory to determine the reward information of that inference trajectory. For example, the model training device may use a programmed evaluator to score the prediction results corresponding to each inference trajectory to obtain the reward information of each inference trajectory.

[0108] Step s22: Construct an advantage reference baseline based on the reward information of each inference trajectory.

[0109] In some embodiments, the model training device can calculate the arithmetic mean of the reward information for each inference trajectory and use it as the advantage reference baseline. For different training epochs, the generated inference trajectories are different, and the corresponding advantage reference baseline is also different; therefore, the advantage reference baseline is a dynamically changing baseline.

[0110] Step s23: Compare the reward information of each inference trajectory with the advantage reference baseline to obtain the advantage information of each inference trajectory.

[0111] In some embodiments, the model training device can calculate the difference between the reward information of each inference trajectory and the advantage reference baseline, and use this difference as the advantage information of the corresponding inference trajectory. This advantage information can be positive or negative.

[0112] In some embodiments, the model training device can calculate the standard deviation of the reward information of the multiple inference trajectories and the dominant reference baseline. The dominance information of each inference trajectory is obtained by dividing the difference between the reward information of each inference trajectory and the dominant reference baseline by the standard deviation. In some embodiments, the dominance information of each inference trajectory can be obtained by dividing the difference between the reward information of each inference trajectory and the dominant reference baseline by the sum of the standard deviation and another stable parameter. This stable parameter has a small value to avoid a denominator of zero and ensure numerical stability.

[0113] Step s24: Based on the advantage information of each inference trajectory, adjust the inference strategy of the inference model so that the inference strategy is close to the inference trajectory with advantage information higher than the reference threshold and far away from the inference trajectory with advantage information lower than the reference threshold.

[0114] The model training device can obtain a reference threshold, which serves as a benchmark for evaluating the dominance information of inference trajectories. Inference trajectories with dominance information above the reference threshold are considered superior, while those with dominance information below the reference threshold are considered inferior. The model training device adjusts the policy parameters of the inference model to make the inference policy gravitate towards inference trajectories with dominance information above the reference threshold and away from those with dominance information below the reference threshold. In other words, it encourages the inference model to favor superior inference trajectories and discard inferior ones.

[0115] In some embodiments, the reference threshold is 0. In this case, a positive dominance value indicates that the corresponding inference trajectory should be encouraged, while a negative dominance value indicates that the corresponding inference trajectory should be suppressed. In some embodiments, the reference threshold can also be other values ​​greater than 0 to encourage faster updates to the inference model. In some embodiments, the reference threshold can also be other values ​​less than 0 to encourage more stable updates to the inference model.

[0116] In some embodiments, the inference model before reinforcement training is a pre-trained model that already possesses a certain level of inference capability. During the reinforcement training of the inference model, the parameter update magnitude of the inference model can also be constrained to reduce the impact on the original performance of the model and prevent excessive updates to the inference model that could lead to performance collapse.

[0117] In some embodiments, the model training device can also determine the difference information between the current inference strategy of the inference model and the original inference strategy, which is the inference strategy before the inference model underwent reinforcement training. In step s24, the model training device can adjust the inference strategy of the inference model based on the advantage information of each inference trajectory and the difference information, wherein the difference information is used to constrain the update magnitude of the inference strategy.

[0118] For example, during reinforcement training, the model training device can determine the differences between the current inference model and the original inference model (which has not undergone reinforcement training) in terms of problem reasoning. These differences can be represented by information about the differences between the current and original inference strategies. The model training device can then use this difference information as a penalty to limit the policy updates of the inference model.

[0119] In some embodiments, the model training device can calculate the KL divergence between the old and new strategies of the inference model, using the KL divergence to represent the difference between the old and new strategies. Then, based on the KL divergence and the advantage information of each inference trajectory, the final loss information is obtained, and the inference strategy of the inference model is adjusted based on the loss information.

[0120] In some embodiments, the model training device can measure the difference between the old and new strategies by pruning the probability ratio, and adjust the inference strategy of the inference model based on this and the advantage information of each inference trajectory.

[0121] Figure 5 A simplified flowchart illustrating a training method for an inference model according to an embodiment of this specification is shown. Figure 5 As shown, at any stage of the reinforcement training process of the inference model, for a sample problem obtained by the model training device, the model training device can input the sample problem into the current inference model and guide the inference model to execute the inference process multiple times to generate multiple inference trajectories. This inference process can correspond to the aforementioned inference steps, and for each inference trajectory, the inference model needs to execute the inference process multiple times in sequence to obtain it.

[0122] During each inference process, the inference model obtains the matching degree between each candidate word in the corresponding vocabulary and the current context information. Then, the mask information of the Bernoulli distribution can be used to determine whether the gating mechanism is triggered, thus determining whether the current inference process uses a standard inference step or a hybrid inference step. When the gating mechanism is triggered (i.e., m=1), a hybrid inference step can be executed. This involves determining the anchor word and its neighboring words and mixing them, using the resulting hybrid vector as the inference result for that step. When the gating mechanism is not triggered (i.e., m=0), a standard inference step can be executed. This involves sampling the target word in the vocabulary to generate the inference result for that step.

[0123] By executing the above reasoning process multiple times in parallel, the reasoning model can generate multiple reasoning trajectories. For example... Figure 5 Each row of small rounded rectangles represents an inference trajectory, with lighter-colored rectangles representing standard inference steps and darker-colored rectangles representing mixed inference steps. The model training device can then calculate the reward and advantage information for each of these multiple inference trajectories.

[0124] The model training device can also compare the current inference model with the original inference model to determine the differences between inference strategies and obtain the KL divergence (corresponding to...). Figure 5 The KL loss is used to adjust the inference model. By integrating this KL loss with the advantage information of each determined inference trajectory, the total loss used to adjust the inference model is obtained. This total loss is then used to adjust the inference strategy of the current inference model, achieving one training iteration. The inference model can then be trained again.

[0125] The training method provided in the above embodiments enhances the overall reasoning ability of the inference model through reinforcement training. Even when using only sample data from one domain during reinforcement training, it can improve the model's reasoning ability on data outside that domain, demonstrating the good applicability and robustness of this training method.

[0126] In summary, the training method for the inference model provided in this specification guides the inference model to generate corresponding prediction results based on multiple inference trajectories during the training process, based on sample questions. The inference strategy of the inference model is then adjusted based on these prediction results. Each inference trajectory includes multiple inference steps, among which a hybrid inference step is performed. In this hybrid inference step, anchor words and their neighboring words with matching degrees higher than a target threshold are identified from the candidate words corresponding to the inference model. The anchor words and neighboring words are then mixed to obtain the corresponding inference result. The inference result of this hybrid inference step can cover more information and can embed perturbations within the effective semantic range in the inference trajectory. While ensuring that the exploration center of the inference model still aligns with a better inference trajectory, the trajectory exploration range of the inference model is expanded. Therefore, it can encourage the inference model to explore a better inference trajectory with minimal impact on the logical coherence of the inference, thereby improving the inference model's inference ability.

[0127] Figure 6 A flowchart of a question-and-answer method provided according to an embodiment of this specification is shown. As before, electronic device 200 can serve as... Figure 1 Server 130 in the electronic device 200 executes the question-and-answer method P600 of this specification. Specifically, processor 220 in the electronic device 200 can read the instruction set stored in its local storage medium and then execute the question-and-answer method P600 of this specification according to the provisions of the instruction set. Figure 6 As shown, method P600 may include the following steps 610 to 630.

[0128] Step 610: Receive the target question sent by the terminal device.

[0129] The terminal device can launch a conversational application upon user triggering and display the application's interface. The user can interact with the application's interface to input a target question. The terminal device then sends the target question to a server, requesting an inference model deployed on the server to respond to it. In some embodiments, the server may be a backend server for the conversational application installed on the terminal device.

[0130] Step 620: Input the target question into the inference model to guide the inference model to answer the target question and obtain the answer information. The inference model is trained based on the aforementioned training method for inference models.

[0131] After obtaining the target question, the server can invoke the inference model deployed on the server, inputting the target question into the inference model to generate the corresponding answer information. In some embodiments, the server can concatenate the target question with corresponding prompts before inputting it into the inference model to guide the inference model to execute the corresponding inference process.

[0132] When reasoning about the target question, the inference model generates the corresponding response information based on only one inference path. In each inference step within this path, the model directly selects appropriate terms from the corresponding vocabulary as the inference result for that step. After the inference path is completed, the inference results from each step are combined sequentially to obtain the response information corresponding to the target question.

[0133] Step 630: Send the reply information to the terminal device.

[0134] After receiving the response, the server sends it to the terminal device for display. For example, the terminal device can display the target question and the response in a dialog box.

[0135] The question-answering method provided in this manual utilizes a reasoning model that, during training, can explore a wider range of trajectories while minimizing the impact on the logical coherence of the reasoning. This allows for the discovery of optimal reasoning trajectories and ultimately, improved reasoning ability. Consequently, the accuracy and reliability of the responses generated based on this reasoning model for the target question can be high, enhancing the reliability of the question-answering process and improving the user's conversational experience.

[0136] This specification, in another aspect, provides a computer-readable non-transitory storage medium storing at least one set of instructions. When the at least one set of instructions is executed by a processor, it instructs the processor to implement the steps of the training method P300 or the question-answering method P600 of the inference model described herein. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on an electronic device 200, the program code causes the electronic device 200 to execute the steps of the training method P300 or the question-answering method P600 of the inference model described herein. The program product for implementing the above methods may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the electronic device 200. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a readable storage medium include: portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include a data signal propagated as part of a carrier wave in baseband, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on electronic device 200, partially on electronic device 200, as a standalone software package, partially on electronic device 200 and partially on a remote computing device, or entirely on a remote computing device.

[0137] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0138] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0139] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0140] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

[0141] Every patent, patent application, publication of a patent application, and other material such as articles, books, specifications, publications, documents, articles, etc., cited herein, except for those inconsistent with or conflicting with this document, or those having a restrictive effect on the widest scope of the claims, may be incorporated herein by reference for all purposes now or hereafter associated with this document. Furthermore, in the event of any inconsistency or conflict between the description, definition, and / or use of relevant terms in any material and the description, definition, and / or use of relevant terms in this document, the terms in this document shall prevail.

[0142] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for training an inference model, comprising: The problem of obtaining samples; The sample problem is input into the inference model, which is then guided to generate corresponding prediction results based on multiple inference trajectories. Each inference trajectory includes multiple inference steps executed sequentially. The prediction result corresponding to each inference trajectory is determined based on the inference results of the multiple inference steps. The mixed inference steps among the multiple inference steps include: Based on the matching degree between the current context information and each candidate word corresponding to the inference model, anchor words with matching degrees higher than the target threshold are determined, as well as the neighboring words of the anchor words in the semantic dimension. In the first inference step, the current context information includes the sample question, and in subsequent inference steps, the current context information includes the sample question and the inference results determined in previous inference steps. The anchor terminology and the neighboring terminology are mixed to obtain the reasoning result of the mixed reasoning step; and Based on the prediction results corresponding to the multiple inference trajectories, the inference strategy of the inference model is adjusted.

2. The method according to claim 1, further comprising: Obtain the embedding vector of each candidate word; as well as The process of mixing the anchor terminology and the neighboring terminology to obtain the reasoning result of the mixed reasoning step includes: Based on the matching degree of the anchor word and the matching degree of the neighbor word, weights are assigned to the anchor word and the neighbor word. as well as Based on the weights of the anchor words and the neighbor words, the embedding vectors of the anchor words and the neighbor words are weighted to obtain the reasoning result of the hybrid reasoning step.

3. The method according to claim 2, wherein, Based on the matching degree of the anchor word and the matching degree of the neighboring word, weights are assigned to the anchor word and the neighboring word, including: The matching degree of the anchor word and the matching degree of the neighbor word are normalized to obtain the weight of the anchor word and the weight of the neighbor word.

4. The method according to claim 1, wherein, The plurality of inference steps also includes a standard inference step, which includes: Based on the current context information and the matching degree of each candidate word, word sampling is performed on each candidate word, and the sampled target word is used as the reasoning result of the standard reasoning step. The sampling probability of each candidate word is positively correlated with the matching degree.

5. The method according to claim 4, further comprising: For each inference step, the inference step is randomly determined to be either the hybrid inference step or the standard inference step based on a preset mixing rate, wherein the preset mixing rate represents the probability of the hybrid inference step being selected.

6. The method according to claim 5, wherein, The step of randomly determining the inference step as either the hybrid inference step or the standard inference step based on a preset mixing rate includes: The mask information of the Bernoulli distribution is obtained based on the preset mixing rate; and Based on the mask information, the inference step is determined to be either the hybrid inference step or the standard inference step.

7. The method according to claim 4, wherein, For each inference trajectory, a corresponding prediction result is generated based on the inference trajectory, including: The anchor words determined in the hybrid inference step of the inference trajectory are combined with the target words sampled in the standard inference step in the order of the inference steps to obtain the prediction result corresponding to the inference trajectory.

8. The method according to claim 7, wherein, Based on the prediction results corresponding to the multiple inference trajectories, the inference strategy of the inference model is adjusted, including: Based on the prediction results corresponding to the multiple inference trajectories, the reward information for each inference trajectory is obtained; A superiority reference baseline is constructed based on the reward information of each reasoning trajectory; The reward information of each inference trajectory is compared with the dominance reference baseline to obtain the dominance information of each inference trajectory; and Based on the advantage information of each inference trajectory, the inference strategy of the inference model is adjusted so that the inference strategy is close to the inference trajectory where the advantage information is higher than the reference threshold, and far away from the inference trajectory where the advantage information is lower than the reference threshold.

9. The method according to claim 8, wherein, The inference model is a pre-trained model, and the method further includes: determining the difference information between the current inference strategy of the inference model and the original inference strategy, wherein the original inference strategy is the inference strategy before the inference model was trained; and The adjustment of the reasoning strategy of the reasoning model based on the advantage information of each reasoning trajectory includes: Based on the advantage information and the difference information of each inference trajectory, the inference strategy of the inference model is adjusted, wherein the difference information is used to constrain the update range of the inference strategy.

10. The method according to claim 1, wherein, Determining the neighborhood lexicons of the anchor lexicon in the semantic dimension includes: Determine the cosine similarity between the embedding vectors of all candidate lexical units other than the anchor lexical unit and the embedding vector of the anchor lexical unit; and Based on the order of similarity from high to low, the top number of candidate words with the highest cosine similarity are determined as the neighborhood words of the anchor word in the semantic dimension.

11. The method according to claim 1, wherein, The target storage space pre-stores the correspondence between each candidate word and its neighboring words in the semantic dimension; Determining the neighborhood lexicons of the anchor lexicon in the semantic dimension includes: Based on the correspondence, the neighboring words of the anchor word are determined.

12. A question-and-answer method, comprising: The target problem sent by the receiving terminal device; The target question is input into the inference model to guide the inference model to answer the target question and obtain the answer information, wherein the inference model is trained based on the method described in any one of claims 1 to 11; as well as The response information is sent to the terminal device.

13. An electronic device, comprising: At least one storage medium storing at least one instruction set; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set and executes the method of any one of claims 1 to 12 according to the instructions of the at least one instruction set.

14. A computer-readable non-volatile storage medium, wherein, The computer-readable non-volatile storage medium stores at least one instruction set, which, when executed by at least one processor, implements the method as described in any one of claims 1 to 12.

Images