A training method, device and equipment of a thinking chain evidence reference industrial large model

Abstract

Embodiments of the present application provide a training method, device and equipment for a thinking chain evidence reference industrial large model, and relate to the field of natural language processing. The method uses training samples including training questions, answers and thinking chains, so that the model can learn the logical links for deriving answers from document content. The training samples are disassembled into multiple word units, weights are configured according to the types of the word units, the importance of different word units in loss calculation is distinguished, a difficulty modulation factor is used to quantify the prediction difficulty of the word units, the generation difficulty of the model for different word units is represented, the training loss is calculated in combination with the weights, the model parameters are continuously optimized through back propagation relying on the training loss, and the final model is determined until the overall training loss meets the preset convergence condition. In application, the model can output answers to questions, and can also present the derivation logic and reference sources of the answers, which facilitates staff to check the derivation process and reference basis of the answers, and solves the technical problem of insufficient traceability.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and in particular to a training method, apparatus and equipment for a large industrial model of thought chain evidence citation. Background Technology

[0002] In scenarios such as industrial quality inspection, equipment maintenance, and safety procedure Q&A, the volume of industrial documents is enormous, and the industry's demand for intelligent Q&A continues to rise. Industrial scenarios have stringent requirements for the authenticity of output content, requiring answers to have clear decision-making basis to support engineering verification and quality traceability. At the same time, industrial document data is sensitive and annotation costs are high, making it difficult for traditional Q&A models to adapt to the needs of these industrial scenarios. Therefore, there is an urgent need for highly reliable Q&A solutions for industrial documents.

[0003] In existing technologies, intelligent question answering is usually achieved by using retrieval-enhanced generation technology. This involves matching and recalling text fragments related to the question from industrial documents, inputting the retrieval results and the user's question into a generation model, and then having the model generate the corresponding answer to complete the question answering task.

[0004] However, existing technologies suffer from poor traceability. When generating answers to questions, operators cannot verify the authenticity of the answers' sources, resulting in poor traceability. Summary of the Invention

[0005] This application provides a training method, apparatus, and equipment for a large industrial model of thought chain evidence citation, in order to solve the problem of poor traceability in the prior art.

[0006] In a first aspect, embodiments of this application provide a training method for a large-scale industrial model of thought chain evidence citation, including:

[0007] Multiple training samples are obtained; each training sample includes a training question, the answer to the training question, and a thought process, wherein the thought process refers to the process of deriving the answer to the training question from the content of a preset document based on the training question.

[0008] Any training sample is split into multiple words, and a weight is assigned to any word based on its word type; wherein the weight is used to represent the importance of the word in the calculation of training loss.

[0009] Obtain the difficulty modulation factor of any word, calculate the training loss of any training sample based on the weight and the difficulty modulation factor, and determine the training loss of the plurality of training samples based on the training loss of any training sample; wherein, the difficulty modulation factor is used to represent the prediction difficulty of the preset model to be trained for the any word.

[0010] The parameters of the preset training model are backpropagated based on the training loss of the multiple training samples. The preset training model whose average training loss of the multiple training samples is less than a preset threshold is determined as the Mind Chain Evidence Reference Industrial Model. The Mind Chain Evidence Reference Industrial Model is used to generate the answer to the question to be processed, the source of the reference of the answer, and the process of deriving the answer to the question to be processed from the content of a preset document.

[0011] In one possible implementation, acquiring multiple training samples includes:

[0012] Obtain multiple training questions and their answers, split any training question into multiple semantic units, and obtain the reference interval of each semantic unit; wherein, the reference interval is used to represent the start and end positions of the evidence fragment corresponding to each semantic unit in the preset document content, and the evidence fragment is the text located in the preset document content according to the reference interval;

[0013] An evidence set for any training question is generated based on the reference interval; wherein, the evidence set is a set of evidence fragments corresponding to the plurality of semantic units;

[0014] Based on the multiple semantic units and the evidence fragments, the multiple training questions are screened to eliminate abnormal questions and obtain multiple effective training questions.

[0015] Input any valid training question, the answer to any valid training question, the evidence set, and the content of the preset document into the preset teacher model to obtain the thought chain of any valid training question.

[0016] The thought chain and any effective training problem are integrated to obtain the target training sample corresponding to the effective training problem, and the plurality of training samples are determined based on the target training sample.

[0017] In one possible implementation, the step of filtering the multiple training questions based on the multiple semantic units and the evidence fragments to eliminate abnormal questions among the multiple training questions and obtain multiple effective training questions includes:

[0018] Based on the comparison between the reference number of each semantic unit and the total number of fragments in the preset document content, the multiple training questions are filtered to remove training questions of semantic units with reference numbers less than one or greater than the total number of fragments, resulting in multiple first questions; wherein, the reference number refers to the sequence number of the evidence fragment corresponding to each semantic unit in the preset document content;

[0019] Based on the content length of the evidence fragment corresponding to each semantic unit, the multiple first questions are filtered out to eliminate the first questions where the content length of the evidence fragment corresponding to a semantic unit is zero, thus obtaining multiple second questions;

[0020] Based on the comparison results between semantic similarity and a preset similarity threshold, the multiple second questions are filtered to remove the second questions containing semantic units with semantic similarity lower than the preset similarity threshold, thereby obtaining the multiple effective training questions; wherein, the semantic similarity refers to the similarity between each semantic unit and the evidence fragment corresponding to each semantic unit.

[0021] In one possible implementation, assigning weights to any lexical based on its lexical type includes:

[0022] The text content of any given word is obtained, and the text content is matched with multiple preset regular expression patterns to obtain a matching result; wherein, the multiple preset regular expression patterns include evidence anchor pattern, citation structure pattern, statement structure pattern and natural language pattern;

[0023] If, based on the matching result, it is determined that the text content successfully matches the evidence anchor pattern, the word type of any word is determined as the evidence anchor class;

[0024] If, based on the matching result, it is determined that the text content successfully matches the reference structure pattern, the word type of any word element is determined as a reference structure class;

[0025] If, based on the matching result, it is determined that the text content successfully matches the statement structure pattern, the word type of any word element is determined to be a statement structure class;

[0026] If, based on the matching result, it is determined that the text content successfully matches the natural language pattern, the word type of any word is determined to be a natural language class;

[0027] Set the weight of any one word to a preset weight value corresponding to the word type.

[0028] In one possible implementation, setting the weight of any lexical unit to a preset weight value corresponding to the lexical unit type includes:

[0029] If the word type of any word is the evidence anchor class, the weight of any word is set to the preset first weight.

[0030] If the word type of any word is the reference structure class, the weight of any word is set to a preset second weight; wherein, the first weight is greater than the second weight;

[0031] If any word element is of the statement structure type, the weight of any word element is set to a preset third weight; wherein, the second weight is greater than the third weight;

[0032] If the word type of any word is the natural language class, the weight of any word is set to a preset fourth weight; wherein the third weight is greater than the fourth weight.

[0033] In one possible implementation, obtaining the difficulty modulation factor of the any given word and calculating the training loss of the any training sample based on the weights and the difficulty modulation factor includes:

[0034] Obtain the prediction probability of any given word, and calculate the difficulty modulation factor of the given word based on the prediction probability; wherein, the prediction probability is used to represent the prediction accuracy of the preset training model for the given word.

[0035] The training loss of any given word is calculated based on the weights and the difficulty modulation factor, and the training loss of any training sample is determined based on the training loss of the given word.

[0036] Secondly, embodiments of this application provide a training device for a large-scale industrial model of evidence citation based on thought chains, comprising:

[0037] The acquisition module is used to acquire multiple training samples; wherein each training sample includes a training question, the answer to the training question, and a thought chain, wherein the thought chain refers to the process of deriving the answer to the training question from the content of a preset document based on the training question;

[0038] The splitting module is used to split any training sample into multiple word units and set weights for any word unit according to its word unit type; wherein, the weights are used to represent the importance of the word unit in the calculation of training loss.

[0039] The calculation module is used to obtain the difficulty modulation factor of any word, calculate the training loss of any training sample according to the weight and the difficulty modulation factor, and determine the training loss of the plurality of training samples according to the training loss of any training sample; wherein, the difficulty modulation factor is used to represent the prediction difficulty of the preset model to be trained for the word.

[0040] The training module is used to backpropagate the parameters of the preset model to be trained based on the training loss of the multiple training samples, and to determine the preset model to be trained when the average value of the training loss of the multiple training samples is less than a preset threshold as the thought chain evidence citation industrial big model; wherein, the thought chain evidence citation industrial big model is used to generate the answer to the question to be processed, the citation source of the answer, and the process of deriving the answer to the question to be processed from the content of a preset document.

[0041] In one possible implementation, the acquisition module includes:

[0042] The first acquisition unit is used to acquire multiple training questions and the answers to the multiple training questions, split any training question into multiple semantic units, and acquire the reference interval of each semantic unit; wherein, the reference interval is used to represent the start and end positions of the evidence fragment corresponding to each semantic unit in the preset document content, and the evidence fragment is the text located in the preset document content according to the reference interval;

[0043] A generation unit is configured to generate an evidence set for any training question based on the reference interval; wherein the evidence set is a set of evidence fragments corresponding to the plurality of semantic units;

[0044] A filtering unit is used to filter the multiple training questions based on the multiple semantic units and the evidence fragments, so as to remove abnormal questions from the multiple training questions and obtain multiple effective training questions;

[0045] The input unit is used to input any valid training question, the answer to any valid training question, the evidence set, and the content of the preset document into the preset teacher model to obtain the thought chain of the any valid training question.

[0046] An integration unit is used to integrate the thought chain and any effective training problem to obtain a target training sample corresponding to any effective training problem, and to determine the plurality of training samples based on the target training sample.

[0047] In one possible implementation, the filtering unit includes:

[0048] The first filtering component is used to filter the multiple training questions based on the comparison results between the reference number of each semantic unit and the total number of fragments in the preset document content, so as to remove the training questions of semantic units with reference numbers less than one or greater than the total number of fragments, and obtain multiple first questions; wherein, the reference number refers to the sequence number of the evidence fragment corresponding to each semantic unit in the preset document content;

[0049] The second filtering component is used to filter the multiple first questions based on the content length of the evidence fragment corresponding to each semantic unit, so as to remove the first questions where the content length of the evidence fragment corresponding to the semantic unit is zero, and obtain multiple second questions.

[0050] The third filtering component is used to filter the plurality of second questions based on the comparison results of semantic similarity and preset similarity threshold, so as to remove the second questions with semantic units whose semantic similarity is lower than the preset similarity threshold, and obtain the plurality of effective training questions; wherein, the semantic similarity refers to the similarity between each semantic unit and the evidence fragment corresponding to each semantic unit.

[0051] In one possible implementation, the splitting module includes:

[0052] The second acquisition unit is used to acquire the text content of any one word and match the text content with a plurality of preset regular expression patterns to obtain a matching result; wherein, the plurality of preset regular expression patterns include evidence anchor pattern, citation structure pattern, statement structure pattern and natural language pattern.

[0053] The first determining unit is configured to determine the word type of any word as the evidence anchor class if it is determined from the matching result that the text content matches the evidence anchor pattern successfully.

[0054] The second determining unit is used to determine the word type of any word as a reference structure class if it is determined that the text content matches the reference structure pattern successfully based on the matching result.

[0055] The third determining unit is used to determine the word type of any word as a statement structure class if it is determined that the text content matches the statement structure pattern successfully based on the matching result.

[0056] The fourth determining unit is used to determine the lexical type of any lexical as a natural language class if it is determined that the text content matches the natural language pattern successfully based on the matching result.

[0057] The fifth determining unit is used to set the weight of any word element to a preset weight value corresponding to the word element type.

[0058] In one possible implementation, the fifth determining unit includes:

[0059] The first setting component is used to set the weight of any word element to a preset first weight if the word element type of any word element is the evidence anchor point class.

[0060] The second setting component is used to set the weight of any word element to a preset second weight if the word element type of any word element is the reference structure class; wherein the first weight is greater than the second weight.

[0061] The third setting component is used to set the weight of any word element to a preset third weight if the word element type of any word element is the statement structure class; wherein the second weight is greater than the third weight.

[0062] The fourth setting component is used to set the weight of any word element to a preset fourth weight if the word element type of any word element is the natural language class; wherein the third weight is greater than the fourth weight.

[0063] In one possible implementation, the computing module includes:

[0064] The third acquisition unit is used to acquire the prediction probability of any word and calculate the difficulty modulation factor of any word based on the prediction probability; wherein, the prediction probability is used to represent the prediction accuracy of the preset training model for any word.

[0065] The calculation unit is configured to calculate the training loss of any word based on the weight and the difficulty modulation factor, and determine the training loss of any training sample based on the training loss of the any word.

[0066] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0067] The memory stores computer-executed instructions;

[0068] When the processor executes the computer execution instructions stored in the memory, it is used to implement the training method of the industrial large model of thought chain evidence citation as described in any of the first aspects.

[0069] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the training method for the industrial large model of thought chain evidence citation as described in any of the first aspects.

[0070] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, is used to implement the training method for the industrial large model of thought chain evidence citation as described in any of the first aspects.

[0071] This application provides a training method, apparatus, and device for a large-scale industrial model of thought chain evidence citation. By integrating training questions, corresponding answers, and thought chains used to reconstruct the logic of the answers into training samples, the model can learn the complete logical chain of deriving answers from document content during the training phase. Subsequently, the training samples are decomposed into multiple word units, and weights are configured differently according to the type of each word unit to distinguish the importance of different word units in loss calculation, allowing the model to focus on learning key content. A difficulty modulation factor is introduced during training, which quantifies the prediction difficulty of each word unit, characterizing the model's generation difficulty for different word units. Combined with the weights, the training loss of each sample is calculated collaboratively, dynamically balancing the model's learning intensity for different word units. Then, backpropagation is performed based on the loss value to continuously optimize the model parameters until the overall training loss meets the preset convergence condition, at which point the final model is determined. When applied, this model can not only output the corresponding question and answer answers, but also simultaneously generate the thought process of how the answers were generated and the sources cited in the answers. It clearly shows the entire derivation logic of the answers based on the document content and the basis of the answers, making it easy for staff to intuitively verify the source and basis of the answers. It solves the technical problems of the inability to trace the source and insufficient traceability of question and answer results in the existing technology. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0073] Figure 1 A schematic diagram illustrating an application scenario of the training method for the industrial large model that references the evidence of the thought chain provided in this application embodiment;

[0074] Figure 2 A flowchart illustrating the training method for the industrial large model of thought chain evidence reference provided in the embodiments of this application;

[0075] Figure 3 A schematic diagram of the large language model processing flow based on document question answering provided in this application embodiment;

[0076] Figure 4 A schematic diagram of the structure of the training device for the industrial large model of thought chain evidence reference provided in the embodiments of this application;

[0077] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0078] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0080] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply difference. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0081] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the training method, apparatus, and device for a large-scale industrial model of thought chain evidence citation provided in the embodiments of this application are merely examples; a training method, apparatus, and device for a large-scale industrial model of thought chain evidence citation may also include more or less content.

[0082] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0083] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0084] To clearly understand the technical solution of this application, we will first provide a detailed introduction to existing solutions. In scenarios such as industrial quality inspection, equipment maintenance, and safety procedure Q&A, the volume of industrial documents is enormous, and the industry's demand for intelligent Q&A continues to increase. Industrial scenarios have stringent requirements for the authenticity of output content, requiring answers to have clear decision-making basis to support engineering verification and quality traceability. At the same time, industrial document data is sensitive and annotation costs are high, making it difficult for traditional Q&A models to adapt to the needs of these industrial scenarios. Therefore, a highly reliable Q&A solution for industrial documents is urgently needed.

[0085] In existing technologies, intelligent question answering typically employs retrieval-enhanced generation techniques. This involves matching and retrieving text fragments relevant to the question from industrial documents, inputting the retrieval results and the user's question into a generation model, which then generates the corresponding answer to complete the question-answering task. However, in existing technologies, operators can only see the final answer text and cannot know which specific fragments or logical steps from the document the answer is based on. Even if operators see the model's output answer, they cannot determine its source's reliability; that is, they cannot verify whether each conclusion in the answer is truly supported by the cited evidence or is the result of model illusion or erroneous associations. Therefore, existing technologies suffer from poor traceability.

[0086] Therefore, addressing the issue of poor traceability in existing technologies, this research found that, to solve this problem: Optionally, the training question, the answer to the question, and the thought chain can be integrated to form training samples. The thought chain describes the intermediate steps of locating evidence from a document and deriving an answer. During training, the model learns to output both the thought chain and the answer simultaneously. Optionally, the training samples can also be split into tokens, with different weights pre-set for each token type, giving higher weights to important tokens in the loss calculation and highlighting the learning signals of key information. Optionally, during training, the model can obtain the prediction probability of each token's current position, calculate a dynamic modulation factor based on this probability, and combine it with static weights to calculate the loss for that token, allowing the model to automatically focus on key tokens that are currently difficult to predict.

[0087] This application provides a training method, apparatus, and device for a large-scale industrial model of thought chain evidence citation. By integrating training questions, corresponding answers, and thought chains used to reconstruct the logic of the answers into training samples, the model can learn the complete logical chain of deriving answers from document content during the training phase. Subsequently, the training samples are decomposed into multiple word units, and weights are configured differently according to the type of each word unit to distinguish the importance of different word units in loss calculation, allowing the model to focus on learning key content. A difficulty modulation factor is introduced during training, which quantifies the prediction difficulty of each word unit, characterizing the model's generation difficulty for different word units. Combined with the weights, the training loss of each sample is calculated collaboratively, dynamically balancing the model's learning intensity for different word units. Then, backpropagation is performed based on the loss value to continuously optimize the model parameters until the overall training loss meets the preset convergence condition, at which point the final model is determined. When applied, this model can not only output the corresponding question and answer answers, but also simultaneously generate the thought process of how the answers were generated and the sources cited in the answers. It clearly shows the entire derivation logic of the answers based on the document content and the basis of the answers, making it easy for staff to intuitively verify the source and basis of the answers. It solves the technical problems of the inability to trace the source and insufficient traceability of question and answer results in the existing technology.

[0088] Based on the above-mentioned inventive discovery, the technical solution of this application is proposed.

[0089] The following describes the application scenarios of the training method for the industrial large model of thought chain evidence citation provided in the embodiments of the present invention. Figure 1 This is a schematic diagram illustrating an application scenario of the training method for the industrial large-scale model that references the evidence of the thought chain provided in this application embodiment. For example... Figure 1 As shown, the application scenario includes a mobile terminal 101 and a server 102. The mobile terminal 101 collects multiple training samples and sends them to the server 102. The server 102 obtains the multiple training samples, splits any training sample into multiple word units, and sets weights for any word unit according to its word unit type. The server 102 obtains the difficulty modulation factor of any word unit and calculates the training loss of any training sample based on the weights and the difficulty modulation factor. Based on the training loss of any training sample, the server 102 determines the training loss of multiple training samples. Based on the training loss of multiple training samples, the server 102 backpropagates the parameters of the preset model to be trained. The preset model to be trained when the average value of the training loss of multiple training samples is less than a preset threshold is determined as the large-scale industrial model for evidence citation of the thought chain.

[0090] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0091] Figure 2 This is a flowchart illustrating the training method for the industrial large-scale model of thought chain evidence cited in the embodiments of this application. Figure 2As shown, in this embodiment, the execution entity of this invention is a server. The training method for the industrial large-scale model of thought chain evidence citation provided in this embodiment includes the following steps:

[0092] S201. Obtain multiple training samples; each training sample includes a training question, the answer to the training question, and a thought chain. The thought chain refers to the process of deriving the answer to the training question from the content of a preset document based on the training question.

[0093] Specifically, this involves collecting documents related to industrial quality inspection, equipment operation and maintenance, and safety procedures. Multiple sets of training questions and corresponding answers are then extracted from these documents. Each training question is broken down into independent semantic units, and the start and end positions of each semantic unit within the document are marked. Based on the positional information, corresponding text fragments are extracted to form an evidence set. Combining semantic units and text fragments, training questions with content anomalies are eliminated to obtain effective training questions. The effective training questions, corresponding answers, evidence sets, and original documents are then input into the teacher model to generate a corresponding content framework. Integrating the effective training questions, answers, evidence sets, and content framework forms a single target training sample. After batch processing, all training samples are obtained. This step is used to gather sufficient sample data that closely matches industrial business, providing basic materials with original textual evidence and a complete content framework for model training.

[0094] Optionally, the training question can be in the form of natural language text, for example, the training question is "Does this part have scratches on its surface?", the answer is in text form, such as "Yes, it has scratches", and the thought chain is in text form, such as "According to the defect judgment standard on page 3 of the quality inspection document, 'a scratch is defined as a continuous linear mark with a length greater than 1mm'. The measurement results in the figure show that there is a linear mark with a length of 2mm on the surface of the part, so it is judged to have scratches."

[0095] Training questions can also be audio or video. When the training question is audio or video, the audio needs to be converted into text first through a speech recognition module, or the text content needs to be extracted from the video through video frame extraction and optical character recognition modules. The converted or extracted text is then used as the training question for subsequent model training. The model outputs the answer and thought process in text form. For example, after the audio question is transcribed as "How to deal with overheating of the motor", the model outputs the answer "Reduce the load and check the cooling fan", and outputs the thought process "According to the record in Chapter 2, Section 1 of the maintenance manual, 'Motor overheating handling process', when the surface temperature of the motor exceeds 85°C, the load should be reduced first. If the temperature continues to rise, check the cooling fan".

[0096] S202. Split any training sample into multiple tokens, and assign weights to any token according to its token type; where the weights are used to represent the importance of any token in the calculation of training loss.

[0097] Specifically, a custom tokenizer for the industrial field may be adopted to split the complete training sample composed of valid training questions, corresponding answers and evidence sets into independent tokens as a whole. Four pre-configured regular expression patterns adapted to industrial document scenarios are retrieved, which are respectively an evidence anchor pattern, a citation structure pattern, a statement structure pattern, and a natural language pattern. The text content of each token is matched with the four regular expression patterns one by one, and the tokens are correspondingly classified into an evidence anchor class, a citation structure class, a statement structure class, and a natural language class according to the matching results. Then, according to a preset hierarchical weight rule, a first weight is configured for the evidence anchor class tokens, a second weight for the citation structure class tokens, a third weight for the statement structure class tokens, and a fourth weight for the natural language class tokens, with the weight values decreasing sequentially, so as to complete the splitting and weight assignment of all tokens. This step is used to accurately distinguish the importance levels of tokens of different structures in the training loss calculation, strengthen the learning proportion of core tokens related to industrial evidence, and reduce the interference of ordinary semantic tokens.

[0098] The token is the minimum calculation unit when a large language model processes text, which is obtained by splitting the text according to fixed rules by the tokenizer. It can be a complete word, a root, or a punctuation mark, and the input, output and training process of the model are all calculated with tokens as basic units. For example, the text "the equipment operates stably" will be split into three tokens, namely "equipment", "operate" and "stable", after being processed by the tokenizer.

[0099] S203: Obtain the difficulty modulation factor of any token, calculate the training loss of any training sample according to the weight and the difficulty modulation factor, and determine the training loss of a plurality of training samples according to the training loss of any training sample. The difficulty modulation factor is used to represent the prediction difficulty of any token by the preset model to be trained.

[0100] Specifically, an industrial question-and-answer sequence generation model may be pre-selected as the preset model to be trained and initial parameters are configured. The complete token sequence obtained by splitting a single training sample is used as the input of the model. The model extracts token features relying on the built-in network structure, and outputs the prediction probability of the target token corresponding to each position bit by bit. The prediction probability corresponding to each token output by the preset model to be trained is read, and the difficulty modulation factor corresponding to each token is calculated in combination with the prediction probability value. The loss value of the token is calculated by multiplying the weight of a single token by the corresponding difficulty modulation factor in combination with a basic loss formula, the loss values of all tokens in the same sample are accumulated to obtain the training loss of a single training sample, and then the loss values of all training samples are aggregated to obtain the training loss of the entire sample set. This step is used to complete the loss calculation by combining the token importance degree and the generation difficulty degree, so as to provide an accurate calculation basis for model parameter updating.

[0101] S204. Backpropagation is performed on the parameters of the preset training model based on the training loss of multiple training samples. The preset training model whose average training loss of multiple training samples is less than a preset threshold is determined as the Mind Chain Evidence Reference Industrial Model. The Mind Chain Evidence Reference Industrial Model is used to generate the answer to the question to be processed, the source of the answer, and the process of deriving the answer to the question to be processed from the content of the preset document.

[0102] Specifically, the gradient descent algorithm can be used to propagate gradient values ​​backward layer by layer based on the overall loss of all training samples. The parameters of each layer inside the model to be trained are adjusted according to the gradient values. The process of sample input, loss calculation and parameter adjustment is repeated continuously. After each iteration, the average value of the training loss of all training samples is calculated, and the relationship between the average value and the set threshold is continuously compared. When the average value meets the condition of being less than the threshold, the parameter update is stopped and the current model is set as the industrial large model for evidence citation of the thought chain. This step is used to complete the iterative optimization of model parameters and obtain a model that is adapted to the industrial document question answering scenario and has the ability to output evidence.

[0103] When using the chain of reasoning evidence to reference the industrial model, the text content of the industrial document and the user's question can be input into the model together. The model generates an output sequence containing both the final answer to the question and a set of derivation steps extracted from the document content and arranged chronologically. Each step explicitly points to a specific location in the document. After receiving the output, the operator can check the original document step by step along the derivation steps to confirm the complete correspondence between the answer and the document's basis. For example, in an industrial quality inspection scenario, if a batch of product inspection data and the question "Is this batch qualified?" are input into the model, the model can output "unqualified" as the answer, along with the following derivation steps: "According to row 3 of the inspection data table, the size of defect A is 0.5mm; according to section 2.1 of the inspection standard, defect A exceeding 0.3mm is considered unqualified; therefore, this batch is unqualified." The operator can then verify this by checking row 3 of the data table and section 2.1 of the standard.

[0104] The training method of the industrial large model based on evidence citation of the thinking chain can be applied to various technical scenarios that rely on professional documents to carry out intelligent question and answer, such as industrial quality inspection question and answer, equipment operation and maintenance consulting, safety procedure interpretation, and industrial standard document retrieval. In these scenarios, there are massive amounts of rigorous and data-sensitive industrial documents stored. Traditional question and answer output content lacks complete source labeling, making it difficult to trace the source of the content. At the same time, different words and professional terms have uneven learning emphasis and different generation difficulties in the content generation process. The training method of the Mind Chain Evidence Citation Industrial Model constructs training samples containing questions, answers, and complete content contexts. It combines word classification weight settings, difficulty modulation factors, refined loss calculations, and parameter iterative optimization to complete model training. When responding to various industrial question-and-answer needs, the trained model can simultaneously output corresponding content and complete content evidence based on original industrial documents, making up for the shortcomings of traditional solutions in content traceability. At the same time, it differentiates the learning focus for different types of words, strengthens the learning of core content such as industrial professional terms and document citation marks, and adapts to the hard requirements of the industrial field for the authenticity of question-and-answer content and the verifiability of content sources, meeting the practical needs of engineering content verification and quality content traceability.

[0105] This embodiment provides a training method for an industrial large-scale model of thought chain evidence citation. By integrating training questions, corresponding answers, and the thought chain used to reconstruct the logic of the answers into training samples, the model can learn the complete logical chain of deriving answers from document content during the training phase. Subsequently, the training samples are decomposed into multiple word units, and weights are configured differently based on the type of each word unit to distinguish the importance of different word units in the loss calculation, allowing the model to focus on learning key content. A difficulty modulation factor is introduced during training. This factor quantifies the prediction difficulty of each word unit, characterizing the model's generation difficulty for different word units. Combined with the weights, the training loss of each sample is calculated collaboratively, dynamically balancing the model's learning intensity for different word units. Then, backpropagation is performed based on the loss values ​​to continuously optimize the model parameters until the overall training loss meets the preset convergence condition, at which point the final model is determined. When applied, this model can not only output the corresponding question and answer answers, but also simultaneously generate the thought process of how the answers were generated and the sources cited in the answers. It clearly shows the entire derivation logic of the answers based on the document content and the basis of the answers, making it easy for staff to intuitively verify the source and basis of the answers. It solves the technical problems of the inability to trace the source and insufficient traceability of question and answer results in the existing technology.

[0106] In one possible design, S201, multiple training samples are acquired, including:

[0107] S2011. Obtain multiple training questions and answers to multiple training questions. Divide any training question into multiple semantic units and obtain the reference range of each semantic unit. The reference range is used to represent the start and end positions of the evidence fragment corresponding to each semantic unit in the preset document content. The evidence fragment is the text located in the preset document content according to the reference range.

[0108] Specifically, various business questions and corresponding standard answers can be extracted from paper and electronic documents related to industrial quality inspection, equipment operation and maintenance, and safety procedures. These can be collected to form batch training questions and corresponding answers. Then, each training question is divided into independent semantic units using industrial text segmentation. The start and end positions of each semantic unit in the original document are located one by one, and the position information is recorded to obtain the reference range of each semantic unit. This step is used to split the question content and lock the original text position corresponding to the semantic unit, providing a basis for subsequent extraction of document text fragments and construction of evidence sets.

[0109] For example, taking the power equipment operation and maintenance scenario as an example, firstly, one hundred daily business questions and corresponding standard answers are collected from electronic documents such as power equipment maintenance manuals and on-site operation specifications to complete the collection of training questions and answers. Then, using a word segmentation tool adapted to the power field, each training question, such as "What are the items included in the daily inspection of high-voltage switchgear?", is broken down into independent semantic units such as "high-voltage switchgear", "daily inspection", "includes", and "what items". The original specification electronic document is retrieved, and the entire text is traversed line by line with the first character of the document as the counting starting point. The text content matching each semantic unit is retrieved in turn. After locating the paragraph and character position of the target content, the sorting number of the first character of the text is accurately recorded as the starting position and the sorting number of the last character as the ending position. The two sets of numbers are bound to the corresponding semantic units for storage to form a complete reference range.

[0110] Semantic units are phrases or clauses into which the training problem is broken down according to semantic integrity. For example, "coolant flow rate is below the threshold and temperature exceeds 80 degrees" is broken down into two semantic units: "coolant flow rate is below the threshold" and "temperature exceeds 80 degrees". The difference between semantic units and word units is that word units are the smallest independent units obtained after word segmentation (such as a single character, sub-word or word), while semantic units are structural blocks composed of multiple word units that have complete expressive meaning. Their boundaries are usually determined by punctuation marks, conjunctions or syntactic components.

[0111] A reference range is a range of character offsets used to identify the specific location of evidence in the original document. It usually records the starting and ending character positions. For example, in the document string, the word "bearing wear" in "according to page 5 of the maintenance manual" starts at character 1045 and ends at character 1049, so the reference range is recorded as [1045, 1049].

[0112] Evidence fragments are continuous text content directly extracted from the original document based on the quoted range, that is, the specific original text that supports the source of the answer. For example, the evidence fragment corresponding to the quoted range [1045, 1049] is "bearing wear". The operator can compare the fragment with the original document word by word to verify the authenticity of the answer.

[0113] S2012. Generate an evidence set for any training question based on the reference interval; wherein, the evidence set is a set of evidence fragments corresponding to multiple semantic units.

[0114] Specifically, text fragments with corresponding start and end positions can be extracted from preset industrial documents according to the reference range corresponding to each semantic unit. The text fragments obtained by matching all semantic units under the same training question are summarized and integrated to form the evidence set corresponding to the training question. This step is used to collect the original text content fragments corresponding to the question, providing complete original text materials for subsequent selection of training questions and generation of content context.

[0115] For example, retrieve the electronic document of power equipment maintenance containing complete text, and according to the start and end numbers of each group of characters recorded in the previous step, extract the original text paragraphs corresponding to semantic units such as "high voltage cabinet", "daily inspection" and "what matters" in the document according to the number range to obtain multiple independent text fragments. Then, integrate all the text fragments related to the same training question into one place according to the original order in the document, classify and store them in a unified manner, and finally form the evidence set specific to the training question.

[0116] After breaking down the training question into semantic units, the corresponding evidence fragments are extracted and an evidence set is generated by locating the reference interval of each semantic unit in the document. For example, if the semantic unit is 'material strength requirement', and its reference interval is [P3.4.2], then the evidence fragment is the original text content of 'material strength must be ≥500MPa' in the document. This process establishes a direct association between the question and the document content through the matching relationship between semantic units and evidence fragments.

[0117] S2013. Based on multiple semantic units and evidence fragments, multiple training questions are screened to eliminate abnormal questions in multiple training questions and obtain multiple effective training questions.

[0118] Specifically, we can first compare the citation number of each semantic unit with the total number of fragments in the preset document, and eliminate training questions with citation numbers exceeding a reasonable range to obtain the first question. Then, we can check the character length of each evidence fragment in the first question and eliminate the first question with blank content fragments to obtain the second question. Finally, we can calculate the semantic similarity between the semantic units in the second question and the corresponding evidence fragments, and eliminate content with similarity that does not meet the standard to obtain the effective training questions. This step is used to filter out unqualified questions with content misalignment, content gaps, and insufficient relevance, and retain high-quality training questions with matching degree.

[0119] For example, the power equipment maintenance documents used in this study were divided into 260 standard text segments. First, the reference number corresponding to each semantic unit in each training question was checked one by one. Training questions with numbers greater than 260 or less than 1 were directly eliminated, resulting in 210 first-category questions to be processed. Then, the characters of the evidence segments associated with these 210 questions were checked one by one, and samples with zero characters or no valid text were eliminated, resulting in 198 second-category questions to be processed. Subsequently, semantic matching calculations were performed on the semantic units in the remaining questions and the corresponding evidence segments. A matching threshold of 0.7 was set, and samples with similarity lower than this value were eliminated, ultimately leaving 185 valid training questions with accurate content matching and complete information.

[0120] S2014. Input any valid training question, the answer to any valid training question, the evidence set, and the content of the preset document into the preset teacher model to obtain the thought chain of any valid training question.

[0121] Specifically, a single valid training question, its corresponding answer, all evidence fragments, and the complete content of a pre-set document can be combined into a unified input text, which is then fed into a pre-set teacher model trained on industrial corpus. The model then combines the input content to output a complete derivation process text as the corresponding thought chain. This step is used to produce a standard content derivation process, supplementing the components of the training samples.

[0122] For example, select a previously selected effective training question, "What are the items included in the daily inspection of high-voltage switchgear?", the corresponding standard answer, a set of evidence consisting of multiple original document texts, and a complete version of the power equipment maintenance document. Combine these four types of content into a complete input text according to a fixed layout order. Then, feed the integrated text into a teacher model that has been trained in advance based on power operation and maintenance and electrical maintenance corpora. The model combines the related content of the question, answer, evidence fragments and original documents, and outputs the complete text content from matching document content to deriving the corresponding knowledge points. This serves as the thought chain corresponding to the question.

[0123] S2015. Integrate the thought chain with any effective training problem to obtain the target training sample corresponding to any effective training problem, and determine multiple training samples based on the target training sample.

[0124] Specifically, a single effective training question and the generated thought chain can be spliced ​​together in a fixed layout order to form a single complete target training sample. This operation is repeated to process all effective training questions, and all generated target training samples are summarized to obtain a complete set of training samples. This step is used to assemble sample data with complete structure and elements, providing standardized input content for subsequent model training.

[0125] For example, continuing with the power equipment operation and maintenance scenario, taking the effective training question "What are the items included in the daily inspection of high-voltage switchgear?" as an example, the question itself and the thought chain generated by the teacher model are filled in according to the preset arrangement order, and combined and packaged into an independent target training sample. Following the same splicing rules, the content of the 185 previously selected effective training questions is combined one by one, and all the completed target training samples are collected and stored in a unified manner, finally forming a complete sample dataset that can be directly used for model training.

[0126] The technical effect of this solution in this embodiment is as follows: by splitting semantic units and extracting reference intervals, the corresponding evidence fragments are accurately located to construct an evidence set. Then, the question is screened according to the matching rules between semantic units and evidence fragments, and abnormal samples are eliminated to ensure the quality of training data. With the help of the teacher model, a standard thinking chain is generated by combining effective questions, answers, evidence sets and document content. Finally, the sample integration is completed, thereby realizing the standardized construction of training samples, improving the effectiveness and regularity of sample data, and laying a data foundation for the subsequent efficient and accurate learning and derivation of logic by the model.

[0127] In one possible design, S2013, multiple training questions are filtered based on multiple semantic units and evidence fragments to eliminate outliers and obtain multiple effective training questions, including:

[0128] S20131. Based on the comparison between the citation number of each semantic unit and the total number of fragments in the preset document content, multiple training questions are screened to remove training questions of semantic units with citation numbers less than one or greater than the total number of fragments, resulting in multiple first questions; where the citation number refers to the sequence number of the evidence fragment corresponding to each semantic unit in the preset document content.

[0129] Specifically, the reference numbers corresponding to all semantic units in each training question can be extracted one by one. Each number is compared with the total number of segments in the preset document content. Training questions containing numbers less than 1 or exceeding the total number of segments are selected and removed. The remaining content is the first question. This step is used to eliminate questions where the document location information is out of a reasonable range, ensuring that semantic units and evidence segments can be matched and extracted normally.

[0130] S20132. Based on the content length of the evidence fragment corresponding to each semantic unit, multiple first questions are filtered out to eliminate first questions where the content length of the evidence fragment corresponding to a semantic unit is zero, thus obtaining multiple second questions.

[0131] Specifically, all evidence fragments associated with each first question can be retrieved one by one, the number of characters in each text segment can be checked, and the first question with evidence fragments containing zero characters can be removed. The remaining content is the second question. This step is used to remove questions with attached blank text fragments to ensure that each sample is accompanied by complete and readable original text content.

[0132] The first type of question is one that survives the initial screening. In this type of question, all semantic units have citation numbers within a reasonable range for document fragment numbers, with no out-of-bounds numbers. The second type of question is one that has undergone a second screening based on the first type of question. Not only are the citation numbers compliant, but all accompanying evidence fragments also contain valid textual content, with no blank fragments. The core difference between the two lies in their screening criteria. The first type of question only verifies the validity of citation numbers and does not check the content of evidence fragments, while the second type of question, while ensuring number compliance, further guarantees that all related evidence fragments are non-empty and valid text.

[0133] S20133. Based on the comparison results of semantic similarity and preset similarity threshold, multiple second questions are screened to remove second questions with semantic units whose semantic similarity is lower than the preset similarity threshold, thereby obtaining multiple effective training questions; where semantic similarity refers to the similarity between each semantic unit and the evidence fragment corresponding to each semantic unit.

[0134] Specifically, the semantic similarity between each semantic unit in each second question and the corresponding evidence fragment can be calculated sequentially. The similarity value is compared with a preset threshold, and the second questions containing semantic units with similarity below the threshold are eliminated. The remaining content is the effective training question. This step is used to retain samples with close semantic matching to ensure that the question content and document fragment have a reliable correspondence.

[0135] The technical effect of this solution in this embodiment is as follows: By sequentially performing multi-level filtering of training questions from three different dimensions—validity of citation numbers, length of evidence fragment content, and semantic matching degree—questions are first eliminated if the citation numbers exceed the reasonable range of the document fragments. Then, invalid questions with empty evidence fragment content are eliminated. Finally, questions with substandard semantic relevance are filtered out, systematically removing various abnormal data. Through multi-dimensional and refined filtering, various defects and errors in the data can be effectively filtered out, purifying the overall training dataset. This ensures that the final retained valid training questions and accompanying evidence fragments are formatted correctly, have complete content, and are semantically closely related. This controls the quality of training data from the source, providing reliable data support for the model's subsequent stable and efficient training and learning.

[0136] In one possible design, S202 assigns weights to any lexical based on its lexical type, including:

[0137] S2021. Obtain the text content of any word, and match the text content with multiple preset regular expression patterns to obtain the matching result; wherein, the multiple preset regular expression patterns include evidence anchor pattern, citation structure pattern, statement structure pattern and natural language pattern.

[0138] Specifically, word-by-word splitting can be performed on word units to extract the original text content corresponding to the word units. Pre-written and stored regular expression rules for four categories—evidence anchors, citation structures, statement structures, and natural language—are then retrieved. The extracted single word unit text is then compared against the four categories of regular expression rules one by one, and character features and sentence structure features are compared and verified one by one. The hit status of each rule is recorded, and all verification information is integrated to form a complete matching result. This step is used to complete word unit content extraction and rule matching determination, providing a basis for subsequent word unit classification and corresponding weight allocation.

[0139] Among them, the regular expression pattern of the evidence anchor pattern is specifically used to identify identifier text in the sample that points to the content of the original document. It mainly matches tokens with features such as document fragment number, position label, and original text reference. This type of text generally serves as a marker for content tracing and is a key identifier connecting the issue and the evidence fragment. For example, texts such as "document fragment 18" and "see line 36" can be accurately matched by this pattern.

[0140] The regular expression pattern for citation structure is used to identify tokens with citation sentence characteristics. Multiple matching is used for transitional statements, guiding expressions, and formatted citations that connect questions, answers, and evidence. This type of text serves to link content and has fixed sentence structures and grammatical features. For example, phrases like "Based on the above content, it can be known" and "Combined with the corresponding specifications, it can be obtained" are both within the matching scope of this pattern.

[0141] The regular expression pattern for the statement structure is used to identify formal text elements such as objective descriptions, rule explanations, and key point lists. It mainly matches statements in industrial specifications and work standards that are used to explain requirements, define content, and describe clauses. The text style is rigorous and well-organized, with objective information as the core. For example, descriptive text such as "the high-voltage cabinet casing is undamaged and has no obvious rust" or "inspections must be completed once a day" can be identified and matched by this pattern.

[0142] The regular expression pattern in natural language modeling is used to identify common colloquial, question-and-answer, and general semantic expressions, covering general text except for the three special structures mentioned above. It mainly matches everyday questions, popular explanations, and general semantic expressions, with flexible sentence structures and no fixed format constraints. For example, common question-and-answer texts such as "Which parts should be checked during the inspection?" and "What precautions should be taken during this operation?" are all matched by this pattern.

[0143] S2022. If the text content is determined to match the evidence anchor pattern successfully based on the matching results, the word type of any word is determined as the evidence anchor class.

[0144] Specifically, the text content of the word element can be compared with four preset regular expression patterns one by one, and the matching results can be checked. When the matching result clearly shows that the current text content meets the character format and content characteristics of the evidence anchor pattern, the match can be determined. Then, the classification identifier corresponding to the evidence anchor pattern is filled in the attribute field of the currently judged word element to clarify the word element class. This step is used to accurately identify words with evidence anchor attributes, and to do a good job of preprocessing for subsequent allocation of corresponding weights according to different categories.

[0145] S2023. If the text content and the reference structure pattern are successfully matched based on the matching results, the word type of any word is determined as the reference structure class.

[0146] Specifically, when the matching result clearly shows that the text content conforms to the sentence structure and writing characteristics of the citation structure pattern, the match can be determined to be successful. Then, the category identifier corresponding to the citation structure pattern is filled in the attribute field of this word to determine the word type. This step is used to accurately identify words with citation connection attributes, and to do a good job of preprocessing for subsequent allocation of corresponding weights according to different categories.

[0147] For example, when comparing the text content of a word with the regular expression patterns, if the comparison results of texts such as "combined with the above clauses" or "refer to the preceding content" show a match with the reference structure pattern, then fill in the reference structure class identifier in the attribute field of that word, and complete the type confirmation of all words of the same type according to this judgment logic.

[0148] S2024. If the text content is determined to match the statement structure pattern successfully based on the matching results, the word type of any word element is determined to be the statement structure class.

[0149] Specifically, when the matching result clearly shows that the current text content conforms to the sentence structure and expression characteristics of the statement structure pattern, the match can be determined. Then, the classification identifier corresponding to the statement structure pattern is filled in the attribute field of the currently judged word element to determine the word element type. This step is used to accurately identify word elements with objective statement attributes, and to do a good job of preprocessing for subsequent allocation of corresponding weights according to different categories.

[0150] For example, when comparing the text content of a word with the regular expression patterns, if the comparison results of texts such as "the equipment is operating normally" and "conduct an on-site inspection once a day" match the statement structure pattern, then fill in the statement structure class identifier in the attribute field of that word. Follow this judgment logic to complete the type confirmation work for all words of the same type.

[0151] S2025. If the text content is determined to match the natural language pattern successfully based on the matching results, the word type of any word is determined to be a natural language class.

[0152] Specifically, when the matching result clearly shows that the current text content matches the expression characteristics of the natural language pattern, and does not match the other three types of patterns, the match can be determined to be successful. Then, the classification identifier corresponding to the natural language pattern is filled in the attribute field of the currently judged word to determine the word type. This step is used to identify words of the regular semantic expression class, which is a preliminary processing for the subsequent allocation of corresponding weights according to different categories.

[0153] For example, when checking the comparison results of text content with various regular expression patterns, if the comparison results of texts such as "which parts need to be paid attention to during daily inspections" or "how to handle equipment abnormalities" match the natural language pattern, then fill in the natural language class identifier in the attribute field of that word. Following this judgment logic, the type confirmation work of all words of the same type is completed in sequence.

[0154] S2026. Set the weight of any word element to a preset weight value corresponding to the word element type.

[0155] Specifically, we can first read the lexical type that has been labeled for a single lexical, compare it with the pre-set weight reference standard for each type of lexical, retrieve the fixed weight value corresponding to that type, and bind the value to the weight field of the current lexical to complete the weight assignment operation for a single lexical. This step is used to match the corresponding weight for different types of lexical, distinguish the importance of each type of lexical, and ensure that the subsequent text processing is carried out in an orderly manner.

[0156] The technical effect of this solution in this embodiment is as follows: By pre-configuring multiple regular expression patterns adapted to different text features, feature matching is performed on the text content corresponding to each word. Based on the matching results, words are automatically classified into types such as evidence anchors, citation structures, statement structures, and natural language, and a matching preset weight is assigned to each word based on its category. Leveraging the text recognition capabilities of regular expressions, the text structure and content attributes of different words can be efficiently identified, achieving automated classification of word types and improving the efficiency and accuracy of type determination. Furthermore, by allocating weights differently according to word type, the importance of each type of word can be fully reflected in the training loss calculation, allowing the model to apply differentiated learning emphasis to different types of words during training, enhancing the targeting of the training process and improving the overall training quality and learning effect of the model.

[0157] In one possible design, S206, the weight of any lexical element is set to a preset weight value corresponding to the lexical element type, including:

[0158] S20261. If any word element is of the evidence anchor type, set the weight of any word element to the preset first weight.

[0159] Specifically, a weight configuration table can be pre-established. Each row in this table records a lexical type and its corresponding preset weight value. Optionally, the weight value for the evidence anchor class is 3.0, the weight value for the citation structure class is 2.0, the weight value for the statement structure class is 1.3, and the weight value for the natural language class is 1.0. After determining that the lexical type is the evidence anchor class, the weight configuration table is looked up based on this type, the corresponding value of 3.0 is retrieved, and assigned to the weight variable of that lexical. This step is used to associate the key lexical that points to a specific location in the document with the highest weight value, so that the model can exert a stronger correction force on the prediction bias of these lexicals in subsequent loss calculations.

[0160] S20262. If any word element is of the reference structure type, set the weight of any word element to the preset second weight; wherein the first weight is greater than the second weight.

[0161] Specifically, based on the current lexical element's type label field value being "reference structure class," the model can use this type as a key to look up a pre-established weight configuration table, retrieve the corresponding second weight value from the table, and assign it to the lexical element's weight variable. This step is used to set a moderately high weight for reference structure lexical elements that indicate the aforementioned content or connect context, so that the model applies a stronger gradient correction to the prediction error of this type of lexical element in the training loss calculation than to natural language lexical elements, but weaker than to evidence anchor lexical elements. For example, after the lexical element "as described above" is determined to be a reference structure lexical element, the second weight 2.0 is retrieved from the weight configuration table and written into its weight field.

[0162] S20263. If any word element is of the statement structure type, set the weight of any word element to the preset third weight; wherein, the second weight is greater than the third weight.

[0163] Specifically, based on the current lexical term's type label field value being "statement structure class," the model can use this type as a key to look up a pre-established weight configuration table, retrieve the corresponding third weight value from the table, and assign it to the lexical term's weight variable. This step is used to set moderately low weights for moderately important lexical terms that express logical conclusions or causal relationships, so that the model applies a correction strength to the prediction error of this type of lexical term in the training loss calculation that is less than that for the citation structure class but greater than that for the natural language class. For example, after the lexical term "therefore" is determined to be a statement structure class, the third weight 1.3 is retrieved from the weight configuration table and written into its weight field.

[0164] S20264. If any word element is of the natural language type, set the weight of any word element to the preset fourth weight; wherein the third weight is greater than the fourth weight.

[0165] Specifically, based on the current word's type label field value being "Natural Language," the model can use this type as a key to look up a pre-established weight configuration table, retrieve the corresponding fourth weight value from the table, and assign it to the word's weight variable. This step is used to set the lowest weight for ordinary text words that do not carry special references or logical identifiers, so that the model applies the weakest correction to the prediction bias of this type of word in the training loss calculation. For example, after the word "device" is determined to be in the Natural Language class, the fourth weight 1.0 is retrieved from the weight configuration table and written into its weight field.

[0166] The technical effect of this scheme in this embodiment is as follows: by configuring gradient-based preset weights for four different lexical types sequentially, the weight values ​​are progressively reduced in the order of evidence anchor, citation structure, statement structure, and natural language. This differentiated weight allocation method can accurately distinguish the contribution ratio of each lexical type in loss calculation, taking into account their different roles in text structure and semantic expression. This allows the model to prioritize learning core lexical types with higher weights during training, rationally allocate training resources, strengthen the learning of key text content, optimize the overall training orientation, and improve the accuracy and efficiency of model learning.

[0167] In one possible design, S203 obtains the difficulty modulation factor of any word, and calculates the training loss of any training sample based on the weights and the difficulty modulation factor, including:

[0168] S2031. Obtain the prediction probability of any word, and calculate the difficulty modulation factor of any word based on the prediction probability; wherein, the prediction probability is used to represent the prediction accuracy of the preset training model for any word.

[0169] Specifically, for each word, the probability value at the corresponding true label position can be extracted from the probability distribution output by the model to be trained as the predicted probability. This predicted probability is then substituted into a preset formula to calculate the difficulty modulation factor. For example, the difficulty modulation factor can be set to one minus the predicted probability, or it can be set to a negative logarithmic probability value. When the predicted probability is high, the difficulty modulation factor approaches zero; when the predicted probability is low, the difficulty modulation factor approaches one or greater. This step quantifies the difficulty level of the model's current prediction for each word, allowing subsequent training loss calculations to dynamically adjust the contribution ratio of each word to the overall loss based on its difficulty level.

[0170] Optionally, for the word "according to Chapter 3", the correct word prediction probability output by the model to be trained is 0.95, then the difficulty modulation factor is calculated as 1 minus 0.95 equals 0.05. For the word "bearing", the model prediction probability is 0.3, then the difficulty modulation factor is 0.7, indicating that this word is currently more difficult to be correctly generated by the model.

[0171] The probability distribution output by the model under training refers to the probability vector formed when the model, after processing the input sequence and preparing to predict the next word, calculates a probability value between 0 and 1 for each candidate word in the vocabulary using the Softmax layer. The sum of the probability values ​​of all candidate words is 1. This probability distribution is obtained by: processing the sequence input into the model from the training samples through the model's multi-layer neural network to obtain the hidden state vector at the last position; multiplying this vector by an output weight matrix to obtain a logical value vector of the same size as the vocabulary; and finally converting the logical values ​​into probability values ​​using the Softmax function, thus forming a probability distribution containing the prediction probability of each word.

[0172] S2032. Calculate the training loss of any word based on the weight and difficulty modulation factor, and determine the training loss of any training sample based on the training loss of any word.

[0173] Specifically, the training loss for any word can be calculated based on its weight and difficulty modulation factor: First, the basic loss value for the word is calculated using the cross-entropy loss function. Then, this basic loss value is multiplied by the word's weight, followed by the difficulty modulation factor, to obtain the weighted loss value for the word. For all words in a training sample, the weighted loss values ​​of each word are summed, and the sum is divided by the total number of words in the training sample to obtain the average training loss for that training sample. This step incorporates both the importance weight and prediction difficulty of the word into the loss calculation, enabling the model to apply stronger parameter update force to high-weight and difficult-to-predict words during backpropagation.

[0174] For example, if the basic cross-entropy loss of a certain word is 0.5, its weight is 3.0, and its difficulty modulation factor is 0.8, then the weighted loss of that word is 0.5 multiplied by 3.0 multiplied by 0.8, which equals 1.2; if the basic loss of another word in the same training sample is 0.2, its weight is 1.0, and its difficulty modulation factor is 0.1, then its weighted loss is 0.02; the training loss of that sample is obtained by summing the weighted losses of all words and dividing by the total number of words.

[0175] The technical effect of this scheme in this embodiment is as follows: By obtaining the model's prediction probability for each word and calculating the difficulty modulation factor accordingly, this factor is then combined with the word's weight to participate in the calculation of the training loss. This allows the loss function to automatically adjust the contribution ratio of that word to the overall loss based on the model's confidence in predicting the current word. When the model's prediction probability for a certain word is low, it means that the word has not been fully mastered in the current training stage, and the difficulty modulation factor relatively enhances the loss contribution of that word. Conversely, when the model's prediction probability for a certain word is high, it indicates that the word has been well-fitted, and its loss contribution is relatively weakened. This approach can adaptively focus on key positions where the model's current learning performance is poor, reducing excessive interference from well-mastered common words in the gradient direction, optimizing the efficiency and direction of model parameter updates, and improving the rationality of the training process.

[0176] Applying large language models to high-reliability fields like industrial quality inspection requires content traceability, verification, and auditability. Existing technologies mainly fall into three categories: retrieval-enhanced question answering, long-context model-based question answering with citations, and derivation enhancement and targeted loss function optimization methods. These technologies lay the foundation for trustworthy question answering, but industrial scenarios present challenges such as scattered evidence in long documents, low fault tolerance, high expert annotation costs, data privacy restrictions, and difficulties in large-scale model training. This solution, based on the aforementioned existing technologies, aims to explore industrial large-scale model training and derivation methods that balance citation accuracy, answer quality, and derivation auditability in scenarios with few samples.

[0177] Existing methods for generating long contextual references suffer from a "derivation black box" problem: traditional supervised fine-tuning typically maps long documents and questions directly to cited answers, without explicitly demonstrating the evidence extraction, comparison, and conclusion derivation steps. In high-credibility scenarios such as industrial quality control and equipment maintenance, this approach, which only provides the final answer without intermediate steps, prevents auditors from verifying the model's decision-making process, thus reducing its credibility.

[0178] The existing training data suffers from gradient dilution, which makes it difficult for the model to transfer to other domains: in long-context question answering, there are far more ordinary semantic words than citation-type structured label words. The standard cross-entropy loss makes the gradient dominated by a large number of semantic words, making it difficult for the model to effectively learn citation-related key signals. This leads to the current methods being highly dependent on large-scale labeled data. However, the cost of labeling samples in industrial vertical domains is high, the cycle is long, and the data is subject to confidentiality restrictions, which makes it difficult to transfer to other domains.

[0179] Existing methods struggle to balance citation accuracy and derivation interpretability: introducing long derivation processes can improve answer interpretability, but it further lengthens the training sequence, leading to sparser citation structure terms, which may reduce citation accuracy; if only citation format is pursued, the factual correctness and logical integrity in complex derivation tasks may be sacrificed.

[0180] Existing Retrieval-Augmented Generation (RAG) or block-level citation methods suffer from coarse evidence granularity and information fragmentation: In unified long documents or multi-source industrial documents, coarse-grained block segmentation may cause semantic fragmentation, and the fragments referenced by the model may not accurately correspond to the specific factual statements in the answer, affecting subsequent compliance review and engineering verification.

[0181] The training method of the Inductive Citing Large Language Model (IndCiteLLM) of this invention yields a credible evidence citation industrial large-scale model (IndCiteLLM) for high-reliability industrial scenarios. It can be applied to low-fault-tolerance scenarios such as industrial quality inspection, equipment maintenance, safety procedure question answering, and test report analysis. It addresses the problems of existing long-context large-scale models in citation generation, including the black box nature of the derivation process, unstable evidence location, sparse structured citation markers, and difficulties in vertical transfer with few samples. The Inductive Citing Large Language Model no longer adopts a single path of "directly mapping long documents and questions to cited answers." Instead, it explicitly establishes a process chain of "question—evidence—derivation—answer—citation" at the data construction level. At the training method level, it strengthens the learning signal of sparse citation markers through a dual compensation objective of evidence perception, ensuring that each factual conclusion generated by the model corresponds to traceable and auditable original text evidence.

[0182] The Industrial Big Model is a deep learning model system with large-scale parameters designed for applications throughout the entire lifecycle of industrial products. It includes model systems of different levels and categories, such as the Industrial Foundation Big Model, the Industrial Task-Oriented Big Model, and the Industrial Sector Big Model. It features industrial data and mechanism knowledge fusion-driven, industrial specialized content generation, highly reliable output, cross-domain task learning and adaptation in multiple industrial scenarios, multi-modal fusion interaction in industry, human-intelligent agent-industrial system collaboration, and flexible adaptation of computing power and efficiency. It has core task capabilities such as intelligent question answering, scenario cognition, process decision-making, terminal control, and content generation. It can adapt to different industry sectors and industrial tasks in discrete and process industries, providing new application paradigms and methods based on big models for the intelligent upgrading of all business domains such as production, manufacturing, and testing.

[0183] The "large" in industrial large model refers to the total amount of industrial data used for model training exceeding a preset data volume threshold, the number of model base parameters exceeding a preset parameter threshold, and the computing power resources required for training and deployment exceeding a preset computing power threshold. These three thresholds can be determined by comprehensively calculating the complexity of the target industrial scenario task, the benchmark indicators of industry-standard industrial large models, the upper limit of local computing power hardware configuration, and real-time requirements. Alternatively, they can be preset with reference to the standards of publicly available industrial large model literature in the same field and the training experience of similar models in the past of enterprises. Furthermore, they can be dynamically and iteratively adjusted according to the accuracy requirements of downstream evidence citation and thought chain generation tasks.

[0184] For example, in a question-and-answer scenario for industrial equipment fault evidence, the initial parameter threshold can be set to 10B, the industrial document data volume threshold to 1 million documents, and the single-node training computing power threshold to 80GB of video memory. Then, considering the complexity of the task in this scenario, which requires parsing equipment time-series sensor data and multimodal evidence from process manuals, the data volume threshold can be increased to 1.5 million documents. At the same time, considering the real-time constraints of local computing hardware being equipped with only 4 A100 graphics cards and the business requirement that the derivation latency be less than 200ms, the computing power threshold can be fixed at 80GB of video memory per card and a total cluster computing power of 200TFLOPS. Finally, based on the initial small sample test, when the accuracy of the thought chain evidence derivation is lower than 92%, iterative optimization can be performed, and the parameter threshold can be updated to 13B. Ultimately, the three sets of fixed thresholds corresponding to this invention are determined: a data volume threshold of 1.5 million industrial document samples, a parameter threshold of 13 billion base parameters, and a computing power threshold of 80GB of video memory per card and a cluster floating-point computing power of not less than 200TFLOPS.

[0185] The IndCiteLLM construction process consists of two parts: the first is the evidence-driven citation inference data construction process, and the second is the evidence-aware dual-compensation training method for fine-grained citation generation. It makes the implicit evidence matching relationships in traditional samples explicit, so that the training data not only contains the final answer and citation number, but also the intermediate inference process of how the model should locate evidence, understand evidence, and form conclusions. Simultaneously, it amplifies the gradient contribution of keyword units (lexical units) such as citation numbers, citation boundaries, and structural labels during long-context training, preventing the model from only learning natural language expressions and ignoring evidence anchoring capabilities. The preset document content D is represented as:

[0186]

[0187] in, This represents the Nth text unit obtained by segmenting according to paragraphs, sentences, or fixed-length windows. The training question is Q, and the answer to training question Q is A. The goal of IndCiteLLM is to learn a conditional generative model that, given Q and D, outputs a target sequence containing the derivation process and the answer A, thereby achieving interpretable and verifiable long-context industrial question answering.

[0188] Figure 3 This is a schematic diagram of the large language model processing flow based on document question answering provided in an embodiment of this application, such as... Figure 3 As shown, the construction process of the training samples of IndCiteLLM includes three steps: precise evidence source extraction, teacher-guided thought chain synthesis, and thought chain sequence integration. The precise evidence source extraction on the left is based on the document, extracting the core expression of the final answer and associating it with the original document ("Graduate Handbook of the Department of Ecology and Evolutionary Biology 2023 Edition"), key citation fragments supporting the answer [315-317], and supplementary citation information used for corroboration

[315] , forming a complete evidence chain. The teacher-guided deduction in the middle simulates the operating logic of the question-and-answer system. It receives the question (the deadline for the doctoral qualification examination) and the preliminary answer, and combines the citation fragments to guide the large language model to generate text containing a detailed deduction process, explaining how the answer is derived from the document. The thought chain sequence integration on the right integrates the output of the first two steps to form the final, standardized answer.

[0189] The construction of model training samples includes the following three steps: The first step is to extract accurate evidence sources. First, the standard answer A is parsed and broken down into several semantic units. and its corresponding reference range , For the j-th semantic unit The starting index of the corresponding reference range, For the j-th semantic unit The corresponding index of the end position of the reference interval, where j is the index of the semantic unit. For each semantic unit... The model directly backtracks and extracts corresponding evidence fragments from the original document D based on the citation number, instead of re-recalling through approximate retrieval, thus avoiding retrieval noise entering the training data. Through semantic unit decomposition and evidence set generation, it ensures that the answers in the training samples have a clear logical correspondence with the document evidence fragments, improving the model's traceability of the answer source. The evidence set is as follows:

[0190]

[0191] in, Let C be the set of evidence for the j-th semantic unit, and C be the text unit. In this step, the correspondence between evidence number, fragment position, fragment text, and answer statement is preserved simultaneously, forming a structured evidence sample:

[0192]

[0193] in, Given structured evidence samples, Q represents the training problem. As a semantic unit, Let J be the evidence set for the j-th semantic unit. If a sample has issues such as out-of-bounds citation numbers, empty citation fragments, or answer statements that do not correspond to the semantics of the evidence, it will be removed or subject to manual review to ensure that subsequent thought chains are strictly generated based on real evidence.

[0194] The second step is the synthesis of thought chains guided by the teacher model. The training question is Q, and the answer to training question Q is A and the evidence set. Input the teacher model. The teacher model needs to generate a natural language thought chain based on the input, explaining what information the question focuses on, which pieces of evidence are relevant to the question, which semantic components in the evidence support the answer conclusion, and why the final quoted range supports the corresponding statement. Let the teacher model be... The process of generating the thought chain can be represented as follows:

[0195]

[0196] Where T represents the thought chain, For the teacher model, Prompt is the constructor for the composite prompt, and Q is the training problem. Let D be the set of evidence and D be the pre-defined document content. To ensure that the derivation process does not introduce illusions, the prompt template restricts the teacher model from using unprovided evidence fragments, changing the semantics of the original answer, expanding conclusions without supporting evidence, and requiring it to explicitly explain the supporting relationship between the evidence fragments and the answer statement in the derivation. This step transforms the semantic matching relationship originally hidden between "answer-citation" into a readable thought chain, enabling the model to learn the evidence selection and logical support process during training, rather than just learning the citation format.

[0197] The third step is the integration of the thought chain sequence. The thought chain T generated by the teacher model is concatenated with the original cited answer A according to a unified template to obtain the training samples for IndCiteLLM:

[0198]

[0199] in, These are the training samples for IndCiteLLM. This serves as the starting marker for the thought chain. This marks the end of the thought chain. This is a sequence concatenation operator.

[0200] When the answer consists of multiple semantic units, the answer portion is further normalized as follows:

[0201]

[0202] Where A is the answer. This indicates that the sequence from the 1st to the mth semantic units is concatenated. As the starting marker for semantic units, This is the end marker for a semantic unit. For the j-th semantic unit, This serves as the starting marker for the referenced range. This is the end marker for the referenced range. For the j-th semantic unit The corresponding reference range, m, represents the total number of semantic units contained in the standard answer A.

[0203] Through the above process, IndCiteLLM explicitly preserves the generation sequence of "understanding the problem, locating evidence, forming conclusions, and finally outputting citations" in the training data. Compared to traditional methods that directly generate citation-based answers, this method gives the model's citation behavior a clear process basis, better meeting the requirements for traceability, verifiability, and accountability in industrial review scenarios.

[0204] After obtaining the above training samples, IndCiteLLM undergoes supervised fine-tuning. Let the target output sequence of IndCiteLLM be:

[0205]

[0206] Where Y is the target output sequence, Let W be the word at position W in the sequence, where W is the total number of words in the target sequence. Let the model parameters be... Then the predicted probability of the true word at position t is:

[0207]

[0208] in, For the real word at position t, For the model at position t, the true word The predicted probability, For parameters The model's conditional probability distribution, where D is the predefined document content and Q is the training problem. Standard supervised fine-tuning typically uses uniform cross-entropy loss.

[0209]

[0210] in, Let W be the uniform cross-entropy loss function, and W be the total number of words in the target sequence. For parameters The conditional probability distribution of the model, For the real word at position t, Here, D represents the model parameters, D represents the preset document content, and Q represents the training question.

[0211] Treating all lexical units equally is suitable for generating plain text, but not for long-context citation question answering. This is because the output sequence of IndCiteLLM contains both a large number of ordinary semantic lexical units and a very small number of structural lexical units that determine the correctness of citations. Let the vocabulary V be divided into a set of ordinary semantic lexical units. and structural lexical set , This includes the boundaries of thought chains, factual statements, citation boundaries, and specific citation numbers. The sparsity ratio of structural lexical units in the complete sequence is defined as:

[0212]

[0213] in, The sparsity ratio of structural lexical units. It is a set of structural lexical units. Let be the real word at position t. In long documents and long thought chains, this is usually... It will be much less than 1. At this point, the total gradient of the standard cross-entropy is mainly determined by ordinary text terms, while the gradient contributions of citation numbers and citation labels are diluted, which can be formally expressed as:

[0214]

[0215] in, Represents the set of positions of structural lexical units. Let W represent the set of positions of ordinary semantic lexical units, and W be the total number of lexical units in the target sequence. The loss term for the t-th word For model parameters gradient, For sets The gradient terms at all positions within the word are summed to represent the total gradient contribution of the structural lexical. For sets The gradient terms at all positions within the word are summed to represent the total gradient contribution of ordinary semantic lexical units.

[0216] This formula describes the gradient dilution phenomenon of structural lexical units in long-context scenarios using standard cross-entropy loss: as the sequence length W approaches infinity, the ratio of the gradient contribution of structural lexical units (such as citation labels and boundary markers) to the gradient contribution of ordinary semantic lexical units approaches 0. This means that the model's parameter updates are almost entirely dominated by the majority of ordinary text lexical units, while the key structural lexical units that determine the correctness of citations are almost not effectively trained, ultimately leading to problems such as missing citations and incorrect numbering. This phenomenon can cause the model to generate fluent answers, but it is prone to problems such as missing citations, citation range deviations, incorrect citation numbering, or incomplete structural label closure.

[0217] To this end, this application proposes an evidence-aware dual-compensation training objective for IndCiteLLM. For the t-th lexical unit, the loss is defined as:

[0218]

[0219] in, As weights, they are used to compensate for the scarcity of structural lexical units by considering their functional roles. This is a dynamic difficulty modulation factor used to adaptively adjust the optimization focus based on the predicted difficulty of the current training phase. For the real word at position t, For the model at position t, the true word The predicted probability, These are the model parameters.

[0220] The lexical units are divided into four categories: the first category is evidence anchor lexical units, including specific citation numbers and citation ranges, which are the core to determine whether the evidence positioning is correct; the second category is citation structure lexical units, including citation start and end labels, used to limit the boundaries of evidence numbers; the third category is statement structure lexical units, including factual statements and thought chain boundary labels, used to maintain the output structure; and the fourth category is ordinary natural language lexical units. The weight function is as follows:

[0221]

[0222] in, As weight, For the real word at position t, The weights assigned to anchor words, The weights assigned to the citation boundary terms, The weights assigned to declarative structural lexical units, For anchor word set, For the set of reference boundary terms, For the set of declarative lexical units, It is a set of ordinary natural semantic lexical units. This design assigns the highest weight to the most critical and sparsest evidence numbers, secondary weight to citation boundaries and statement boundaries, and maintains the basic weight for ordinary natural language, thereby enhancing the fine-grained evidence anchoring capability at the loss level. The difficulty modulation factor is defined as:

[0223]

[0224] in, This is the difficulty modulation factor. The modulation coefficient, Greater than or equal to 0, For the model at position t, the true word The predicted probability. When the model's predicted probability for a certain word is low, When the value is close to 1, the training signal for that word is preserved; when the model can stably predict ordinary semantic words, Approaching 1, Rapid decay reduces the interference of simple lexical units on the overall gradient direction. This mechanism encourages the model to continuously focus on hard-to-learn citation numbers, structural closures, and evidence boundaries, rather than repeatedly optimizing already mastered general language patterns.

[0225] To ensure the stability of the loss scale across different samples and batches, IndCiteLLM uses a normalized overall training objective:

[0226]

[0227]

[0228] in, Let IndCiteLLM be the overall training loss function. Let be the overall weight of the t-th word element. As weight, This is the difficulty modulation factor. For the model at position t, the true word The predicted probability, For model parameters, Let D be the modulation coefficient, D be the preset document content, and Q be the training question. To prevent extremely small constants with a denominator of zero, the overall training loss function retains the basic form of autoregressive language modeling. Simultaneously, through two compensation mechanisms—weight and difficulty modulation factors—the model can still learn stable fine-grained referencing capabilities under conditions of long context, few samples, and extremely sparse structural labels.

[0229] This technical solution enables large industrial models to not only provide natural language answers when answering complex questions, but also output evidence derivation processes and precise reference locations. Therefore, it is suitable for application scenarios that require high credibility and verifiability, such as industrial manual Q&A, maintenance record verification, quality document review, and safety procedure traceability.

[0230] This application employs precise evidence source extraction, teacher model thought chain synthesis, and thought chain sequence integration to enable IndCiteLLM to learn the evidence screening and logical deduction process before generating the final answer, avoiding the black-box problem caused by traditional models relying solely on implicit mapping to generate citations. During the training phase, this application sets higher optimization weights for citation numbers, citation labels, and answer structure markers, and introduces a dynamic difficulty modulation mechanism to enhance the learning signals of sparse evidence anchors and unpredictable citation terms, mitigating the gradient dilution problem of structured citation markers and improving the stability and accuracy of citation localization in long contexts and low-sample scenarios. IndCiteLLM output includes both the answer and the thought chain, facilitating manual review and evidence verification. This method is applicable to scenarios such as industrial procedure question answering, equipment maintenance record analysis, and safety standard verification, reducing manual annotation and verification costs and improving the accuracy, security, and credibility of large-scale industrial model applications.

[0231] Figure 4 This is a schematic diagram of the structure of the training device for the industrial large-scale model of thought chain evidence reference provided in the embodiments of this application. (See attached diagram.) Figure 4 As shown, the evidence for this chain of thought cites a training apparatus for a large industrial model, including:

[0232] The acquisition module 401 is used to acquire multiple training samples; each training sample includes a training question, the answer to the training question, and a thought chain, which refers to the process of deriving the answer to the training question from the content of a preset document based on the training question.

[0233] The splitting module 402 is used to split any training sample into multiple tokens and set weights for any token according to the token type; wherein, the weights are used to represent the importance of any token in the calculation of training loss.

[0234] The calculation module 403 is used to obtain the difficulty modulation factor of any word, calculate the training loss of any training sample based on the weight and the difficulty modulation factor, and determine the training loss of multiple training samples based on the training loss of any training sample; wherein, the difficulty modulation factor is used to represent the prediction difficulty of the preset model to be trained for any word.

[0235] The training module 404 is used to backpropagate the parameters of the preset training model based on the training loss of multiple training samples, and to determine the preset training model when the average value of the training loss of multiple training samples is less than a preset threshold as the thinking chain evidence citation industrial big model; wherein, the thinking chain evidence citation industrial big model is used to generate the answer to the question to be processed, the source of the answer, and the process of deriving the answer to the question to be processed from the content of the preset document.

[0236] In one possible implementation, the acquisition module 401 includes:

[0237] The first acquisition unit is used to acquire multiple training questions and answers to multiple training questions, split any training question into multiple semantic units, and acquire the reference range of each semantic unit; wherein, the reference range is used to represent the start and end positions of the evidence fragment corresponding to each semantic unit in the preset document content, and the evidence fragment is the text located in the preset document content according to the reference range.

[0238] The generation unit is used to generate an evidence set for any training question based on the reference interval; wherein, the evidence set is a set of evidence fragments corresponding to multiple semantic units.

[0239] The filtering unit is used to filter multiple training questions based on multiple semantic units and evidence fragments, so as to eliminate abnormal questions among the multiple training questions and obtain multiple effective training questions.

[0240] The input unit is used to input any valid training question, the answer to any valid training question, the evidence set, and the content of a preset document into the preset teacher model to obtain the thought chain of any valid training question.

[0241] The integration unit is used to integrate the thought chain and any effective training problem to obtain the target training sample corresponding to any effective training problem, and to determine multiple training samples based on the target training sample.

[0242] In one possible implementation, the screening unit includes:

[0243] The first filtering component is used to filter multiple training questions based on the comparison between the citation number of each semantic unit and the total number of fragments in the preset document content. This process removes training questions for semantic units with citation numbers less than one or greater than the total number of fragments, resulting in multiple first questions. Here, the citation number refers to the sequence number of the evidence fragment corresponding to each semantic unit in the preset document content.

[0244] The second filtering component is used to filter multiple first questions based on the content length of the evidence fragments corresponding to each semantic unit, so as to remove the first questions where the content length of the evidence fragments corresponding to the semantic units is zero, and obtain multiple second questions.

[0245] The third filtering component is used to filter multiple second questions based on the comparison results of semantic similarity and preset similarity threshold, so as to remove second questions with semantic units whose semantic similarity is lower than the preset similarity threshold, and obtain multiple effective training questions; wherein, semantic similarity refers to the similarity between each semantic unit and the evidence fragment corresponding to each semantic unit.

[0246] In one possible implementation, the split module 402 includes:

[0247] The second acquisition unit is used to acquire the text content of any word and match the text content with multiple preset regular expression patterns to obtain the matching result; wherein, the multiple preset regular expression patterns include evidence anchor pattern, citation structure pattern, statement structure pattern and natural language pattern.

[0248] The first determining unit is used to determine the lexical type of any lexical as the evidence anchor class if the text content is successfully matched with the evidence anchor pattern based on the matching result.

[0249] The second determining unit is used to determine the lexical type of any lexical as the reference structure class if the text content is successfully matched with the reference structure pattern based on the matching result.

[0250] The third determining unit is used to determine the word type of any word as the statement structure class if the text content is successfully matched with the statement structure pattern based on the matching result.

[0251] The fourth determining unit is used to determine the lexical type of any lexical unit as a natural language class if the text content is successfully matched with the natural language pattern based on the matching result.

[0252] The fifth determining unit is used to set the weight of any word element to a preset weight value corresponding to the word element type.

[0253] In one possible implementation, the fifth determining unit includes:

[0254] The first setting component is used to set the weight of any word element to the preset first weight if the word element type is evidence anchor.

[0255] The second setting component is used to set the weight of any word element to a preset second weight if the word element type is a reference structure class; wherein the first weight is greater than the second weight.

[0256] The third setting component is used to set the weight of any word element to a preset third weight if the word element type is a statement structure type; wherein the second weight is greater than the third weight.

[0257] The fourth setting component is used to set the weight of any word element to a preset fourth weight if the word element type is natural language; wherein the third weight is greater than the fourth weight.

[0258] In one possible implementation, the computing module 403 includes:

[0259] The third acquisition unit is used to acquire the prediction probability of any word and calculate the difficulty modulation factor of any word based on the prediction probability; wherein, the prediction probability is used to represent the prediction accuracy of the preset training model for any word.

[0260] The computation unit is used to calculate the training loss of any word based on the weight and difficulty modulation factor, and to determine the training loss of any training sample based on the training loss of any word.

[0261] The training device for the industrial large-scale model of thought chain evidence provided in this embodiment can execute... Figure 2 The technical solution of the embodiment of the training method of the industrial large model of evidence reference in the thought chain is shown, and its implementation principle and technical effect are similar to those of the present invention. Figure 2 The example of the training method for the industrial large model of evidence citing the thought chain is similar and will not be described in detail here.

[0262] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0263] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-mentioned training method of the industrial large model of thought chain evidence reference.

[0264] The specific implementation process of processor 501 can be found in the above-mentioned embodiment of the training method of the industrial large model of evidence reference in the thinking chain. Its implementation principle and technical effect are similar, and will not be repeated here.

[0265] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0266] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0267] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0268] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described training method for the industrial large model of thought chain evidence citation.

[0269] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-mentioned training method for the industrial large-scale model of thought chain evidence citation.

[0270] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0271] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0272] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0273] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0274] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0275] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0276] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0277] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A training method for a large-scale industrial model of evidence citation based on thought chains, characterized in that, include: Multiple training samples are obtained; each training sample includes a training question, the answer to the training question, and a thought process, wherein the thought process refers to the process of deriving the answer to the training question from the content of a preset document based on the training question. Any training sample is split into multiple tokens, and the text content of any token is obtained. The text content is then matched against multiple preset regular expression patterns to obtain matching results. These preset regular expression patterns include evidence anchor patterns, citation structure patterns, statement structure patterns, and natural language patterns. If the text content matches the evidence anchor pattern based on the matching results, the token type is determined to be an evidence anchor class. If the text content matches the citation structure pattern based on the matching results, the token type is determined to be a citation structure class. If the text content matches the statement structure pattern based on the matching results, the token type is determined to be a statement structure class. If the text content matches the natural language pattern based on the matching results, the token type is determined to be a natural language class. Weights are assigned to any given word based on its word type; wherein the weights represent the importance of the given word in the training loss calculation; wherein, if the word type is the evidence anchor class, the weight of the given word is set to a preset first weight; if the word type is the reference structure class, the weight of the given word is set to a preset second weight; wherein the first weight is greater than the second weight; if the word type is the statement structure class, the weight of the given word is set to a preset third weight; wherein the second weight is greater than the third weight; if the word type is the natural language class, the weight of the given word is set to a preset fourth weight; wherein the third weight is greater than the fourth weight. The process involves: obtaining the difficulty modulation factor of any given word; calculating the training loss of any training sample based on the weights and the difficulty modulation factor; determining the training loss of the plurality of training samples based on the training loss of any training sample; wherein the difficulty modulation factor represents the prediction difficulty of the preset model to be trained for the given word; obtaining the prediction probability of the given word; calculating the difficulty modulation factor of the given word based on the prediction probability; wherein the prediction probability represents the prediction accuracy of the preset model to be trained for the given word; calculating the training loss of the given word based on the weights and the difficulty modulation factor; and determining the training loss of any training sample based on the training loss of the given word. The parameters of the preset training model are backpropagated based on the training loss of the multiple training samples. The preset training model whose average training loss of the multiple training samples is less than a preset threshold is determined as the Mind Chain Evidence Reference Industrial Model. The Mind Chain Evidence Reference Industrial Model is used to generate the answer to the question to be processed, the source of the reference of the answer, and the process of deriving the answer to the question to be processed from the content of a preset document.

2. The method according to claim 1, characterized in that, The acquisition of multiple training samples includes: Obtain multiple training questions and their answers, split any training question into multiple semantic units, and obtain the reference interval of each semantic unit; wherein, the reference interval is used to represent the start and end positions of the evidence fragment corresponding to each semantic unit in the preset document content, and the evidence fragment is the text located in the preset document content according to the reference interval; An evidence set for any training question is generated based on the reference interval; wherein, the evidence set is a set of evidence fragments corresponding to the plurality of semantic units; Based on the multiple semantic units and the evidence fragments, the multiple training questions are screened to eliminate abnormal questions and obtain multiple effective training questions. Input any valid training question, the answer to any valid training question, the evidence set, and the content of the preset document into the preset teacher model to obtain the thought chain of any valid training question. The thought chain and any effective training problem are integrated to obtain the target training sample corresponding to the effective training problem, and the plurality of training samples are determined based on the target training sample.

3. The method according to claim 2, characterized in that, The process of filtering the multiple training questions based on the multiple semantic units and the multiple evidence fragments to eliminate abnormal questions and obtain multiple effective training questions includes: Based on the comparison between the reference number of each semantic unit and the total number of fragments in the preset document content, the multiple training questions are filtered to remove training questions of semantic units with reference numbers less than one or greater than the total number of fragments, resulting in multiple first questions; wherein, the reference number refers to the sequence number of the evidence fragment corresponding to each semantic unit in the preset document content; Based on the content length of the evidence fragment corresponding to each semantic unit, the multiple first questions are filtered out to eliminate the first questions where the content length of the evidence fragment corresponding to a semantic unit is zero, thus obtaining multiple second questions; Based on the comparison results between semantic similarity and a preset similarity threshold, the multiple second questions are filtered to remove the second questions containing semantic units with semantic similarity lower than the preset similarity threshold, thereby obtaining the multiple effective training questions; wherein, the semantic similarity refers to the similarity between each semantic unit and the evidence fragment corresponding to each semantic unit.

4. A training device for using a large-scale industrial model of evidence citing thought chains, characterized in that, include: The acquisition module is used to acquire multiple training samples; wherein each training sample includes a training question, the answer to the training question, and a thought chain, wherein the thought chain refers to the process of deriving the answer to the training question from the content of a preset document based on the training question; The splitting module is used to split any training sample into multiple tokens, obtain the text content of any token, and match the text content with multiple preset regular expression patterns to obtain matching results. The multiple preset regular expression patterns include evidence anchor patterns, citation structure patterns, statement structure patterns, and natural language patterns. If the matching result determines that the text content matches the evidence anchor pattern successfully, the token type of the token is determined to be an evidence anchor class; if the matching result determines that the text content matches the citation structure pattern successfully, the token type of the token is determined to be a citation structure class; if the matching result determines that the text content matches the statement structure pattern successfully, the token type of the token is determined to be a statement structure class; if the matching result determines that the text content matches the natural language pattern... Successfully, the lexical type of any given lexical unit is determined to be a natural language class; a weight is assigned to any given lexical unit based on its lexical type; wherein the weight is used to represent the importance of the given lexical unit in the training loss calculation; wherein, if the lexical type of any given lexical unit is the evidence anchor class, the weight of the given lexical unit is set to a preset first weight; if the lexical type of any given lexical unit is the reference structure class, the weight of the given lexical unit is set to a preset second weight; wherein the first weight is greater than the second weight; if the lexical type of any given lexical unit is the statement structure class, the weight of the given lexical unit is set to a preset third weight; wherein the second weight is greater than the third weight; if the lexical type of any given lexical unit is the natural language class, the weight of the given lexical unit is set to a preset fourth weight; wherein the third weight is greater than the fourth weight. A calculation module is configured to obtain the difficulty modulation factor of any given word, calculate the training loss of any training sample based on the weights and the difficulty modulation factor, and determine the training loss of the plurality of training samples based on the training loss of any training sample; wherein, the difficulty modulation factor is used to represent the prediction difficulty of the preset model to be trained for the given word; obtain the prediction probability of the given word, and calculate the difficulty modulation factor of the given word based on the prediction probability; wherein, the prediction probability is used to represent the prediction accuracy of the preset model to be trained for the given word; calculate the training loss of the given word based on the weights and the difficulty modulation factor, and determine the training loss of any training sample based on the training loss of the given word; The training module is used to backpropagate the parameters of the preset model to be trained based on the training loss of the multiple training samples, and to determine the preset model to be trained when the average value of the training loss of the multiple training samples is less than a preset threshold as the thought chain evidence citation industrial big model; wherein, the thought chain evidence citation industrial big model is used to generate the answer to the question to be processed, the citation source of the answer, and the process of deriving the answer to the question to be processed from the content of a preset document.

5. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; When the processor executes the computer execution instructions stored in the memory, it is used to implement the training method of the industrial large model of thought chain evidence citation as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the training method for the industrial large model of thought chain evidence citation as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, is used to implement the training method for the industrial large model of thought chain evidence citation as described in any one of claims 1 to 3.

Images