Domain large model construction method based on reverse fact disassembly and fact perception fine-tuning

Abstract

The application discloses a field large model construction method based on reverse fact disassembly and fact perception fine-tuning, and belongs to the technical field of artificial intelligence and natural language processing. The method comprises the following steps: calling a multi-modal document analysis engine to extract full-quantity continuous text flow; constructing a dynamic regular expression matrix for heuristic anchor point matching; filtering layout noise and performing semantic error correction by using a pre-trained large language model; configuring a reverse extraction instruction to perform de-privatization and de-logicalization feature dimension reduction on target data; constructing a fact-consistent instruction fine-tuning triple data set; introducing a fact perception loss mask strategy in the calculation of the cross-entropy loss of model fine-tuning, and dynamically amplifying the loss weight of the core fact identifier; dynamically merging the adapter weight and the main weight and performing low-bit quantization compression, and deploying an output high-precision financial field large model. The application effectively eliminates the fact illusion in large model fine-tuning, and significantly improves the fitting capability for key numerical values.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method for constructing a large domain model based on reverse fact decomposition and fact perception fine-tuning. Background Technology

[0002] In recent years, artificial intelligence technologies, represented by large language models, have made groundbreaking progress, achieving unprecedented results in general fields such as natural language understanding, logical reasoning, and content generation. With technological evolution, large models are gradually penetrating vertical industries. In the financial investment research field, high-quality industry analysis reports and financial statement interpretations demand extremely high levels of accuracy, timeliness, and professional logic. Therefore, building "large financial models" with professional investment research capabilities has become a key direction for fintech development. Modern financial institutions have accumulated massive amounts of unstructured documents (such as PDF financial reports and brokerage research reports), providing a rich foundation of corpora for training large-scale models.

[0003] However, current methods for transforming general-purpose large models into specialized financial large models still have significant shortcomings, mainly in the following two aspects: First, acquiring high-quality fine-tuning data is extremely costly and prone to creating factual illusions. Existing domain fine-tuning typically relies on experienced analysts with specialized backgrounds manually writing input-output question-and-answer pairs, which is time-consuming, labor-intensive, and difficult to scale up. Some studies attempt to use large models to automatically generate data, i.e., directly inputting financial indicators and letting the large model generate research reports freely. However, since large models are essentially probability-based autoregressive generators, this positive generation is highly susceptible to creating factual illusions by fabricating financial indicators out of thin air or confusing subject and object, and such factual errors are extremely fatal in rigorous financial business.

[0004] Secondly, traditional fine-tuning algorithms lack sensitivity to core financial facts. Most mainstream supervised fine-tuning methods currently employ the standard cross-entropy loss function, treating all generated terms equally during gradient updates. This results in the model incurring the same error penalty weight when predicting ordinary logical connectors and when predicting core financial data (such as "net profit increased by 136% year-on-year"). This mechanism leads to the fine-tuned model merely learning the linguistic appearance and writing style of financial research reports, lacking the ability to deeply constrain and accurately fit key financial figures and specialized terms.

[0005] In summary, the industry urgently needs a low-cost, fully automated, high-quality data synthesis paradigm, as well as a fine-tuning mechanism that can enhance the sensitivity of large models to core figures and facts, thereby completely solving the problems of high labor costs and factual bias in the process of building large domain models. Summary of the Invention

[0006] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a method for constructing a large domain model based on reverse fact decomposition and fact perception fine-tuning.

[0007] Technical solution: The present invention provides a method for constructing a large domain model based on reverse fact decomposition and fact-aware fine-tuning, comprising the following steps: Step 1: Receive the original unstructured financial document and call the multimodal document parsing engine to extract the full continuous text stream; Step 2: Construct a dynamic regular expression matrix to perform heuristic anchor matching, extract and generate coarse-grained original text with typographic noise from the full continuous text stream; Step 3: Use a pre-trained large language model to filter out typesetting noise and perform semantic error correction, and reconstruct and generate professional text that conforms to the preset analysis structure as the target data. Step 4: Configure reverse extraction instructions to perform feature reduction on the target data by removing viewpoints and logic, extract the underlying financial indicators and entity names, and generate a structured list of discrete fact data. Step 5: Automatedly concatenate the domain role instructions, discrete fact data list, and target data to construct a fact-consistent instruction fine-tuning triple dataset; Step 6: Construct the loss function in the iterative process of the large language model, use the instruction fine-tuning triple dataset to perform low-rank adaptive iterative optimization on the large language model, and train to obtain the domain adapter weights. Step 7: Dynamically merge the domain adapter weights and the backbone weights, and perform low-bit quantization compression on the merged model to deploy and obtain a large financial domain model. Use the large financial domain model to realize the business function of automatically generating professional financial research reports based on input structured financial data.

[0008] Preferably, step 1 includes: The original unstructured financial documents are input into a multimodal document parsing engine, eliminating the physical separation of multi-column layout and cross-page sentence breaks. Layout analysis and optical character recognition are performed, and the output is a set of text blocks containing spatial coordinate information. Each text block contains text attributes and a corresponding bounding box feature vector. ,in The coordinates of the top left corner of the bounding box. , These are the width and height, respectively. Calculate the spatial Euclidean distance and vertical alignment between any two text blocks based on the spatial bounding box feature vectors; Construct a directed graph model to represent the reading order of document layout. ,in, A set of nodes that corresponds one-to-one with the set of text blocks. Let be the set of directed edges used to connect different text block nodes, and The edge weight of each directed edge is used to represent the semantic connection probability between corresponding text blocks; For directed graph models The topological sorting algorithm is executed to find the one-dimensional traversal path that maximizes the global weight. This path is then used as the globally optimal traversal path, and the data is serialized and concatenated to obtain the full continuous text stream.

[0009] Preferably, step 2 includes: Step 21: Predefine a hash set consisting of financial road sign words as a dictionary space. This dictionary space is used to provide indexes for matching road sign words and is released after matching is completed. Step 22: Construct a dynamic regular expression matrix, which contains a starting feature vector set and a terminating feature vector set; Step 23: Set the sliding window length and step size. Use the sliding window to traverse the entire continuous text stream and calculate the matching confidence score between the text substring within the window and the starting and ending feature vector sets. The formula is: , in, This represents a multidimensional word embedding feature vector representing a substring of text within a sliding window. This represents the target multidimensional word embedding feature vector in the start or end feature vector set. This indicates the computational norm of a vector; Step 24, based on the activation threshold With cutoff threshold Determine the degree of matching between the text substring and the target features, and determine the starting index. With Termination Index To obtain the globally optimal index range ,in ; Step 25, based on the globally optimal index range The entire continuous text stream is sliced ​​and extracted, and the text content within the interval is used as coarse-grained raw text.

[0010] Preferably, step 3 includes: Step 31, Construct the forward formatting constraint instruction space It contains a prior template matrix used to define the output structure. and the set of penalty hyperparameters used to suppress divergence generation ; Step 32: Convert the coarse-grained raw text into a sequence of terms. ,in Indicates the first Each word element, along with positive formatting constraint instructions, is input into a pre-trained large language model; Step 33: The large language model, based on an autoregressive decoding mechanism, decodes a given sequence of lexical terms. and instruction space Under the given conditions, calculate the target word sequence. The joint conditional probability distribution, whose objective function is expressed by the chain rule, is as follows: , in, Indicates the first Each generated word element The total length of the generated sequence; In each decoding step, the large language model selects the word with the highest probability for output based on the local conditional probability distribution of the current step using a greedy decoding strategy. Step 34: The target word sequence of the decoded output is processed using the set structured regularization function. Perform format validation; if it meets the preset three-segment topological structure rules, then the target word sequence will be... Target data after denoising Otherwise, discard the currently generated target word sequence. and increase local temperature parameters This re-triggers the autoregressive decoding and probability calculation process to generate new candidate sequences.

[0011] Preferably, step 4 includes: Establish reverse delogization mapping function This mapping function is a semantic recognition-based information extraction operator used to identify and extract entity feature words with objective attributes from text, map these entity feature words into structured sets, and construct a stop feature dictionary containing subjective adjectives, predictive verbs, and causal conjunctions. ; target data Input is a large language model configured with reverse extraction of prompt words, combined with a stop feature dictionary. Perform attention masking and information entropy dimensionality reduction; During feature reduction, the model is forced to assign high attention weights only to numerical values, time nodes, and proper noun entities with objective attributes. The extracted entities are symbolized as a set of discrete fact triples. The expression is: ,in, The total number of facts extracted, Indicates the first Individual indicator entity name, This represents the numerical or state characteristics of the entity. This indicates the time dimension characteristic of the entity; set Clustering and concatenation based on business logic categories generates a tree-like hierarchical text containing category labels, serving as a structured list of discrete fact data. .

[0012] Preferably, step 5 includes: Obtain system-level role instructions for defining the boundaries of large domain model tasks. And initialize a special control character set for model-supervised fine-tuning. ,in, This is the system prompt. For user prompt, For model response prompts, This is the sequence terminator; Define string concatenation operators Based on preset cross-entropy mask alignment rules, system-level role instructions are... Discrete fact data list With target data Automated sequence assembly is performed to construct a fact-consistent triplet supervised fine-tuning dataset containing instruction, input, and output structures. The assembly formula is: .

[0013] Preferably, step 5 further includes: After splicing, the triple-supervised fine-tuning dataset was performed. Perform byte-pair-based lexicalization to generate the input tensor matrix and mark the corresponding target data in the concatenated full sequence. The complete sequence interval formed by the start index and the end index , which serves as the effective target domain for calculating the loss function during backpropagation.

[0014] Preferably, step 6 includes: Step 61: For large language models based on the Transformer architecture, freeze the backbone network parameters. A low-rank adaptive dimensionality reduction matrix is ​​injected into the bypass of the key layer of the self-attention mechanism. With increasing dimension matrix During the iteration process, only the dimensionality reduction matrix is ​​updated. and the increasing dimension matrix The parameters, and the rank , Let be the dimension of the hidden layer of the model, and rank be... For the dimension reduction matrix With increasing dimension matrix Common internal low-rank dimension; Step 62: Fine-tune the target word sequence in the triplet dataset according to the instructions. The sequence is scanned using a named entity recognition algorithm and a regularization engine to construct a binary fact indicator vector of the same length as the sequence. When word element When it pertains to financial figures, time points, or proper nouns, Otherwise ; Step 63, construct the dynamic loss weight coefficient sequence The weighting formula is as follows: , in, For the first The dynamic loss weight coefficients corresponding to each word element As a factor amplifying the punishment of facts, and ; Step 64: Define the fact-aware cross-entropy loss function. In each forward propagation of the autoregressive training, the prediction loss of each word is weighted based on the dynamic loss weight coefficient sequence; the loss function formula is: , in, For the pre-generated word sequence, For trainable low-rank matrix parameters, This represents the conditional probability distribution of the model's output target word under given conditions. The length of the target word sequence. The position index in the target word sequence; Step 65: Minimize the loss function using the backpropagation algorithm until the large language model converges, and output the trained domain adapter weights. .

[0015] Preferably, step 7 includes: Step 71, based on the trained domain adapter weights Calculate the weight increment matrix The calculation formula is: , in, For scaling hyperparameters; Step 72, adjust the weight increment matrix. Compared with the backbone network parameters of the original large language model Perform linear addition to merge the parameters to obtain the weights of the domain-wide large model after parameter merging, as follows: ; Step 73: Set the scaling factor based on the maximum absolute value of the weight tensor. and zero offset Mapping floating-point domain-specific large model weights to integer weights The quantification formula is: , in, This is a truncation function. For nearest neighbor floor function, and These are the minimum and maximum allowed values ​​for the quantization bit width, respectively; Step 74, quantize the integer weights Deployment to large-scale models in the financial sector.

[0016] Beneficial effects: Compared with the prior art, the significant advantages of this invention are: 1. By utilizing inverse feature decomposition, the illusion of factuality in the data construction process is eliminated to a certain extent: This invention first uses a large model to denoise and reconstruct the original document to obtain the target output, and then forces the large model to perform de-logicization and dimensionality reduction to extract purely objective factual data. Since the list of discrete factual data is strictly physically extracted from the high-quality target data, this ensures 100% factual consistency between the input conditions and the output results from the underlying logic of data generation. This eliminates the need for post-verification using massive computing power, thus obtaining a high-quality SFT training dataset free of illusions.

[0017] 2. This invention proposes a fact-aware loss mask to overcome the shortcomings of standard autoregressive fine-tuning: By intervening at the algorithmic level, this invention introduces a fact indicator vector into the loss function. This causes the model to be dynamically penalized for incorrect predictions of core indicators, values, or timeframes during backpropagation weight updates. This technique fundamentally enhances the logical constraints and sensitivity of large-scale domain models to financial numerical data through a robust mathematical optimization approach.

[0018] 3. Achieved a low-cost, automated closed loop from unstructured documents to high-quality instruction data: This invention effectively replaces the inefficient model in traditional financial large model construction that heavily relies on expensive industry experts to manually write SFT question-answer pairs by integrating multimodal parsing, dynamic regularization matching, and large model collaboration, significantly reducing the R&D cost and cycle of domain large models. Attached Figure Description

[0019] Figure 1 The flowchart shows a method for constructing a large domain model based on reverse fact decomposition and fact perception fine-tuning. Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0020] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.

[0021] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0026] Combination Figure 1 and Figure 2 As shown in this embodiment, the domain-wide model construction method based on reverse fact decomposition and fact-aware fine-tuning includes the following steps: Step 1: Receive the original unstructured financial document and call the multimodal document parsing engine to extract the full continuous text stream.

[0027] Further, step 1 includes: The original unstructured financial documents are input into a multimodal document parsing engine, eliminating the physical separation of multi-column layout and cross-page sentence breaks. Layout analysis and optical character recognition are then performed, outputting a set of text blocks containing spatial coordinate information. ,in The total number of text blocks, and each text block Includes text attributes and the corresponding spatial bounding box feature vector ,in The coordinates of the top left corner of the bounding box. , These are the width and height, respectively; the multimodal document parsing engine adopts a layout analysis model based on the existing open-source framework LayoutLM architecture and an optical character recognition (OCR) component, specifically by extracting document features and recognizing text content through a convolutional neural network; Based on spatial bounding box feature vectors Calculate any two text blocks and Spatial Euclidean distance between and vertical alignment The calculation formulas are as follows: , , Construct a directed graph model to represent the reading order of document layout. ,in, This is the set of nodes that correspond one-to-one with the set of text blocks B. Let be the set of directed edges used to connect different text block nodes, and The edge weight of each directed edge is used to represent the semantic connection probability between corresponding text blocks; For directed graph models The topological sorting algorithm is executed to find the one-dimensional traversal path with the largest global weight, which is then used as the globally optimal traversal path. This path is then serialized and concatenated to obtain the complete continuous text stream. .

[0028] The specific search process uses Dijkstra's algorithm or depth-first search algorithm to calculate the path with the highest global semantic coherence probability weight, finds the globally optimal traversal path, serializes and concatenates it, and outputs a full continuous text stream.

[0029] Step 2: Construct a dynamic regular expression matrix for heuristic anchor matching, extract and generate coarse-grained original text with typographic noise from the full continuous text stream.

[0030] Further, step 2 includes: Step 21: Predefine a hash set consisting of financial road sign words as a dictionary space. This dictionary space is used to provide indexes for matching road sign words and is released after matching is completed. Step 22, Construct a dynamic regular expression matrix The matrix Includes the initial feature vector set and terminating feature vector set ,in This refers to financial structured roadmap terms determined by expert experience. , It is represented as a multi-dimensional word embedding vector derived from the mapping of financial structured road sign words determined by expert experience; Step 23, Set the length of the sliding window With step size Use a sliding window to traverse the entire continuous text stream Calculate the text substring within the window and the initial feature vector set. and terminating feature vector set The matching confidence score is calculated using the following formula: , in, This represents a multidimensional word embedding feature vector representing a substring of text within a sliding window. This represents the target multidimensional word embedding feature vector in the start or end feature vector set. This indicates the calculation of the L2 norm of a vector, which is the magnitude of the vector; Step 24, based on the activation threshold With cutoff threshold Determine the degree of matching between the text substring and the target features, and determine the starting index. With Termination Index To obtain the globally optimal index range ,in ; Specifically, when a substring is detected that matches the initial feature vector set The matching confidence score is greater than the set activation threshold. At that time, record the current text sequence index as the starting point for interception. When a subsequent substring is detected that matches the terminating feature vector set The matching confidence score is greater than the set cutoff threshold. At that time, record the current text sequence index as the termination point. .

[0031] Step 25, based on the globally optimal index range For the full continuous text stream Perform slice extraction to extract the text content within the specified interval as coarse-grained raw text. .

[0032] Extracting the text content within this range as the original text effectively filters out irrelevant information from the beginning and end of the document. This generates coarse-grained original text with layout noise. And output it to the next processing flow.

[0033] Step 3: Use a pre-trained large language model to filter out typesetting noise and perform semantic error correction, and reconstruct professional text that conforms to the preset analysis structure. This text is used as the target data and is denoted as Output.

[0034] Furthermore, step 3 includes: Step 31, Construct the forward formatting constraint instruction space It contains a prior template matrix used to define the output structure. and the set of penalty hyperparameters used to suppress divergence generation ; Step 32, convert the coarse-grained original text Transform into a word sequence Together with the positive formatting constraint instructions, they are input into a pre-trained large language model, such as a general large-scale pre-trained model with a known structure, like GPT-4, Llama3, or DeepSeek-V3. Step 33: The large language model, based on an autoregressive decoding mechanism, decodes a given sequence of lexical terms. and instruction space Under the given conditions, calculate the target word sequence. The joint conditional probability distribution, whose objective function is expressed by the chain rule, is as follows: , in, Indicates the first Each generated word element The total length of the generated sequence; In each decoding step, the large language model selects the word with the highest probability for output based on the local conditional probability distribution of the current step using a greedy decoding strategy. Step 34: The target word sequence of the decoded output is processed using the set structured regularization function. Perform format validation. If the preset three-segment topology structure rule is met, where the generated text must be strictly formatted as three natural segments, with each segment explicitly physically isolated and logically segmented by preset feature delimiters, then the word sequence will be... Target data after denoising Otherwise, discard the currently generated target word sequence. and increase local temperature parameters This re-triggers the autoregressive decoding and probability calculation process to generate new candidate sequences. Among these, the local temperature parameter... Used to adjust the smoothness of the word prediction probability distribution, by increasing This can improve the randomness and diversity of the output results, thereby seeking alternative paths that meet the verification conditions.

[0035] In this step, the large model acts as a high-level cleaning filter. During the autoregressive generation process, to prevent the model from generating factual illusions—that is, fabricating financial data that doesn't exist in the PDF—the sampling strategy must be strictly controlled. Typically, the local temperature parameter T is set... The lower range between these values ​​is used to reduce the randomness of the generation; the Top-P parameter is set to... Left and right, restricting the sampling of low-probability words.

[0036] The calculation of conditional probability distributions is highly dependent on the token length. If a segment of text truncated from the original PDF exceeds the upper limit of the context window for a single inference by the large model (e.g., exceeding 4096 tokens), it will lead to computational truncation (OOM error). Therefore, in practical engineering, it is necessary to pre-set a sliding window for block input, and then have the large model merge and denoise the output.

[0037] Step 4: Configure reverse extraction instructions to perform feature reduction on the target data by removing viewpoints and logic, extract the underlying financial indicators and entity names, and generate a structured list of discrete fact data, Input.

[0038] Furthermore, step 4 includes: Establish reverse delogization mapping function This mapping function is a semantic recognition-based information extraction operator used to identify and extract entity feature words with objective attributes from text, map these entity feature words into structured sets, and construct a stop feature dictionary containing subjective adjectives, predictive verbs, and causal conjunctions. ; target data The input is a large language model configured with reverse extraction prompts (e.g., "Please extract the core financial indicators, values, and times from the following research report text, remove all subjective evaluations and logical connectors, and generate independent sentences that are unrelated to each other"), combined with a deactivation feature dictionary. Perform attention masking and information entropy dimensionality reduction; During feature reduction, the model is forced to assign high attention weights only to numerical values, time nodes, and proper noun entities with objective attributes. The extracted entities are symbolized as a set of discrete fact triples. The expression is: ,in, The total number of facts extracted, Indicates the first Individual indicator entity name, This represents the numerical or state characteristics of the entity. This indicates the time dimension characteristic of the entity; set Clustering and concatenation based on business logic categories generates a tree-like hierarchical text containing category labels, serving as a structured list of discrete fact data. .

[0039] Step 4 is the core of this invention, namely, reverse delogization and feature dimensionality reduction. Traditional natural language processing tasks mostly involve divergent generation, while this step involves convergent extraction. In the financial context, these correspond to professional terms such as "operating revenue," "net profit attributable to the parent company," and "market share." The corresponding absolute value or growth percentage (e.g., "105 billion yuan", "year-on-year +12%)"). The financial reporting period must be precisely aligned (e.g., "Q3 2023").

[0040] To prevent the model from incorporating subjective biases into the extracted input list, it is necessary to... The dictionary pre-loads highly subjective words such as "eye-catching," "exceeding expectations," and "under pressure." In the attention mechanism computation, negative mask weights are applied to these words, forcing the model to output only purely discrete facts (i.e., (Set), this reverse physical extraction mechanism is the underlying guarantee to ensure that the final fine-tuning of the large model does not produce factual bias.

[0041] Step 5: Automatedly concatenate the domain role instructions, the list of discrete fact data, and the target data to construct a fact-consistent instruction fine-tuning triplet dataset.

[0042] Furthermore, step 5 includes: Obtain system-level role instructions for defining the boundaries of large domain model tasks. ( Figure 2 (referred to as Prompt), and initializes a special control character set for model-supervised fine-tuning. ,in, For system prompt, For user prompt, For model response prompts, This is the sequence terminator; Define string concatenation operators Based on a preset cross-entropy mask alignment rule, when concatenating multiple text segments into a complete sequence, a binary mask vector of the same length as the sequence is simultaneously constructed. This aligns the mask values ​​of system-level role instructions and discrete factual data intervals to 0, ensuring that only the target data... Align the mask values ​​of the intervals to 1; align system-level role commands. Discrete fact data list With target data Automated sequence assembly is performed to construct a fact-consistent triplet supervised fine-tuning dataset containing instruction, input, and output structures. ( Figure 2 (referred to as SFT in Chinese) The assembly formula is: .

[0043] Furthermore, step 5 also includes: After splicing, the triple-supervised fine-tuning dataset was performed. Perform byte-pair-based lexicalization to generate the input tensor matrix and mark the corresponding target data in the concatenated full sequence. The complete sequence interval formed by the start index and the end index , which serves as the effective target domain for calculating the loss function during backpropagation.

[0044] Step 5 completes the transformation and encapsulation of unstructured data into standard large-model fine-tuning data. In the concatenation formula, the system prompt... User prompt These act as special separators. When generating a tensor matrix, if the input part (...) is processed during backpropagation... Calculating gradients would lead to the model "memorizing" the input question. Therefore, this invention strictly defines the target domain. This ensures that the calculation of cross-entropy loss applies only to the target data. This allows the model to truly learn "reasoning and analytical abilities" rather than rote memorization.

[0045] Step 6: Construct the loss function for the iterative process of the large language model, and use the instruction fine-tuning triple dataset to perform low-rank adaptive iterative optimization on the large language model to obtain the domain adapter weights.

[0046] Furthermore, step 6 includes: Step 61: For large language models based on the Transformer architecture, freeze the parameters of their backbone network. A low-rank adaptive dimensionality reduction matrix is ​​injected into the bypass of the key layer of its self-attention mechanism. With increasing dimension matrix During the iteration process, only the dimensionality reduction matrix is ​​updated. and the increasing dimension matrix The parameters, and the rank , Let be the dimension of the hidden layer of the model, where rank For the dimension reduction matrix With increasing dimension matrix Common internal low-rank dimension; Step 62: Fine-tune the target word sequence in the triplet dataset according to the instructions. The sequence is scanned using a named entity recognition algorithm and a regularization engine to construct a binary fact indicator vector of the same length as the sequence. When word elements When it pertains to financial figures, time points, or proper nouns, Otherwise ; Step 63, construct the dynamic loss weight coefficient sequence The weighting formula is as follows: , in, For the first The dynamic loss weight coefficients corresponding to each word element As a factor amplifying the punishment of facts, and ; Step 64: Define the fact-aware cross-entropy loss function. In each forward propagation of the autoregressive training, the prediction loss of each word is weighted based on the dynamic loss weight coefficient sequence; the loss function formula is: , in, The input consists of a sequence of instructions and discrete factual data. For the pre-generated word sequence, For trainable low-rank matrix parameters, This represents the conditional probability distribution of the model's output target word under given conditions. The length of the target word sequence. The position index in the target word sequence; Step 65: Minimize the loss using the backpropagation algorithm. Continue until the large language model converges, then output the trained domain adapter weights. .

[0047] In traditional deep learning fine-tuning (such as standard Cross-Entropy), the error generated by the model when predicting stop words such as "de" (of) and "le" (past tense marker) contributes exactly the same to the model gradient as the error when predicting numbers in "net profit increased by 130%". This "equal treatment" is the fundamental reason why large financial models are insensitive to numbers and produce "factual hallucinations". In the weight distribution formula, the fact penalty amplification factor is a key valve for regulating the "rigor degree" of the model. In actual engineering implementation, if , it degenerates into traditional fine-tuning; in the present invention, it is preferably set as . This means that when the model deviates in predicting a "financial number (such as 130%)", the loss penalty it receives is 3 to 6 times that of predicting ordinary text.

[0048] By introducing weights , the model is forced to allocate more model capacity to fitting the mapping of financial logic when performing gradient descent to update the matrix and , thereby rigidly suppressing the behavior of fabricating numbers in the mathematical dimension.

[0049] Step 7: dynamically merge the domain adapter weights with the backbone weights, perform low-bit quantization compression on the merged model, and deploy to obtain a large financial domain model, which is used to realize the business function of fully automatically generating professional financial research reports based on input structured financial data.

[0050] Further, step 7 includes: Step 71: based on the trained domain adapter weights calculate the weight increment matrix , the calculation formula is: , wherein, is a scaling hyperparameter; Step 72: combine the weight increment matrix with the backbone network parameters of the original large language model perform linear additive merging to obtain the domain large model weight after parameter merging, expressed as: ; Step 73: set a scaling factor according to the maximum absolute value of the weight tensor and a zero offset , map the floating-point domain large model weights to 8-bit or 4-bit integer weights , the quantization formula is: , wherein, For nearest neighbor floor function, This is a truncation function. and These are the minimum and maximum allowed values ​​for the quantization bit width, respectively; Step 74, quantize the integer weights Deployed to large-scale financial models, it provides standardized API interfaces to enable the automatic generation of professional financial research reports based on input structured financial data.

[0051] To verify the effectiveness of this invention, in actual engineering implementation, the system performed automated data synthesis verification on a batch of real brokerage research report PDFs.

[0052] In the formula for calculating the weight increment matrix, since this example directly adds the result of the two matrix multiplications to the weight matrix of the original model during the inference phase, the fine-tuned model does not incur any additional computational delay during inference.

[0053] Financial institutions typically have limited computing power on their local servers, so they use... The truncation formula in this example achieves almost no loss of accuracy in financial logic reasoning (accuracy degradation). Under the premise of [missing information], models that originally required a large amount of video memory (such as 32GB of video memory under FP16 precision) can be compressed to run smoothly with only 8GB or even less video memory (INT4 precision).

[0054] To further verify the effectiveness and feasibility of the data assembly mechanism of this invention, in the actual engineering verification stage, the system, based on the automated splicing rules of the aforementioned steps, finally encapsulated the data and exported a high-quality SFT corpus that meets industrial-grade standards. As shown in Table 1, this invention actually generates a high-quality instruction fine-tuning (SFT) dataset sample structure table.

[0055] Table 1. SFT Data Samples The generated data strictly follows a structured triplet format, where: The Instruction field is obtained from the system-level role instruction (Prompt) acquired in step 5 and written into the triplet supervised fine-tuning dataset when automatically concatenating the system-level role instruction, the list of discrete fact data, and the target data. It is used to carry out system-level role assignment and clarify the task boundaries of the model. The input field fully encapsulates the purely objective financial data (such as operating revenue, net profit attributable to the parent company, and market value range) extracted in reverse physical steps 4, ensuring that there is no subjective logical noise at the input end; The output field precisely corresponds to the target data reconstructed from step 3, showcasing a well-structured and professionally written analysis result.

[0056] Experimental results show that using traditional positive prompts to directly generate financial statement analysis SFT data from large models results in a high proportion of numerical fabrication and subject confusion errors in the generated complex financial analysis text. However, the dual-loop data synthesis engine with forward reconstruction and reverse decomposition proposed in this invention successfully and automatically outputs batches of high-quality instruction fine-tuning triplet datasets.

[0057] The actual data structure shown in Table 1 perfectly matches the underlying data loading specifications of mainstream parameter-efficient fine-tuning frameworks (such as LLaMA-Factory). This fully demonstrates that the forward reconstruction and reverse decomposition data synthesis paradigm proposed in this invention has successfully established a fully automated technical loop from unstructured PDF research reports to standard training materials for large models, possessing extremely high engineering application value.

[0058] During the manual verification of the generated SFT dataset, it was found that, since the Input (discrete facts) in this method is strictly extracted from the Output (gold target data) by dimensionality reduction through masking, the logical causal relationship and numerical accuracy between the two are highly consistent with the theory. No "factual illusion" samples caused by the free divergence of the large model were detected. This fully proves that the present invention can realistically and efficiently construct a high-quality domain fine-tuning dataset that meets the rigor requirements of finance without relying on expensive manual annotation and using existing open source large models.

Claims

1. A method for constructing a large-scale domain model based on reverse fact decomposition and fact perception fine-tuning, characterized in that, Includes the following steps: Step 1: Receive the original unstructured financial document and call the multimodal document parsing engine to extract the full continuous text stream; Step 2: Construct a dynamic regular expression matrix to perform heuristic anchor matching, extract and generate coarse-grained original text with typographic noise from the full continuous text stream; Step 3: Use a pre-trained large language model to filter out typesetting noise and perform semantic error correction, and reconstruct and generate professional text that conforms to the preset analysis structure as the target data. Step 4: Configure reverse extraction instructions to perform feature reduction on the target data by removing viewpoints and logic, extract the underlying financial indicators and entity names, and generate a structured list of discrete fact data. Step 5: Automatedly concatenate the domain role instructions, discrete fact data list, and target data to construct a fact-consistent instruction fine-tuning triple dataset; Step 6: Construct the loss function in the iterative process of the large language model, use the instruction fine-tuning triple dataset to perform low-rank adaptive iterative optimization on the large language model, and train to obtain the domain adapter weights. Step 7: Dynamically merge the domain adapter weights and the backbone weights, and perform low-bit quantization compression on the merged model to deploy and obtain a large financial domain model. Use the large financial domain model to realize the business function of automatically generating professional financial research reports based on input structured financial data. Step 3 includes: Step 31, Construct the forward formatting constraint instruction space It contains a prior template matrix used to define the output structure. and the set of penalty hyperparameters used to suppress divergence generation ; Step 32: Convert the coarse-grained raw text into a sequence of terms. ,in Indicates the first Each word element, along with positive formatting constraint instructions, is input into a pre-trained large language model; Step 33: The large language model, based on an autoregressive decoding mechanism, decodes a given sequence of lexical terms. and instruction space Under the given conditions, calculate the target word sequence. The joint conditional probability distribution, whose objective function is expressed by the chain rule, is as follows: ， in, Indicates the first Each generated word element The total length of the generated sequence; In each decoding step, the large language model selects the word with the highest probability for output based on the local conditional probability distribution of the current step using a greedy decoding strategy. Step 34: The target word sequence of the decoded output is processed using the set structured regularization function. Perform format validation; if it meets the preset three-segment topological structure rules, then the target word sequence will be... Target data after denoising Otherwise, discard the currently generated target word sequence. and increase local temperature parameters This re-triggers the autoregressive decoding and probability calculation process to generate new candidate sequences; Step 4 includes: Establish reverse delogization mapping function This mapping function is a semantic recognition-based information extraction operator used to identify and extract entity feature words with objective attributes from text, map these entity feature words into structured sets, and construct a stop feature dictionary containing subjective adjectives, predictive verbs, and causal conjunctions. ; target data Input is a large language model configured with reverse extraction of prompt words, combined with a stop feature dictionary. Perform attention masking and information entropy dimensionality reduction; During feature reduction, the model is forced to assign high attention weights only to numerical values, time nodes, and proper noun entities with objective attributes. The extracted entities are symbolized as a set of discrete fact triples. The expression is: ,in, The total number of facts extracted, Indicates the first Individual indicator entity name, This represents the numerical or state characteristics of the entity. This indicates the time dimension characteristic of the entity; set Clustering and concatenation based on business logic categories generates a tree-like hierarchical text containing category labels, serving as a structured list of discrete fact data. .

2. The domain-wide large-scale model construction method based on reverse fact decomposition and fact perception fine-tuning according to claim 1, characterized in that, Step 1 includes: The original unstructured financial documents are input into a multimodal document parsing engine, eliminating the physical separation of multi-column layout and cross-page sentence breaks. Layout analysis and optical character recognition are performed, and the output is a set of text blocks containing spatial coordinate information. Each text block contains text attributes and a corresponding bounding box feature vector. ,in The coordinates of the top left corner of the bounding box. , These are the width and height, respectively; Calculate the spatial Euclidean distance and vertical alignment between any two text blocks based on the spatial bounding box feature vectors; Construct a directed graph model to represent the reading order of document layout. ,in, A set of nodes that corresponds one-to-one with the set of text blocks. Let be the set of directed edges used to connect different text block nodes, and The edge weight of each directed edge is used to represent the semantic connection probability between corresponding text blocks; For directed graph models The topological sorting algorithm is executed to find the one-dimensional traversal path that maximizes the global weight. This path is then used as the globally optimal traversal path, and the data is serialized and concatenated to obtain the full continuous text stream.

3. The domain-wide large-scale model construction method based on reverse fact decomposition and fact perception fine-tuning according to claim 1, characterized in that, Step 2 includes: Step 21: Predefine a hash set consisting of financial road sign words as a dictionary space. This dictionary space is used to provide indexes for matching road sign words and is released after matching is completed. Step 22: Construct a dynamic regular expression matrix, which contains a starting feature vector set and a terminating feature vector set; Step 23: Set the sliding window length and step size. Use the sliding window to traverse the entire continuous text stream and calculate the matching confidence score between the text substring within the window and the starting and ending feature vector sets. The formula is: ， in, This represents a multidimensional word embedding feature vector representing a substring of text within a sliding window. This represents the target multidimensional word embedding feature vector in the start or end feature vector set. This indicates the computational norm of a vector; Step 24, based on the activation threshold With cutoff threshold Determine the degree of matching between the text substring and the target features, and determine the starting index. With Termination Index To obtain the globally optimal index range ,in ; Step 25, based on the globally optimal index range The entire continuous text stream is sliced ​​and extracted, and the text content within the interval is used as coarse-grained raw text.

4. The domain-wide large-scale model construction method based on reverse fact decomposition and fact perception fine-tuning according to claim 1, characterized in that, Step 5 includes: Obtain system-level role instructions for defining the boundaries of large domain model tasks. And initialize a special control character set for model-supervised fine-tuning. ,in, This is the system prompt. For user prompt, For model response prompts, This is the sequence terminator; Define string cascading operators Based on preset cross-entropy mask alignment rules, system-level role instructions are... Discrete fact data list With target data Automated sequence assembly is performed to construct a fact-consistent triplet supervised fine-tuning dataset containing instruction, input, and output structures. The assembly formula is: 。 5. The domain-wide large-scale model construction method based on reverse fact decomposition and fact perception fine-tuning according to claim 4, characterized in that, Step 5 also includes: After splicing, the triple-supervised fine-tuning dataset was performed. Perform byte-pair-based lexicalization to generate the input tensor matrix and mark the corresponding target data in the concatenated full sequence. The complete sequence interval formed by the start index and the end index , which serves as the effective target domain for calculating the loss function during backpropagation.

6. The domain-wide large-scale model construction method based on reverse fact decomposition and fact perception fine-tuning according to claim 5, characterized in that, Step 6 includes: Step 61: For large language models based on the Transformer architecture, freeze the backbone network parameters. A low-rank adaptive dimensionality reduction matrix is ​​injected into the bypass of the key layer of the self-attention mechanism. With increasing dimension matrix During the iteration process, only the dimensionality reduction matrix is ​​updated. and the increasing dimension matrix The parameters, and the rank , Let be the dimension of the hidden layer of the model, and rank. For dimensionality reduction matrix With increasing dimension matrix Common internal low-rank dimension; Step 62: Fine-tune the target word sequence in the triplet dataset according to the instructions. The sequence is scanned using a named entity recognition algorithm and a regularization engine to construct a binary fact indicator vector of the same length as the sequence. When the word element When it pertains to financial figures, time points, or proper nouns, Otherwise ; Step 63, Construct the dynamic loss weight coefficient sequence The weighting formula is as follows: ， in, For the first The dynamic loss weight coefficients corresponding to each word element As a factor amplifying the punishment of facts, and ; Step 64: Define the fact-aware cross-entropy loss function. In each forward propagation of the autoregressive training, the prediction loss of each word is weighted based on the dynamic loss weight coefficient sequence; the loss function formula is: ， in, For the pre-generated word sequence, For trainable low-rank matrix parameters, The model outputs the conditional probability distribution of the target word under given conditions. The length of the target word sequence. The position index in the target word sequence; Step 65: Minimize the loss function using the backpropagation algorithm until the large language model converges, and output the trained domain adapter weights. .

7. The domain-wide large-scale model construction method based on reverse fact decomposition and fact perception fine-tuning according to claim 6, characterized in that, Step 7 includes: Step 71, based on the trained domain adapter weights Calculate the weight increment matrix The calculation formula is: ， in, For scaling hyperparameters; Step 72, adjust the weight increment matrix. Compared with the backbone network parameters of the original large language model Perform linear addition to merge the parameters to obtain the weights of the domain-wide large model after parameter merging, as follows: ； Step 73: Set the scaling factor based on the maximum absolute value of the weight tensor. and zero offset Mapping floating-point domain-specific large model weights to integer weights The quantification formula is: ， in, This is a truncation function. For nearest neighbor floor function, and These are the minimum and maximum allowed values ​​for the quantization bit width, respectively. Step 74, quantize the integer weights Deployment to large-scale models in the financial sector.

Images