Model training method, model, program product, and storage medium

Abstract

The application provides a model training method, the model is a pre-trained base large model for processing data of a specific industry, and the base large model reserves a timestamp embedding interface. The method comprises: labeling training data to form a structured training data set, the training data set comprising original data, labeling of the original data, entity labels, and timestamp metadata calibration; performing noise filtering operation based on entity conditional probability on the training data set by the base large model, and normalizing entities of the industry by entity alignment to obtain a cleaned data set; processing the cleaned data set by the base large model to output an initial text embedding; determining a time decay factor according to the timestamp metadata, and applying the determined time decay factor to the initial text embedding to obtain a time-adjusted embedding vector; triggering event enhancement item calculation by an indicator function, combining the event enhancement item with the time-adjusted embedding vector to generate a training embedding vector enhanced by time-sensitive features; and inputting the training embedding vector into the base large model to train the base large model with the labeling of the original data as a supervision label. The application also provides a corresponding large model, an electronic device, a program product, and the like.

Description

Technical Field

[0001] This application relates to big data processing, and more specifically, to the training of large models for processing data. Background Technology

[0002] With the digitalization and intelligentization of the financial industry, massive amounts of financial text data are constantly being generated, including corporate announcements, research reports, customer service Q&As, and investment community discussions. This data contains rich professional knowledge and market information, but it also has high timeliness, domain specificity, and semantic complexity.

[0003] Traditional natural language processing models are mostly trained on general corpora, lacking a deep understanding of financial terminology, entities, and event logic, making accurate parsing difficult. Especially in question-answering scenarios, user questions often involve multiple entities, multiple time dimensions, and strong contextual dependencies. Existing general models are prone to problems such as entity recognition errors, semantic breaks, delayed answers, or distorted confidence assessments.

[0004] Furthermore, financial data suffers from uneven distribution and a long-tail effect, with a high proportion of sparse samples and noisy data, which conventional data cleaning and augmentation methods cannot effectively address, thus limiting the quality of model training. With the development of large-scale industry models, how to construct specialized large-scale models for the financial field that can understand the relationships between financial entities, perceive temporal dynamics, and possess knowledge transfer and self-learning capabilities has become a critical technical problem that urgently needs to be solved. This invention is proposed against this backdrop, aiming to address the pain points of noise control, temporal modeling, knowledge distillation, and adaptive optimization in financial question-answering tasks by constructing a systematic and iteratively optimizable method for training large-scale financial industry models. Summary of the Invention

[0005] According to one aspect of this application, a model training method is provided to at least address one of the aforementioned problems. In this model training method, the model is a pre-trained base model for processing industry-specific data, and the base model reserves a timestamp embedding interface. The model training method may include: labeling training data to form a structured training dataset, the training dataset including original data, annotations of the original data, entity labels, and timestamp metadata calibration; performing noise filtering based on entity conditional probabilities on the training dataset using the base model, and dynamically standardizing industry entities through entity alignment to obtain a cleaned dataset; processing the cleaned dataset using the base model to output initial text embeddings; determining a time decay factor based on the timestamp metadata, and applying the determined time decay factor to the initial text embeddings to obtain a time-adjusted embedding vector; triggering event enhancement term calculation through an indicator function, combining the event enhancement term with the time-adjusted embedding vector to generate a time-sensitive feature-enhanced training embedding vector; and inputting the training embedding vector into the base model, using the annotations of the original data as supervisory labels, to train the base model.

[0006] According to another approach, a large model is provided, based on a base large model, for processing industry-specific data. The large model includes an input layer, a hidden layer, and an output layer, and is trained using the methods described herein.

[0007] An electronic device is also provided, comprising a memory and a processor. The memory stores the large model described herein, implemented through program instructions. The processor executes the program instructions to cause the large model to process industry-specific data.

[0008] A program product is also provided, the program product including instructions that, when executed, enable the implementation of the methods described herein.

[0009] A non-volatile storage medium is also provided, on which instructions are stored, which, when executed, enable the implementation of the methods described herein.

[0010] According to the method in this application, dynamic noise filtering combining entity conditional probability and contextual similarity, along with entity standardization based on embedding similarity, effectively filters out noise in financial data, eliminates entity ambiguity, and improves the structuring quality of financial data. Furthermore, the dynamic embedding method combining time decay factors and financial event enhancement terms improves the model's time-awareness, contributing to higher accuracy in responding to long-range dependencies and event-driven problems. Attached Figure Description

[0011] This application will be more fully understood by referring to the following detailed description of specific embodiments in conjunction with the accompanying drawings, wherein the same reference numerals in the drawings refer to the same elements, wherein: Figure 1 This is a flowchart of a model training method according to some embodiments of this application; Figure 2 yes Figure 1 An exemplary execution process of step S10; Figure 3 yes Figure 1 An exemplary execution process of step S12; Figure 4 This is a comparative illustration of the effectiveness of noise filtering and entity standardization methods based on entity conditional probability in financial question-and-answer data processing according to embodiments of this application. Figure 5 According to the embodiments of this application Figure 1 A schematic diagram of a specific example of step S16; Figure 6 This is a comparison illustration of time-sensitive embedding and static embedding; Figure 7 yes Figure 1 A schematic diagram of a specific example of step S22; Figure 8 This is a comparative illustration of the effects of stratified distillation; Figure 9 This illustrates the confidence calibration using the normalized exponential function (softmax) probability directly; Figure 10 This illustrates the confidence calibration based on the generation probability combined with entity standardization according to the example in this application; Figure 11 This is a schematic diagram of the basic large model according to an embodiment of this application; Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described implementation methods are only a part of the implementation methods of the technical solutions of this application, and not all of them.

[0013] Figure 1 This is a flowchart of a model training method according to some embodiments of this application. For example, the model is a pre-trained base model that at least reserves a timestamp embedding interface.

[0014] In step S10, the training data is labeled to form a structured training dataset, which includes the original data, annotations of the original data, entity labels, and timestamp metadata calibration. In step S12, noise filtering based on entity conditional probabilities is performed on the training dataset using the base model, and industry-specific entities are dynamically standardized through entity alignment to obtain a cleaned dataset. In step S14, the cleaned dataset is processed by the base model to output initial text embeddings. In step S16, a time decay factor is determined based on the timestamp metadata, and the determined time decay factor is applied to the initial text embeddings to obtain a time-adjusted embedding vector. In step S18, an event enhancement term is calculated using an indicator function, and the event enhancement term is combined with the time-adjusted embedding vector to generate a time-sensitive feature-enhanced training embedding vector. In step S20, the training embedding vector is input into the base model, using the annotations of the original data as supervisory labels, to train the base model.

[0015] The method implemented in this embodiment significantly improves the purity and structure of question-and-answer data for a specific industry (e.g., the financial industry) through noise filtering and entity standardization mechanisms based on entity conditional probability. By embedding enhancement methods using time decay factors and event triggering mechanisms, the model gains the ability to express time-sensitive features, dynamically monitor recent financial information and capture long-term event dependencies, and maintain stable question-and-answer accuracy and response consistency when facing market changes and timeliness issues.

[0016] The model training method according to embodiments of this application, and the large model, are models that can be used to process data from a specific industry. In the example of this application, the specific industry is the financial industry, but this is only an example and not a limitation. For example, by focusing the data in the training dataset on other industries, the large model can be trained to be applicable to other industries, such as the transportation industry. In the examples below, the financial industry will be used as an example for illustration.

[0017] In the example of this application, the large model uses a pre-trained Transformer architecture as its base model. More specifically, a Transformer-based Architecture for Finance can be used as the base model. This model is pre-trained on billions of financial corpora (including financial reports, research reports, and news). Its word embedding layer initializes financial entity features, and its hidden layers have the ability to capture long-distance dependencies in financial text. The word embedding layer is part of the input layer of the large Transformer model and is the first computational layer of the model. It receives the raw financial text data, i.e., it receives and processes the raw data from step S10. The hidden layers are the intermediate computational layers of the large Transformer model, located after the word embedding layer and before the output layer. The self-attention mechanism and feedforward network of the Transformer model are both implemented in the hidden layers. It should be noted that there may be multiple hidden layers, such as 6 or 12 layers, which can also be called multi-layer hidden layers. The model's initial parameters are inherited from general pre-trained weights in the general financial domain. Its multi-head attention mechanism can handle the ambiguity of financial terms (such as "leverage" referring to financial leverage or trading leverage in different contexts). By capturing different semantic dimensions through different heads of multi-head attention, it achieves ambiguity resolution by combining with financial pre-trained weights and provides a compatible vector space for subsequent time-sensitive embedding enhancement. The model's input layer reserves a timestamp embedding interface, and the output layer is adapted to financial entity classification heads to ensure seamless integration with subsequent fine-tuning modules (described below).

[0018] Figure 2 This is an exemplary process of step S10. For example... Figure 2 As shown, in step S100, raw data related to the industry from multiple sources is collected. In step S102, the collected raw data is labeled, including, for example, industry-specific entity standardization labeling, semantic relationship labeling, and question type classification labeling for the financial industry. In step S104, the collected raw data and the labels are combined to form a structured training dataset. For example, industry entity standardization labeling includes standardized naming, type labeling, and ambiguity resolution of industry entities in the raw data; semantic relationship labeling includes causal chain and long text dependency labeling of question-and-answer text in the raw data; and question type classification labeling classifies user questions in the raw data according to their context.

[0019] For example, the raw data collected mainly comes from professional financial platforms, historical customer service dialogue records, listed company announcements, and Q&A texts from financial communities. During the collection process, web crawlers can be used to selectively crawl multi-source heterogeneous financial texts to obtain user Q&A data containing relevant scenarios. For the acquired raw data, a three-level verification mechanism is used to complete the annotation process: financial experts formulate annotation standards, a pre-trained entity recognition model performs initial annotation, and the annotation team corrects ambiguities and adds labels. The pre-trained entity recognition module is a sub-module of the basic large model. Automated initial annotation is achieved through the initialization of financial entity features and entity classification head capabilities of this sub-module. More specifically, the pre-trained entity recognition module can reuse functional modules from the input, hidden, and output layers of the basic large model for annotation. Initial annotation can include standardized annotation of financial entities, such as unified standardized naming, type labeling, and ambiguity resolution, such as standardizing Apple Inc. as Apple Inc.; semantic relationship annotation, such as the causal chain in "the impact of interest rate hikes on government bond yields"; long text dependency annotation; and question type classification annotation. The feature rules, entity classification dimensions, and semantic relationship capture logic of these annotations are all based on the pre-trained weights, multi-head attention mechanism, and long-distance dependency capture capabilities of the basic large-scale model's financial corpus. The manual annotation process precisely corrects the aforementioned financial semantic logic. Finally, the collected raw data, various annotation results, and calibrated timestamp metadata are integrated and encapsulated to form a structured training dataset containing the original question, annotated answers, entity labels, and timestamp metadata.

[0020] Figure 3 This is an exemplary process of step S12. For example... Figure 3 As shown, in step S120, using a predefined industry entity library, the sum of the products of the frequency of the industry entity appearing in the question and the contextual similarity of the answer in each question-answer pair of the original data is calculated. In step S122, the conditional probability of the entity is determined based on the size of the industry entity library. In step S124, the conditional probability of the entity is compared with a noise threshold. In step S126, question-answer pairs with entity conditional probabilities lower than the noise threshold are dynamically filtered out. In step S128, based on the filtered question-answer pairs, the canonical entity with the highest similarity is selected by calculating the word embedding similarity between the token in the question and the entity in the industry entity library. In step S130, the entity in the question is replaced with the canonical entity.

[0021] Taking financial entity processing as an example, a predefined financial entity database is used to calculate the sum of the product of the frequency of an entity appearing in the question and the contextual similarity between the entity and the answer in each question-answer pair. This sum is then combined with the size of the entity database to obtain the entity conditional probability. The entity conditional probability is compared with a noise threshold, and question-answer pairs with entity conditional probabilities below the threshold are dynamically filtered out, retaining only those with high contextual consistency. This reduces noise interference and improves the purity of the training data. The retained question-answer pairs with high contextual consistency are represented by formula (1): In formula (1), D filt This indicates filtering the question-and-answer pair dataset to ensure an initial improvement in data quality; {·|·} are set construction symbols, representing a set of elements that satisfy a given condition, where the left side is the element and the right side is the condition. (q i ,а i ) represents the i-th original question-answer pair, where q i For the question text, а i The answer text is the basic unit of financial question-and-answer data; P ent (q i ,а i P represents the entity conditional probability, characterizing the contextual consistency of financial entities in a question-and-answer pair, and is used to filter noise. ent (q i ,а i It can be calculated using formula (2): Where E is a predefined financial entity database, such as containing stock codes, company names, etc., and e is a single financial entity index in the entity database E; fe(q i Let ) be the e-th entity in problem q i Frequency of occurrence in; Sim ctx (e,а i ) represents the e-th entity and the answer a. i The context cosine similarity is calculated based on pre-trained word vectors; |E| is the size of the entity database E, i.e., the total number of entities; τ noi The noise threshold is the critical value for determining noise; for example, it can be set to 0.6.

[0022] Based on the filtered question-answer pairs, the word embedding similarity between the token in the question and the entity in the financial entity database is calculated. The canonical entity with the highest similarity is selected, and the entity in the question is replaced with the canonical form to achieve entity alignment, thereby eliminating entity ambiguity, ensuring semantic consistency, and balancing data distribution. The cleaned question-answer pair dataset can be represented by formula (3): (3) Among them, D cleAlign represents the cleaning of question-answer pair datasets. ent (q i ,а i `()` is the entity alignment function, which returns standardized question-answer pairs. Here, `Align`... ent (q i ,а i This can be expressed as formula (4): Here, token refers to the token in the question-and-answer text, that is, a word or sub-word unit, which is the basic processing unit for entity alignment; For word embedding similarity function, Calculate token and entity e ent The cosine similarity between them is used to select the most similar canonical entities; E lib To distinguish it from the predefined financial entity library of E; ent For entity library E lib Individual financial entities; Represents taking the entity e that has the highest similarity. ent .

[0023] The effectiveness of the entity conditional probability-based noise filtering and entity standardization method in financial question-answering data processing can be verified by comparing three different data processing methods in this embodiment. Figure 4 This is a schematic illustration of the comparison. Three different data processing methods include a general text cleaning method that directly applies stop word removal and simple normalization techniques; a filtering method based on the Transformer-based Bidirectional Encoder Representations from Transformers (BERT) model that utilizes a pre-trained language model for context-aware noise filtering; and, according to embodiments of this application, a method that dynamically identifies and normalizes financial entities using entity conditional probabilities. Figure 4 As shown, data group 40 uses a general text cleaning method to process financial Q&A data, data group 42 uses BERT-based processing of financial Q&A data, and data group 44 uses the method described in this application to process financial Q&A data. In the experimental configuration, the financial Q&A data used in the three processing methods is the same, including raw text collected from financial professional platforms, customer service dialogue records, and other channels. The evaluation metrics cover three dimensions: data quality score, entity consistency, and noise reduction rate. Figure 4As can be seen, the method described in this application significantly outperforms the other two methods across all three evaluation metrics. Regarding data quality (A) scoring, the method in this application demonstrates a clear advantage, indicating that the entity conditional probability mechanism can effectively identify and retain question-answer pairs with high contextual consistency, avoiding the accidental deletion of important financial information. In terms of entity consistency (B) metric, the method in this application performs particularly well because the entity alignment operation replaces financial entities in the question with their canonical forms, effectively resolving ambiguities such as the word "Apple" between a company and a fruit. Regarding noise reduction rate (C), the method in this application also performs best, demonstrating that a dynamic filtering threshold based on the entity database can more accurately identify and remove low-quality data.

[0024] Reference Figure 1 The cleaned dataset obtained in step S12 is fed into the basic large model to be trained. The basic large model's word embedding layer, hidden layer, etc., perform basic semantic encoding on the dataset, thereby outputting a pure text semantic embedding vector without any temporal feature adjustment, i.e., the output initial text embedding.

[0025] Figure 5 This is a schematic diagram of a specific example of step S16 according to an embodiment of this application. For example... Figure 5 As shown, in step S160, a time decay factor is calculated based on the timestamp of the question-answer pair and the current time. In step S162, the time decay factor is applied to the initial text embedding by element-wise multiplication to obtain a time-adjusted embedding vector.

[0026] Taking financial entity processing as an example, based on the timestamp of the question-answer pair and the current time, a time decay factor is calculated. This time decay factor is then applied to the initial text embedding output of the pre-trained basic large model on the cleaned dataset through element-wise multiplication. This reduces the weight of older data, enhances the temporal relevance of new data, and makes the time-adjusted embedding vector more focused on recent information. This time-adjusted embedding vector can be represented by formula (5): (5) Among them, E temp Tran represents the time-adjusted embedding vector. base (D cle ) indicates that the pre-trained large model is trained on the cleaned dataset D. cle The initial embedded output is the raw language representation of the text; This represents element-wise multiplication, used to apply the time decay factor to each dimension of the initial embedding; T dec (t i () represents the time decay factor, used to dynamically adjust the embedded time weights to ensure that new data has higher importance. T dec (t iIt can be calculated according to formula (6), and the absolute value of the time interval can be taken to avoid the problem of future data weight amplification: (6) Where exp(·) represents the natural exponential function, used to simulate the time decay effect; λ time This is the decay rate hyperparameter, controlling the rate of decay over time; for example, it can be set to 0.05. i The timestamps for the question-and-answer pairs are obtained from the data metadata and represent the point in time when the data was generated; t cur The current time is used to calculate the time interval and represents a reference time point.

[0027] Furthermore, based on the financial event database and the question text, an indicator function is used to check whether the event appears in the question. Then, an event enhancement term is calculated, and by combining the event enhancement term with a time-adjusted embedding, long-range dependency and event triggering features are captured, generating a time-sensitive embedding vector to enhance event-related features. The time-enhanced embedding vector can be calculated using formula (7): (7) Among them, E time (q i ,а i ) represents the temporally augmented embedding vector; Eve boost (q i ,а i The term ) represents an event enhancement term, used to strengthen the financial event-related features in the problem. It is calculated as follows: ;e eve For the financial event database E eve For individual events, such as "interest rate adjustment," this symbol is used for distinction; E eve A predefined financial event library, containing common financial events and their pre-embedded representations; I eve (e eve, q i ) is an indicator function, which is triggered when event e occurs. eve In question q i The value is 1 when it appears in the event queue, and 0 otherwise; it is used to trigger event enhancement. eve (e eve ) for event e eve The pre-embedded vectors, obtained from the pre-trained base model, represent the semantic representation of the event; q i The question text is the object of event enhancement processing; a i The answer text is part of the context.

[0028] Figure 6This diagram illustrates a comparison between time-sensitive embedding and static embedding, verifying the time-sensitive embedding module's effect on improving the timeliness of financial question answering. The experiment simulates market event scenarios across different time spans (1-36 months), comparing static embedding (directly using pre-trained Transformer embedding) with the time decay factor combined with the event triggering mechanism of this embodiment. The horizontal axis represents the historical time span of the question; the vertical axis represents the question-answering accuracy, which is dimensionless. The line graph data comes from a test set evaluation of time-sensitive questions such as interest rate adjustments and financial report updates. The significant difference between the two lines 70 and 72 in the figure indicates that the static embedding curve 70 shows a stable downward trend over time, due to the decay of accuracy caused by the expiration of market events; the time-sensitive embedding line 72 remains high and stable within 36 months, with only slight fluctuations. Specifically, the difference between the two is small at 1 month, about 3%, but by 36 months, the accuracy according to this embodiment is still above 80%, while the static embedding drops to 55%. This advantage stems from the dynamic suppression of old data weights by the time decay factor, and the capture of long-term dependencies such as the impact of interest rate hikes on government bonds by the event enhancement term.

[0029] Based on some examples in this application, the model is fine-tuned after being trained on a basic large model. Returning to... Figure 1 The method may further include step S22, which involves fine-tuning the trained base model by using the time-sensitive feature-enhanced training embedding vector as input. Figure 7 This is a schematic diagram of a specific example of step S22. In step S220, the context gating module is integrated into the basic large model, and relevant features are dynamically weighted. In step S222, an entity-context hierarchical distillation mechanism is adopted to calculate the entity-level distillation loss and the context-level distillation loss separately and combine them with weights. In step S224, the data density is calculated based on the k-nearest neighbor algorithm, and the weights of the hierarchical distillation loss and the question-answering task loss are dynamically adjusted to ensure the task accuracy in sparse data regions. In step S226, the gradient update path is smoothed with a gradient variance regularization term to suppress gradient oscillations on the long-tail data of the industry. In step S228, the confidence of the question-answering results generated by the model is calibrated, low-confidence samples are screened, and synthetic data is generated based on the industry entity library and templates to update the training dataset. In step S230, multiple rounds of incremental fine-tuning-distillation joint training are started based on the updated dataset to verify the industry entity recognition accuracy and confidence calibration error as the convergence criterion, and the iteration terminates after the model stabilizes.

[0030] Taking financial entity processing as an example, financial question answering requires handling complex contexts, such as questions implying multi-hop reasoning. However, conventional fine-tuning methods directly fine-tune all parameters, ignoring irrelevant feature interference, which can easily lead to overfitting to sparse data. In the example of this application, by integrating the context gating module into the Transformer layer, relevant features are dynamically weighted to suppress noise and focus on important contexts.

[0031] As in step S220, the context gating weights are calculated. Specifically, the temporal augmentation embedding is linearly transformed using trainable weights and biases, and the sigmoid function is applied to generate context gating weights, thereby dynamically adjusting feature importance and generating gating weights for weighting features. The context gating weights can be calculated using formula (8): (8) Among them, G con represents the context-gated weights, with values ​​ranging from [0,1], used to dynamically weight the output features of the transform layer; sig(·) represents the Sigmoid activation function, which maps input values ​​to the [0,1] interval, used to generate smooth gating weights; W g b is a trainable weight matrix used to perform a linear transformation on the input embedding to learn the feature importance mapping; g E is a trainable bias vector used to adjust the bias terms in the gating computation. time Enhance the embedding vector for time.

[0032] Furthermore, gating weights are applied to generate enhanced hidden states. Specifically, the context gating weights are multiplied element-wise with the standard output of the model layer for temporal enhanced embedding during fine-tuning to obtain the gated enhanced hidden states. This suppresses irrelevant features, preserves important context, and generates enhanced hidden states that serve as inputs for subsequent distillation steps. The gated enhanced hidden states can be represented by formula (9): (9) Among them, H gate This indicates the gated enhancement hidden state, which serves as the input for subsequent distillation steps; This represents element-wise multiplication, used to apply gated weights to each feature dimension; Tran layer (E time ) indicates that during fine-tuning, the Tran layer responds to the input E time The standard output is the ungated, raw hidden state.

[0033] Financial question-answering tasks require distillation from a large model (teacher) to a small model (student). However, conventional distillation methods use a single soft-label loss, which cannot transfer the hierarchical knowledge specific to finance, such as entity-level and context-level knowledge. This application's embodiments employ an entity-context hierarchical distillation mechanism (such as step S222) to match the intermediate representations of the teacher and student, thereby transferring the hierarchical knowledge specific to finance.

[0034] Specifically, the entity-level distillation loss is first calculated. Entity-related features in the gated augmented hidden states of the teacher and student models are extracted through the entity projection layer, and their mean squared error loss is calculated to enhance the consistency between the teacher and student models at the entity feature level and transfer entity-level knowledge. The entity-level distillation loss can be expressed by formula (10): Among them, L ent This represents the entity-level distillation loss, used to measure the consistency between the teacher and student models at the entity feature level; N is the number of samples in the batch. Proj represents the square of the Euclidean norm, used to calculate characteristic differences; ent (·) represents the entity projection layer, which is a linear transformation layer used to extract entity-related features from the hidden state; This represents the gated augmented hidden state of the teacher model, derived from the output of the teacher model; This represents the gated augmented hidden state of the student model, derived from the output of the student model.

[0035] Subsequently, the context-level distillation loss is calculated. The Kullback-Leibler divergence of the attention weight matrices of the teacher and student models is calculated to capture the differences in contextual dependencies between the teacher and student models, thus conveying contextual knowledge. The context-level distillation loss can be expressed by formula (11): (11) Among them, L ctx represents the context-level distillation loss, used to measure the consistency of the teacher and student models at the attention distribution level; KL(· ||· ) represents the Kulback-Leibler divergence, used to calculate the difference between two probability distributions; Attn tea This represents the attention weight matrix of the teacher model, which contains the contextual attention patterns of the teacher model; Attn stu This represents the attention weight matrix of the student model, which contains the contextual attention patterns of the student model.

[0036] Then, the total loss of stratified distillation is calculated. By balancing the entity-level distillation loss and the context-level distillation loss through a weighted combination of hyperparameters, the total loss of stratified distillation is obtained. This balances the transfer of entity-level and context-level knowledge, ensuring the effectiveness of stratified knowledge distillation. The total loss of stratified distillation can be expressed by formula (13): (13) Among them, L dis α represents the total loss of stratified distillation; α represents the equilibrium hyperparameter, used to adjust the relative importance of entity-level loss and context-level loss, preferably set to 0.7.

[0037] Figure 8 This study compares the effects of stratified distillation to verify the effectiveness of the stratified knowledge distillation strategy in model compression. The experiment compares three types of models: a complete teacher model (100% parameter count), a single-distillation student model (25% parameter count), and a stratified distillation student model (25% parameter count). The left vertical axis represents the accuracy of financial question answering (bar chart), and the right vertical axis represents the relative size of the models (line 80, teacher model as 100% baseline). Data comes from the same test set and includes complex questions such as stock price queries and risk assessments. The bar chart reveals key phenomena: the accuracy of the single-distillation student model (middle bar) is significantly lower than that of the teacher model (left bar), proving that conventional distillation loses financial knowledge; while the accuracy of the stratified distillation student model (right bar) is close to that of the teacher model, with a difference of less than 5%. The line 80 clearly shows that the two types of student models have the same size (both 25%). The advantage stems from the entity-level distillation loss enhancing "stock code recognition," and the context-level loss conveying "multi-hop reasoning" capability. The column texture (no texture / diagonal texture / dot texture) distinguishes the model type, and the error bar shows that the stratified distillation has less fluctuation, proving that it achieves about 85% of the teacher model performance at 25% volume.

[0038] Gradients are prone to oscillations on long-tailed financial data during training. For example, Adam's conventional optimizer uses a fixed learning rate, which easily leads to gradient oscillations on long-tailed financial data, resulting in slow model convergence or getting stuck in local optima. The gradient variance regularization term (as in step S226) in this embodiment smooths the optimization path and suppresses gradient oscillations. Specifically, the gradient variance regularization term is first calculated. Based on mini-batch training data, the variance of the total loss gradient is calculated, and the gradient variance regularization term is obtained by scaling with a regularization coefficient, thereby suppressing gradient oscillations and smoothing the optimization path. This gradient variance regularization term is expressed by formula (14): Among them, R var λ represents the gradient variance regularization term, used to suppress gradient oscillations during weight updates; λvar represents the regularization coefficient, controlling the degree of influence of the variance term on weight updates, preferably set to 0.1. This represents the variance of the total loss gradient, used to quantify the degree of gradient variability. The gradient represents the total loss, which is the partial derivative of the loss function with respect to the model parameters.

[0039] Further, the regularized gradient update amount is calculated. The regularized gradient update amount is obtained by adding the original total loss gradient to the gradient variance regularization term, which stabilizes the weight update process and reduces gradient fluctuations. The regularized gradient update amount is expressed by formula (15): (15) in, This represents the amount of regularized gradient update.

[0040] Furthermore, the model weights are updated. Based on the base learning rate and the regularized gradient update amount, the model weights are updated according to the gradient descent rule. This achieves stable training optimization, avoids local optima, and the updated model weights are represented by formula (16): (16) Among them, W upd The updated model weights are used as the initial weights for the next round of training; W represents the current model weights, which include all trainable parameters; η represents the base learning rate, which controls the step size of the weight updates.

[0041] In financial question-answering generation tasks, the final output needs to generate answers and evaluate confidence levels. However, conventional techniques do not consider the semantic ambiguity of financial questions and answers and the uncertainty of market predictions. They directly use softmax probability as confidence level, which can easily lead to bias in confidence level calculation and fail to reflect the entity consistency and semantic reliability of the answer. Conventional methods lack consideration for the context alignment of financial entities and are prone to generating erroneous confidence levels that are too high or too low on fuzzy or long-tailed questions.

[0042] According to the example in this application, the confidence level of the generated answer is calibrated by combining the generation probability and the entity consistency score.

[0043] Specifically, based on the gated enhanced hidden state, the model decoder is used to generate the answer text, thereby ensuring the semantic coherence of the output and generating an answer consistent with the input question. The generated answer text is represented by formula (17): (17) Among them, a out Represents the generated answer text, ensuring semantic coherence in the output; Dec(·) represents the basic large model decoder function, used to generate sequential text based on the hidden states; H gate This indicates a gated enhanced hidden state, which includes contextual features of the focus.

[0044] The calibration confidence score is calculated by combining the generation probability and the entity consistency score. The entity consistency score is obtained by calculating the average embedding similarity between the entities in the answer and the entities in the question, thereby quantifying the reliability of the answer, solving the ambiguity problem in financial question answering, and improving the accuracy of confidence estimation. The calibration confidence score is expressed by formula (18): (18) Where conf represents the calibration confidence level, used to evaluate the reliability of the generated answer, with a value range of [0,1]; P gen The generation probability is represented by the softmax output of the decoder, and it characterizes the model's original confidence in the generated answer; C ent (а out ) represents the entity consistency score, calculated by formula (19): Among them, E out To get the answer a out The entity set extracted from; |E out | indicates the number of entities; e out For entity set E out Entities in C; ent A higher score indicates a better alignment between the answer and the entity semantics of the question, which is used to correct for deviations in the generation probability.

[0045] Figure 9 This illustrates the conventional confidence calibration using softmax probabilities directly. Figure 10 This illustrates the confidence calibration based on the generation probability combined with entity standardization according to the example in this application. Figure 9 and Figure 10 The horizontal axis represents prediction confidence, and the vertical axis represents actual accuracy. The color depth of the scatter plots indicates data density. The experiment configuration used 500 fuzzy questions, such as uncertain market predictions, and the distribution characteristics were presented through kernel density estimation. Figure 9 The data shows that the low confidence region (<0.6) scatter points are widely dispersed below 90 degrees of the diagonal, indicating that the confidence level overestimates the actual accuracy; the high confidence region (>0.8) has abnormally high points (actual accuracy is less than 0.7). Figure 10 In the figure, the scatter plots are significantly improved, closely adhering to the diagonal, especially forming a concentrated distribution band in the low confidence region. Color mapping shows that the present invention has the highest density in the 0.4-0.6 range (high-risk issues), proving its accurate identification of uncertainty. Experimental results show that the quantitative correction of entity semantic alignment by entity consistency score reduces the confidence error by about 40%, and the reference diagonal 92 in the figure indicates the ideal calibration state.

[0046] In the closed loop of financial question answering training, new data needs to be processed to iteratively improve the model. However, conventional data augmentation techniques rely on external data sources, are prone to introducing noise, and cannot effectively augment sparse areas of data, resulting in a decline in model performance on sparse or difficult problems.

[0047] The example in this application dynamically filters low-confidence question-answer pairs through a confidence feedback mechanism and generates a synthetic data augmentation dataset based on an entity library and templates. Specifically, based on the calibration confidence score of the generated answers, question-answer pairs with confidence scores below a preset threshold are dynamically selected, regions where the model is uncertain or makes incorrect predictions are identified, and filtering is performed on sparse data. The filtering criteria are as follows: Where conf represents the calibration confidence of the generated answer, used to evaluate the model's certainty about the answer, with a value range of [0,1]. This represents the confidence threshold, the critical value for judging samples with low confidence, and can be set to 0.7.

[0048] For the low-confidence samples selected, a template generation function based on the entity library is used. The original generated answer and question text are input, and the synthesized answer is output. The new question-answer pair is merged into the clean dataset, which can specifically supplement the samples in the sparse data area, while avoiding the introduction of external noise data and ensuring the purity of the dataset. The generated synthesized answer text is represented by formula (20): Among them, a syn This represents the generated synthetic answer text, used to supplement the original data; This represents a template generation function based on an entity library, which generates semantically consistent synthetic answers by matching financial entities in the question and answer using predefined templates; а out This represents the answer text generated by the model.

[0049] The augmented dataset is shown in Equation (21): Among them, D new Indicates augmented dataset; D cle This represents the output of the clean question-answer pair dataset, derived from noise filtering and entity standardization steps in financial question-answer data. This represents the set union operation, used to combine raw and composite data.

[0050] Based on the example in this application, a new round of fine-tuning-distillation joint training is initiated using a new dataset generated by the feedback loop. Specifically, the model is first incrementally trained using steps S220 to S228. After each iteration, the task accuracy and confidence calibration error are evaluated on an independent validation set. When the accuracy of financial entity identification on the validation set improves by less than, for example, 1% and the fluctuation range of the confidence calibration error is less than, for example, 0.5% over three consecutive iterations, the model is considered to have converged and the iteration is terminated. This threshold is an adjustable parameter that can be set according to industry task requirements. The best checkpoint is retained in each iteration, and the final output is a large industry model that performs stably in both sparse data regions and common scenarios.

[0051] Following the example in this application, the trained large model can be deployed to a financial business system via API services. During inference, the user-input question first passes through... Figure 1 The noise filtering and entity standardization process in step S12, followed by, for example... Figure 1 Steps S16 to S18 involve a series of processing steps, such as injecting real-time financial event features into the time-sensitive embedding module. The large model can focus on key information based on a context-gating mechanism, generating an initial answer and then performing confidence calibration. Specifically, it calculates the semantic consistency score between the answer entity and the question entity, and calibrates the confidence level by combining the generated probability output. Finally, it returns an answer with a confidence label. Low-confidence results trigger a manual review process. The manually reviewed, real-labeled samples are prioritized for inclusion in the data feedback loop system, and then supplemented with synthetic data for subsequent augmentation training.

[0052] Figure 11 This is a schematic diagram of the basic large-scale model according to an embodiment of this application. This basic large-scale model is used in a specific industry; here, the financial industry is taken as an example. Figure 11 As shown, the basic large model includes an input layer 110, a hidden layer 112, and an output layer 114.

[0053] The input layer 110 includes a word embedding layer 1101, a positional encoding layer 1102, and an interface reservation layer 1103. Unlike a general Transformer, the word embedding layer 1101 initializes its embedding vectors with financial entity features, directly identifying basic financial entities (such as stock codes and company names), reducing subsequent training costs. The positional encoding layer 1102 is used to capture the sequence relationships of long financial texts such as research reports and announcements. The interface reservation layer 1103 can be used as a timestamp embedding interface for "time-sensitive feature enhancement" in this large model, enabling the generation of time-adjusted embedding vectors.

[0054] Hidden layer 112 includes a multi-attention layer 1120, a feedforward network layer 1121, and layer normalization and residual connections 1122. The multi-attention layer 1120 handles the ambiguity of financial terms, for example, by capturing the contextual differences of "leverage" in "financial leverage" and "trading leverage" through different attention heads, achieving accurate semantic distinction. The feedforward network layer 1121 enhances the extraction of key features from financial texts through nonlinear transformations, such as extracting core financial elements like "interest rate adjustment" and "financial statement profit" from research reports. Layer normalization and residual connections 1122 are used to ensure the stability of large models during multi-layer training, while also enhancing the capture of dependencies in long financial texts (such as multi-paragraph financial reports and multi-round question-and-answer sessions), avoiding gradient vanishing over long sequences.

[0055] The output layer 114 is configured to adapt to the financial entity classification header 1140 for subsequent fine-tuning. When the fine-tuning is implemented as a software module, the module used for fine-tuning is connected to the output layer 114 through the financial entity classification header 1140.

[0056] According to the example in this application, the large model may also include a pre-training initialization module 109, which includes billions of pre-trained corpora related to the financial industry, including financial reports, research reports, news, and other textual data from the financial industry. During the pre-training process, the initial implantation of entity features from the financial industry is completed, enabling the model to naturally possess the basic semantic understanding ability of relevant texts in this industry, while providing a compatible vector space for subsequent multi-head attention mechanisms to handle the polysemy of industry terminology.

[0057] for Figure 11 The basic model shown can be trained using the model training method described above. In short, this basic model is a pre-trained transformer-based model for the financial domain, referred to as the Tran model. This Tran model is a model initialized and trained by the pre-training initialization module 109. In other words, this model is pre-trained on billions of financial corpora (including financial reports, research reports, and news), enabling the word embedding layer 1101 of the input layer 110 to include financial entity feature initialization and giving the hidden layer 112 the ability to capture long-distance dependencies in financial text. The model's parameters inherit from general financial pre-trained weights, allowing the multi-head attention mechanism of the hidden layer 112 to handle the ambiguity of financial terms (such as "leverage" referring to financial leverage or trading leverage in different contexts), and providing a compatible vector space for subsequent time-sensitive embedding enhancements. Furthermore, the input layer 110 of this model also reserves a timestamp embedding interface, and the output layer is adapted to a financial entity classification head, ensuring seamless integration with subsequent fine-tuning modules.

[0058] According to the model training method of this application, the basic large model can automatically label the training data. For example, the word embedding layer 1101 of the input layer initializes the financial entity features of the training data. Specifically, the original data (financial Q&A text, announcements, research reports) first enters the input layer, and the word embedding sub-layer directly identifies the basic financial entities (stock codes, company names, financial terms) in the text, converting the unstructured text into vectors with financial entity features. Entity ambiguity resolution, semantic relationship labeling (e.g., causal chains), long text dependency labeling, and question type classification labeling of the original data can be performed through the hidden layer. For example, the multi-head attention layer 1120 handles the ambiguity of financial terms and makes judgments for ambiguity resolution in the labeling process; the feedforward network sub-layer 1121 extracts the causal chain and long-distance dependency features of the financial text and performs feature extraction for semantic relationship labeling in the labeling process; layer normalization and residual connections 1122 ensure the stability of long text labeling and prevent feature loss due to excessive text length. The input layer 110 initializes features, the hidden layer 112 performs feature processing, and the final labeling result is output through the output layer. When manual annotation is required, data annotations can be corrected manually. This annotation process greatly improves annotation efficiency. During training, the function of processing annotations in the input layer 110, hidden layer 112, and output layer 114 is called the pre-trained entity recognition model, used to execute... Figure 1 Step S10 in the process.

[0059] Furthermore, the basic large model executes step S12. Specifically, the basic large model utilizes the financial entity feature processing capabilities of hidden layer 112 and the semantic understanding capabilities of pre-trained weights, combined with the externally dependent predefined financial entity library 150, to perform a full-process cleaning and standardization process on the structured training dataset generated in step S10, ultimately outputting the cleaned dataset. Step S14 can be implemented by combining the word embedding sublayer 1101 of input layer 110, the multi-head attention layer 1120 of hidden layer 112, and output layer 114. Step S16 can be executed by combining the timestamp embedding interface reserved in input layer 110, the feedforward network sublayer of hidden layer 112, and output layer 114. Step S18 can be executed by combining the timestamp embedding interface reserved in input layer 110, the multi-head attention layer of hidden layer 112, and output layer 114. Step S20 can be executed through output layer 114. In addition, the parameters of the large model can be fine-tuned after training. For example, the software-implemented fine-tuning module can be adapted to the large model through the financial entity classification head to perform the fine-tuning process.

[0060] The model training method according to the embodiments of this application employs an entity conditional probability calculation model combined with contextual similarity to achieve dynamic noise filtering. Furthermore, it embeds a similarity-driven entity alignment and standardization method, which can automatically identify and retain effective data with high contextual consistency in multi-source heterogeneous financial corpora. This method can accurately filter out invalid noise in financial question-and-answer data and effectively eliminate semantic ambiguity in financial entities, significantly improving the structured quality and semantic consistency of financial data, thus laying a high-quality data foundation for model training. In addition, the model training method according to the embodiments of this application specifically addresses the timeliness and long-range dependency issues of financial question-and-answer data. It combines a time decay factor with financial event enhancement terms to construct dynamic text embeddings. By weighting older data embeddings with weaker timeliness and injecting exclusive semantic enhancements into financial event-related features, the basic large model possesses accurate time awareness capabilities, enabling it to continuously adapt to the dynamic changes in the financial market, effectively capture the long-range dependencies and event-driven logic of financial texts, and ensure high accuracy in responding to time-sensitive financial question-and-answer questions.

[0061] Furthermore, the model training method according to the embodiments of this application, on the one hand, achieves dynamic weighting of financial text-related features by integrating a context gating module into the basic large model, effectively suppressing irrelevant feature noise, focusing on core context information, and avoiding overfitting problems in model training; on the other hand, it adopts a hierarchical knowledge distillation strategy at the entity level and context level, which can accurately transfer the hierarchical knowledge structure unique to the financial field between teacher-student models, taking into account both the lightweight compression of the model and the complete transfer of financial professional knowledge, and achieving dual optimization of model performance and efficiency.

[0062] The embodiments of this application also complete the accurate determination of the model output confidence through confidence calibration rules, complete the automatic identification of low confidence samples by combining the consistency characteristics of financial entities, and generate compliant synthetic data by geometric financial entity library and standardized templates to incrementally supplement and optimize the training dataset. This closed-loop training system does not require the introduction of external noisy data sources and can specifically solve the problems of sparsity and imbalance of financial question answering training data. Through iterative incremental training, the model performance is continuously optimized, so that the trained financial industry large model can maintain stable response performance in both common financial question answering scenarios and long-tail sparse financial question scenarios.

[0063] Figure 12 This is a structural diagram of an electronic device according to an embodiment of this application. For example... Figure 12 As shown, the electronic device includes a memory 1200 and a processor 1202. The memory stores instructions that enable the implementation of the above-described combination of... Figure 11The described large model is a model trained according to the methods described in the embodiments of this application. When the processor executes these instructions, the model is enabled to process industry-specific data. This industry is, for example, the financial industry. The electronic device can be a server, computer, tablet, smart terminal, etc.

[0064] The technical features in the various embodiments of this application can be combined with each other to form new implementation methods without departing from the spirit of this application and without conflicting with each other. Although specific embodiments of this application have been shown and described in detail to illustrate the principles of this application, it should be understood that this application can be implemented in other ways without departing from such principles.

Claims

1. A model training method, characterized in that, The model is a pre-trained, large-scale model used to process industry-specific data. This large-scale model reserves a timestamp embedding interface. The method includes: The training data is labeled to form a structured training dataset, which includes the original data, labels of the original data, entity labels, and timestamp metadata calibration. The training dataset is subjected to noise filtering based on entity conditional probability using the base model, and the entities in the industry are dynamically standardized by entity alignment to obtain a cleaned dataset. The cleaned dataset is processed using the aforementioned basic large model to output the initial text embedding; The time decay factor is determined based on the timestamp metadata, and the determined time decay factor is applied to the initial text embedding to obtain a time-adjusted embedding vector; The event enhancement term is calculated by triggering the event enhancement term calculation through the indicator function, and the event enhancement term is combined with the time-adjusted embedding vector to generate a training embedding vector with time-sensitive feature enhancement. The training embedding vector is input into the basic large model, and the annotation of the original data is used as the supervision label to train the basic large model.

2. The method according to claim 1, characterized in that, The method further includes fine-tuning the trained base model by using the time-sensitive feature-enhanced training embedding vector as input.

3. The method according to claim 2, characterized in that, Fine-tuning the trained base model includes: The context gating module is integrated into the basic large model, and relevant features are dynamically weighted. A hierarchical distillation mechanism based on entity and context is employed to calculate entity-level distillation loss and context-level distillation loss separately and combine them in a weighted manner. Data density is calculated based on the k-nearest neighbor algorithm, and the weights of hierarchical distillation loss and question-answering task loss are dynamically adjusted to ensure task accuracy in sparse data regions. The gradient update path is smoothed by using a gradient variance regularization term to suppress gradient oscillations on the long-tail data of the industry. The confidence level of the question-and-answer results generated by the model is calibrated, low-confidence samples are filtered out, and synthetic data is generated based on the industry entity library and templates of the industry to update the training dataset; Multiple rounds of incremental fine-tuning-distillation joint training are initiated based on the updated dataset to verify the accuracy of the industry entity recognition and the confidence calibration error as the convergence criteria. The iteration ends after the model is stable.

4. The method according to claim 1, characterized in that, The training data is labeled to form a structured training dataset, including: Collect raw data related to the industry from multiple sources; The collected raw data is labeled, including industry entity standardization labeling, semantic relationship labeling, and question type classification labeling; The collected raw data and the annotations are combined to form a structured training dataset.

5. The method according to claim 4, characterized in that, The industry entity standardization annotation includes standardized naming, type labeling, and ambiguity removal of industry entities in the original data; the semantic relationship annotation includes causal chain and long text dependency annotation of question and answer text in the original data; and the question type classification annotation classifies user questions in the original data according to scenarios.

6. The method according to claim 1, characterized in that, Performing entity conditional probability-based noise filtering on the training dataset using the aforementioned base model includes: By using a predefined industry entity library, the sum of the products of the frequency of the industry entity appearing in the question and the contextual similarity of the answer in each question-answer pair of the original data is calculated. The conditional probability of the entity is determined by combining the size of the industry entity database; Compare the entity conditional probability with a noise threshold; The question-answer pairs with entity conditional probabilities lower than the noise threshold are dynamically filtered out.

7. The method according to claim 6, characterized in that, Dynamically standardize industry entities through entity alignment, including: Based on the filtered question-answer pairs, the canonical entity with the highest similarity is selected by calculating the word embedding similarity between the token in the question and the entity in the industry entity library; Replace the entity in the problem with the specified entity.

8. The method according to claim 1, characterized in that, A time decay factor is determined based on the timestamp metadata, and the determined time decay factor is applied to the initial text embedding to obtain a time-adjusted embedding vector, including: Calculate the time decay factor based on the timestamp of the question-and-answer pair and the current time; The time decay factor is applied to the initial text embedding by element-wise multiplication to obtain a time-adjusted embedding vector.

9. The method according to claims 1 to 8, characterized in that, The industry mentioned is the financial industry.

10. A large model, characterized in that, The large model is based on a basic large model and is used to process data from a specific industry. The large model includes an input layer, a hidden layer, and an output layer. The large model is trained by the method described in any one of claims 1 to 8. The input layer includes a word embedding layer and reserves a timestamp embedding interface.

11. The large model according to claim 10, characterized in that, The specific industry mentioned is the financial industry.

12. An electronic device, characterized in that, The electronic device includes: A memory for storing the large model as described in claim 10, implemented via program instructions; A processor for executing the program instructions to cause the large model to process industry-specific data.

13. A program product, characterized in that, The program product includes instructions that, when executed, enable the implementation of the method according to any one of claims 1 to 9.

14. A non-volatile storage medium, characterized in that, Instructions are stored on the non-volatile storage medium, and when executed, the instructions are capable of implementing the method according to any one of claims 1 to 9.

Images