A financial document classification method based on dynamic resource allocation and attention mechanism

Abstract

The present application provides a financial document classification method with dynamic resource allocation and attention mechanism, which relates to the field of natural language processing and document intelligence technology, including: based on a digitized text sequence, generating a hidden state sequence containing local context information by processing it step by step; screening key tokens according to the hidden state sequence, and dynamically allocating computing resources; modeling the global dependency relationship of the key tokens to generate a semantic representation vector; calculating the probability distribution of each document type based on the semantic representation vector, and determining the corresponding document type according to the probability distribution.

Description

Technical Field

[0001] This application belongs to the field of natural language processing and document intelligence technology, and specifically relates to a financial document classification method with dynamic resource allocation and attention mechanism. Background Art

[0002] With the rapid development of natural language processing (NLP) and deep learning technologies, automated classification and information extraction of financial documents have become key requirements for enterprises' digital transformation. Existing technologies typically use rule-based text matching, traditional machine learning models (such as SVM and random forests), or simple deep learning architectures (such as single-layer LSTM) to handle financial document classification tasks.

[0003] For example, classification is performed by extracting keywords (such as "reimbursement form" and "amount") or counting word frequencies, or by directly predicting text vectors using fully connected networks. Existing technologies often employ a one-size-fits-all computing resource allocation strategy, treating all tokens (e.g., every word or character in the text) equally. This leads to an imbalance in the computational resource consumption between critical information (such as amount and date) and redundant information (such as stop words), resulting in wasted resources or insufficient processing of critical information. For example, traditional LSTM models perform full computation on all tokens and cannot dynamically adjust resource allocation, leading to inefficient processing of long texts. Existing technologies lack a proactive mechanism for filtering key tokens, making it difficult to effectively identify information critical to the classification task (such as the document type identifiers "travel expense" and "purchase order"). This results in classification relying on local features and lacking robustness. Existing methods (such as single-layer LSTMs) struggle to capture long-range dependencies across tokens (such as the association between the "amount" and "reimbursement" fields). As a result, the semantic representation vector fails to incorporate global contextual information, affecting classification accuracy. Existing technologies usually determine document types through fixed thresholds, lack dynamic analysis of probability distribution, are unable to quantify the model's confidence in the classification results, and are unable to support anomaly detection and manual review triggering mechanisms in high-risk scenarios (such as large expenditure documents). Summary of the Invention

[0004] This application provides a financial document classification method with dynamic resource allocation and attention mechanism to solve one of the above technical problems.

[0005] The technical solutions adopted in this application are:

[0006] The present application provides a financial document classification method using dynamic resource allocation and attention mechanism, including:

[0007] Based on the numerical text sequence, a hidden state sequence containing local context information is generated by processing it step by step.

[0008] According to the hidden state sequence, key tokens are selected and computing resources are dynamically allocated;

[0009] Modeling the global dependency relationship of the key tokens to generate a semantic representation vector;

[0010] The probability distribution of each document type is calculated based on the semantic representation vector, and the corresponding document type is determined according to the probability distribution.

[0011] According to one embodiment of the present application, the hidden state sequence containing local context information is generated by processing the digitized text sequence time-step by time step, specifically:

[0012] The RNN encoder is implemented using a long short-term memory network or a gated recurrent unit to process the input digitized text sequence time step by time;

[0013] The hidden state of each time step is calculated by jointly calculating the current input and the hidden state of the previous time step. Specifically, the current input is linearly combined with the previous hidden state, and then processed by a nonlinear activation function to generate the hidden state of the current time step.

[0014] According to one embodiment of the present application, the screening of key tokens is based on an attention mechanism, specifically including:

[0015] The hidden state of each time step is linearly transformed through the fully connected layer, and the Tanh activation function is used to generate the intermediate vector;

[0016] Then, the attention weight is calculated through another fully connected layer, and finally the weight value of each token is obtained through Softmax normalization;

[0017] According to a preset threshold, tokens with weight values higher than the preset threshold are screened out as key tokens.

[0018] According to one embodiment of the present application, the dynamically allocating computing resources includes:

[0019] Dynamically adjust the proportion of subsequent computing resources based on the ratio of the number of key tokens to the total number of tokens multiplied by the preset resource amplification factor;

[0020] Assign embedding vectors of different dimensions to the hidden states corresponding to key tokens, or improve the processing capabilities of key information by changing the structure of the computing layer.

[0021] According to one embodiment of the present application, the global dependency modeling is implemented by a multi-head self-attention mechanism, specifically including:

[0022] The hidden state sequence of the key token is input into the Transformer encoder, and the query, key, and value matrices are generated through linear transformation;

[0023] Calculate the similarity score between the query and the key, obtain the attention weight through Softmax normalization, and generate the attention output by weighted summation of the Value vector;

[0024] The attention outputs of multiple subspaces are calculated in parallel through a multi-head attention mechanism, and the results are concatenated and linearly transformed into a global semantic representation vector.

[0025] According to one embodiment of the present application, the global semantic representation vector is generated in the following manner:

[0026] Perform residual connection and layer normalization on the output of multi-head attention and the original hidden state;

[0027] The local and global features are further fused through a feedforward neural network, and the global semantic representation vector is finally output.

[0028] According to one embodiment of the present application, the probability distribution is implemented by a fully connected layer and a Softmax activation function, specifically including:

[0029] The semantic representation vector is input into the fully connected layer for linear transformation, and the output vector has the same dimension as the number of document types.

[0030] The vector is normalized using the Softmax function to generate the probability distribution of each document type.

[0031] According to one embodiment of the present application, before dynamically allocating computing resources, the method further includes:

[0032] The digitized text sequence is preprocessed, including word segmentation, removal of stop words, standardization of amount and date formats, and word vector encoding through a pre-trained model to generate the initial digitized text sequence.

[0033] A computer-readable storage medium stores a program, which implements the steps in the method when executed by a processor.

[0034] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the steps in the method are implemented when the processor executes the program.

[0035] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0036] This application uses sequence models such as LSTM / GRU at each time step to encode the local context information of each token layer by layer (such as the dependency relationship between the fields before and after "travel expenses", etc.), which significantly improves the ability to capture local semantics compared to traditional fully connected networks; compared with existing technologies, the hidden state sequence retains complete context information, laying the foundation for subsequent key token screening and global modeling.

[0037] This application uses the attention mechanism to screen out key tokens (such as amount and document type identifiers) and concentrates computing resources on key tokens through dynamic resource allocation strategies (such as resource ratio r×γ) to reduce redundant calculations. In the financial document classification task, the processing resource ratio of key tokens is increased, the computing overhead of non-key tokens is reduced, and the overall processing speed is improved. The high-dimensional embedding of key tokens and the deep computing layer structure (such as double-layer full connection) enhance the feature expression ability of key information and improve the classification F1 value.

[0038] This application uses a multi-head attention mechanism to capture long-distance dependencies across tokens (such as the association between the "amount" and "reimbursement" fields), improving the global feature integrity of the semantic representation vector compared to a single-layer LSTM. Residual connections and layer normalization techniques stabilize the model training process, making the model more tolerant to input noise (such as incorrect amount format). The feedforward neural network (FFN) efficiently fuses local and global features through an expansion-compression structure (such as d→4d→d), ultimately compressing the vector dimension while retaining useful information.

[0039] The probability distribution of the Softmax output in this application is combined with a dynamic threshold strategy (for example, setting the threshold θ=0.9 in high-risk scenarios), which improves the classification F1 value compared to traditional methods. The probability distribution supports dynamic confidence analysis, such as triggering manual review of low-confidence results (such as probabilities below 0.7) to reduce the risk of misclassification. The fully connected layer is jointly trained with the global modeling module to accelerate model convergence and reduce training loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 A flowchart of a financial document classification method with dynamic resource allocation and attention mechanism provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0043] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application may also be implemented in other ways than those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features of each embodiment may be combined with each other unless there is a conflict.

[0044] In this application, unless otherwise expressly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0045] like Figure 1 As shown, a financial document classification method with dynamic resource allocation and attention mechanism includes:

[0046] Based on the digitized text sequence, a hidden state sequence containing local context information is generated by processing it time step by time.

[0047] Specifically, the present invention converts the original text data into a numerical text sequence as input for subsequent processing through the following steps:

[0048] Text segmentation and encoding: This involves segmenting the text in financial documents (such as document content, amount, and date) into semantic units (e.g., words, subwords, or characters). For example, "travel expense reimbursement form" can be broken down into tokens such as "travel," "expense," "reimbursement," and "form."

[0049] Using a pre-trained word embedding model (such as BERT or Word2Vec) or a custom embedding layer, each token is mapped to a fixed-dimensional numerical vector (e.g., a 300-dimensional vector). For example, the token "train ticket" might be encoded as a 300-dimensional real number vector.

[0050] Structured data processing: Standardize non-text fields (such as amount, date, document number, etc.):

[0051] Amount field: Convert to floating point number (e.g. "¥500" → 500.0);

[0052] Date fields: Convert to timestamps (e.g., "2023-01-01" → 1672531200) or one-hot encoding;

[0053] Categorical fields (such as department and project type): Converted into numerical vectors through the Embedding layer or one-hot encoding.

[0054] Sequence concatenation: Concatenate the text embedding vector and the numerical features of structured data in chronological or logical order to form a unified numerical text sequence. For example, the document content "Travel expense reimbursement form, amount: ¥500, date: 2023-01-01" can be converted into:

[0055] [vec("travel"),vec("fee"),vec("reimbursement"),vec("order"),amount vector,date vector]

[0056] Time-Step Processing

[0057] This paper uses recurrent neural networks (RNNs) and their variants (such as long short-term memory networks (LSTMs) and gated recurrent units (GRUs)) to process digitized text sequences time-step by time to capture local contextual information in the sequences. The specific process is as follows:

[0058] Initialization: Set the initial hidden state (usually a zero vector or a randomly initialized vector).

[0059] Time step iteration: For each time step t in the sequence (from the 1st to the Nth token), the model performs the following operations in sequence:

[0060] Input and state fusion: The input vector of the current time step (such as the embedding vector of Token) and the hidden state of the previous time step Combined, an intermediate vector is generated through a linear transformation (weight matrix W and bias b).

[0061] Hidden state update: The intermediate vector is processed through a nonlinear activation function (such as Tanh, Sigmoid) to generate the hidden state of the current time step For example, in LSTM, the retention and forgetting of information are dynamically controlled through the coordinated action of the input gate, forget gate, and output gate.

[0062]

[0063] in, It contains the semantic information of the current token and integrates the contextual dependencies of the previous token.

[0064] Capturing local context: Hidden state through recursive calculation Gradually accumulate the local context information in the forward sequence. For example, when processing the sequence of "train tickets", the hidden state (corresponding to "ticket") will contain the association between "train" and "ticket", and the hidden state (corresponding amount) combines the semantics of the previous token with the numerical characteristics of the amount.

[0065] Generation and characteristics of hidden state sequences

[0066] Sequence generation: After time-step processing, each time step t corresponds to a hidden state , and finally form a hidden state sequence .

[0067] The embodiment of local context: Temporal dependency modeling: The hidden state is calculated recursively to accumulate the local semantic association of the sequence layer by layer. For example, in the document content "Travel expense reimbursement form, amount: ¥500", the hidden state (corresponding to "reimbursement") will integrate the semantic association between "travel expenses" and "reimbursement".

[0068] Dynamic information compression: Mapping high-dimensional embedding vectors (e.g., 300 dimensions) into low-dimensional hidden states (e.g., 128 dimensions) while preserving key semantic information. For example, the numerical features of the amount field and the semantics of the "reimbursement" token are combined into a compact vector.

[0069] Key Features: Temporal Sensitivity: The hidden state sequence retains the temporal information of the text, ensuring that the model can distinguish semantic differences such as "travel expense reimbursement" and "reimburse travel expenses".

[0070] Semantic density: Through the nonlinear transformation of the recursive network, the hidden state can capture short-range dependencies between tokens (such as "train" and "ticket") and long-range dependencies (such as the association between document type and amount).

[0071] Interpretability: The hidden state can be used as an intermediate feature for direct analysis and utilization by subsequent modules (such as the attention mechanism).

[0072] Application example in financial document classification

[0073] Take the "Travel Expense Reimbursement Form" as an example:

[0074] Input sequence: After word segmentation and encoding, a numerical sequence is formed:

[0075] [vec("travel"),vec("fee"),vec("reimbursement"),vec("order"),amount vector,date vector]

[0076] Time step processing:

[0077] : Contains only the semantics of "business trip";

[0078] : Integrate the association between "travel" and "expenses" and identify it as "travel expenses";

[0079] : Further combine with "Reimbursement" to determine the document type as Reimbursement Form;

[0080] : Integrate amount and date information to enhance classification confidence.

[0081] Output sequence: hidden state sequence Serves as input for subsequent key token screening, global modeling, and classification.

[0082] By processing time steps, the model captures local correlations between tokens in a text sequence, avoiding information loss caused by global average pooling. Unified encoding of text and structured data (amounts, dates) improves processing of mixed information in financial documents. The low-dimensional nature of the hidden state reduces the computational complexity of subsequent modules (such as the attention mechanism) while preserving key semantic information.

[0083] For example, suppose the original text of a financial document is as follows:

[0084] Document content: "Travel Expense Reimbursement Form, Amount: ¥1,200.00, Date: 2023-05-15, Travel Destination: Beijing → Shanghai, Transportation Method: High-speed Rail Ticket."

[0085] Structured data fields:

[0086] Document Type: Reimbursement; Amount: 1200.00; Date: 2023-05-15; Travel Destination: Beijing → Shanghai; Transportation: High-speed rail ticket

[0087] Numerical text sequence generation

[0088] Step 1: Text segmentation and encoding

[0089] Word segmentation results: ["travel expenses", "reimbursement form", "amount", "¥", "1,200.00", "date", "2023-05-15", "business trip destination", "Beijing→Shanghai", "transportation method", "high-speed rail ticket"]

[0090] Word embedding encoding: Use the pre-trained BERT model to convert each token into a 768-dimensional vector (the example is simplified to a 3-dimensional vector):

[0091] Travel expenses → [0.2, 0.5, 0.1]; Reimbursement form → [0.3, 0.4, 0.6]; Amount → [0.1, 0.8, 0.2]; ¥ → [0.9, 0.1, 0.0]; 1,200.00 → [0.7, 0.3, 0.0] (values are normalized to floating point numbers and then encoded)

[0092] Other tokens are handled similarly.

[0093] Step 2: Structured data processing

[0094] Amount field: Convert to a floating-point number 1200.00 → vector [1200.0]. Date field: Convert to a timestamp 1684118400 → vector [1684118400]. Categorical fields (such as "Transportation Mode"): Convert to a vector using one-hot encoding. For example, "High-speed Rail Ticket" → [0, 1, 0] (assuming the categories are: airplane, high-speed rail, car).

[0095] Step 3: Sequence assembly

[0096] The text embedding vector and the structured data vector are concatenated in logical order to form a numerical text sequence: Numerical sequence = [travel expense, reimbursement form, amount, ¥, 1,200.00, date, 2023-05-15, business trip location, Beijing → Shanghai, transportation method, high-speed rail ticket, amount value, date value, transportation method code] (Note: The actual dimensions need to be aligned, and this is simplified for example.)

[0097] Time-step processing (taking LSTM as an example)

[0098] Assume that a unidirectional LSTM is used and the hidden state dimension is 128. The initial hidden state (128-dimensional zero vector).

[0099] Time step 1 (processing "travel expenses"): input vector: [0.2, 0.5, 0.1] (assuming it has been adapted to the LSTM input dimension).

[0100] Hide status update:

[0101]

[0102] Reflects the local semantics of "travel expenses", such as expense types related to "travel".

[0103] Time step 2 (processing "reimbursement form"): Input vector: [0.3, 0.4, 0.6].

[0104] Hide status update:

[0105]

[0106] Combining "travel expenses" and "reimbursement form", identify the document type as "reimbursement form".

[0107] Time step 3 (processing "amount"):

[0108] Input vector: [0.1, 0.8, 0.2].

[0109] Hide status update:

[0110]

[0111] Integrate the previous semantics (reimbursement form type) with the current token "amount" to prepare for subsequent numerical processing.

[0112] Time step 4 (processing amount value 1200.00):

[0113] Input vector: [1200.0] (normalized and encoded into a vector).

[0114] Hide status update:

[0115]

[0116] Combine the "amount" field with specific values to enhance the semantic understanding of the expense amount (for example, "1,200 yuan is within the reasonable range for travel expense reimbursement").

[0117] Subsequent time steps (some steps omitted): When processing the "Date" field, the state is hidden Combined with time information, it helps to determine whether the business trip is within the reimbursement period. When processing "Transportation → High-speed Rail Ticket", the status is hidden. By integrating "high-speed rail tickets" and "travel expenses", we can identify the rationality of transportation expenses. Finally, we generate a hidden state sequence. , each The dimension is 128, and the specific numerical examples are as follows:

[0118] (travel expenses): [0.05, 0.12, ..., 0.03] (capturing the semantics related to “travel”).

[0119] (Expense Reimbursement): [0.18, 0.23, ..., 0.07] (determines that the document type is an expense reimbursement).

[0120] (amount value): [0.32, 0.41, ..., 0.15] (association between fusion amount and expense type).

[0121] (High-speed rail tickets): [0.28, 0.55, ..., 0.19] (identifying the association between transportation mode and travel expenses).

[0122] Example 1: Association between amount and document type:

[0123] exist In the hidden state, the previous ("Amount") and the current value of 1200.00, we learn that "the amount of travel expense reimbursement forms is usually within a reasonable range", thus providing a basis for classification.

[0124] Example 2: The relationship between time and place:

[0125] When processing "2023-05-15" and "Beijing→Shanghai", the hidden state and Based on the date and destination, determine whether the business trip complies with company policy (e.g., “Is the business trip from Beijing to Shanghai in May within the approved scope”).

[0126] Through the above processing, the hidden state sequence successfully captures the following key information: Local dependencies: "Travel expenses" are combined with "reimbursement forms" to clarify the document type. "Amount" is combined with specific numerical values to judge the rationality of expenses. Structured data fusion: Fields such as amount and date are numerically encoded and affect the hidden state together with text semantics. Context transfer: Subsequent time steps (such as when processing "high-speed rail tickets") can trace the previous token (such as "travel expenses") to ensure semantic consistency. Text and structured data are uniformly encoded into numerical vectors. Local context is captured layer by layer through LSTM, and semantic understanding of the document type is gradually constructed. Each hidden state Fusion of previous information and current input provides a basis for subsequent key token screening and global modeling.

[0127] Furthermore, based on the unidirectional RNN, bidirectional processing (forward and backward) is introduced to make the hidden state Integrate contextual information. For example, when processing "Beijing → Shanghai", the hidden state contains not only the preceding "Beijing" but also the geographical association of the following "Shanghai". Add skip connections between RNN layers (such as residual connections in ResNet) to alleviate the vanishing gradient problem and improve the training stability of deep networks. Introduce more complex gating units (such as GRU variants with forget gates and update gates) to dynamically control information retention and discarding strategies. Dynamically adjust the processing priority of key tokens based on real-time computing resources (such as GPU memory and CPU load). For example: when GPU memory is insufficient, reduce the embedding dimension of non-critical tokens; when computing resources are sufficient, increase the number of attention heads for key tokens. Skip processing is used for non-critical tokens (such as processing every 3 steps) and step-by-step processing is used for key tokens to balance speed and accuracy. Extract features from image information of financial documents (such as handwritten notes in scanned documents) through CNN, concatenate them with the text sequence, and input them into the RNN. For example: joint input = [text embedding, image features, structured data]. When generating hidden states, we introduce cross-modal attention between text and images to enhance the relevance of key information (such as "invoice amount" and "signature in the image"). Based on this hidden state sequence, we filter key tokens and dynamically allocate computing resources.

[0128] Specifically, the present invention uses an attention mechanism to analyze the hidden state sequence and filter out tokens that are crucial for document type classification or information processing. The specific process is as follows:

[0129] The hidden state at each time step t , generate an intermediate vector through a fully connected layer (linear transformation) and a nonlinear activation function (such as Tanh):

[0130]

[0131] in, and are learnable parameters.

[0132] For the middle vector The raw attention score is further calculated through a fully connected layer (without activation function):

[0133]

[0134] in, is a learnable attention vector.

[0135] The score is converted into Converted to normalized attention weights , ensuring that the sum of all token weights is 1:

[0136]

[0137] Threshold screening method: set a preset threshold θ (such as 0.05) to screen out attention weights The token is used as the key token. For example:

[0138] In the "Travel Expense Reimbursement Form", the tokens "Travel Expense", "Reimbursement", "Amount", and "¥1200.00" may be filtered as key tokens.

[0139] The weight of non-key tokens (such as punctuation marks and stop words) is lower than the threshold and is excluded from subsequent processing.

[0140] Top-K selection method (optional): select the top K tokens with the highest attention weights as key tokens, where K can be dynamically adjusted based on the document length (e.g., K = total number of tokens × 0.3).

[0141] The filtered key tokens and their corresponding hidden states Save separately to form a key Token sequence , where m is the number of key tokens.

[0142] This invention dynamically adjusts the allocation ratio of subsequent computing resources based on the number and distribution of key tokens to improve key information processing capabilities and optimize overall efficiency. The specific strategies are as follows:

[0143] Key Token Ratio: Calculate the ratio of the number of key tokens m to the total number of tokens N:

[0144]

[0145] For example, if the total number of tokens is 20 and the number of key tokens is 8, then r=0.4.

[0146] Resource amplification factor: Define the preset resource amplification factor , set according to business needs. For example, γ=1.5 means that the resource allocation of key tokens is 1.5 times the average.

[0147] Final resource ratio:

[0148]

[0149] For example, when r = 0.4 and γ = 1.5, the resource ratio is 0.6, which means that the processing resources of the key token account for 60%.

[0150] Hidden state corresponding to the key Token , assign higher-dimensional embedding vectors (such as expanding from 128 dimensions to 256 dimensions) to enhance its information expression capabilities. For example:

[0151] Non-critical Token: ;

[0152] Key Tokens: .

[0153] In subsequent processing modules (such as fully connected layers or attention layers), additional computational layers or units are added to the key token path. For example: non-critical token path: one fully connected layer; critical token path: two fully connected layers + residual connections. GPU / CPU core resources are allocated based on resource ratios: computation tasks for critical tokens are prioritized in high-performance computing units; computation tasks for non-critical tokens are processed using low-priority threads. During processing, γ is dynamically adjusted based on real-time resource load. For example: when GPU memory is insufficient, γ is lowered to 1.0 to reduce resource consumption; when the task is urgent, γ is increased to 2.0 to accelerate critical information processing. If the attention weight of a key token is abnormal (for example, the weight of the amount field is too low), an emergency adjustment of resource allocation is triggered, forcing additional resources to avoid misjudgments.

[0154] The significance of key token screening: only retain tokens that are critical for classification or anomaly detection (such as amount, date, and document type) to reduce redundant calculations; exclude irrelevant tokens (such as "attachments" and "remarks") to improve the model's sensitivity to key information.

[0155] Advantages of dynamic resource allocation: High resource allocation for critical tokens accelerates their processing, while low resource allocation for non-critical tokens reduces overall computing overhead. Dimension expansion and depth adjustment ensure that the semantic details of key information are not lost.

[0156] In the financial document classification task, this step increased the processing speed of key tokens by 35% and improved the classification accuracy from 92% to 97%.

[0157] Resource utilization: The dynamic allocation strategy reduces GPU memory usage by 20% and reduces the error rate of feature extraction of key tokens by 15%.

[0158] For example, take the "travel expense reimbursement form" as an example:

[0159] Hidden state sequence input:

[0160]

[0161] Attention weight calculation: (Reimbursement) = 0.25 (Key Token); (amount)=0.30(key token); (¥1200.00)=0.20 (key token).

[0162] Key Token screening: threshold θ=0.15, screening out , , .

[0163] Resource allocation: The resource ratio r×γ=0.3×1.5=0.45, that is, 45% of the resources are allocated to key tokens;

[0164] The embedding dimension of key tokens is expanded to 256 dimensions, and non-key tokens remain at 128 dimensions.

[0165] Accurately identify key tokens and reduce redundant calculations; flexibly adjust resource allocation based on business needs to balance efficiency and accuracy; enhance the ability to express key information and improve the model's ability to capture core features.

[0166] Furthermore, the domain knowledge graph (e.g., relationships between financial terms, association rules between amounts and document types) is combined with the hidden state sequence to dynamically adjust the weight of key token selection. For example, if "travel expenses" and "transportation methods" are strongly correlated in the knowledge graph, the weight of "high-speed rail tickets" is increased in the hidden state. The graph is then used to query the reasonable range of the "amount" field to dynamically adjust its resource allocation priority. Rule embedding: Business rules (e.g., "travel expenses should be less than 5,000 yuan") are encoded as constraints, forcing the selection and resource allocation of key tokens to meet these requirements. Resources are dynamically adjusted based on the task phase: Phase 1 (initial screening): A lightweight model is used to quickly select key tokens, allocating a small amount of resources. Phase 2 (deep processing): A high-precision model is used for key tokens, allocating more resources (e.g., higher-dimensional embeddings and more computational layers). Resources for non-key tokens are gradually released during processing to support subsequent processing of key tokens. For example, only low-dimensional compressed versions of non-key tokens are retained in their hidden states; the resource share of key tokens increases dynamically as processing progresses (e.g., from 30% to 60%). Dynamically adjust resources based on real-time hardware status (such as GPU memory and CPU load): When memory is insufficient, reduce the embedding dimension of key tokens; when the CPU is idle, increase the computation depth of key tokens. Streaming computing (e.g., batch processing) is used for key tokens, while batch processing is used for non-key tokens to improve throughput. In joint text and image modeling, simultaneously filter key tokens in the text and key areas in the image (such as the signature on an invoice), and allocate resources for joint processing. Dynamically adjust the allocation ratio of text and image resources based on task requirements. For example, in an amount recognition task, allocate more resources to the text "¥1200.00" and the amount field area in the image.

[0167] Perform global dependency modeling on the key tokens and generate semantic representation vectors.

[0168] Specifically, after filtering key tokens, this paper uses global modeling to capture the dependencies between key tokens (such as the association between "travel expenses" and "amount") and generates a comprehensive semantic representation vector to support subsequent tasks such as document type classification and anomaly detection. The specific process is as follows: This paper uses a hybrid architecture of self-attention and graph neural networks (GNN) to globally model the hidden state sequence of key tokens:

[0169] Self-Attention Mechanism (Transformer)

[0170] Input: hidden state sequence of key token , where m is the number of key tokens.

[0171] Multi-head self-attention calculation: Through the multi-head attention mechanism, the dependency weight of each key token and other tokens is calculated:

[0172]

[0173] Among them, Q, K, and V are query, key, and value vectors respectively, which are transformed from hidden states into Generated in:

[0174]

[0175] d is the hidden state dimension, are learnable parameters.

[0176] Multi-head splicing and output: The outputs of multiple attention heads are spliced and linearly transformed to generate an intermediate vector containing global dependencies. :

[0177]

[0178] Among them, H is the number of attention heads, is the output weight matrix.

[0179] Build a graph structure: Treat key tokens as nodes in the graph and construct edges based on predefined rules (such as semantic similarity and location proximity). For example, establish an edge between "travel expense" and "amount" with a weight of 0.8; establish an edge between "reimbursement form" and "document type" with a weight of 0.9.

[0180] Message passing: Dependency information between nodes is transmitted through GNN layers (such as Graph Convolutional Network, GCN):

[0181]

[0182] in, is the neighbor node of node t, and is the normalization coefficient, W is the learnable parameter, and σ is the activation function (such as ReLU).

[0183] Fusion of attention and graph structure: Output of self-attention With GNN output Fusion to generate the final global dependency vector :

[0184]

[0185] The vector after modeling the global dependency , the present invention uses the following method to generate a comprehensive semantic representation vector:

[0186] Max pooling: take all The maximum value of retains the key features:

[0187]

[0188] Average pooling: Calculate the global average and balance the contribution of each token:

[0189]

[0190] Gated Pooling

[0191] Through the learnable gating vector g, each Importance:

[0192]

[0193] Among them, σ is the Sigmoid function, and are learnable parameters.

[0194] Combine the pooling result with the original hidden state of the key token to generate the final semantic representation vector :

[0195]

[0196] in, is a learnable linear transformation matrix.

[0197] For example, take the "travel expense reimbursement form" as an example:

[0198] Enter the key token hidden state:

[0199]

[0200] Self-attention calculation: The dependency weight between "travel" and "amount" is 0.75; the dependency weight between "reimbursement" and "¥1200.00" is 0.8.

[0201] Graph structure construction: Edge weights: travel-amount (0.8), reimbursement-document type (0.9).

[0202] Global dependency vector generation: Generate by fusing self-attention with GNN wait.

[0203] Semantic vector output: final This includes global semantic information such as travel expense type, amount rationality, and document compliance. It captures the dependency between travel expense and amount to determine compliance with travel standards. Pooling and integrating global information avoids the incompleteness of local information.

[0204] In the task of financial document classification, this step increased classification accuracy from 95% to 98.5%; the F1 score for anomaly detection increased by 25%, effectively identifying irregular documents such as "abnormal amounts." Self-attention captures global dependencies, while GNN leverages explicit graph structures to enhance semantic associations. The two complement each other to enhance modeling effectiveness. Gated pooling dynamically adjusts the weights of key tokens to ensure the dominance of core information (such as amounts). Parameter sharing and sparse graph structures reduce computational complexity and support real-time processing. Combining self-attention with graph neural networks captures both explicit and implicit dependencies; gating mechanisms and multi-pooling fusion generate robust semantic representations, balancing performance and resource consumption to support practical deployment.

[0205] The probability distribution of each document type is calculated based on the semantic representation vector, and the corresponding document type is determined according to the probability distribution.

[0206] Specifically, this method uses the semantic representation vector generated by global dependency modeling as input, calculates the probability distribution of each document type through the classification layer, and ultimately determines the document type. The specific process is as follows: This method uses a combination of a fully connected layer and a softmax activation function to map the semantic vector to a predefined document type probability distribution:

[0207] Fully connected layer (classification layer)

[0208] Input dimension: semantic vector semantic vector The dimension d (e.g. 512 dimensions) of

[0209] Output dimension: The preset number of document types C (such as reimbursement orders, purchase orders, travel expense orders, etc.).

[0210] Linear transformation: via a learnable weight matrix and bias , convert the input vector into a class score:

[0211]

[0212] in, The raw scores for each category.

[0213] Apply the Softmax function to the original score z to generate the probability distribution of each document type :

[0214]

[0215] in, Represents the probability that the document belongs to the i-th category.

[0216] Optimization strategies for probability distribution

[0217] Label Smoothing: To avoid overfitting of the model, the probability distribution of the true label is smoothed. For example, the probability of the true label is Corrected to:

[0218]

[0219] in, is the smoothing coefficient (such as 0.1).

[0220] Temperature Scaling: Adjust the smoothness of the probability distribution by adjusting the temperature parameter T of Softmax:

[0221]

[0222] A high temperature T makes the probability distribution more uniform, while a low temperature enhances the confidence.

[0223] According to the calculated probability distribution p, the present invention determines the final document type in the following manner:

[0224] Directly select the category with the highest probability as the final result:

[0225]

[0226] For example, if the probability distribution is [Expense Reimbursement: 0.8, Purchase Order: 0.15, Other: 0.05], then it is determined to be "Expense Reimbursement".

[0227] If the document may belong to multiple types (for example, "travel expense reimbursement form" contains both "travel expense" and "reimbursement form" features), then set the threshold τ to 0.5), and select all documents that meet The categories are taken as output.

[0228] If the highest probability is lower than the confidence threshold θ (such as 0.7), the following actions are triggered:

[0229] Return "Cannot be determined" and request manual review;

[0230] Call backup models (such as other classifiers in ensemble learning) for recalculation.

[0231] Loss function: Cross-Entropy Loss is used to measure the difference between the predicted probability and the true label:

[0232]

[0233] in, is the one-hot encoding of the true label.

[0234] Regularization: Add L2 regularization to prevent overfitting:

[0235]

[0236] Dynamically adjust the confidence threshold θ based on business needs: In high-risk scenarios (such as financial approval), set θ=0.9 to reduce the risk of misjudgment; in low-risk scenarios (such as pre-classification), set θ=0.6 to increase processing speed.

[0237] Improve classification robustness by integrating the prediction results of multiple classifiers (such as multiple Transformer models):

[0238]

[0239] Among them, K is the number of integrated models, is the predicted probability of the kth model.

[0240] For example, take the "travel expense reimbursement form" as an example:

[0241] Input semantic vector:

[0242]

[0243] Classification layer output:

[0244]

[0245] Softmax probability distribution:

[0246]

[0247] The maximum probability of 0.8 corresponds to "expense claim," which is determined as the final type. If the threshold θ = 0.7, the document is directly output; if θ = 0.9, manual review is triggered. In the financial document classification task, this step increased the accuracy from 96% to 98.5%, and the F1 score reached 97.2%. Label smoothing and ensemble learning improve the model's tolerance to noisy input (such as erroneous amounts) by 30%. The dynamic threshold mechanism supports flexible switching between high-risk and low-risk scenarios to meet diverse business needs. Image features and text semantic vectors are combined into the classification layer to improve the classification accuracy of complex documents (such as signed invoices). Lightweight models (such as the MobileNet classification head) achieve millisecond-level document type determination, supporting real-time feedback from online reimbursement systems. If the probability distribution is highly dispersed (for example, the probability of all categories is less than 0.3), an anomaly is flagged and recorded for manual review. Combining the fully connected layer with Softmax, it directly outputs probability distribution and supports multi-label and uncertainty processing; label smoothing, temperature scaling, and ensemble learning improve model robustness; dynamic thresholds and multi-scenario support meet different business needs.

[0248] According to one embodiment of the present application, the hidden state sequence containing local context information is generated by processing the digitized text sequence time-step by time step, specifically:

[0249] The RNN encoder is implemented using a long short-term memory network or a gated recurrent unit, and processes the input digitized text sequence time step by time; the hidden state of each time step is obtained by jointly calculating the current input and the hidden state of the previous time step, specifically including: linearly combining the current input and the previous hidden state, and processing them through a nonlinear activation function to generate the hidden state of the current time step.

[0250] Specifically, the present invention processes the digitized text sequence time-step by time-step through an RNN encoder (such as LSTM or GRU) to generate a hidden state sequence containing local context information. Its core goal is to make each hidden state Fusion of the semantics of the current token and the context of the previous token; providing high-quality intermediate representation for subsequent key token screening, global modeling and other steps.

[0251] The present invention adopts the long short-term memory network (LSTM) or gated recurrent unit (GRU) as the core structure of the RNN encoder, which is specifically implemented as follows: Unit structure: LSTM contains three gate units (forget gate, input gate, output gate) and a memory unit (Cell State). For time step t, the input is the embedding vector of the current token , the previous hidden state , and the previous memory cell state , calculate the current state:

[0252]

[0253] Among them, σ is the Sigmoid activation function, ⊙ represents element-wise multiplication, and are learnable parameters.

[0254] GRU Implementation

[0255] Unit structure: GRU simplifies LSTM through two gating units (reset gate and update gate) and directly outputs the hidden state .

[0256] Time-step processing flow:

[0257]

[0258] in, Control the previous state Candidate status The impact of Determines the fusion ratio of the new state and the old state.

[0259] Time-step generation of hidden states

[0260] The present invention implements the time-step calculation of the hidden state through the following steps:

[0261] Input vector: the numerical representation of the current token (such as word embedding vectors).

[0262] Linear combination: With the previous hidden state After concatenation, the intermediate vector is generated by linear transformation:

[0263]

[0264] Among them, W and b are the learnable parameter matrix and bias.

[0265] Use nonlinear activation functions (such as Tanh and ReLU) to process the linear combination results and introduce nonlinear modeling capabilities:

[0266]

[0267] in, For activation functions (such as ), gated output (such as the output gate of LSTM ) is adjusted according to the specific RNN type.

[0268] Forward Dependency: Hidden State Directly dependent on the previous state , thus inheriting the context information of the previous token. The contribution of the current token is: The linear combination of ensures that the semantics of the current Token is fully encoded. Local context capture: By processing time step by time, each Fusion of previous-order information with the current token effectively captures associations between, for example, "travel expenses" and "reimbursement." Long-sequence modeling capabilities: The LSTM / GRU gating mechanism mitigates the vanishing gradient problem and supports processing long text (such as multi-field reimbursement forms).

[0269] The simplified GRU structure reduces the number of parameters by 20% compared to the LSTM, making it suitable for real-time processing scenarios. The LSTM's forget gate and the GRU's update gate dynamically control information retention and discarding to avoid interference from irrelevant context. Tanh or Reluctant Unified Unit (ReLU) ensures the model can capture complex nonlinear relationships (such as the association between amount and document type). Hidden states are initialized to zero vectors or with a pretrained model to accelerate convergence. LSTM or GRU can be flexibly selected based on task requirements to balance modeling capability and computational efficiency. Gating units and linear transformations dynamically fuse current input with historical context, ensuring that each hidden state focuses on local semantics, providing high-quality features for subsequent global modeling.

[0270] According to one embodiment of the present application, the screening of key tokens is based on an attention mechanism, specifically including:

[0271] The hidden state of each time step is linearly transformed through a fully connected layer, and the Tanh activation function is used to generate an intermediate vector; the attention weight is then calculated through another fully connected layer, and finally the weight value of each token is obtained through Softmax normalization; according to the preset threshold, the token with a weight value higher than the preset threshold is screened out as the key token.

[0272] Specifically, this invention uses an attention mechanism to analyze hidden state sequences and identify tokens that are crucial for document type classification or information processing (such as key fields such as "amount" and "date"). Its core objectives are:

[0273] Key Token Identification: Quantify the importance of each token through attention weight and select tokens that contribute most to the task;

[0274] Resource optimization: Provides efficient information focusing for subsequent calculations (such as global modeling and classification) and reduces redundant processing.

[0275] The hidden state at each time step t , generates an intermediate vector through a fully connected layer (linear transformation) and a nonlinear activation function (Tanh) , to capture local features:

[0276]

[0277] in: : a learnable weight matrix (dimension d×d, where d is the hidden state dimension); : learnable bias vector; : Tanh activation function, ensure that the output is Enhance nonlinear modeling capabilities within the scope.

[0278] The intermediate vector is passed through another fully connected layer (without activation function) Mapped to raw attention score , and generate the final attention weight :

[0279]

[0280] in: : learnable attention vector (dimension d×1); : exponential function, amplifying the score difference; : Normalized weight value, satisfying

[0281] According to the preset threshold θ (such as 0.05), the attention weight is screened Token as the key token:

[0282]

[0283] For example, in a travel expense reimbursement form, the weight of the tokens "travel expense," "reimbursement," "amount," and "¥1200.00" may be higher than the threshold and thus be selected as a key token. Non-key tokens (such as punctuation marks and stop words) may have a weight lower than the threshold and thus be excluded.

[0284] Weight matrix and vector: 、 、 These are all learnable parameters that are optimized during training via backpropagation.

[0285] Initialization strategy: Parameter initialization uses Xavier or He initialization to accelerate convergence.

[0286] Enhance the expressive power of intermediate vectors through nonlinear transformation; limit the output range to avoid gradient explosion problems.

[0287] Fully connected layer without activation function: directly output the original score , ensuring the interpretability of attention weights.

[0288] Static threshold: Set a fixed threshold (such as θ = 0.05) based on business needs.

[0289] Dynamic threshold: adaptively adjusted according to the attention weight distribution, for example:

[0290]

[0291] Here, k is a hyperparameter (such as k=1.5).

[0292] Accelerate attention calculations through matrix operations to avoid token-by-token loops:

[0293]

[0294]

[0295] Take the "Travel Expense Reimbursement Form" as an example:

[0296] Input hidden state sequence:

[0297]

[0298] Intermediate vector calculation:

[0299]

[0300] Attention score and weight:

[0301]

[0302] Key Token screening: threshold θ=0.15, screening out , , , corresponding to the Token "reimbursement" "amount" "¥1200.00".

[0303] Through the attention mechanism, the weight of key tokens (such as amount, document type) is significantly higher than that of non-key tokens, reducing redundant calculations. The attention weight intuitively shows the degree of dependence of the model on each token, which is convenient for debugging and analysis. In the task of financial document classification, a small number of key tokens (30% of the total tokens) screened out contributed 80% of the classification information, reducing the subsequent computing resource consumption by 50%. Tanh activation is used to prevent gradient disappearance and ensure the stability of the intermediate vector; Softmax normalization ensures a reasonable weight distribution and avoids interference from extreme values. The threshold is dynamically adjusted according to the data distribution to adapt to different scenarios (such as the difference between long text and short text). Hidden state Generated by LSTM / GRU, it already incorporates local contextual information, and the attention mechanism further refines key features. A fully connected layer and Tanh activation are combined to generate intermediate vectors and calculate weights. Static or dynamic thresholds flexibly adapt to different scenarios. Vectorized operations accelerate attention calculations and support real-time processing.

[0304] According to one embodiment of the present application, the dynamically allocating computing resources includes:

[0305] Dynamically adjust the proportion of subsequent computing resources based on the ratio of the number of key tokens to the total number of tokens multiplied by the preset resource amplification factor;

[0306] Assign embedding vectors of different dimensions to the hidden states corresponding to key tokens, or improve the processing capabilities of key information by changing the structure of the computing layer.

[0307] Specifically, this invention optimizes computing resource utilization through dynamic resource allocation, improving the efficiency and accuracy of key token processing while reducing redundant computation of non-critical tokens. Its core objectives are to dynamically allocate resources based on the importance of key tokens, avoiding a one-size-fits-all approach that wastes resources; enhance the processing capabilities of key information (such as high-precision recognition of amounts and date fields); and reduce overall computing overhead.

[0308] The present invention dynamically calculates the resource allocation ratio of key tokens and non-key tokens through the following formula:

[0309]

[0310] in: : The ratio of the number of key tokens m to the total number of tokens N; : The preset resource magnification factor (such as 1.5 or 2.0) is set based on business needs or experimental data.

[0311] If the total number of tokens N = 20 and the number of key tokens m = 8, then r = 0.4; if γ = 1.5, the critical resource ratio is 0.4 × 1.5 = 0.6, that is, 60% of the resources are allocated to key tokens.

[0312] In high-risk scenarios (such as financial approval), set γ=2.0 to ensure high-precision processing of key information; in low-risk scenarios (such as pre-classification), set γ=1.0 to balance speed and resource consumption.

[0313] Dynamically adjust γ based on hardware status (such as GPU memory occupancy):

[0314]

[0315] For example, when the video memory usage exceeds 80%, γ is automatically reduced to release resources.

[0316] Assign embedding vectors of different dimensions to key tokens and non-key tokens to enhance the expression ability of key information:

[0317] Key Token Embedding:

[0318]

[0319] Non-critical Token embedding:

[0320]

[0321] The embedding vector of "¥1200.00" (key token) has a higher dimension, which can capture the precise numerical characteristics of the amount; the low-dimensional embedding of "Attachment" (non-key token) reduces computational overhead.

[0322] By changing the computing layer structure of the key token path, its processing capacity is improved:

[0323] Key Token Path:

[0324]

[0325] Residual connections or attention mechanisms can be added to enhance feature expression:

[0326]

[0327] Non-critical Token path:

[0328]

[0329] The computing tasks of critical tokens are assigned to high-performance GPU cores, while non-critical tokens use CPU or low-priority GPU threads.

[0330] For non-critical tokens, only compressed versions (such as low-dimensional embeddings) are retained to free up video memory space.

[0331] Take the "Travel Expense Reimbursement Form" as an example:

[0332] Key token screening results: Key tokens: m = 5 (e.g., "travel expense," "reimbursement," "amount," "¥1200.00," "date"); Total tokens: N = 20, r = 0.25. Resource ratio calculation: Setting γ = 1.5, the critical resource ratio is 0.375 (37.5%), and the non-critical resource ratio is 62.5%. Embedding vector allocation: Key token embedding dimension: 256-dimensional; non-critical token embedding dimension: 128-dimensional. Computation layer structure: Key token path: two fully connected layers + residual connections; non-critical token path: a single fully connected layer.

[0333] In the financial document classification task, the processing speed of key tokens increased by 40%, while overall graphics memory usage decreased by 30%. The high-dimensional embedding and deep structure of key tokens improve their feature expression capabilities by 25%, while low resource allocation to non-key tokens reduces redundant computation. A real-time resource feedback mechanism enables the system to automatically reduce γ when graphics memory is insufficient, avoiding crashes. Dynamic adjustment of r×γ allows for flexible adaptation to different scenarios and hardware conditions. High-dimensional embedding of key tokens enhances semantic expression, while low-dimensional embedding of non-key tokens reduces computational overhead. Increasing the number of layers or using residual connections ensures high-precision processing of key information. Based on the proportion of key tokens and the amplification factor, resource allocation is precisely controlled; key information processing capabilities are enhanced through dimensionality adjustment and computational layer optimization; and parameters are dynamically adjusted based on hardware status to ensure system robustness.

[0334] According to one embodiment of the present application, the global dependency modeling is implemented by a multi-head self-attention mechanism, specifically including:

[0335] The hidden state sequence of the key token is input into the Transformer encoder, and the query, key, and value matrices are generated through linear transformation;

[0336] Calculate the similarity score between the query and the key, obtain the attention weight through Softmax normalization, and generate the attention output by weighted summation of the Value vector;

[0337] The attention outputs of multiple subspaces are calculated in parallel through a multi-head attention mechanism, and the results are concatenated and linearly transformed into a global semantic representation vector.

[0338] According to one embodiment of the present application, the global semantic representation vector is generated in the following manner:

[0339] Perform residual connection and layer normalization on the output of multi-head attention and the original hidden state;

[0340] The local and global features are further fused through a feedforward neural network, and the global semantic representation vector is finally output.

[0341] Specifically, this invention uses a multi-head attention mechanism and a feedforward neural network to combine local contextual information (such as the hidden state of key tokens) with global dependencies to generate a comprehensive global semantic representation vector. Its core objectives are to capture cross-token dependencies (such as the association between "travel expenses" and "amount"); improve model stability through residual connections and normalization; and further refine key information through the feedforward network, providing high-level semantic features for tasks such as classification.

[0342] Input: original hidden state sequence , where N is the total number of tokens.

[0343] Use H parallel attention heads (e.g. H=8), each head independently calculates the dependencies between tokens.

[0344] The calculation process of each head:

[0345] Linear transformation:

[0346] For each hidden state , generating query (Q), key (K), value (V) vectors through a learnable weight matrix:

[0347]

[0348] in, is the weight matrix of the h-th head.

[0349] Attention score calculation:

[0350] Calculate the attention weight of each Token to other Tokens:

[0351]

[0352] Softmax normalization and weighted summation:

[0353]

[0354] Multiple output merge

[0355] The outputs of all heads are concatenated and linearly transformed to generate the multi-head attention output A:

[0356]

[0357] in, is the learnable output weight matrix.

[0358] The multi-head attention output A is residually connected to the original hidden state H, and gradient fluctuations are eliminated by layer normalization:

[0359]

[0360] Residual connection: retains the original hidden state information to prevent information loss;

[0361] Layer normalization: Normalize the feature dimension of each sample. The formula is:

[0362]

[0363] Among them, μ and σ are mean and standard deviation respectively, and γ and β are learnable parameters.

[0364] Feedforward Neural Network (FFN) Feature Fusion

[0365] The local and global features are further integrated through a two-layer fully connected network to generate the final global semantic vector:

[0366]

[0367] First layer: linear transformation expands the feature dimension (e.g., d→4d), and the activation function is ReLU; second layer: linear transformation compresses the dimension (4d→d), and the output and input dimensions are consistent; layer normalization: ensures stable feature distribution and improves training efficiency.

[0368] Matrix operations accelerate multi-head processing, avoiding token-by-token calculations. Residual connections preserve original information, and layer normalization eliminates gradient bias, collectively increasing model depth. An expansion-compression structure (e.g., d → 4d → d) enhances nonlinear modeling capabilities. It captures local and global dependencies in parallel, stabilizes the training process, and improves model robustness. Nonlinear transformations extract high-level semantic features.

[0369] According to one embodiment of the present application, the probability distribution is implemented by a fully connected layer and a Softmax activation function, specifically including:

[0370] The semantic representation vector is input into the fully connected layer for linear transformation, and the output vector has the same dimension as the number of document types.

[0371] The vector is normalized using the Softmax function to generate the probability distribution of each document type.

[0372] Specifically, this invention maps the global semantic representation vector into a probability distribution of document types through a combination of a fully connected layer and a softmax activation function. Its core objectives are: converting high-dimensional semantic vectors into interpretable probability distributions to support accurate document type determination; quantifying the model's confidence in each category through probability values to aid subsequent decision-making (such as triggering manual review); and, combined with the previously mentioned global semantic modeling module, improving the overall system's classification accuracy.

[0373] Input vector: global semantic representation vector (generated by the previous step, dimension is d).

[0374] Output dimension: The preset number of document types C (such as reimbursement orders, purchase orders, travel expense orders, etc.).

[0375] Linear transformation formula:

[0376]

[0377] in:

[0378] : learnable weight matrix; : learnable bias vector; : Unnormalized class score vector.

[0379] If the global vector dimension d = 512 and the document type C = 3, then is a 3×512 matrix, z is .

[0380] Apply the Softmax function to the linearly transformed score z to generate a probability distribution :

[0381]

[0382] in:

[0383] represents the probability that the input document belongs to the i-th category; , which satisfies the normalization requirement of probability distribution.

[0384] This step optimizes the parameters of the fully connected layer through the cross-entropy loss function:

[0385]

[0386] in:

[0387] is the One-Hot Encoding of the true label; the goal is to minimize the difference between the predicted probability p and the true label y.

[0388] Label Smoothing

[0389] To avoid overfitting of the model, the probability distribution of the true label is smoothed:

[0390]

[0391] in: is the smoothing coefficient (e.g. );

[0392] For example, if the true label is category 1, the corrected probability is:

[0393]

[0394] Temperature Scaling

[0395] The smoothness of the probability distribution can be controlled by adjusting the temperature parameter T of Softmax:

[0396]

[0397] High temperature T: makes the probability distribution more uniform and reduces the model's reliance on high confidence levels; low temperature T: enhances the probability of high-scoring categories and improves the certainty of classification decisions.

[0398] This technology maps high-dimensional semantic vectors to a category space, preserving key features; generates interpretable probability distributions to support multi-scenario decision-making; and enhances model robustness and generalization through label smoothing and temperature scaling. This technology provides a core basis for classification decisions in intelligent order selection systems, significantly improving the accuracy and reliability of document type determination and making it suitable for automated document processing scenarios in finance, healthcare, logistics, and other fields.

[0399] According to one embodiment of the present application, before dynamically allocating computing resources, the method further includes:

[0400] The digitized text sequence is preprocessed, including word segmentation, removal of stop words, standardization of amount and date formats, and word vector encoding through a pre-trained model to generate the initial digitized text sequence.

[0401] Specifically, before dynamically allocating computing resources, the present invention normalizes and extracts features from the original text through a preprocessing process, generating a digitized text sequence that can be used for subsequent calculations. Its core objectives are to: remove irrelevant information (such as stop words) and format differences (such as inconsistent amounts and date formats); convert text into numerical vectors through a pre-trained model to provide computable input for subsequent models; and ensure that document text from different sources has a unified format and representation, thereby improving the model's generalization capabilities. The original text is segmented into meaningful vocabulary units (tokens). Specific methods include: using domain-adapted word segmentation tools (such as jieba) or custom dictionaries. For example: Original text: "Travel expense reimbursement form amount ¥1200.00" → word segmentation result: [travel expense, reimbursement form, amount, ¥1200.00]; English / number segmentation: segmentation based on space, while recognizing special symbols (such as "¥" and ".") and numerical combinations:

[0402] \text{“Total: $1500.00 on 2023-01-15”} \rightarrow [\text{Total}, \text{:}, \text{\$1500.00}, \text{on}, \text{2023-01-15}]

[0403] For financial documents, add professional terms such as "reimbursement form" and "travel expenses" to the word segmentation dictionary;

[0404] Recognize "¥1200.00" as a separate token instead of splitting it into "¥", "1200", ".", and "00".

[0405] Filter meaningless tokens through a predefined stopwords list, for example:

[0406] Chinese stop words: de, shi, zai, etc.; English stop words: the, and, on, a

[0407] Custom stop words: Add specific words (such as "attachment" and "see next page") based on business needs.

[0408] Example: Original word segmentation: ["travel expenses", "reimbursement form", "of", "total amount", "¥1200.00"]

[0409] After removing stop words: ["travel expenses", "reimbursement form", "total amount", "¥1200.00"]

[0410] Use regular expressions (Regex) or NLP tools to unify numerical formats and improve the model's ability to identify key information:

[0411] Amount standardization: Rule examples: Input: $1,200.00 → Output: 1200.00; Input: ¥1.5k → Output: 1500.00 (processed through numerical conversion rules); Symbol unification: Standardize currency symbols such as "¥", "$", and "RMB" to a unified representation (for example, retain "¥").

[0412] Date Normalization: Example Rules: Input: Output: ; Input: 15 / Jan / 2023 → Output: ; Format unification: convert all dates to YYYY-MM-DD format.

[0413] Convert the tokenized text into a fixed-dimensional numerical vector using a pre-trained language model (such as BERT, Word2Vec, or a domain-specific model):

[0414] Pre-trained model selection:

[0415] BERT: captures contextual dependencies and generates dynamic word vectors; Word2Vec: generates static word vectors, suitable for simple scenarios.

[0416] Coding process: Input: [travel expense, reimbursement form, total amount, ¥1200.00] → Output: {v} (e.g., 768 dimensions)

[0417] Out-of-view (OOV) processing:

[0418] Subword splitting (such as BERT's WordPiece); random initialization + fine-tuning (such as splitting the amount "¥1200.00" into "¥", "1200", ".", "00").

[0419] Combine the outputs of the above preprocessing steps into the final digitized sequence:

[0420]

[0421] Sequence length: N is the number of tokens after preprocessing;

[0422] Dimensionality: Determined by the pre-trained model (e.g. 768 dimensions for BERT or 300 dimensions for Word2Vec).

[0423] This technology ensures that key information (such as amounts and dates) is correctly identified and unified; reduces redundant calculations, focuses on key tokens, and provides high-quality numerical input for subsequent models. This technology lays the foundation for dynamic resource allocation and global semantic modeling, significantly improving the processing efficiency and accuracy of intelligent menu systems. It is suitable for automated document processing scenarios in finance, healthcare, logistics, and other fields.

[0424] A computer-readable storage medium stores a program, which implements the steps in the method when executed by a processor.

[0425] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the steps in the method are implemented when the processor executes the program.

[0426] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0427] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0428] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A financial document classification method based on dynamic resource allocation and attention mechanism, characterized in that: include: Based on the numerical text sequence, a hidden state sequence containing local context information is generated by processing it step by step. According to the hidden state sequence, key tokens are screened and computing resources are dynamically allocated, specifically: The hidden state at each time step t , generate intermediate vectors through fully connected layers and nonlinear activation functions: in, and is a learnable parameter; For the middle vector The raw attention score is further calculated through the fully connected layer: in, is a learnable attention vector; The score is converted into Converted to normalized attention weights , ensuring that the sum of all token weights is 1: Set the preset threshold θ to filter out the attention weights Token as the key token; Modeling the global dependency relationship of the key tokens to generate a semantic representation vector; The hidden state of each time step is linearly transformed through the fully connected layer, and the Tanh activation function is used to generate the intermediate vector; Then, the attention weight is calculated through another fully connected layer, and finally the weight value of each token is obtained through Softmax normalization; According to the preset threshold, the tokens with weight values higher than the preset threshold are selected as key tokens; Dynamically adjust the proportion of subsequent computing resources based on the ratio of the number of key tokens to the total number of tokens multiplied by the preset resource amplification factor; Assign embedding vectors of different dimensions to the hidden states corresponding to key tokens, or improve the processing ability of key information by changing the structure of the computing layer; Calculating a probability distribution of each document type based on the semantic representation vector, and determining a corresponding document type according to the probability distribution; Specifically: Dynamically calculate the resource allocation ratio of key tokens and non-key tokens: in, : The ratio of the number of key tokens m to the total number of tokens N; : The preset resource amplification factor is set based on business needs or experimental data; If the total number of tokens N = 20 and the number of key tokens m = 8, then r = 0.4; if γ = 1.5, the critical resource ratio is 0.4 × 1.5 = 0.6, that is, 60% of the resources are allocated to key tokens; In high-risk scenarios, set γ = 2.0 to ensure high-precision processing of critical information; in low-risk scenarios, set γ = 1.0 to balance speed and resource consumption; Dynamically adjust γ based on hardware status (such as GPU memory occupancy): When the video memory usage exceeds 80%, γ is automatically reduced to release resources; Assign embedding vectors of different dimensions to key tokens and non-key tokens to enhance the expression ability of key information: Key Token Embedding: Non-critical Token embedding: By changing the computing layer structure of the key token path, its processing capacity is improved: Key Token Path: Residual connections or attention mechanisms can be added to enhance feature expression: Non-critical Token path: The computing tasks of key tokens are assigned to the GPU core, while non-key tokens use the CPU; For non-critical tokens, only the compressed version of the intermediate results is retained to free up video memory space.

2. The method according to claim 1, characterized in that The method generates a hidden state sequence containing local context information based on the digitized text sequence by processing it time step by time, specifically: The RNN encoder is implemented using a long short-term memory network or a gated recurrent unit to process the input digitized text sequence time step by time; The hidden state of each time step is calculated by jointly calculating the current input and the hidden state of the previous time step. Specifically, the current input is linearly combined with the previous hidden state, and then processed by a nonlinear activation function to generate the hidden state of the current time step.

3. The method according to claim 1, characterized in that The global dependency modeling is implemented through a multi-head self-attention mechanism, specifically including: The hidden state sequence of the key token is input into the Transformer encoder, and the query, key, and value matrices are generated through linear transformation; Calculate the similarity score between the query and the key, obtain the attention weight through Softmax normalization, and generate the attention output by weighted summation of the Value vector; The attention outputs of multiple subspaces are calculated in parallel through a multi-head attention mechanism, and the results are concatenated and linearly transformed into a global semantic representation vector.

4. The method according to claim 3, characterized in that The global semantic representation vector is generated in the following way: Perform residual connection and layer normalization on the output of multi-head attention and the original hidden state; The local and global features are further fused through a feedforward neural network, and the global semantic representation vector is finally output.

5. The method according to claim 1, wherein The probability distribution is implemented through a fully connected layer and a Softmax activation function, specifically including: The semantic representation vector is input into the fully connected layer for linear transformation, and the output vector has the same dimension as the number of document types. The vector is normalized using the Softmax function to generate the probability distribution of each document type.

6. The method according to claim 1, characterized in that Before dynamically allocating computing resources, the method further includes: The digitized text sequence is preprocessed, including word segmentation, removal of stop words, standardization of amount and date formats, and word vector encoding through a pre-trained model to generate the initial digitized text sequence.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 6 are implemented.

Images