Big language model-based receipt identification method, apparatus and device, and medium

Abstract

The invention relates to the technical field of large language models, and discloses a large language model-based receipt recognition method, device and equipment and a medium, and the method comprises the steps of carrying out domain term training on a large language model by utilizing corpus data to obtain a business domain basic model; training an adapter layer in the business field basic model by using historical electronic document data to obtain an enterprise business model; performing model optimization on the enterprise business model by using the historical document exception type data to obtain an exception identification optimization model of the target enterprise; identifying a text modal anomaly probability of the target business document through the anomaly identification optimization model, and analyzing a structural modal anomaly probability of the target business document; and performing bimodal weighted fusion on the text modal anomaly probability and the structural modal anomaly probability to obtain a target anomaly probability of the target business document, and identifying a document type of the target financial document according to the target anomaly probability. According to the invention, the accuracy of bill identification can be improved.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a document recognition method, apparatus, device and medium based on a large language model. Background Technology

[0002] As businesses expand, the number of financial documents grows exponentially, and the types of documents (such as invoices, expense reports, and payment vouchers) and abnormal scenarios (such as fake invoices, duplicate reimbursements, and amount tampering) become increasingly diverse. Therefore, it is necessary to identify different types of documents to meet diverse identification needs.

[0003] Existing document recognition technologies are mostly based on rule engines or simple machine learning models. Rule engines require manual pre-setting of a large number of judgment conditions, which makes it difficult to cover complex and ever-changing abnormal patterns. Simple machine learning models have weak parsing ability for unstructured document information (such as handwritten notes and blurred seals), and their generalization ability is limited by the sample distribution, resulting in missed detection of document anomalies and thus low accuracy in document recognition. Summary of the Invention

[0004] This invention provides a document recognition method, apparatus, device, and medium based on a large language model to solve the technical problem of low accuracy in document recognition.

[0005] Firstly, a document recognition method based on a large language model is provided, including: Collect corpus data of the target business domain, and use the corpus data to train a pre-set large language model with domain terms to obtain a basic model of the business domain. Collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the business domain basic model to obtain the enterprise business model; The positive and negative sample pairs in the historical document anomaly type data are converted into feature representation vectors. The feature distance between similar and dissimilar samples in the positive and negative sample pairs is calculated based on the feature representation vectors. The model parameters in the enterprise business model are optimized based on the feature distance to obtain an anomaly identification optimization model. The anomaly detection optimization model analyzes the semantic and coding anomaly modes between text fields in the preset target business document, and outputs the text mode anomaly probability of the target business document for the semantic and coding anomaly modes through the activation function in the anomaly detection optimization model. Based on the document entities of the target business document and the logical relationships between the document entities, a graph structure of the target business document is constructed. A preset graph neural network is used to perform node feature propagation and edge weight learning on the graph structure to obtain node abnormal modes and edge abnormal modes. The classifier in the graph neural network outputs the structural mode abnormality probability of the target business document for the node abnormal modes and the edge abnormal modes. The text modal anomaly probability and the structural modal anomaly probability are fused using a dual-modal weighted method to obtain the target anomaly probability of the target business document. The target anomaly probability is then compared with a preset risk probability threshold, and the risk type of the target financial document is determined based on the comparison result.

[0006] Secondly, a document recognition device based on a large language model is provided, comprising: The domain terminology training module is used to collect corpus data of the target business domain, and use the corpus data to train the domain terminology of the preset large language model to obtain the basic model of the business domain. The adapter layer training module is used to collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the basic business domain model to obtain the enterprise business model. The model optimization module is used to convert positive and negative sample pairs in the historical document anomaly type data into feature representation vectors, calculate the feature distance between similar and dissimilar samples in the positive and negative sample pairs based on the feature representation vectors, and optimize the model parameters in the enterprise business model based on the feature distance to obtain an anomaly identification optimization model. The text modality anomaly probability analysis module is used to analyze the semantic anomaly modality and the coding anomaly modality between text fields in a preset target business document through the anomaly recognition optimization model, and output the text modality anomaly probability of the target business document for the semantic anomaly modality and the coding anomaly modality through the activation function in the anomaly recognition optimization model. The structural modality anomaly probability analysis module is used to construct a graph structure of the target business document based on the document entities and the logical relationships between the document entities, and to use a preset graph neural network to perform node feature propagation and edge weight learning on the graph structure to obtain node anomaly modalities and edge anomaly modalities. The classifier in the graph neural network outputs the structural modality anomaly probability of the target business document for the node anomaly modalities and the edge anomaly modalities. The risk type identification module is used to perform dual-modal weighted fusion of the text modal anomaly probability and the structural modal anomaly probability to obtain the target anomaly probability of the target business document, and compare the target anomaly probability with a preset risk probability threshold to determine the risk type of the target financial document based on the comparison result.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the document recognition method based on a large language model described above.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the document recognition method based on a large language model described above.

[0009] In this invention, domain terminology training is conducted by collecting target business domain corpus data, enabling a pre-set large language model to accurately grasp the meaning of domain professional terms and document fields, laying the foundation for business adaptation. Then, the adapter layer of the business domain basic model is trained using historical electronic document data from the target enterprise. By freezing most model parameters, training costs and overfitting risks are reduced, achieving personalized adaptation of the model to the enterprise's unique document styles and approval processes. Model optimization is performed based on historical document anomaly type data, enhancing the model's sensitivity to subtle anomaly features and effectively reducing missed and false positives. A three-level fine-tuning mechanism strengthens the model's ability to learn new anomaly patterns from the enterprise. Anomaly recognition optimization model and graph neural network are used to extract text modality and structural modality anomaly probabilities respectively, comprehensively covering the needs of unstructured information parsing and cross-field logical conflict identification. The target anomaly probability is obtained through dual-modal weighted fusion, and document type determination is completed, dynamically adapting to the identification focus of different document types, significantly improving the identification accuracy of complex anomaly patterns, and accelerating the response speed to new anomaly patterns. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a document recognition method based on a large language model according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a document recognition method based on a large language model in one embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 4 This is a schematic diagram of a document recognition device based on a large language model in one embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The document recognition method based on a large language model provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can collect corpus data of the target business domain from the client, train the large language model with domain terminology, and obtain a basic business domain model adapted to the professional expression of the business. It can also collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic documents to train the adapter layer of the basic model, obtaining an enterprise business model adapted to the enterprise's unique document styles and processes. The server can then optimize the enterprise model using historical document anomaly type data, obtaining an anomaly recognition optimization model that can sensitively identify subtle anomalies. This optimization model identifies the text modal anomaly probability of the target business documents, and simultaneously analyzes its structural modal anomaly probability using a graph neural network. These two probabilities are weighted and fused to obtain the target anomaly probability. Finally, based on this probability, the type of the target document is identified, and the result is fed back to the client. Through three-layer model training, the model is adapted to the business domain and enterprise characteristics, while also strengthening the anomaly recognition capability. Combined with dual-modal fusion analysis, this effectively improves the accuracy of document anomaly and type recognition, and also accelerates the response speed to new anomaly patterns. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0014] Please see Figure 2 As shown, Figure 2A flowchart illustrating the document recognition method based on a large language model provided in this embodiment of the invention includes the following steps: S1. Collect corpus data of the target business domain, and use the corpus data to train the preset large language model with domain terms to obtain the basic model of the business domain.

[0015] In this embodiment of the invention, the corpus data refers to a set of publicly available professional data in the industry to which the target business belongs. For the financial field, this specifically includes accounting standard texts, tax regulations, and publicly available electronic document format specifications.

[0016] In detail, corpus data of the target business domain can be collected from a pre-stored storage area using computer statements (such as Java statements, Python statements, etc.) with data scraping capabilities, where the storage area includes, but is not limited to, databases and blockchains.

[0017] In this embodiment of the invention, the business domain basic model refers to a model that, after being trained with business domain-specific terminology, can accurately understand the meaning of professional terms and fields in the target business domain.

[0018] In this embodiment of the invention, the step of using the corpus data to train a preset large language model with domain terminology to obtain a basic business domain model includes: The corpus data is segmented into words, and each word in the segmented corpus data is replaced with a unique integer sequence number corresponding to each word in a preset vocabulary. The unique integer sequence number is used as the row index of the embedding matrix in the embedding layer of the large language model. The word vector corresponding to the unique integer sequence number in the embedding matrix is ​​located by the row index. The word vector is analyzed by self-attention mechanism and feedforward neural network to obtain a hidden state vector sequence containing contextual semantics. Perform linear classification on each word vector in the hidden state vector sequence to obtain the probability distribution of each word vector; Calculate the training loss value between the probability distribution of each word vector and the true probability value of the corresponding position of the word vector in the digit sequence; The large language model is backpropagated based on the training loss value, and the model parameters of the large language model are updated based on the backpropagation. When the training loss value is less than the preset loss threshold, the large language model corresponding to the updated model parameters is used as the basic model for the business domain.

[0019] In detail, the large language model symbolizes text through a pre-defined vocabulary. The collected financial domain corpus is segmented sentence by sentence by a word segmenter, and each unit is mapped to a unique numerical ID. After segmenting the original financial domain corpus, a predefined financial domain vocabulary is queried. This vocabulary assigns a unique integer ID to each known word after segmentation (e.g., {“company”: 101, “revenue”: 205, “amount”: 307}). At this point, each word is simply replaced with its corresponding ID, thus converting the text sequence (company revenue amount) into a numerical sequence [101, 205, 307] constructed from unique integer sequences. For example, professional expressions in accounting standards, key clauses in tax regulations, and fixed fields in electronic documents are converted into discrete unique integer sequences (tokens) that the model can recognize. Similarly, input tax deductions are split into corresponding token combinations, generating a numerical sequence that the model can directly read and process. This allows the model to receive structured domain data input and initially establishes the ability to recognize financial text.

[0020] Specifically, the embedding layer converts unique integer indices into continuous, semantically rich vector representations. The embedding layer is essentially a trainable parameter matrix with rows equal to the vocabulary size and columns equal to the dimension of the word vectors. During model processing, each unique integer indice (e.g., 101) is treated as a row index of this matrix, directly locating and extracting the vector of the corresponding row. For example, the unique integer indice 101 extracts the D-dimensional vector of the 101st row of the matrix, serving as the distributed representation of the word "company". Thus, a sequence of numbers of length L is transformed into a sequence of real-valued vectors of dimension [L, D], achieving a mapping from discrete symbols to a continuous vector space. Ultimately, the entire sequence of numbers is converted into a vector sequence, transforming discrete symbolic input into a continuous numerical form that can be used for complex mathematical operations by subsequent neural network layers. The parameters in this matrix are continuously optimized during model training, ensuring that the vectors corresponding to semantically similar words are also close to each other in space. This sequence then flows into the core computational layer, which consists of a self-attention mechanism and a feedforward neural network. Each word vector is associated with all other words in the sequence to fully understand the contextual relationships, ultimately outputting a sequence of hidden state vectors containing the complete semantics of the sentence. The linear classification layer at the top of the large language model calculates the probability distribution across the entire vocabulary for each position in this sequence, predicting "what word is most likely to be at this position in this context." For example, when inputting the accounting principle text "Enterprises should recognize revenue according to the accrual basis," the model can identify core domain terms such as "accrual basis" and "revenue recognition" through forward propagation and generate a corresponding probability distribution for each domain term. Key domain terms are selected from the massive corpus, initially mining professional information within the corpus to provide data support for the model to learn the meaning of domain terms. The predicted probability distribution output by the model is compared with the actual words at each position (in self-supervised learning, a "cloze test" task is constructed by pre-masking some words, and the masked words themselves are the actual answers). The difference between the two is quantified into a specific value by the cross-entropy loss function, namely the training loss value, which represents the overall prediction error of the model. The loss value is the core feedback signal of the entire learning process. The larger the difference, the lower the model's recognition accuracy of domain terms.

[0021] Furthermore, backpropagation refers to calculating the gradient layer by layer along the backward path of the neural network based on the calculated training loss value, and adjusting the model's weights and biases using the gradient descent algorithm. The gradient indicates how each parameter should be slightly adjusted to most effectively reduce the loss. The optimizer (such as Adam) adjusts all the hundreds of millions of parameters in the model simultaneously based on this gradient information and a preset learning rate. The general form of the update is: New parameter = Old parameter - Learning rate × Gradient. This process iterates repeatedly on large amounts of data, with each iteration aiming to make the model's predictions closer to reality, thereby incorporating financial language patterns into the model's parameters. For example, when the loss value is high, the model increases the weight of features related to financial terminology, strengthening its ability to recognize professional expressions such as "invoice code" and "input tax deduction," gradually aligning the model with the needs of understanding financial terminology and solving the problem of insufficient recognition ability of general-purpose language models for professional terminology. The loss threshold is an empirical value set according to the accuracy requirements of financial terminology recognition, for example, set to 0.05. When the training loss value is lower than this value, it indicates that the difference between the probability distribution output by the model and the real data is within an acceptable range. At this point, the updated model has the basic ability to understand financial terms and electronic document fields, and can accurately identify the core expressions in "Accounting Standards" and "Tax Regulations". It has become the basic model for the business domain, enabling the model to shift from general semantic understanding to financial scenario adaptation, and providing a reliable foundation for subsequent personalized fine-tuning by enterprises.

[0022] For example, domain pre-training is performed on the basic large language model to optimize the model's basic understanding of financial terms (such as "input tax deduction" and "accrual basis") and electronic document fields (such as "invoice code" and "verification code"), and outputs a "basic financial domain model" to solve the problem of adaptability of the model's general semantic understanding to financial scenarios.

[0023] Furthermore, traditional models lack an understanding of financial terminology and document specifications, resulting in insufficient recognition accuracy. Therefore, it is necessary to build a business domain-based model through domain pre-training. However, this basic model only completes a preliminary adaptation from general to professional, and has not yet incorporated the specific needs of individual enterprises. Therefore, it is necessary to fine-tune the model with enterprise data to achieve an accurate match between the model and the actual business operations of the enterprise.

[0024] S2. Collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the business domain basic model to obtain the enterprise business model.

[0025] In this embodiment of the invention, historical electronic document data refers to a collection of internal electronic documents generated by the target company within the past five years. This includes compliant normal document samples and manually annotated abnormal document samples. Sample formats include PDF, Excel, and scanned documents (converted to text via OCR), such as compliant invoices, approved expense reports, fraudulent invoices, and duplicate expense reports within the company. Historical document anomaly type data refers to document data corresponding to various abnormal payment scenarios that have occurred in the target company's past, such as sample data of high-frequency abnormal scenarios like inconsistent electronic invoice headers and contract subjects, and conflicts between capitalized and numeral amounts.

[0026] In detail, corpus data of the target business domain can be collected from a pre-stored storage area using computer statements (such as Java statements, Python statements, etc.) with data scraping capabilities, where the storage area includes, but is not limited to, databases and blockchains.

[0027] In this embodiment of the invention, the enterprise business model refers to a personalized model that, after being trained with historical enterprise data, can adapt to the unique document styles and approval processes of the target enterprise.

[0028] In this embodiment of the invention, training the adapter layer in the business domain basic model using the historical electronic document data to obtain the enterprise business model includes: Freeze the target parameters in the business domain basic model, and use the parameters corresponding to the unfrozen adapter layer in the business domain basic model as model training parameters; The historical electronic document data is used to perform forward propagation calculation on the basic model of the business domain to obtain the probability distribution of each document in the historical electronic document data for different document categories, and the category with the highest probability distribution is selected as the category label of each document. The loss value between the category label and the true category label of each document is calculated by a preset loss function, and the model training parameters are updated by a preset low-rank adaptation algorithm. When the loss value is less than the preset convergence threshold, the adapter layer is updated with the updated model training parameters, and the updated adapter layer is integrated into the business domain basic model to obtain the enterprise business model.

[0029] In detail, target parameters refer to the core parameters in the business domain foundation model responsible for general semantic understanding and domain knowledge storage, accounting for over 95% of the total model parameters. Freezing target parameters means keeping these parameters unchanged during training, using only the parameters of the adapter layer as model training parameters. By using the parameter freezing function of the model training framework, the main parameters of the foundation model are locked, and only the adapter layer's parameter update permissions are opened, preventing the domain knowledge of the foundation model from being overwritten. This significantly reduces the scale of training parameters, lowers training costs and the risk of overfitting, and solves the problems of high cost and poor adaptability in training with all parameters. Category labels are the labels of the model's prediction results for historical electronic document data, including document type labels, compliance labels, etc. The structured / textual data of a document is converted into an input format acceptable to the model. The data propagates forward through the frozen backbone network of the business domain foundation model, undergoing transformations at each layer, and finally reaching the classification head (a fully connected layer) at the top of the model. The classification head outputs a probability distribution (such as probabilities in normal and abnormal classes). The class with the highest probability value is selected as the model's prediction result for this document, i.e., the category label. For example, when the model performs forward propagation calculations on an internal expense report, it outputs category labels such as expense report, compliant expense report, or duplicate expense report based on features such as the header format and field content of the document. By combining domain knowledge from the business domain-based model with the personalized characteristics of enterprise documents, predictions are made to classify and predict the compliance of enterprise documents, providing a basis for subsequent parameter updates.

[0030] Specifically, the true category label refers to the accurate label manually assigned to each document in the historical electronic document data, such as labels for normal invoices and counterfeit invoices verified by finance personnel. The loss value represents the deviation between the target label and the true category label. A pre-defined loss function calculates the difference between the output category label and the inherent true category label of the document, outputting a quantified loss value. This loss value is input into the LoRA (Low-Rank Adaptation) algorithm, which updates the adapter layer parameters by constructing a low-rank matrix without modifying the main model structure. Low-rank matrix decomposition reduces the complexity of parameter updates, calculating the gradient of the loss value with respect to the adapter layer parameters and updating only these parameters. Decomposing into two low-rank matrices efficiently represents and updates the adapter parameters, significantly reducing training overhead. This iterative process drives the adapter to learn how to adjust the output of the base model, making its predictions increasingly aligned with the company's actual business classification standards, achieving efficient optimization of the adapter layer parameters. While maintaining the model's domain knowledge, the adapter layer quickly learns the personalized characteristics of the company's documents, solving the problem of traditional models being difficult to adapt to specific business scenarios. When the loss value remains consistently below the preset convergence threshold, it indicates that the adapter has completed its learning. The system integrates the trained adapter parameters back into the original base model, forming a seamless unified whole. The resulting enterprise business model uses the convergence threshold, which is defined as the loss value falling below a preset deviation threshold. For example, if the deviation threshold is set to 0.03, a loss value less than this value indicates that the model's prediction accuracy for enterprise documents meets the requirements. At this point, the updated adapter layer parameters are integrated into the business domain base model, enabling the model to recognize enterprise-specific document styles (such as internal expense report header formats) and approval processes (such as two-level approval from department manager to CFO), forming the enterprise business model. This achieves personalized model adaptation, allowing the model to accurately identify various documents from the target enterprise and solving the problem of model generalization being limited by sample distribution.

[0031] In addition to LoRA low-rank adaptation, Adapter technology can be used as an alternative. By inserting small adapter modules (such as bottleneck structures) between model layers, freezing the basic model parameters, and training only the adapter parameters, efficient parameter fine-tuning can also be achieved to achieve enterprise data adaptation.

[0032] Furthermore, since the basic model only has general financial domain recognition capabilities and cannot adapt to the unique document styles and approval processes of different enterprises, it is necessary to collect historical data from within the enterprise to train the adapter layer to achieve accurate adaptation. While the enterprise business model can recognize regular enterprise documents, its sensitivity to high-frequency abnormal scenarios is insufficient. Therefore, model optimization is needed to further improve the model's anomaly recognition capabilities.

[0033] S3. Convert the positive and negative sample pairs in the historical document anomaly type data into feature representation vectors, calculate the feature distance between similar and dissimilar samples in the positive and negative sample pairs based on the feature representation vectors, and optimize the model parameters in the enterprise business model based on the feature distance to obtain an anomaly identification optimization model.

[0034] In this embodiment of the invention, the feature representation vector is a high-dimensional vector obtained by the anomaly detection model after extracting features from the document sample. It can quantitatively represent the textual and structural features of the document. For example, for an anomaly document with altered amounts, its feature representation vector will show a significant difference from that of a normal document in the "amount field feature" dimension.

[0035] In detail, the historical document anomaly data includes manually labeled normal documents and various types of abnormal documents. For example, compliant purchase invoices are used as positive samples, and abnormal invoices where the electronic invoice header does not match the contract subject are used as negative samples. These are combined in a 1:1 ratio to form positive and negative sample pairs. A positive and negative sample pair refers to a training sample combination consisting of normal and abnormal document samples, with normal documents as positive samples and abnormal documents as negative samples. The sample size for each high-frequency anomaly scenario (such as conflicts between uppercase and lowercase amounts, approval dates earlier than reimbursement dates, etc.) is no less than 1000 entries to ensure training effectiveness. Using the feature extraction module of the enterprise business model, feature parsing is performed on each document in the positive and negative sample pairs to extract textual features (such as supplier name and invoice header) and structural features (such as amount and approval date), and these features are mapped into high-dimensional feature representation vectors.

[0036] Specifically, feature distance refers to the similarity measure between the feature representation vectors of different samples, specifically two measures: average similarity (the average closeness between similar sample documents) and average difference (the average distance between dissimilar sample documents). These two indicators together quantify the model's current discriminative ability. The feature distance between similar samples (both normal or both abnormal documents) should be as small as possible, while the feature distance between dissimilar samples (normal and abnormal documents) should be as large as possible. Feature distance is calculated using a contrastive learning loss function. The feature representation vectors of positive and negative sample pairs are input into the loss function, and the model parameters are adjusted using a gradient descent algorithm to narrow the feature distance between similar samples and widen the feature distance between dissimilar samples. For example, for abnormal samples of "duplicate reimbursement," the feature distance between them and normal reimbursement documents will significantly increase after optimization, thereby strengthening the model's ability to distinguish abnormal features and solving the problem of traditional models being insensitive to subtle abnormal features. In addition to the contrastive learning loss function, the triplet loss function can be used to construct sample triplets with "normal documents - similar abnormal documents - different abnormal documents" to optimize the model's ability to distinguish different types of abnormalities, which can also improve the sensitivity to subtle abnormal features.

[0037] In this embodiment of the invention, the anomaly identification optimization model refers to a model that, after model optimization, can accurately identify high-frequency anomaly scenarios in enterprises.

[0038] In this embodiment of the invention, reference is made to Figure 3 As shown, the step of using the historical document anomaly type data to optimize the model parameters in the enterprise business model based on the feature distance to obtain an anomaly identification optimization model includes: S31. Calculate the average similarity between the samples of the same type based on the feature representation vectors of the samples of the same type, and calculate the average difference between the samples of different types based on the feature representation vectors of the samples of different types. S32. When the average similarity is less than a preset similarity threshold and the average difference is greater than a preset difference threshold, a convergence signal for the feature distance is generated. S33. After the convergence signal is generated, the gradient of the feature distance with respect to the model parameters is calculated using a preset backpropagation algorithm, and the model parameters are updated using a preset optimizer based on the gradient. S34. Use the enterprise business model corresponding to the updated model parameters as the anomaly identification optimization model.

[0039] In detail, for known samples of the same type (e.g., all normal documents), the average similarity of their feature vectors is calculated using cosine similarity. A higher value indicates that samples of the same type are more clustered in the feature space. For samples of different types (one normal and one abnormal), the average difference between them is calculated using Euclidean distance. A higher value indicates that the two types of samples are separated better. These two indicators together quantify the separation quality of the model's feature space. Based on preset convergence conditions, it is determined whether further optimization is needed. Two thresholds are set: a similarity threshold represents how dispersed samples of the same type can be, and a difference threshold represents how far apart samples of different types should be separated at least. Only when both sufficient clustering of samples of the same type (average similarity less than the similarity threshold) and sufficient separation of samples of different types (average difference greater than the difference threshold) are met is the model's feature space considered to have reached an ideal state, and a convergence signal is issued to terminate this stage of optimization.

[0040] Specifically, if the convergence condition is not met, parameter updates are initiated to improve the model. At this point, the average similarity and average difference are combined into a differentiable loss function (e.g., the loss function encourages reducing the distance between similar samples and increasing the distance between dissimilar samples). Through backpropagation, the algorithm automatically calculates the gradient of this loss with respect to each parameter of the model, i.e., the direction and magnitude of adjustment for each parameter. Subsequently, the optimizer updates the model parameters based on these gradients, ensuring that in the next calculation, the feature vectors of similar samples are closer together and the feature vectors of dissimilar samples are further apart. This process iterates until the convergence condition is triggered. Ultimately, a set of optimized model parameters is obtained, defining a model that can clearly map different categories of documents to different regions of the feature space, thereby significantly improving discrimination accuracy. This model is then solidified as an anomaly detection optimization model that can be directly used in a production environment. The updated enterprise business model shows significantly improved sensitivity to subtle anomalies such as "semantic contradictions" and "cross-field logical conflicts," becoming an anomaly detection optimization model that achieves accurate identification of anomaly patterns, reducing the false negative and false positive rates to a reasonable range, and avoiding the problem of model lag in responding to new anomaly patterns.

[0041] Furthermore, while the constructed enterprise business model can adapt to the enterprise's routine business, it lacks the ability to identify high-frequency abnormal scenarios and cannot effectively deal with risks such as false invoices and duplicate reimbursements. Therefore, it is necessary to improve the model's abnormal sensitivity through specialized model optimization. The abnormal identification optimization model needs to comprehensively judge the abnormal situation of the document through a dual-modal fusion algorithm. Therefore, it is necessary to extract the abnormal probability of text and structural modalities separately to provide a basis for the final decision.

[0042] S4. Analyze the semantic and coding anomalous modalities between text fields in the preset target business document using the anomaly recognition optimization model, and output the text modal anomaly probability of the target business document for the semantic and coding anomalous modalities using the activation function in the anomaly recognition optimization model.

[0043] In this embodiment of the invention, the target business document refers to the enterprise document to be reviewed that requires anomaly identification, including various payment-related documents such as invoices, expense reports, and payment vouchers. Semantic anomaly modality refers to anomalies where there are logical conflicts between text fields, such as a supplier named "Company A" but an invoice header of "Company B"; coding anomaly modality refers to anomalies where the document coding does not conform to industry standards or enterprise standards, such as an electronic invoice's region code not matching the actual invoicing location.

[0044] In this embodiment of the invention, the step of analyzing the semantic and coding anomaly modalities between text fields in a preset target business document using the anomaly identification optimization model includes: Extract the content of multiple text fields from the target business document, and use the word segmenter of the anomaly recognition optimization model to segment the content of the multiple text fields to obtain multiple target word segments; The logical correlation between the multiple target word segments is calculated based on their vector representations in the semantic space of the anomaly detection optimization model. The target word segmentation sequence whose logical correlation is lower than a preset correlation threshold is determined to be a semantically abnormal modality; The code field containing the preset encoding format rules is extracted from the target business document. The code field is parsed according to the encoding structure knowledge learned by the anomaly recognition optimization model to obtain the conflict result between the constituent elements of the code field and the field information of the target business document. The code field with the conflict result is determined as the encoding anomaly mode.

[0045] In detail, key textual description fields, such as "supplier name" and "invoice header," are extracted from the target document. The tokenizer built into the anomaly detection optimization model is used to segment the text content (such as supplier name, invoice header, and handwritten remarks) in the electronic document and convert it into a token sequence that the model can recognize. The text content such as supplier name, invoice header, handwritten remarks, and coding information in the document is split into segments one by one. For example, "XX Technology Co., Ltd." is split into corresponding token combinations to form the target word segmentation sequence. In this way, unstructured text content is transformed into structured input data, enabling the model to directly process natural language text.

[0046] Specifically, the target word segmentation sequence is mapped into the model's high-dimensional semantic space, transforming it into vectors rich in contextual information. The model quantifies the semantic consistency of these vectors by calculating the logical correlation between them (such as cosine similarity). For example, the vector correlation between the fields "supplier name" and "invoice header" would be very low. Alternatively, the model's self-attention mechanism can be used to analyze the logical consistency between text fields. The self-attention mechanism is the core module in the model used to capture the correlation between text fields. It can calculate the attention weights of different text fields and then analyze whether their logic is consistent. For example, it can check whether "supplier name" and "invoice header" match, or whether "contract number in remarks" and "related contract subject" correspond. If the correlation is lower than a preset correlation threshold, it indicates that there is a descriptive contradiction or logical inconsistency between the fields, and the system will mark it as a semantically abnormal modality, such as "the supplier and the payee do not match".

[0047] Furthermore, parallel execution of structured coding compliance verification locates and extracts coded fields with fixed formats from the documents, such as invoice numbers and social credit codes. The model, pre-trained on massive amounts of data, has internalized the general rules governing these codes (e.g., invoice numbers contain implicit information such as region, year, and batch number). For coding information specific to electronic documents (e.g., the "region code + year code + sequence code" rule for electronic invoice numbers), the model fine-tunes its learned coding parsing capabilities to identify "coding format anomalies" (e.g., region code not matching the actual invoice location). The anomaly identification optimization model, based on parsing syntax, decomposes the constituent elements of the code and compares them with other explicit fields in the document (e.g., invoice address, registration date). If a conflict is found (e.g., the region code implicit in the invoice code does not match the province of the transaction address filled in on the document), a conflict result is generated, and the coded field is identified as a coding anomaly modality, such as "invoice region code conflicts with transaction location." The anomaly detection optimization model completes a three-dimensional audit of a business document in terms of content logic and format specifications by simultaneously outputting diagnostic results in two dimensions: semantic anomaly modality and coding anomaly modality. This enables automated auditing to not only detect explicit textual errors, but also identify hidden false information that is concealed by compliant formats.

[0048] In this embodiment of the invention, the text modality anomaly probability refers to the possibility of judging its anomaly from the dimension of document text content. The value ranges from 0 to 1, and the higher the value, the greater the risk of anomaly in the text dimension.

[0049] In detail, semantic anomalies (such as logical contradictions between fields) and coding anomalies (such as coding rule conflicts) are not simply yes / no flags, but are transformed into numerical feature representations of strength or confidence. For example, the degree of deviation of logical relevance from the threshold, the severity level of coding conflicts, etc., these numerical values ​​together constitute a comprehensive feature vector characterizing text anomalies. This comprehensive feature vector is input into a dedicated neural network layer (usually a fully connected layer) in the anomaly detection optimization model. The weight parameters learned internally by the dedicated neural network layer assign different importance to anomaly evidence from different sources and of different types. For example, the model may consider "mismatch between supplier and invoice header" to be more serious than "slightly non-standard remarks format".

[0050] Specifically, by performing a weighted summation and nonlinear transformation on the input features corresponding to the comprehensive feature vector, this layer calculates a comprehensive score representing the degree of text anomaly. This comprehensive score is processed by an activation function (usually the Sigmoid function). The core function of the Sigmoid function is to compress and smoothly map the input score from any range to a fixed probability interval between 0 and 1. Its output value is the text modality anomaly probability. The closer this value is to 1, the higher the probability that the model judges it to be an anomaly after integrating all text clues; the closer this value is to 0, the more normal and credible the text content appears. Thus, the model achieves intelligent decision-making from identifying specific problems to outputting an overall risk assessment.

[0051] Furthermore, based on the severity and frequency of identified semantic contradictions and encoding format anomalies, a text modality anomaly probability P_text (ranging from 0 to 1, with higher values ​​indicating a greater likelihood of text dimension anomalies) is generated. For example, the probability of a single minor semantic contradiction might be 0.4, while the probability of a severe encoding format anomaly accompanied by multiple semantic contradictions might be 0.9. This quantifies the anomaly risk of the text dimension, providing accurate numerical basis for subsequent bimodal fusion.

[0052] S5. Construct a graph structure for the target business document based on the document entities and the logical relationships between the document entities. Use a preset graph neural network to perform node feature propagation and edge weight learning on the graph structure to obtain node abnormal modes and edge abnormal modes. Output the structural mode abnormality probability of the target business document for the node abnormal modes and the edge abnormal modes through the classifier in the graph neural network.

[0053] In this embodiment of the invention, the graph structure is composed of nodes (representing entities or attributes) and edges (representing relationships between entities), which can intuitively and formally express the interrelationships of various elements in the document.

[0054] In detail, key document entities are identified and extracted from the target business documents as nodes in the graph. These entities include crucial information such as amount, date, approver, and associated contract number; these entities are the graph nodes. Based on pre-defined business rules and logic, relationships are established between these entities as edges in the graph. These relationships define the logical constraints that need to be verified during the approval process. For example, a "should be earlier than" relationship edge can be established between the "reimbursement date" node and the "approval date" node; a "should conform to" relationship edge can be established between the "payment amount" node and the "contract amount" node. An edge not only connects two nodes but can also carry the attributes of the relationship; that is, the logical relationship between document entities is the target edge.

[0055] Specifically, using a Graph Neural Network (GNN), the document entities (amount, date, approver, associated contract number) of electronic documents are treated as nodes, and the logical relationships between document entities are treated as edges (e.g., "approval date later than reimbursement date" is a normal edge, and "approval date earlier than reimbursement date" is an abnormal edge). This transforms the structural features of the document into a node-edge structure that can be processed by the GNN. Based on the determined graph nodes and target edges, a graph structure modeling tool is used to construct the topology graph of the target business document, forming a graph structure containing multiple nodes and edges, thereby integrating the scattered structural fields into structured graph data.

[0056] In this embodiment of the invention, the Graph Neural Network (GNN) is an algorithm used to process structured data and model the logical relationships between nodes and edges, capable of analyzing the logical consistency between document entities. A node anomalous modality refers to a significant deviation between the value or state of a document entity and the overall pattern of all other similar fields in the graph; an edge anomalous modality refers to a deviation between the expected business logic relationship between two fields.

[0057] In this embodiment of the invention, the step of using a preset graph neural network to perform node feature propagation and edge weight learning on the graph structure to obtain node anomalous modes and edge anomalous modes includes: Through the multi-layer information transmission mechanism of the graph neural network, for each node in the graph structure, the feature information of the edge associated with the node and the feature information of the neighboring nodes of the node are iteratively aggregated, and the edge weights corresponding to the associated edges are updated synchronously. After the information transmission process is completed, the final state vector of each node in the graph structure is obtained. The feature cluster center of the node is calculated based on the final state vector of all nodes. The final state vector of each node is compared with the feature cluster center to obtain the deviation of each node. Nodes whose deviation exceeds a preset node deviation threshold are identified, and the state of the nodes is determined to be an abnormal node mode; After the weight learning process is completed, the final weight value of each edge in the graph structure is obtained, and the final weight value of each edge is compared with a preset edge weight threshold. Edges whose final weight values ​​are lower than the preset edge weight threshold are identified, and the state of the edge is determined to be an edge abnormal mode.

[0058] In detail, based on the predefined initial feature vectors of each node and the predefined initial relational attributes of each edge in the graph structure, the initial feature vectors and the initial relational attributes are input into a preset graph neural network to complete the input state initialization of the graph neural network. Through the multi-layer information transmission mechanism in the graph neural network, the nodes and edges in the graph engage in multi-round dialogues. In each layer iteration, the following operations are performed: for each node in the graph structure, the node aggregates feature information from the edges connected to it and the neighboring nodes adjacent to it. That is, each node aggregates information from its connected edges and neighboring nodes, thereby updating its own state so that its feature vector is no longer isolated but integrated into the global relational context.

[0059] The edge weight values ​​corresponding to the edges and used to characterize the strength of the relationship are updated synchronously. After the information transmission process is completed, the final state vector of each node in the graph structure is obtained. Based on the final state vectors of all nodes of the same type, the feature cluster center of that type of node is calculated. The final state vector of each node is compared with the feature cluster center. Nodes whose deviation exceeds the preset node deviation threshold are identified, and the state of the node is determined as an abnormal node mode. After the weight learning process is completed, the final weight value of each edge in the graph structure is obtained. The final weight value of each edge is compared with the preset edge weight threshold. Edges whose final weight value is lower than the preset edge weight threshold are identified, and the logical conflict represented by the edge is determined as an abnormal edge mode.

[0060] Specifically, graph neural networks calculate the structural consistency score of documents through node feature propagation and edge weight learning. The structural consistency score is a comprehensive score calculated based on the consistency of the target propagation node's features and the reasonableness of the target edge features; a higher score indicates a more compliant structure. For example, a 5% deviation rate between the payment amount and the contractually agreed amount results in a higher structural consistency score, while a 20% deviation rate results in a lower score. For instance, when the "deviation rate between the payment amount and the contractually agreed amount" exceeds a preset threshold (e.g., 10%), the risk of structural anomalies increases. Graph neural networks use a node feature propagation mechanism to transmit the feature information of each node to adjacent nodes, achieving global fusion of node features to obtain the target propagation node. Simultaneously, through an edge weight learning algorithm, different weights are assigned based on the importance of the logical relationships between fields; for example, the edge weight for "amount relationship" is higher than that for "approver hierarchy relationship," resulting in target edge features. Based on the message passing mechanism of graph neural networks, the potential correlations between structural fields are deeply mined, improving the comprehensiveness of structural feature analysis.

[0061] In this embodiment of the invention, the structural modality anomaly probability refers to the possibility of judging its anomaly from the dimension of document structural features, and the value range is also 0 to 1.

[0062] In detail, the graph neural network aggregates discrete node and edge anomaly evidence into a global graph-level representation, which is then comprehensively evaluated by a classifier. The final output is a standardized probability value to quantify the anomaly risk of the entire document in the structural logic dimension. This involves pooling and converging (e.g., global average pooling) the final state vectors of all nodes, the final weights of edges, and the identified anomaly modalities (such as the proportion of anomalous nodes, the degree of weight decay on key edges), generating a fixed-length, high-dimensional graph-level feature vector. This graph-level feature vector is then fed into a dedicated classifier (typically one or more fully connected layers) at the top of the graph neural network. The classifier's internal parameters have learned during training how to assign different levels of attention or weight to different types of structural anomalies. For example, it might determine that a break in the core approval process (a serious edge anomaly) contributes more risk than a slight deviation in a single auxiliary field (a minor node anomaly). By performing complex weighted calculations and nonlinear transformations on the input features, this layer outputs a comprehensive score representing the overall degree of structural anomaly (i.e., a structural consistency score). This comprehensive score is mapped to a value between 0 and 1 using a sigmoid activation function, forming the final structural modality anomaly probability P_struct. The structural modality anomaly probability is based on the logical relationships and process rules between all fields, allowing the model to determine the likelihood of structural anomalies in the document. A probability closer to 1 indicates more severe process contradictions and logical conflicts (such as reversed approval flows or significant discrepancies between the amount and the contract); a probability closer to 0 indicates that all field relationships conform to the expected business rules.

[0063] Furthermore, extracting anomaly probabilities using both modalities is necessary because document anomalies may manifest in a single dimension (text content or structural features) or simultaneously in both dimensions. A single modality alone cannot comprehensively cover all anomaly scenarios. Therefore, it is necessary to capture semantic and encoded anomalies using the text modality and capture logical anomalies in the fields using the structural modality. This provides comprehensive anomaly data support for dual-modal fusion, ensuring the accuracy of the final recognition result.

[0064] S6. Perform dual-modal weighted fusion on the text modal anomaly probability and the structural modal anomaly probability to obtain the target anomaly probability of the target business document, and compare the target anomaly probability with a preset risk probability threshold to determine the risk type of the target financial document based on the comparison result.

[0065] In this embodiment of the invention, the target anomaly probability is the final anomaly probability obtained after dual-modal fusion, which comprehensively reflects the overall anomaly risk of the document.

[0066] In this embodiment of the invention, the step of performing bimodal weighted fusion of the text modality anomaly probability and the structural modality anomaly probability to obtain the target anomaly probability of the target business document includes: The text modality weight and structure modality weight corresponding to the target business document are determined based on the business type of the target business document; The text modality weights and text modality anomaly probabilities are weighted by a preset weighted voting mechanism to obtain the first target data, and the structural modality weights and structural modality anomaly probabilities are weighted by a preset weighted voting mechanism to obtain the second target data; The first target data and the second target data are linearly combined to obtain the target anomaly probability of the target business document.

[0067] In detail, the importance of text modal and structural modal is determined based on the business type of the target business document, and the weight allocation is adjusted accordingly. The model automatically adjusts the text modal weights based on the electronic document type. With structural modal weights ,satisfy For example, in payment vouchers involving contractual relationships, structural features (such as the matching degree between payment amount and contract amount) are more critical; therefore, the text modality weight is set to 0.4 and the structural modality weight to 0.6. In ordinary invoice recognition, textual features (such as invoice header and encoding format) are more important; therefore, the text modality weight is set to 0.6 and the structural modality weight to 0.4. Based on the preset document type-weight mapping rules, the model automatically matches the corresponding weights to adapt to the recognition needs of different types of documents, solving the problem that fixed weights cannot handle cross-type document recognition.

[0068] Specifically, the weighted voting mechanism refers to the rule for calculating the product of weights and corresponding modal anomaly probabilities. The first target data is the product of text modal weights and text modal anomaly probabilities; the second target data is the product of structural modal weights and structural modal anomaly probabilities. The linear combination, i.e., the summation of the first and second target data, yields the final anomaly probability. , By combining the anomaly risks from both textual and structural dimensions, a comprehensive and accurate final anomaly probability is obtained, thus solving the problem of the one-sidedness of single-modal recognition.

[0069] Furthermore, in addition to the weighted voting mechanism, an attention fusion mechanism can be adopted, which learns the feature importance of text and structural modalities through the attention layer and dynamically allocates fusion weights instead of fixed weight coefficients, making it suitable for scenarios with more complex relationships between document fields.

[0070] In this embodiment of the invention, the risk types include three categories: high-confidence abnormal documents, low-confidence documents pending review, and normal documents.

[0071] In this embodiment of the invention, determining the risk type of the target financial document based on the comparison results includes: When the comparison result shows that the target anomaly probability is higher than a preset first threshold, the risk type of the target financial document is determined to be a first confidence level anomaly document. When the comparison result shows that the target anomaly probability is greater than a preset second threshold and less than a preset first threshold, the risk type of the target financial document is determined to be a document requiring review at the second confidence level. When the comparison result shows that the target anomaly probability is lower than a preset second threshold, the risk type of the target financial document is determined to be a normal document.

[0072] Specifically, the first threshold is 0.9, and the second threshold is 0.3. These thresholds are set based on the enterprise's financial risk control requirements and empirical data from model training to ensure the rigor of the identification results. When the target anomaly probability is higher than 0.9, it indicates that the document has high-risk anomalies in both textual and structural dimensions, and is classified as a high-confidence anomaly document (first-confidence anomaly document). For example, a document with an anomaly probability of 0.95 can be directly classified as high-risk. When the target anomaly probability is between 0.3 and 0.9, it indicates that the document has some anomaly risk but does not meet the high-confidence standard, and is classified as a low-confidence document requiring further review (second-confidence document requiring further review), requiring manual review. For example, a document with an anomaly probability of 0.6 requires review by financial personnel. When the target anomaly probability is lower than 0.3, it indicates that the document is compliant in both textual and structural dimensions, and is classified as a normal document. For example, a document with an anomaly probability of 0.2 can be directly approved. Clearly defining the risk type of the document provides a clear decision-making basis for financial review.

[0073] Specifically, in addition to the semi-supervised pseudo-labeling strategy, an active learning strategy can be adopted, in which the model selects documents with the "highest uncertainty" (such as low confidence samples with P close to 0.5) for priority manual labeling. This can also expand the sample library with less manual cost and improve the generalization of the model.

[0074] Furthermore, based on bimodal anomaly probability, unified identification of cross-type documents is achieved through dynamic weighted fusion, ultimately outputting a clear document type and completing the closed loop of identification and judgment. This comprehensively solves the core problems in existing technologies, such as incomplete rule engine coverage, poor generalization of machine learning models, difficulty in identifying cross-type documents, and delayed anomaly response, achieving accurate and efficient identification of abnormal payment financial documents.

[0075] As can be seen, in the above solution, domain terminology training is conducted by collecting target business domain corpus data, enabling the pre-set large language model to accurately grasp the meaning of domain professional terms and document fields, laying the foundation for business adaptation. Then, the adapter layer of the business domain basic model is trained using the target enterprise's historical electronic document data. By freezing most of the model parameters, the training cost and overfitting risk are reduced, achieving personalized adaptation of the model to the enterprise's unique document styles and approval processes. Model optimization is carried out based on historical document anomaly type data, enhancing the model's sensitivity to subtle anomaly features and effectively reducing missed and false judgments. The three-level fine-tuning mechanism enhances the model's ability to learn new anomaly patterns of the enterprise. The anomaly recognition optimization model and graph neural network extract text modality and structural modality anomaly probabilities respectively, comprehensively covering the needs of unstructured information parsing and cross-field logical conflict recognition. The target anomaly probability is obtained through dual-modal weighted fusion and the document type is determined, dynamically adapting to the recognition focus of different types of documents, significantly improving the recognition accuracy of complex anomaly patterns, and accelerating the response speed to new anomaly patterns.

[0076] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] In one embodiment, a document recognition device based on a large language model is provided. This document recognition device 100 based on a large language model corresponds one-to-one with the document recognition method based on a large language model in the above embodiments. For example... Figure 4 As shown, the document recognition device 100 based on a large language model includes a domain terminology training module 101, an adapter layer training module 102, a model optimization module 103, a text modality anomaly probability analysis module 104, a structural modality anomaly probability analysis module 105, and a risk type recognition module 106. Detailed descriptions of each functional module are as follows: Domain terminology training module 101 is used to collect corpus data of the target business domain, and use the corpus data to train domain terminology on a preset large language model to obtain a basic business domain model. The adapter layer training module 102 is used to collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the business domain basic model to obtain the enterprise business model. Model optimization module 103 is used to convert positive and negative sample pairs in the historical document anomaly type data into feature representation vectors, calculate the feature distance between similar samples and dissimilar samples in the positive and negative sample pairs based on the feature representation vectors, and optimize the model parameters in the enterprise business model based on the feature distance to obtain the anomaly identification optimization model of the target enterprise. The text modality anomaly probability analysis module 104 is used to analyze the semantic anomaly modality and the coding anomaly modality between text fields in a preset target business document through the anomaly identification optimization model, and output the text modality anomaly probability of the target business document for the semantic anomaly modality and the coding anomaly modality through the activation function in the anomaly identification optimization model. The structural modality anomaly probability analysis module 105 is used to construct a graph structure of the target business document based on the document entities and the logical relationships between the document entities, and to use a preset graph neural network to perform node feature propagation and edge weight learning on the graph structure to obtain node anomaly modalities and edge anomaly modalities. The classifier in the graph neural network outputs the structural modality anomaly probability of the target business document for the node anomaly modalities and the edge anomaly modalities. The risk type identification module 106 is used to perform dual-modal weighted fusion of the text modal anomaly probability and the structural modal anomaly probability to obtain the target anomaly probability of the target business document, and compare the target anomaly probability with a preset risk probability threshold to determine the risk type of the target financial document based on the comparison result.

[0078] In one embodiment, the domain terminology training module 101, when performing domain terminology training on a preset large language model using the corpus data to obtain a business domain basic model, is used to: The corpus data is segmented into words, and each word in the segmented corpus data is replaced with a unique integer sequence number corresponding to each word in a preset vocabulary. The unique integer sequence number is used as the row index of the embedding matrix in the embedding layer of the large language model. The word vector corresponding to the unique integer sequence number in the embedding matrix is ​​located by the row index. The word vector is analyzed by self-attention mechanism and feedforward neural network to obtain a hidden state vector sequence containing contextual semantics. Perform linear classification on each word vector in the hidden state vector sequence to obtain the probability distribution of each word vector; Calculate the training loss value between the probability distribution of each word vector and the true probability value of the corresponding position of the word vector in the digit sequence; The large language model is backpropagated based on the training loss value, and the model parameters of the large language model are updated based on the backpropagation. When the training loss value is less than the preset loss threshold, the large language model corresponding to the updated model parameters is used as the basic model for the business domain.

[0079] In one embodiment, the adapter layer training module 102, when training the adapter layer in the business domain basic model using the historical electronic document data to obtain the enterprise business model, is used to: Freeze the target parameters in the business domain basic model, and use the parameters corresponding to the unfrozen adapter layer in the business domain basic model as model training parameters; The historical electronic document data is used to perform forward propagation calculation on the basic model of the business domain to obtain the probability distribution of each document in the historical electronic document data for different document categories, and the category with the highest probability distribution is selected as the category label of each document. The loss value between the category label and the true category label of each document is calculated by a preset loss function, and the model training parameters are updated by a preset low-rank adaptation algorithm. When the loss value is less than the preset convergence threshold, the adapter layer is updated with the updated model training parameters, and the updated adapter layer is integrated into the business domain basic model to obtain the enterprise business model.

[0080] In one embodiment, the model optimization module 103, when performing optimization of the model parameters in the enterprise business model based on the feature distance to obtain an anomaly identification optimization model, is used to: The average similarity between the samples of the same type is calculated based on the feature representation vectors of the samples of different types, and the average difference between the samples of different types is calculated based on the feature representation vectors of the samples of different types. When the average similarity is less than a preset similarity threshold and the average difference is greater than a preset difference threshold, a convergence signal for the feature distance is generated. After the convergence signal is generated, the gradient of the feature distance with respect to the model parameters is calculated using a preset backpropagation algorithm, and the model parameters are updated using a preset optimizer based on the gradient. The updated model parameters correspond to the enterprise business model, which is then used as the anomaly detection optimization model.

[0081] In one embodiment, the text modality anomaly probability analysis module 104, when performing the analysis of semantic anomaly modalities and encoding anomaly modalities between text fields in a preset target business document using the anomaly identification optimization model, is used to: Extract the content of multiple text fields from the target business document, and use the word segmenter of the anomaly recognition optimization model to segment the content of the multiple text fields to obtain multiple target word segments; The logical correlation between the multiple target word segments is calculated based on their vector representations in the semantic space of the anomaly detection optimization model. The target word segmentation sequence whose logical correlation is lower than a preset correlation threshold is determined to be a semantically abnormal modality; The code field containing the preset encoding format rules is extracted from the target business document. The code field is parsed according to the encoding structure knowledge learned by the anomaly recognition optimization model to obtain the conflict result between the constituent elements of the code field and the field information of the target business document. The code field with the conflict result is determined as the encoding anomaly mode.

[0082] In one embodiment, the structural modality anomaly probability analysis module 105, when performing node feature propagation and edge weight learning on the graph structure using a preset graph neural network to obtain node anomaly modes and edge anomaly modes, is further configured to: Through the multi-layer information transmission mechanism of the graph neural network, for each node in the graph structure, the feature information of the edge associated with the node and the feature information of the neighboring nodes of the node are iteratively aggregated, and the edge weights corresponding to the associated edges are updated synchronously. After the information transmission process is completed, the final state vector of each node in the graph structure is obtained. The feature cluster center of the node is calculated based on the final state vector of all nodes. The final state vector of each node is compared with the feature cluster center to obtain the deviation of each node. Nodes whose deviation exceeds a preset node deviation threshold are identified, and the state of the nodes is determined to be an abnormal node mode; After the weight learning process is completed, the final weight value of each edge in the graph structure is obtained, and the final weight value of each edge is compared with a preset edge weight threshold. Edges whose final weight values ​​are lower than the preset edge weight threshold are identified, and the state of the edge is determined to be an edge abnormal mode.

[0083] In one embodiment, the risk type identification module 106, when performing bimodal weighted fusion of the text modality anomaly probability and the structural modality anomaly probability to obtain the target anomaly probability of the target business document, is used to: The text modality weight and structure modality weight corresponding to the target business document are determined based on the business type of the target business document; The text modality weights and text modality anomaly probabilities are weighted by a preset weighted voting mechanism to obtain the first target data, and the structural modality weights and structural modality anomaly probabilities are weighted by a preset weighted voting mechanism to obtain the second target data; The first target data and the second target data are linearly combined to obtain the target anomaly probability of the target business document.

[0084] This invention provides a document recognition device based on a large language model. It trains the device on domain terminology by collecting target business domain corpus data, enabling the pre-set large language model to accurately grasp the meaning of domain-specific terms and document fields, laying the foundation for business adaptation. Then, it trains the adapter layer of the business domain basic model using historical electronic document data from the target enterprise. By freezing most model parameters, it reduces training costs and the risk of overfitting, achieving personalized adaptation of the model to the enterprise's unique document styles and approval processes. Based on historical document anomaly type data, it optimizes the model, enhancing its sensitivity to subtle anomaly features and effectively reducing missed and false positives. A three-level fine-tuning mechanism strengthens the model's ability to learn new anomaly patterns from the enterprise. Through anomaly recognition optimization model and graph neural network, it extracts textual and structural modal anomaly probabilities respectively, comprehensively covering the needs of unstructured information parsing and cross-field logical conflict recognition. The target anomaly probability is obtained through dual-modal weighted fusion, and document type determination is completed, dynamically adapting to the recognition focus of different document types, significantly improving the recognition accuracy of complex anomaly patterns and accelerating the response speed to new anomaly patterns.

[0085] Specific limitations regarding the document recognition device based on large language models can be found in the limitations of the document recognition method based on large language models mentioned above, and will not be repeated here. Each module in the aforementioned document recognition device based on large language models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0086] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side document recognition method based on a large language model.

[0087] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a document recognition method based on a large language model.

[0088] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Collect corpus data of the target business domain, and use the corpus data to train a pre-set large language model with domain terms to obtain a basic model of the business domain. Collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the business domain basic model to obtain the enterprise business model; The positive and negative sample pairs in the historical document anomaly type data are converted into feature representation vectors. The feature distance between similar and dissimilar samples in the positive and negative sample pairs is calculated based on the feature representation vectors. The model parameters in the enterprise business model are optimized based on the feature distance to obtain an anomaly identification optimization model. The anomaly detection optimization model analyzes the semantic and coding anomaly modes between text fields in the preset target business document, and outputs the text mode anomaly probability of the target business document for the semantic and coding anomaly modes through the activation function in the anomaly detection optimization model. Based on the document entities of the target business document and the logical relationships between the document entities, a graph structure of the target business document is constructed. A preset graph neural network is used to perform node feature propagation and edge weight learning on the graph structure to obtain node abnormal modes and edge abnormal modes. The classifier in the graph neural network outputs the structural mode abnormality probability of the target business document for the node abnormal modes and the edge abnormal modes. The text modal anomaly probability and the structural modal anomaly probability are fused using a dual-modal weighted method to obtain the target anomaly probability of the target business document. The target anomaly probability is then compared with a preset risk probability threshold, and the risk type of the target financial document is determined based on the comparison result.

[0089] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Collect corpus data of the target business domain, and use the corpus data to train a pre-set large language model with domain terms to obtain a basic model of the business domain. Collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the business domain basic model to obtain the enterprise business model; The positive and negative sample pairs in the historical document anomaly type data are converted into feature representation vectors. The feature distance between similar and dissimilar samples in the positive and negative sample pairs is calculated based on the feature representation vectors. The model parameters in the enterprise business model are optimized based on the feature distance to obtain an anomaly identification optimization model. The anomaly detection optimization model analyzes the semantic and coding anomaly modes between text fields in the preset target business document, and outputs the text mode anomaly probability of the target business document for the semantic and coding anomaly modes through the activation function in the anomaly detection optimization model. Based on the document entities of the target business document and the logical relationships between the document entities, a graph structure of the target business document is constructed. A preset graph neural network is used to perform node feature propagation and edge weight learning on the graph structure to obtain node abnormal modes and edge abnormal modes. The classifier in the graph neural network outputs the structural mode abnormality probability of the target business document for the node abnormal modes and the edge abnormal modes. The text modal anomaly probability and the structural modal anomaly probability are fused using a dual-modal weighted method to obtain the target anomaly probability of the target business document. The target anomaly probability is then compared with a preset risk probability threshold, and the risk type of the target financial document is determined based on the comparison result.

[0090] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0091] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0093] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0094] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A document recognition method based on a large language model, characterized in that, include: Collect corpus data of the target business domain, and use the corpus data to train a pre-set large language model with domain terms to obtain a basic model of the business domain. Collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the business domain basic model to obtain the enterprise business model; The positive and negative sample pairs in the historical document anomaly type data are converted into feature representation vectors. The feature distance between similar and dissimilar samples in the positive and negative sample pairs is calculated based on the feature representation vectors. The model parameters in the enterprise business model are optimized based on the feature distance to obtain an anomaly identification optimization model. The anomaly detection optimization model analyzes the semantic and coding anomaly modes between text fields in the preset target business document, and outputs the text mode anomaly probability of the target business document for the semantic and coding anomaly modes through the activation function in the anomaly detection optimization model. Based on the document entities of the target business document and the logical relationships between the document entities, a graph structure of the target business document is constructed. A preset graph neural network is used to perform node feature propagation and edge weight learning on the graph structure to obtain node abnormal modes and edge abnormal modes. The classifier in the graph neural network outputs the structural mode abnormality probability of the target business document for the node abnormal modes and the edge abnormal modes. The text modal anomaly probability and the structural modal anomaly probability are fused using a dual-modal weighted method to obtain the target anomaly probability of the target business document. The target anomaly probability is then compared with a preset risk probability threshold, and the risk type of the target financial document is determined based on the comparison result.

2. The document recognition method based on a large language model as described in claim 1, characterized in that, The step of using the corpus data to train a pre-defined large language model with domain terminology to obtain a basic business domain model includes: The corpus data is segmented into words, and each word in the segmented corpus data is replaced with a unique integer sequence number corresponding to each word in a preset vocabulary. The unique integer sequence number is used as the row index of the embedding matrix in the embedding layer of the large language model. The word vector corresponding to the unique integer sequence number in the embedding matrix is ​​located by the row index. The word vector is analyzed by self-attention mechanism and feedforward neural network to obtain a hidden state vector sequence containing contextual semantics. Perform linear classification on each word vector in the hidden state vector sequence to obtain the probability distribution of each word vector; Calculate the training loss value between the probability distribution of each word vector and the true probability value of the corresponding position of the word vector in the digit sequence; The large language model is backpropagated based on the training loss value, and the model parameters of the large language model are updated based on the backpropagation. When the training loss value is less than the preset loss threshold, the large language model corresponding to the updated model parameters is used as the basic model for the business domain.

3. The document recognition method based on a large language model as described in claim 1, characterized in that, The process of training the adapter layer in the business domain foundation model using the historical electronic document data to obtain the enterprise business model includes: Freeze the target parameters in the business domain basic model, and use the parameters corresponding to the unfrozen adapter layer in the business domain basic model as model training parameters; The historical electronic document data is used to perform forward propagation calculation on the basic model of the business domain to obtain the probability distribution of each document in the historical electronic document data for different document categories, and the category with the highest probability distribution is selected as the category label of each document. The loss value between the category label and the true category label of each document is calculated by a preset loss function, and the model training parameters are updated using a preset low-rank adaptation algorithm. When the loss value is less than the preset convergence threshold, the adapter layer is updated with the updated model training parameters, and the updated adapter layer is integrated into the business domain basic model to obtain the enterprise business model.

4. The document recognition method based on a large language model as described in claim 1, characterized in that, The step of optimizing the model parameters in the enterprise business model based on the feature distance to obtain an anomaly identification optimization model includes: The average similarity between the samples of the same type is calculated based on the feature representation vectors of the samples of different types, and the average difference between the samples of different types is calculated based on the feature representation vectors of the samples of different types. When the average similarity is less than a preset similarity threshold and the average difference is greater than a preset difference threshold, a convergence signal for the feature distance is generated. After the convergence signal is generated, the gradient of the feature distance with respect to the model parameters is calculated using a preset backpropagation algorithm, and the model parameters are updated using a preset optimizer based on the gradient. The updated model parameters correspond to the enterprise business model, which is then used as the anomaly detection optimization model.

5. The document recognition method based on a large language model as described in claim 1, characterized in that, The step of analyzing the semantic and encoding anomaly modalities between text fields in the preset target business document using the anomaly identification optimization model includes: Extract the content of multiple text fields from the target business document, and use the word segmenter of the anomaly recognition optimization model to segment the content of the multiple text fields to obtain multiple target word segments; The logical correlation between the multiple target word segments is calculated based on their vector representations in the semantic space of the anomaly detection optimization model. The target word segmentation sequence whose logical correlation is lower than a preset correlation threshold is determined to be a semantically abnormal modality; The code field containing the preset encoding format rules is extracted from the target business document. The code field is parsed according to the encoding structure knowledge learned by the anomaly recognition optimization model to obtain the conflict result between the constituent elements of the code field and the field information of the target business document. The code field with the conflict result is determined as the encoding anomaly mode.

6. The document recognition method based on a large language model as described in claim 1, characterized in that, The step of using a preset graph neural network to perform node feature propagation and edge weight learning on the graph structure to obtain node anomalous modes and edge anomalous modes includes: Through the multi-layer information transmission mechanism of the graph neural network, for each node in the graph structure, the feature information of the edge associated with the node and the feature information of the neighboring nodes of the node are iteratively aggregated, and the edge weights corresponding to the associated edges are updated synchronously. After the information transmission process is completed, the final state vector of each node in the graph structure is obtained. The feature cluster center of the node is calculated based on the final state vector of all nodes. The final state vector of each node is compared with the feature cluster center to obtain the deviation of each node. Nodes whose deviation exceeds a preset node deviation threshold are identified, and the state of the nodes is determined to be an abnormal node mode; After the weight learning process is completed, the final weight value of each edge in the graph structure is obtained, and the final weight value of each edge is compared with a preset edge weight threshold. Edges whose final weight values ​​are lower than the preset edge weight threshold are identified, and the state of the edge is determined to be an edge abnormal mode.

7. The document recognition method based on a large language model as described in claim 1, characterized in that, The step of performing bimodal weighted fusion of the text modal anomaly probability and the structural modal anomaly probability to obtain the target anomaly probability of the target business document includes: The text modality weight and structure modality weight corresponding to the target business document are determined based on the business type of the target business document; The text modality weights and text modality anomaly probabilities are weighted by a preset weighted voting mechanism to obtain the first target data, and the structural modality weights and structural modality anomaly probabilities are weighted by a preset weighted voting mechanism to obtain the second target data; The first target data and the second target data are linearly combined to obtain the target anomaly probability of the target business document.

8. A document recognition device based on a large language model, characterized in that, include: The domain terminology training module is used to collect corpus data of the target business domain, and use the corpus data to train the domain terminology of the preset large language model to obtain the basic model of the business domain. The adapter layer training module is used to collect historical electronic document data and historical document anomaly type data of the target enterprise, and use the historical electronic document data to train the adapter layer in the basic business domain model to obtain the enterprise business model. The model optimization module is used to convert positive and negative sample pairs in the historical document anomaly type data into feature representation vectors, calculate the feature distance between similar and dissimilar samples in the positive and negative sample pairs based on the feature representation vectors, and optimize the model parameters in the enterprise business model based on the feature distance to obtain an anomaly identification optimization model. The text modality anomaly probability analysis module is used to analyze the semantic anomaly modality and the coding anomaly modality between text fields in a preset target business document through the anomaly recognition optimization model, and output the text modality anomaly probability of the target business document for the semantic anomaly modality and the coding anomaly modality through the activation function in the anomaly recognition optimization model. The structural modality anomaly probability analysis module is used to construct a graph structure of the target business document based on the document entities and the logical relationships between the document entities, and to use a preset graph neural network to perform node feature propagation and edge weight learning on the graph structure to obtain node anomaly modalities and edge anomaly modalities. The classifier in the graph neural network outputs the structural modality anomaly probability of the target business document for the node anomaly modalities and the edge anomaly modalities. The risk type identification module is used to perform dual-modal weighted fusion of the text modal anomaly probability and the structural modal anomaly probability to obtain the target anomaly probability of the target business document, and compare the target anomaly probability with a preset risk probability threshold to determine the risk type of the target financial document based on the comparison result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the document recognition method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the document recognition method based on a large language model as described in any one of claims 1 to 7.

Images