A financial invoice review decision-making method, system, terminal and medium

Abstract

This application relates to the field of invoice review, specifically providing a financial invoice review decision-making method, system, terminal, and medium. The method includes: collecting multimodal financial and tax data such as text, images, and structured data, and preprocessing and aligning them; extracting features from the preprocessed data using a multimodal large-scale model, and performing feature fusion using a dynamic weight allocation algorithm based on the credibility of each modality, business priority, and historical feedback; obtaining risk probabilities through a risk identification neural network; comparing the risk probabilities with preset thresholds and rules in the review rule base, triggering review actions such as automatic approval, approval after recording risk anomalies, push to manual review, or activation of high-risk emergency plans; and finally, adjusting the model weight parameters based on manual feedback to achieve system self-optimization. This invention improves review decision-making efficiency, accuracy, and business adaptability.

Description

Technical Field

[0001] This application relates to the field of invoice review, specifically to a financial invoice review decision-making method, system, terminal, and medium. Background Technology

[0002] The auditing of financial and tax invoices naturally involves multimodal data, including text, images, and structured data. Traditional technical solutions mostly process and analyze single-modal data independently. For example, some systems focus on OCR recognition of invoice images, while others process structured expense reimbursement data. The lack of a unified framework to cross-validate these heterogeneous data sources leads to a heavy reliance on manual verification between different systems, resulting in low efficiency and a high risk of errors. Traditional automated auditing systems typically hard-code or embed business rules in configuration files. When business rules need adjustment, technical personnel must modify the code or configuration and redeploy the system, preventing quick responses from business personnel. Furthermore, these rule engines can usually only handle simple logical judgments and are ill-equipped to handle complex business scenarios requiring comprehensive assessment. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a financial invoice review and decision-making method, system, terminal, and medium, which improves review and decision-making efficiency, accuracy, and business adaptability.

[0004] In a first aspect, the technical solution of the present invention provides a financial document review and decision-making method, comprising the following steps:

[0005] Collect multimodal data related to the pending financial and tax invoices business, and establish a unified association identifier for the multimodal data, which includes text data, image data, and structured data;

[0006] The collected multimodal data is preprocessed to generate standardized text feature data, image feature data, and structured feature data;

[0007] The preprocessed multimodal data is input into a multimodal large model for processing, including: extracting feature vectors of corresponding feature data using encoders corresponding to the modalities to obtain text feature vectors, image feature vectors, and structured feature vectors; using a dynamic weight allocation algorithm to weight and fuse various feature vectors to generate a fused feature vector; and inputting the fused feature vector into a risk identification neural network to obtain the risk probability of the financial and tax invoices to be audited. Among these, the dynamic weight allocation algorithm dynamically calculates and allocates the fusion weights of each modality based on the credibility of each modality's data, predefined business risk priorities, and historical audit feedback results.

[0008] The risk probability is compared with the preset risk threshold in the audit rule base, and corresponding audit actions are triggered based on the comparison results, including one or more of the following: automatic approval, approval after recording risk anomalies, push to manual review, and triggering high-risk emergency plans.

[0009] Receive manual feedback on the review results and adjust the weight parameters in the multimodal large model based on the manual feedback.

[0010] Secondly, the technical solution of the present invention provides a financial invoice review and decision-making system, comprising:

[0011] The multimodal data acquisition module is used to collect multimodal data related to the financial and tax invoices to be reviewed, and to establish a unified association identifier for the multimodal data. The multimodal data includes text data, image data, and structured data.

[0012] The multimodal data preprocessing module is used to preprocess the collected multimodal data to generate standardized text feature data, image feature data, and structured feature data;

[0013] The risk probability generation module is used to input preprocessed multimodal data into a multimodal large model for processing. This includes: extracting feature vectors of corresponding feature data using encoders corresponding to each modality to obtain text feature vectors, image feature vectors, and structured feature vectors; using a dynamic weight allocation algorithm to weight and fuse various feature vectors to generate a fused feature vector; and inputting the fused feature vector into a risk identification neural network to obtain the risk probability of the financial and tax invoices to be audited. The dynamic weight allocation algorithm dynamically calculates and allocates the fusion weights of each modality based on the credibility of each modality's data, predefined business risk priorities, and historical audit feedback results.

[0014] The audit action triggering module is used to compare the risk probability with the preset risk threshold in the audit rule base, and trigger the corresponding audit action according to the comparison result, including one or more of the following: automatic approval, approval after recording risk anomalies, push to manual review, and triggering high-risk emergency plan.

[0015] The large model optimization and adjustment module is used to receive manual feedback on the review results and adjust the weight parameters in the multimodal large model based on the manual feedback.

[0016] Thirdly, the technical solution of the present invention provides a terminal, comprising:

[0017] The memory is used to store the decision-making process for reviewing financial documents;

[0018] A processor is configured to implement the steps of the financial document review decision method as described above when executing the financial document review decision procedure.

[0019] Fourthly, the present invention provides a computer-readable storage medium storing a financial invoice review decision program, wherein the financial invoice review decision program, when executed by a processor, implements the steps of the financial invoice review decision method as described in any of the above claims.

[0020] As can be seen from the above technical solutions, this application has the following advantages: By constructing a unified processing framework based on a multimodal large model, it can automatically collect, align, and fuse key information from different modalities synchronously, achieve cross-validation, reduce the reliance on manual verification between different systems, and improve audit efficiency and accuracy; by placing the audit logic post-processed on the quantitative risk probability output by the large model and combining it with a configurable rule base, financial personnel can directly adjust the mapping relationship between risk thresholds and audit actions according to business needs through a visual interface, without modifying the underlying code or restarting the system, thus enhancing responsiveness; by utilizing the feature extraction and semantic understanding capabilities of the multimodal large model, it can integrate the features and correlations of multi-source heterogeneous data and adapt to complex scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a financial invoice review and decision-making method provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic block diagram of a financial invoice review and decision-making system provided in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0027] Figure 1 This is a schematic flowchart of a financial invoice review and decision-making method provided by an embodiment of the present invention. Figure 1 The executing entity can be a financial document review and decision-making system. The financial document review and decision-making method provided in this embodiment of the invention is executed by a computer device; correspondingly, the financial document review and decision-making system runs on the computer device. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0028] like Figure 1 As shown, the method includes the following steps.

[0029] S1. Collect multimodal data related to the financial and tax invoices to be reviewed, and establish a unified association identifier for the multimodal data. The multimodal data includes text data, image data, and structured data.

[0030] S2 preprocesses the collected multimodal data to generate standardized text feature data, image feature data, and structured feature data.

[0031] S3, the preprocessed multimodal data is input into a multimodal large model for processing, including: extracting feature vectors of corresponding feature data using encoders corresponding to the modal to obtain text feature vectors, image feature vectors and structured feature vectors; using a dynamic weight allocation algorithm to weight and fuse various feature vectors to generate a fused feature vector; inputting the fused feature vector into a risk identification neural network to obtain the risk probability of the financial and tax invoices to be audited; wherein, the dynamic weight allocation algorithm dynamically calculates and allocates the fusion weights of each modality based on the credibility of each modal data, the predefined business risk priority and the historical audit feedback results.

[0032] S4. Compare the risk probability with the preset risk threshold in the audit rule base, and trigger the corresponding audit action according to the comparison result, including one or more of the following: automatic approval, approval after recording risk anomalies, push to manual review, and triggering high-risk emergency plan.

[0033] S5 receives manual feedback on the review results and adjusts the weight parameters in the multimodal large model based on the manual feedback.

[0034] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the specific implementation process in this embodiment, another financial document review and decision-making method is provided, which includes the following steps.

[0035] S101, build and train a multimodal large model.

[0036] The multimodal large model in this embodiment is a deep neural network, including a feature encoding layer, a feature fusion layer, and a risk identification layer, which is used to convert multimodal inputs into risk probabilities.

[0037] The feature encoding layer comprises three independent modal encoders to transform standardized feature data into high-dimensional feature vectors: a text encoder, an image encoder, and a structured data encoder. The text encoder employs a pre-trained model based on the Transformer architecture, such as BERT. It takes standardized text feature data as input, encodes it through multiple layers of self-attention, and outputs a fixed-dimensional text feature vector. The image encoder uses a Visual Transformer (ViT) model. It takes standardized image feature data as input, treats the image as a sequence of image patches, performs linear projection and positional encoding, and then extracts global visual features through Transformer layers, outputting an image feature vector. The structured data encoder uses a Multilayer Perceptron (MLP). It takes standardized structured feature data as input, maps the structured data to a feature space of the same dimension as other modalities through several fully connected layers and activation functions, and outputs a structured feature vector.

[0038] The feature fusion layer adaptively weights and fuses the feature vectors from the three modalities. It takes text feature vectors, image feature vectors, and structured feature vectors from the feature encoding layer as input. Based on the overall credibility of the data from each modality, predefined business risk priorities, and historical feedback, the fusion algorithm dynamically calculates and outputs the fusion weights for each modality. Then, the three feature vectors are weighted and summed according to the dynamic fusion weights to generate the final fused feature vector. This will be discussed in detail later and will not be elaborated upon here.

[0039] The risk identification layer is a multilayer perceptron classifier used for final risk assessment based on fused features. It contains at least one fully connected hidden layer and uses the sigmoid function as the activation function of the output layer. The input is the fused feature vector output by the feature fusion layer. The fused feature vector undergoes a non-linear transformation through the first fully connected layer to extract high-order risk features; then, it passes through the second fully connected layer to map the feature dimension to 1D; finally, the output value is compressed to the (0,1) interval by the sigmoid function to obtain the risk probability.

[0040] The constructed multimodal large model is trained by supervised learning, which optimizes all weight parameters in feature encoding, fusion, and risk identification. The specific steps include the following.

[0041] Step 1: Collect a large amount of historical tax and financial invoice case data to form a training dataset.

[0042] Each sample contains multimodal raw data of a complete case. Each case is reviewed and a true risk label is determined. This label is quantified according to the degree of risk and used as the regression target.

[0043] Step 2 involves preprocessing the multimodal data of the training samples before inputting it into the multimodal model to be trained. The data passes through a feature encoding layer, a feature fusion layer, and a risk identification layer, ultimately calculating the risk probability predicted by the model. The data preprocessing will be detailed later and will not be elaborated upon here.

[0044] Step 3: Calculate the difference between the model's predicted values ​​and the actual risk labels using the binary cross-entropy loss function. The loss function is defined as follows:

[0045]

[0046] in, This is the predicted value, i.e., the predicted risk probability output in step 2. This is a true risk label.

[0047] For batch training, calculate the average loss of all samples within a batch.

[0048] Step 4, backpropagation and parameter update.

[0049] Using the backpropagation algorithm, starting from the output of the loss function, the gradient of the loss with respect to all trainable parameters in the model is calculated layer by layer along the model computation graph. The trainable parameters include the weights and biases of each encoder, the dynamic weight calculation unit of the fusion layer, and the risk identification MLP.

[0050] Use a gradient descent optimization algorithm (such as the Adam optimizer) to iteratively update the model parameters based on the calculated gradients. Repeat steps 2 and 4 multiple times across the training dataset until the model loss function converges to a predetermined threshold or the performance no longer significantly improves on the validation set, thus completing the initial training of the model.

[0051] S102, collect multimodal data related to the financial and tax invoices to be reviewed, and establish a unified association identifier for the multimodal data.

[0052] Multimodal data includes text data, image data, and structured data.

[0053] The text data is connected to the OA system via HTTP API to parse the expense report JSON. The JSON fields include reimburse_id (unique identifier of expense report), emp_name (employee name), dept (department), amount (expense amount), and travel_date (travel date).

[0054] Image data: Invoice images uploaded via FTP to the OA system, named according to the rule "reimburse_id_file type.format", and stored in the MySQL index table data_collect_xx.

[0055] Structured data: The ERP system's financial ledger is synchronized periodically (every 10 minutes) via JDBC, and transaction data is obtained by calling the bank's API. The ERP system's financial ledger fields include emp_id (employee ID) and remaining_budget (monthly remaining budget), while the transaction data fields include trans_amount (transaction amount) and payee (payee).

[0056] S103 performs preprocessing on multimodal data.

[0057] The collected multimodal data is preprocessed, including data cleaning, format standardization, key entity extraction and anomaly detection, and the multimodal data is aligned to finally generate standardized text feature data, image feature data and structured feature data.

[0058] S103.1, Data Cleaning and Format Standardization.

[0059] For text data, remove meaningless special characters, and standardize the date and amount formats.

[0060] For image data, Gaussian filtering is used for noise reduction, Hough transform-based image tilt correction is performed, and the image resolution is unified to a preset standard.

[0061] For structured data, handle null values, standardize date and amount formats, and ensure the correctness of data types.

[0062] S103.2, Key Entity Extraction and Confidence Output.

[0063] This step calls the corresponding recognition model to extract the key entities necessary for business review from each modal data. These key entities are the target entities, and the confidence level output by the model is obtained simultaneously.

[0064] For text-based data, the cleaned text data is input into the Named Entity Recognition (NER) model to extract key entities, including expense reimbursement amounts, business trip dates, department names, and possibly project codes. The NER model employs a BERT-tuned NER model, which outputs a recognition confidence score for each successfully identified entity. The NER model includes a Softmax layer at the end of the sequence tags to calculate the probability that each token belongs to each entity tag. The average probability is then used to aggregate the probabilities into a single entity score composed of consecutive identified tokens.

[0065] For image modalities, the preprocessed image is input into the OCR engine to extract key entities, including invoice amount, invoice date, seller name, and seal clarity. The OCR engine simultaneously outputs a recognition confidence score for each successfully recognized entity; this confidence score indicates the OCR engine's degree of certainty regarding the entity's recognition result. While performing character recognition, the OCR engine generates a confidence score for each recognized text field, i.e., the entity, based on the probability output of its internal classification or sequence model.

[0066] For structured data modalities, predefined data validation rules are invoked for logical validation. For example, for "budget balance data consistency," the confidence level is calculated based on the ratio or degree of difference between the budget amount and the reimbursement amount using a preset mapping function. The mapping function is: Budget Consistency Confidence = max(0, min(1, (absolute value of the difference between the reimbursement amount in the reimbursement form and the current available budget balance / preset tolerance threshold))). This means that when the reimbursement amount equals the budget balance, the difference is 0 and the confidence level is 1. When the reimbursement amount exceeds the budget balance, but the excess is within the tolerance threshold, the confidence level linearly decreases from 1 to a value greater than 0. When the amount by which the reimbursement amount exceeds the budget balance reaches or exceeds the tolerance threshold, the confidence level drops to 0. For example, for "bank statement matching status," the confidence level is calculated based on the similarity of the amount, time, and payee name of the matching items in the bank statement records. The similarity is calculated using cosine similarity.

[0067] S103.3, Multimodal data alignment.

[0068] Using the unique identifier of tax invoices as the association key, data records of three modalities—text, images, and structured data—after cleaning and entity extraction are associated to generate a unified data alignment table.

[0069] S103.4, Generate standardized feature data.

[0070] The data processed and aligned in the above steps is organized into a standardized structure that includes the key entities of each modality, as well as the confidence level of each key entity generated during the extraction and verification process, for use in the subsequent calculation of the overall confidence level.

[0071] S104 inputs the preprocessed multimodal data into the multimodal large model for processing and output of risk probabilities.

[0072] The multimodal large model's processing of multimodal data includes: extracting feature vectors of corresponding feature data using encoders corresponding to the modalities to obtain text feature vectors, image feature vectors, and structured feature vectors; using a dynamic weight allocation algorithm to weight and fuse various feature vectors to generate a fused feature vector; and inputting the fused feature vector into a risk identification neural network to obtain the risk probability of the tax invoices to be audited.

[0073] The process of extracting feature vectors using the encoder is described in step S101 and will not be repeated here.

[0074] After obtaining the feature vectors, a dynamic weight allocation algorithm is used to weight and fuse various feature vectors to generate a fused feature vector. The dynamic weight allocation algorithm dynamically calculates and allocates the fusion weights of each modality based on the credibility of each modality data, the predefined business risk priority, and the historical audit feedback results. Specifically, it includes the following steps.

[0075] S104.1, a basic weight is preset for each modality based on the predefined business risk priority.

[0076] S104.11, based on predefined rules related to the review of financial and tax invoices, generates an evaluation vector for text, image and structured data modalities respectively.

[0077] The assessment vector is used to quantify the business risk priority of each modality, and its dimensions include at least accuracy criticality, tampering difficulty, and historical error rate. Accuracy criticality indicates the degree to which the modality data is decisive for the final audit conclusion; tampering difficulty indicates how difficult it is for the modality data to be modified without authorization; and historical error rate indicates the statistical error rate of the modality data in historical audits.

[0078] In the initial phase, historical audit data and its final manual audit results are collected. By analyzing this data, the contribution of each modality to the correct identification of risks is quantified.

[0079] Accuracy Criterion The score is calculated as the proportion of audit results that match the actual labels when the modality data is judged to be "highly reliable," and is expressed as .

[0080]

[0081] In other words, a molecule refers to a mode. The total number of cases judged as "highly reliable" by the system and whose final review results are correct, with the denominator being modality. The total number of cases that were judged as "highly reliable" by the system.

[0082] For example, in 1000 cases where the image is clear, the system correctly judges 980 times, then the image modality... Among them, , These represent text modality, image modality, and structured data modality, respectively.

[0083] Historical error rate The score is calculated by directly determining the error rate of the modality data recognition result and converting it into a percentage score, expressed as follows:

[0084]

[0085] in, finger mode The statistical error rate in historical audits is calculated as follows: within the statistical period, the proportion of the number of cases in which the data or judgment provided by the modality was ultimately proven to be wrong or led to wrong decisions to the total number of cases in which the modality was used or referenced by the system.

[0086] Historical error rate It's a reverse scoring system; the lower the error rate, the higher the score. The higher the value, the better the historical performance of the mode and the more reliable it is.

[0087] Difficulty of tampering The scoring is calculated by combining factors such as the system authentication level of the data source and the integrity of the data logs, using a fixed classification scoring table. For example: direct data interface source = 95 points, encrypted third-party transmission = 80 points, user-uploaded files = 60 points.

[0088] For each modality, a percentage score is applied to the aforementioned dimensions to form an evaluation vector, denoted as follows: .

[0089] S104.12, based on a preset dimension weight vector, the evaluation vector of each modality is weighted and linearly combined to calculate the risk priority score of each modality; wherein, each weight component in the dimension weight vector corresponds to an evaluation dimension, and the sum of all weight components is 1.

[0090] Using a weighted summation method, the scores from each dimension are aggregated to calculate the comprehensive risk priority score for each modality, expressed as follows:

[0091]

[0092] in, , , These are the weighting coefficients for each dimension.

[0093] Accuracy criticality weight reflects the company's requirements for the correctness of audit results and is generally given the highest or higher priority.

[0094] The weighting of the difficulty of tampering reflects a company's requirements for data authenticity and fraud prevention capabilities.

[0095] Historical error rate weights reflect a company's requirements for system stability and predictability.

[0096] The specific values ​​of the weights are initially predefined based on expert experience, and then optimized based on data. That is, after running for a period of time, the optimal weight combination can be found using grid search cross-validation. The optimization goal is to maximize the overall performance of the intelligent review system, such as minimizing the overall false positive loss. The historical dataset is divided into training and validation sets. Different weight combinations are used to run the system on the training set, and the review performance is evaluated on the validation set. Finally, the weight set that optimizes the performance metrics is selected as the final configuration.

[0097] S104.13, normalize the risk priority, including calculating the ratio of the risk priority score of each modality to the sum of the risk priority scores of all modalities. The resulting ratio is the preset basic weight of that modality, and the sum of the preset basic weights of all modalities is 1.

[0098] The risk priority scores for each modality are normalized to obtain the preset basic weights for each modality, as follows:

[0099]

[0100] S104.2, set a credibility threshold for each modality, which is configured based on the current historical audit feedback results.

[0101] S104.21 Collect all audit case data generated within the operating cycle or within a preset historical period, wherein each data record includes at least the modality type, the overall credibility output by the modality when processing the corresponding case, and the audit action of the case.

[0102] Collect data from all audit cases. The dataset includes a unique case identifier, modality type, overall credibility, and audit action.

[0103] S104.22, from the full volume of audit case data, select cases whose audit action is to push them to manual review to form a strongly labeled dataset.

[0104] S104.23 For each modality in the strongly labeled dataset, it is divided into different confidence intervals according to its overall confidence level; for each confidence interval, the misclassification rate of the modality in that interval is calculated, and the misclassification rate is the ratio of the number of misclassified cases in the interval to the total number of cases.

[0105] The confidence interval [0,1] is divided into H equal-width intervals. For the h-th interval... Calculate the misclassification rate of the data within this interval. , is represented as ,

[0106]

[0107] in, It is modal The overall credibility falls within the range Total number of cases within, This refers to the number of cases that were manually reviewed and determined to be "misjudged" out of the total number of cases.

[0108] S104.24, based on the preset acceptable maximum false positive rate, traverse each confidence interval, and configure the lower limit of the minimum confidence interval that satisfies the condition that the statistical false positive rate is not greater than the acceptable maximum false positive rate as the confidence threshold of this modality.

[0109] Based on business requirements, set a maximum acceptable false positive rate for the system. Modality Credibility threshold Configured as:

[0110]

[0111] In other words, the boundary value of the minimum confidence interval that satisfies the condition that its statistical misclassification rate is no greater than the highest acceptable misclassification rate is configured as the updated confidence threshold of this modality, where the boundary value refers to the minimum value of this confidence interval.

[0112] S104.3 Calculate the overall credibility of each modality in the current audit case. This overall credibility is the aggregation of the confidence of each target identification item within the modality.

[0113] S104.31, for each modality in the current audit case, obtain the confidence level of its internal target recognition items after processing by the corresponding recognition model during its preprocessing; the target recognition items of the image modality include invoice amount, invoice date, seller name, and seal clarity, and the recognition model is an OCR engine; the target recognition items of the text modality include reimbursement amount, business trip date, and department name, and the recognition model is a named entity recognition model; the target recognition items of the structured data modality include budget balance data consistency and bank transaction matching status, and the recognition model is data verification rules.

[0114] The calculation of confidence level is detailed in step S103 of the preprocessing procedure, and will not be repeated here.

[0115] S104.32, For each modality, assign weights to each target identification item based on its business importance in the financial and tax audit.

[0116] An optional implementation method is to use a structured approach based on the analytic hierarchy process (AHP) to assign weights. First, several experts in finance and risk control are invited to conduct pairwise importance comparisons of all target identification items within the same modality using a 1-9 scale to construct a judgment matrix. Eigenvectors are then calculated from the judgment matrix, and a consistency check is performed. Once passed, the normalized weight vectors are obtained.

[0117] S104.33, the confidence scores of each target identification item are weighted and averaged to generate the overall confidence score of the modality.

[0118] S104.4 compares the overall reliability of each modality with its corresponding reliability threshold to calculate the reliability deviation coefficient, which is used to quantify the reliability of the modality data, and is expressed as follows:

[0119]

[0120] in, This is the credibility deviation coefficient. To assess overall credibility, , where is the credibility threshold. , Represents text modality, Represents image modality, Represents structured data modes.

[0121] S104.5, using the calculated reliability deviation coefficient The base weights are adjusted to obtain the final dynamic fusion weights.

[0122] The basic weights are calculated using the following formula. After making corrections, the final dynamic fusion weights are obtained. , is represented as ,

[0123]

[0124] S104.6, The feature vectors of each modality are weighted and summed according to the dynamic fusion weights to generate the final fusion feature vector.

[0125] The final fused feature vector is then input into a risk identification neural network to obtain the risk probability of the tax invoices to be audited.

[0126] S105, Match audit actions based on risk probability.

[0127] The risk probability is compared with the preset risk threshold in the audit rule base, and the corresponding audit action is triggered based on the comparison result, including one or more of the following: automatic approval, approval after recording risk anomalies, push to manual review, and triggering high-risk emergency plan.

[0128] Specifically, the risk probability is matched with a predefined risk threshold range in the audit rule base and mapped to the corresponding risk level. The risk threshold range includes: a first range [0, 0.2), mapped to a level with no significant risk; a second range [0.2, 0.5), mapped to a low risk level; a third range [0.5, 0.7), mapped to a medium risk level; and a fourth range [0.7, 1], mapped to a high risk level. Based on the mapped risk level, a pre-defined audit action uniquely corresponding to the risk level in the audit rule base is triggered. Among these actions, a level with no significant risk triggers automatic approval; a low risk level triggers approval after recording the risk anomaly; a medium risk level triggers submission to manual review; and a high risk level triggers activation of the high-risk emergency plan.

[0129] S106, Model Optimization.

[0130] Receive manual feedback on the review results, and adjust the weight parameters in the multimodal large model based on the manual feedback. The specific steps include the following steps.

[0131] S106.1 The received manual feedback information is processed in a structured manner to generate a standardized feedback record for each misjudged case. The record includes a unique case identifier, the correct risk level confirmed by manual review, and the actual risk probability mapped based on the correct risk level.

[0132] S106.2, Based on the unique case identifier, retrieve the multimodal data corresponding to the case, input it again into the multimodal large model to perform forward propagation calculation, and obtain the current predicted risk probability of the multimodal large model for the case.

[0133] S106.3, Based on the true risk probability and the current predicted risk probability of the case, calculate the loss value of the multimodal large model on the current case using the cross-entropy loss function.

[0134] S106.4, using the backpropagation algorithm, starting from the output of the cross-entropy loss function, calculates the gradient of the loss value with respect to at least one trainable weight parameter matrix in the multimodal large model layer by layer along the model computation graph; the trainable weight parameter matrix includes the weights in the feature fusion network and the risk recognition neural network.

[0135] S106.5, based on the calculated gradient and the preset learning rate, the trainable weight parameter matrix is ​​iteratively updated using the gradient descent method.

[0136] Specifically, the "misjudgment cases" marked by finance personnel after reviewing on the web platform contain three core types of information:

[0137] Basic information: reimburse_id (reimburse form ID), audit_id (audit record ID), feedback_time (feedback time);

[0138] False positive type: Preset enumeration value;

[0139] Correct label: Revised risk level.

[0140] Only "highly credible feedback" is retained, requiring cross-confirmation of misjudgment results by two finance managers, and specific risk characteristics must be identified. Invalid feedback such as "operational error" and "incomplete information" is eliminated.

[0141] By associating the original multimodal features of the reimburse form with reimburse_id, a "feedback-feature mapping table" is generated, clarifying the core impact feature dimension corresponding to each feedback.

[0142] A pre-defined mapping relationship between "misjudgment type" and "weight dimension" is established. Based on the misjudgment types reported by users, the dimensions of the large model are identified. The weights of these dimensions are then adjusted using a gradient descent fine-tuning method.

[0143] The foregoing has described in detail an embodiment of a financial invoice review and decision-making method. Based on the financial invoice review and decision-making method described in the above embodiment, this invention also provides a financial invoice review and decision-making system corresponding to the method.

[0144] Figure 2 This is a schematic block diagram of a financial invoice review and decision-making system provided in an embodiment of the present invention. In this embodiment, the financial invoice review and decision-making system 200 can be divided into multiple functional modules according to the functions it performs, such as... Figure 2As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory.

[0145] The multimodal data acquisition module 210 is used to acquire multimodal data related to the financial and tax invoices to be reviewed, and to establish a unified association identifier for the multimodal data. The multimodal data includes text data, image data, and structured data.

[0146] The multimodal data preprocessing module 220 is used to preprocess the collected multimodal data to generate standardized text feature data, image feature data, and structured feature data.

[0147] The risk probability generation module 230 is used to input the preprocessed multimodal data into a multimodal large model for processing, including: extracting feature vectors of corresponding feature data using encoders corresponding to the modal to obtain text feature vectors, image feature vectors, and structured feature vectors; using a dynamic weight allocation algorithm to weight and fuse various feature vectors to generate a fused feature vector; and inputting the fused feature vector into a risk recognition neural network to obtain the risk probability of the financial and tax invoices to be audited. Among them, the dynamic weight allocation algorithm dynamically calculates and allocates the fusion weights of each modality based on the credibility of each modality data, the predefined business risk priority, and the historical audit feedback results.

[0148] The audit action triggering module 240 is used to compare the risk probability with the preset risk threshold in the audit rule base, and trigger the corresponding audit action according to the comparison result, including one or more of the following: automatic approval, approval after recording risk anomalies, push to manual review, and triggering high-risk emergency plan.

[0149] The large model optimization and adjustment module 250 is used to receive manual feedback information on the review results and adjust the weight parameters in the multimodal large model based on the manual feedback information.

[0150] The financial invoice review and decision system of this embodiment is used to implement the aforementioned financial invoice review and decision method. Therefore, the specific implementation of this system can be found in the embodiment section of the financial invoice review and decision method above. Thus, the specific implementation can be referred to the description of the corresponding embodiments, and will not be elaborated here.

[0151] Furthermore, since the financial document review and decision system of this embodiment is used to implement the aforementioned financial document review and decision method, its function corresponds to the function of the above method, and will not be described again here.

[0152] Figure 3This is a schematic diagram of a terminal 300 provided in an embodiment of the present invention, including: a processor 310, a memory 320, and a communication unit 330. The processor 310 is used to implement the process steps of the above-described embodiment of the financial document review and decision-making method when implementing the financial document review and decision-making program stored in the memory 320.

[0153] This invention also provides a computer storage medium, which may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The computer storage medium stores a financial invoice review and decision-making program, which, when executed by a processor, implements the process steps of the above-described financial invoice review and decision-making method embodiment.

[0154] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of financial document review decision making, characterized by, The method comprises the following steps: Collecting multi-modal data associated with the to-be-audited financial and tax document business and establishing a unified association identifier for the multi-modal data, wherein the multi-modal data comprises text data, image data and structured data; Preprocessing the collected multi-modal data to generate standardized text feature data, image feature data and structured feature data; Inputting the preprocessed multi-modal data into a multi-modal large model for processing, comprising: extracting feature vectors of corresponding feature data using an encoder corresponding to each modality to obtain text feature vectors, image feature vectors and structured feature vectors; performing weighted fusion of each type of feature vector using a dynamic weight distribution algorithm to generate a fusion feature vector; inputting the fusion feature vector into a risk identification neural network to obtain a risk probability of the to-be-audited financial and tax document; wherein the dynamic weight distribution algorithm dynamically calculates and distributes the fusion weight of each modality based on the credibility of each modality data, a pre-defined business risk priority and historical audit feedback results; Comparing the risk probability with a pre-set risk threshold in an audit rule library, and triggering one or more of the following according to the comparison result: automatic passing, passing after recording risk abnormalities, pushing to manual audit, triggering a high-risk emergency plan; Receiving manual feedback information for the audit result, and adjusting the weight parameters in the multi-modal large model based on the manual feedback information, comprising: structuring the received manual feedback information to generate a standardized feedback record for each misjudgment case, which contains a case unique identifier, a correct risk level confirmed by manual review, and a real risk probability mapped based on the correct risk level; according to the case unique identifier, the multi-modal data corresponding to the case is retrieved and inputted into the multi-modal large model again to perform forward propagation calculation to obtain the predicted risk probability of the multi-modal large model for the case; based on the real risk probability and the predicted risk probability of the case, the loss value of the multi-modal large model on the current case is calculated through a cross-entropy loss function; using a back propagation algorithm, starting from the output end of the cross-entropy loss function, the loss value is calculated layer by layer along the model calculation graph to obtain the gradient of at least one trainable weight parameter matrix in the multi-modal large model; the trainable weight parameter matrix includes the weights in the feature fusion network and the risk identification neural network; according to the calculated gradient and a pre-set learning rate, the trainable weight parameter matrix is iteratively updated through a gradient descent method; The dynamic weight distribution algorithm is used to perform weighted fusion of each type of feature vector to generate a fusion feature vector, specifically comprising: A base weight is pre-set for each modality based on the pre-defined business risk priority; A credibility threshold is set for each modality, which is configured based on the current historical audit feedback results; The comprehensive credibility of each modality in the current audit case is calculated, which is the aggregation of the confidence levels of each target identification item in the modality; The comprehensive credibility of each modality is compared with its corresponding credibility threshold to calculate a credibility deviation coefficient, which is used to quantify the reliability of the modality data and is represented as wherein, is a trustworthiness bias coefficient, is a comprehensive trustworthiness, is a trustworthiness threshold, wherein , denotes a text modality, denotes an image modality, denotes a structured data modality; using the calculated confidence bias coefficient correcting the base weight to obtain a final dynamic fusion weight; The feature vectors of each modality are weighted and summed according to dynamic fusion weights to generate a final fusion feature vector.

2. The method of claim 1, wherein, A basic weight is preset for each modality based on a predefined business risk priority, which specifically includes: Based on predefined rules related to the audit of financial and tax documents, an evaluation vector is generated for each of the text, image and structured data modalities. The evaluation vector is used to quantify the business risk priority of each modality, and its dimensions at least include accuracy criticality, tampering difficulty and historical error rate. The accuracy criticality represents the decisive degree of the modality data on the final audit conclusion. The tampering difficulty represents the degree of difficulty of unauthorized modification of the modality data. The historical error rate represents the statistical error rate of the modality data in historical audits. The evaluation vector of each modality is linearly combined based on a preset dimension weight vector to calculate the risk priority score of each modality. Each weight component in the dimension weight vector corresponds to an evaluation dimension, and the sum of the weight components is 1. The risk priority is normalized, including calculating the ratio of the risk priority score of each modality to the sum of the risk priority scores of all modalities. The obtained ratio is the preset basic weight of the modality, and the sum of the preset basic weights of all modalities is 1.

3. The method of claim 1, wherein, A credibility threshold is set for each modality, which specifically includes: Collect all audit case data generated within a running period or a preset historical period, where each data record at least includes the modality type, the comprehensive credibility output by the modality when processing the corresponding case, and the case audit action. From the full amount of audit case data, the cases whose audit actions are pushed to manual audit are selected to constitute a strong label data set. For each modality in the strong label data set, it is divided into different credibility intervals according to the size of its comprehensive credibility. For each credibility interval, the false positive rate of the modality in the interval is calculated, which is the ratio of the number of false positive cases in the interval to the total number of cases. According to the preset acceptable maximum false positive rate, traverse each credibility interval, and configure the lower limit value of the smallest credibility interval that meets the condition that the statistical false positive rate is not greater than the acceptable maximum false positive rate as the credibility threshold of the modality.

4. The method of claim 1, wherein, Calculate the comprehensive credibility of each modality in the current audit case, which specifically includes: For each modality in the current audit case, obtain the confidence of the internal target recognition item in its preprocessing process after being processed by the corresponding recognition model. The target recognition items of the image modality include invoice amount, invoice date, seller name and seal clarity, and the recognition model is an OCR engine. The target recognition items of the text modality include reimbursement amount, business trip date and department name, and the recognition model is a named entity recognition model. The target recognition items of the structured data modality include budget balance data consistency and bank flow matching state, and the recognition model is a data verification rule. For each modality, assign weights to each target recognition item based on its business importance in the financial and tax audit. The confidence of each target recognition item is weighted and averaged to generate the comprehensive confidence of the modality.

5. The method of claim 1, wherein, The risk probability is compared with a preset risk threshold in an audit rule library, and a corresponding audit action is triggered according to a comparison result, specifically including: The risk probability is matched with a predefined risk threshold interval in the audit rule library, and is mapped to a corresponding risk level; the risk threshold interval includes: a first interval [0, 0.2), which is mapped to a no significant risk level, a second interval [0.2, 0.5), which is mapped to a low risk level, a third interval [0.5, 0.7), which is mapped to a medium risk level, and a fourth interval [0.7, 1], which is mapped to a high risk level; According to the risk level obtained by mapping, the audit action corresponding to the risk level in the audit rule library is triggered; wherein the no significant risk level triggers the automatic pass action, the low risk level triggers the record risk exception and pass action, the medium risk level triggers the push to manual audit action, and the high risk level triggers the high risk emergency plan action.

6. A financial instrument review decision system characterized by, Including: A multi-modal data acquisition module for acquiring multi-modal data associated with the to-be-audited financial and tax document business and establishing a unified association identifier for the multi-modal data, the multi-modal data including text data, image data and structured data; A multi-modal data preprocessing module for preprocessing the acquired multi-modal data to generate standardized text feature data, image feature data and structured feature data; A risk probability generation module for inputting the preprocessed multi-modal data into a multi-modal large model for processing, including: using an encoder corresponding to each modality to extract a feature vector of the corresponding feature data to obtain a text feature vector, an image feature vector and a structured feature vector; using a dynamic weight distribution algorithm to weight and fuse each type of feature vector to generate a fused feature vector; inputting the fused feature vector into a risk identification neural network to obtain a risk probability of the to-be-audited financial and tax document; wherein the dynamic weight distribution algorithm dynamically calculates and distributes the fusion weight of each modality based on the credibility of each modality data, the predefined business risk priority and the historical audit feedback result; An audit action triggering module for comparing the risk probability with a preset risk threshold in an audit rule library, and triggering a corresponding audit action according to a comparison result, including one or more of automatic pass, record risk exception and pass, push to manual audit, and trigger high risk emergency plan. The large model optimization and adjustment module is configured to receive artificial feedback information for the audit result, and adjust weight parameters in the multi-modal large model based on the artificial feedback information. The method includes: structurally processing the received artificial feedback information to generate a standardized feedback record for each misjudgment case, the record including a case unique identifier, a correct risk level confirmed by artificial review, and a real risk probability obtained based on the correct risk level mapping; according to the case unique identifier, the multi-modal data corresponding to the case is retrieved and input into the multi-modal large model again to perform forward propagation calculation to obtain a predicted risk probability of the multi-modal large model for the case; based on the real risk probability and the predicted risk probability of the case, a loss value of the multi-modal large model on the current case is calculated through a cross-entropy loss function; using a back propagation algorithm, the loss value is calculated layer by layer in reverse along the model calculation graph starting from the output end of the cross-entropy loss function to obtain the gradient of the loss value with respect to at least one trainable weight parameter matrix in the multi-modal large model; the trainable weight parameter matrix includes weights in the feature fusion network and the risk identification neural network; according to the calculated gradient and a preset learning rate, the trainable weight parameter matrix is iteratively updated through a gradient descent method; The dynamic weight distribution algorithm is used to weight and fuse each type of feature vector to generate a fused feature vector, specifically including: A base weight is preset for each modality based on a predefined business risk priority; A credibility threshold is set for each modality, which is configured based on the current historical audit feedback result; The comprehensive credibility of each modality in the current audit case is calculated, which is an aggregation of the confidence of each target identification item in the modality; The comprehensive credibility of each modality is compared with its corresponding credibility threshold to calculate a credibility deviation coefficient, which is used to quantify the reliability of the modality data and is represented as wherein, is a trustworthiness bias coefficient, is a comprehensive trustworthiness, is a trustworthiness threshold, wherein , denotes a text modality, denotes an image modality, denotes a structured data modality; using the calculated confidence bias coefficient correcting the base weight to obtain a final dynamic fusion weight; The feature vectors of each modality are weighted and summed according to the dynamic fusion weight to generate the final fused feature vector.

7. A terminal, characterized by comprising: The memory is configured to store a financial document audit decision program. The processor is configured to execute the financial document audit decision program to implement the steps of the financial document audit decision method according to any one of claims 1 to 5. The readable storage medium stores a financial document audit decision program, and the financial document audit decision program is executed by the processor to implement the steps of the financial document audit decision method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, ​

Images