Financial analysis method based on multi-modal large model

Abstract

The invention relates to the technical field of financial management, and discloses a multi-modal large model-based financial analysis method, which comprises the following steps of S1, acquiring and identifying multi-modal accounting data; s2, extracting features based on modal types, and converting the data into feature tensors of a unified feature space; s3, calculating a confidence coefficient weight according to the data quality, and weighting the fusion feature tensor according to the weight to obtain a fusion semantic representation; s4, constructing a semantic similarity map, generating an amount residual tensor in combination with the amount information, and generating a leveling score tensor; and S5, determining an accounting record matching relationship by using the leveling score tensor, and completing automatic leveling. By combining feature extraction and fusion of the multi-modal accounting data, the accuracy and automation degree of accounting matching are effectively improved. By constructing the leveling score tensor and the optimized matching algorithm, the association between the accounting records can be accurately identified, manual intervention is reduced, and the financial processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of financial management, and in particular to a financial analysis method based on a multimodal large model. Background Art

[0002] With the advancement of enterprise informatization, financial systems have gradually achieved the electronicization of accounting records and process automation. However, during the account reconciliation process, especially when involving a large number of heterogeneous data sources or multiple types of transaction records, matching accuracy and processing efficiency still face significant challenges. Traditional account reconciliation relies on static rule matching methods that primarily focus on amount features. While feasible for structured data, it struggles to cope with the semantic differences caused by unstructured information such as summaries, project names, and transaction descriptions in actual business. Although the amounts are similar, the semantic disconnect often leads to misjudgment, which in turn leads to reconciliation failures or increased subsequent audit costs.

[0003] Existing account reconciliation technologies generally lack in-depth modeling of semantic information. Their reconciliation logic relies heavily on fixed rules or manual experience, resulting in a lack of adaptive capabilities. Existing algorithms often struggle to effectively identify potential matching relationships, especially when accounting data is sparsely distributed, with small monetary errors but large semantic deviations. Furthermore, most existing systems fail to globally optimize matching solutions and lack systematic modeling of path weights or scoring mechanisms. This results in locally reasonable but overall suboptimal reconciliation results, impacting the accuracy and consistency of financial decisions.

[0004] While some studies have attempted to incorporate machine learning methods to aid in identification, most have focused solely on feature selection, failing to construct an interpretable matching scoring system or incorporate optimization algorithms to infer optimal paths in high-dimensional spaces. Furthermore, human intervention remains frequent in the matching process, resulting in a low level of automation and limiting the system's practicality.

[0005] Therefore, the present invention proposes a financial analysis method based on a multimodal large model to address the deficiencies of the prior art. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a financial analysis method based on a multimodal large model, which solves the problems of insufficient semantic understanding and low degree of automation in account matching in the existing technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a financial analysis method based on a multimodal large model, comprising the following steps: S1. Acquire multimodal account data including text data, image data, and structured data, and perform modality type identification on the multimodal account data; S2. Performing feature extraction on the text data, image data, and structured data based on the identified modality types, respectively, to convert the multimodal account data into a feature tensor in a unified feature space; S3. Calculating corresponding confidence weights based on the quality of each modal data, and performing weighted fusion on the feature tensors according to the weights to obtain a fused semantic representation for subsequent account matching processing; S4. Constructing a semantic similarity graph between the account records based on the fused semantic representation, generating an amount residual tensor based on the amount information of each account record, and generating a leveling score tensor for account leveling based on the amount residual tensor; S5. Determine the leveling matching relationship between each account record using the leveling score tensor to complete automatic account leveling processing.

[0008] Preferably, the step of identifying the modality type includes: Perform linguistic analysis on text data to identify it as a natural language document, financial report, or contract text; Perform image recognition on image data to confirm whether it is an invoice, receipt or bill; Perform format analysis on structured data to determine whether it is a table, data list or database record.

[0009] Preferably, the feature extraction step includes: Use natural language processing technology on text data to extract key information such as company name, transaction amount, and date; Apply optical character recognition technology to image data to extract digital, text and table information from images; Extract fields from structured data to identify and encode numerical information, row and column relationships, and header information in tables.

[0010] Preferably, the step of weightedly fusing the feature tensors according to the weights includes: The weighting coefficient is calculated according to the quality assessment results of each modal data. Let the quality assessment results of each modal data be Q text , Q img , Q tab , where Q text , Q img , Q tab are the quality assessment values ​​for text data, image data, and table data respectively; Calculate the weighting coefficient α text , α img , α tab ,in: The characteristic tensor of each modal data is represented as T text, T img , T tab , where T text , T img , T tab are the feature tensors of text data, image data, and table data respectively; According to the weighting coefficient α text , α img , α tab Perform weighted fusion on the feature tensors to obtain the fused feature tensor F, where: F=α text ·T text +α img ·T img +α tab ·T tab .

[0011] Preferably, the step of constructing the leveled score tensor includes: Based on the fused semantic feature vectors, the semantic similarity between the account records is calculated to generate the semantic similarity matrix M, whose formula is: Among them, F i and F j represent the fusion feature representation of the i-th and j-th accounting records respectively, ||F i || and ||F j || represent F respectively i and F j The Euclidean norm of ; According to the amount information of the accounting record A i and A j , calculate the amount residual R(i,j), the formula is: Among them, A i and A j Represent the amounts of the i-th and j-th accounting records respectively; The leveling score tensor J is generated by the semantic similarity matrix M and the amount residual matrix R. Its formula is: J(i,j)=λ·M(i,j)-(1-λ)·R(i,j); Among them, λ is the balance coefficient, 0≤λ≤1, which is used to adjust the weights of semantic similarity and amount residual.

[0012] Preferably, the step of generating the leveled score tensor further comprises: The semantic similarity matrix M and the amount residual matrix R are standardized so that they are in the same dimensional range.

[0013] Preferably, the step of determining the leveling matching relationship includes: Solve the leveled score tensor using a sparse optimization algorithm; Based on the result of leveling the score tensor, the account record pairs are automatically matched and a matching score is generated for each pair of records.

[0014] Preferably, the step of determining the leveling matching relationship further includes: Constrain the matching results of accounting records to ensure that each accounting record is matched with only one other accounting record; determine the optimal accounting reconciliation path based on the matching score.

[0015] The present invention also provides a financial analysis system based on a multimodal large model, comprising: A modality recognition module, configured to acquire multimodal account data including text data, image data, and structured data, and perform modality type recognition on the multimodal account data; a feature extraction module, configured to extract features from the text data, image data, and structured data based on the identified modality types, and uniformly encode the extracted features into feature tensors in the same feature space; A feature fusion module is used to calculate corresponding confidence weights based on the quality of each modal data, and perform weighted fusion on the feature tensors according to the weights to obtain a fused semantic representation for subsequent account matching processing; A scoring construction module is used to construct a semantic similarity map between account records based on the fused semantic representation, and to generate an amount residual tensor based on the amount information of each account record, and to generate a leveling score tensor for account leveling based on the amount residual tensor; The leveling matching module is used to use the leveling scoring tensor to determine the leveling matching relationship between each account record to complete the automatic leveling process of the account.

[0016] The present invention provides a financial analysis method based on a multimodal large model. It has the following beneficial effects: 1. This invention significantly improves the efficiency of the account reconciliation process by automatically constructing a leveling score tensor and using a sparse optimization algorithm to find the optimal match. Traditional manual verification of account records is time-consuming and error-prone, but this invention can quickly and accurately match account records, saving significant time in the financial processing process and reducing the need for manual intervention.

[0017] 2. This invention combines semantic similarity with amount residuals to construct a leveling score tensor, enabling precise quantification of the degree of matching between accounting records. Previous approaches often relied solely on amount features, which were prone to misjudgment. The scoring mechanism of this invention offers greater discriminatory power, significantly improving leveling accuracy.

[0018] 3. The sparse optimization algorithm and path weight maximization strategy employed by this invention not only handle large amounts of accounting data but also flexibly adjust to the characteristics of different types of accounting records. Whether it's simple accounting records or complex multimodal data, the system can effectively balance them, demonstrating the invention's strong adaptability across diverse application scenarios.

[0019] 4. By fully automating the matching and reconciliation of accounting records, this invention reduces the need for manual intervention and effectively mitigates accounting discrepancies caused by human error. By precisely calculating reconciliation scores and matching paths, the system avoids omissions and errors in manual review processes, thereby improving the accuracy and consistency of accounting data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Please see the attached Figure 1 , an embodiment of the present invention provides a financial analysis method based on a multimodal large model, comprising the following steps: S1, acquiring multimodal account data including text data, image data, and structured data, and performing modality type identification on the multimodal account data; S2. Performing feature extraction on the text data, image data, and structured data based on the identified modality types, respectively, to convert the multimodal account data into a feature tensor in a unified feature space; S3. Calculating corresponding confidence weights based on the quality of each modal data, and performing weighted fusion on the feature tensors according to the weights to obtain a fused semantic representation for subsequent account matching processing; S4. Constructing a semantic similarity graph between the account records based on the fused semantic representation, generating an amount residual tensor based on the amount information of each account record, and generating a leveling score tensor for account leveling based on the amount residual tensor; S5. Determine the leveling matching relationship between each account record using the leveling score tensor to complete automatic account leveling processing.

[0023] Regarding step S1, in practical applications, the types of data recorded in enterprise or financial systems typically include, but are not limited to, the following three modalities: text data, image data, and structured data. To achieve unified modeling for subsequent multimodal fusion and semantic matching, the input raw accounting data must first be automatically identified and labeled by modality type.

[0024] Specifically, the system preferably employs a modular data acquisition component to acquire raw accounting data from accounting data sources (e.g., financial system interfaces, batch import channels, file directories, etc.). The acquired data can be input into the modality recognition module in the form of files, streams, or objects. Data sources can include electronic invoice systems, bank receipt images, contract documents, scanned business documents, or transaction records exported from spreadsheet systems.

[0025] The modality recognition module preferably combines the metadata and content features of the data to identify the data type. For input data format identification, the following strategies can be used for identification: When a data file has a standard format identifier (such as MIME type, file extension, HTTP Content-Type header, etc.), the system can prioritize preliminary modality prediction based on the format information; For data that lacks clear formatting or has confusing formats, the system can assist in determining the authenticity of the data by analyzing its internal content structure. This process may include analyzing the degree of structure, assessing character density, and detecting visual layout.

[0026] On this basis, the system further applies modal feature analysis methods to data from different modalities to achieve more accurate classification and labeling. The specific technical mechanism of this analysis process is described below.

[0027] For data that is initially judged to be text modal, the system further uses a language analysis model to understand the semantic content. Preferably, the system introduces a pre-trained natural language processing model (such as a language encoder based on the Transformer architecture) to embed the text paragraphs and further identify whether they belong to natural language documents, standardized financial reports or contract documents by judging whether they contain typical accounting terms, table titles, amount figures, time expressions, company entities and other elements. The specific discrimination strategy can be based on rule template matching or training classifiers (such as softmax classifiers). For example, when keywords such as "payment application", "invoice number", "signing time" appear in a text segment and conform to the contract language style, they can be marked as contract text modal.

[0028] For data identified as image modalities, the system preferably introduces image recognition and optical character recognition (OCR) modules. The OCR module can identify printed or handwritten text content in the image based on a deep convolutional neural network, and extract text fragments that can be used for structural analysis. When specific invoices, receipts, bills and other structural features (such as headers, QR code areas, seal areas, etc.) appear in the image content, the system can determine whether the image data is a standardized financial document image based on this. In order to improve recognition accuracy, the image recognition module can also use a visual distribution feature analysis algorithm to divide the entire image into regions to detect typical invoice layouts.

[0029] For content that is determined to be structured data, the system preferably uses format analysis and field extraction algorithms. Structured data is usually presented in the form of JSON, XML, CSV or Excel tables. The system determines whether the data belongs to a tabular account record by parsing the field name, field type and value distribution pattern. This process may include the following operations: field keyword matching (such as "amount", "summary", "debit", "credit"), header alignment analysis, missing value density assessment, etc. The system further extracts the data type of each field (such as floating point, date, text) and constructs a structural semantic tree to support subsequent feature encoding and semantic fusion operations.

[0030] In order to ensure the accuracy of the modality type identification process, the system preferably generates a modality classification confidence score for each account data, which is recorded as: γ (k) =P(modality=k|x); Among them, x represents the current account data to be identified, k∈{text,image,table} represents the possible modal type γ (k) Represents the posterior probability under this mode. The system finally outputs the mode type of the data with the highest confidence.

[0031] The modal identification module can be run independently as a pre-processing module or embedded into the overall multimodal accounting processing pipeline. The system can also be configured with a manual review interface to correct the modal labels of a small number of borderline data.

[0032] Through the above method, the system can effectively perform modal recognition on accounting information from heterogeneous data sources, provide accurate modal labeling basis for subsequent semantic modeling and leveling scoring, and ensure that the overall processing flow has modal perception and adaptability.

[0033] In step S2, in this embodiment, feature extraction is performed on the account data of the identified modality type obtained in step S1, and the multimodal account data is converted into a feature tensor in a unified feature space. This process extracts key information from each modality through specific processing techniques for each modality and converts it into a feature representation of the same dimension, laying the foundation for subsequent semantic fusion and analysis.

[0034] For each modality type of data, the feature extraction process specifically includes the following aspects.

[0035] First, for the account information identified as text data, this embodiment uses natural language processing (NLP) technology to extract key information through the following steps: Text preprocessing: De-noising the input text data, including removing unstructured characters (such as special symbols, noise text) and word segmentation.

[0036] Key Information Extraction: Leveraging natural language processing technology, specifically the named entity recognition (NER) module, we identify and extract key information from text, such as company name, transaction amount, transaction date, invoice number, and other financial data fields. This process leverages a named entity recognizer trained using a deep learning model (such as BERT or RoBERTa) to perform sequence annotation on the text, extracting meaningful entity information.

[0037] For example, if the input text is: "Company A paid RMB 20,000 on April 20, 2025, invoice number A123456", the system can extract: Company Name: "Company A" Transaction amount: "20,000 RMB" Transaction date: "April 20, 2025" Invoice number: "A123456" Vectorized representation: After extracting this key information, it is converted into feature vectors using vectorization techniques. Generally speaking, each entity in the text is mapped to a high-dimensional vector and represented using a context-sensitive embedding generated by a pre-trained language model (such as BERT or the GPT series). The overall feature vector of the text is the concatenation or weighted average of the individual entity vectors.

[0038] For image data, in this embodiment, optical character recognition (OCR) technology and image recognition methods are used to extract information from the image. The specific steps are as follows: Image preprocessing: Preprocess the input image data, including denoising, grayscale conversion, edge detection, etc., to ensure that OCR technology can better recognize text information in the image.

[0039] OCR Recognition: Uses OCR technology (such as Tesseract and PaddleOCR) to extract text, numbers, tables, and other information from images. The OCR module converts character information in the image into parseable text and recognizes specific financial information such as table layouts, amounts, and dates.

[0040] For example, an invoice in an image may include "Payment amount: RMB 20,000, Date: April 20, 2025". After OCR extraction, the system can identify the relevant amount and date information.

[0041] Visual feature extraction: In addition to the text information extracted by OCR, the system can also identify visual features in the image through image feature extraction methods such as convolutional neural networks (CNN). For example, the invoice layout, invoice header, seal and other visual symbols. These features help to further improve the accurate understanding of financial images such as invoices.

[0042] For structured data, such as bills, account books, or financial statements exported from spreadsheets or databases, this embodiment uses data format parsing and feature encoding technology to process them. The steps are as follows: Field parsing: By parsing table fields (such as column names and row data), key fields of financial data such as amount, date, transaction type, account number, etc. are identified.

[0043] Data normalization: Clean and normalize each field. For example, converting an amount field to a standard currency format, or converting a date field to a unified date standard (such as ISO 8601). This normalizes the values ​​and dates in the data into a standard format that is easy for computers to process.

[0044] Feature encoding: Convert the content of each field into a feature vector of uniform dimension. For text fields, embedding techniques (such as Word2Vec and GloVe) can be used to convert text into vectors. For numeric fields (such as amount and transaction number), numerical normalization can be used to convert them into a standardized value range and further embed them into the feature space.

[0045] After the above processing, the features of text data, image data and structured data are converted into their own feature vectors or tensors. These feature vectors will be mapped to a unified feature space for subsequent multimodal fusion. Specifically, assuming that the text feature vector is T text , the image feature vector is T img , the structured data feature vector is T tab , they can be mapped into a unified feature space as follows: F i =f(T text ,T img ,T tab ); Among them, f represents the feature fusion function, which is responsible for appropriately transforming the feature vectors of different modalities so that they can be compared and fused in the same feature space.

[0046] Ultimately, through this feature extraction and fusion processing, the features of text data, image data, and structured data will be unified into a multimodal feature tensor F, which has the same dimension and representation, providing a reliable foundation for subsequent multimodal analysis and matching.

[0047] Through the above steps, this embodiment effectively extracts features from accounting data of different modalities and uniformly maps them into the same feature space, thereby enabling more accurate and efficient multimodal data fusion and analysis in subsequent processing.

[0048] In step S3, in this embodiment, weighting coefficients are further calculated based on the quality assessment results of each modal data for the multimodal feature tensor extracted in step S2, and a weighted fusion is performed on the feature tensor. The core goal of this step is to rationally adjust the contribution of different modalities in subsequent analysis based on the reliability and quality differences of the modal data, thereby improving the accuracy and effectiveness of the overall feature fusion.

[0049] In practice, the system first evaluates the quality of each modality using different quality assessment methods. Different quality assessment techniques are used for text data, image data, and structured data. The results of these quality assessments reflect the completeness and accuracy of the data source and its contribution to the final analysis task.

[0050] First, for text data, quality assessment methods include but are not limited to: Language quality assessment: This analyzes the grammatical structure, clarity of expression, and information density of a text. For example, the text comprehension capabilities of deep learning models (such as BERT or the GPT series) can be used to assess the completeness and effective information content of a text.

[0051] Named Entity Recognition Accuracy: Evaluates the accuracy of extracting key information from text (such as company name, amount, date, etc.).

[0052] For image data, the main indicators of quality assessment include: OCR accuracy: The accuracy of text information extracted using OCR technology. A low OCR recognition rate indicates poor image quality, which affects the reliability of information extraction.

[0053] Image clarity: Evaluate the clarity of the image to determine if there is blur or distortion. Poor image quality may affect subsequent image recognition and information extraction.

[0054] For structured data, quality assessment focuses on: Field completeness: Determine whether the fields in the table are complete and whether there are missing values ​​or abnormal data.

[0055] Data format consistency: Check whether the structured data conforms to the expected format, for example, whether fields such as amount and date comply with the specifications.

[0056] The system finally obtains the quality evaluation results of each modal data according to the above evaluation method, which are Q text , Q img , Q tab , which are the quality assessment values ​​of text data, image data and table data respectively. The quality assessment results reflect the credibility of the information that the modality can provide during the feature extraction process.

[0057] When calculating the weighting coefficient, the system calculates the weighting coefficient of each mode based on the quality evaluation results of each mode. The specific weighting coefficient calculation formula is as follows: Among them, α text , α img , α tab These represent the weighting coefficients for text data, image data, and structured data, respectively. The magnitude of these coefficients is determined by the data quality of the corresponding modality. The higher the quality of the modality, the larger its corresponding weighting coefficient, indicating that the modality should occupy a more important position in the subsequent fusion process.

[0058] Next, the system represents the data feature tensor of each modality as T text , T img , T tab These feature tensors correspond to the feature representations of text data, image data, and structured data respectively. These feature tensors are obtained through the aforementioned feature extraction steps and will be used as input for weighted fusion.

[0059] According to the weighting coefficient α text , α img , α tab The system performs weighted fusion on the feature tensors of each modality and ultimately obtains a fused feature tensor F, which represents the fused semantic representation of multimodal accounting data. The specific calculation formula for the fusion process is as follows: F=α text ·T text +α img ·T img +αtab ·T tab ; Through this weighted fusion step, the system can automatically adjust the contribution of different modalities in the fusion result according to the quality of each modality data, ensuring that the final fused feature tensor can fully reflect the important information of each modality while reducing the negative impact of low-quality modalities on the results.

[0060] Ultimately, the fused feature tensor F is used in subsequent accounting matching, enabling more accurate comparison and reconciliation of accounting records. Because the weighted fusion of each modality accounts for quality differences in the data, the fused features are more accurate and stable, providing reliable support for subsequent financial analysis and decision-making.

[0061] Through the above steps, this embodiment achieves quality-weighted fusion based on multimodal data, thereby providing more accurate and efficient feature representation for subsequent account matching and analysis. This weighted fusion method is highly robust and can adapt to input data of varying quality levels, ensuring the accuracy and efficiency of the system when processing complex account information.

[0062] In step S4, in this embodiment, the semantic similarity between the account records is further calculated for the fused feature tensor F obtained in step S3. This is combined with the amount information of each account record to generate a leveling score tensor. The core goal of this step is to calculate the semantic similarity and amount residuals between the account records to generate a score tensor that comprehensively evaluates the matching degree of the account records, thereby achieving account leveling.

[0063] First, based on the fused semantic feature vector F, the system calculates the semantic similarity between each pair of account records. This process utilizes the common vector space model principle. Specifically, the system measures the similarity between each pair of account records by calculating the cosine similarity between the fused feature vectors. Assume that the fused feature of the i-th account record is represented by F i , the fusion feature of the jth account record is expressed as F j , then the semantic similarity matrix M(i,j) between them is calculated as follows: Among them, F i and F j represent the fusion feature representation of the i-th and j-th accounting records respectively, ||F i || and ||F j || represent F respectively i and F j The Euclidean norm of the vector (i.e., the length of the vector), F i ·F j Represents the fused feature vector Fi and F j This formula is based on the cosine similarity calculation method and is used to measure the similarity between two vectors in the feature space. Each element of the semantic similarity matrix M reflects the semantic similarity between account records.

[0064] Next, the system combines the amount information of the account records to calculate the amount residual for each pair of account records. The amount residual R(i,j) is used to measure the difference in amount between two account records. The specific calculation formula is: Among them, A i and A j Represent the amounts in the i-th and j-th accounting records, respectively. This formula calculates the relative difference between the amounts in two accounting records and normalizes them to ensure that the difference is within the same dimensional range. The normalized residual R(i,j) typically ranges between 0 and 1. Smaller values ​​indicate closer amounts in the two accounting records, while larger values ​​indicate a greater difference.

[0065] Finally, the system generates a leveling score tensor J by weighted combination based on the semantic similarity matrix M and the amount residual matrix R. The leveling score tensor J(i,j) comprehensively considers the semantic similarity and amount residual between account records. The calculation formula is as follows: J(i,j)=λ·M(i,j)-(1-λ)·R(i,j); Where λ is the balancing coefficient, 0≤λ≤1, which adjusts the weighting of semantic similarity and residual amount. The balancing coefficient λ controls the relative importance of semantic similarity and residual amount in the leveling score. When λ is large, the system places greater emphasis on semantic matching between account records; when λ is small, the system prioritizes matching amounts.

[0066] Each element of the reconciliation score tensor J(i,j) represents the reconciliation score between two account records. A higher score indicates a greater likelihood of a match between the two accounts. Using this score tensor, the system can identify the most appropriate matching relationship between the accounts, thereby supporting subsequent reconciliation operations.

[0067] To ensure consistent dimensioning of the calculated results of the leveled scoring tensor J, the system also normalizes the semantic similarity matrix M and the residual amount matrix R. This standardization aims to bring the values ​​of both into the same range, thereby preventing a single metric from significantly influencing the final score. Normalization typically involves normalizing M and R to the range of 0 and 1, ensuring their influence remains within a manageable range, resulting in a more balanced scoring result.

[0068] Through the above steps, the system can generate a leveling score tensor based on the semantic similarity and amount differences of the account records, providing an accurate basis for subsequent account record matching and leveling. The generation process of the leveling score tensor J combines multimodal feature information and amount information, making account matching more intelligent and efficient.

[0069] Regarding step S5, in this embodiment, the step of determining the leveling matching relationship is based on automatic analysis of the numerical results of the aforementioned constructed leveling score tensor J(i, j), and is combined with a sparse optimization algorithm to solve the optimal account pairing relationship.

[0070] The leveled scoring tensor J(i, j) has been clearly given in step S4. Specifically, it is a composite scoring result calculated by combining the semantic similarity M(i, j) and the amount residual R(i, j) between the account records. Its construction method has been detailed in the implementation part of the aforementioned step S4 and will not be repeated here.

[0071] In this embodiment, the system first obtains the constructed tensor J(i, j), where each element J(i, j) represents the degree of leveling match between account record i and account record j. This tensor serves as the basic input for the subsequent leveling match relationship determination process.

[0072] To extract matching relationships from the aforementioned scoring tensor, this embodiment employs a sparse optimization algorithm. This sparse optimization process uses the leveled scoring tensor as a weighted input to establish a matching model for the account record set. Under set constraints, it selects the account record pairs with the highest scores to construct a preliminary leveled pairing set.

[0073] The sparse optimization process preferably employs a matching model from graph theory. Specifically, accounting records are abstracted as nodes in a graph structure, and the leveling scores J(i, j) between accounting records constitute the edge weights in the graph. Using algorithms such as maximum matching or maximum weighted matching, the maximum scoring matching set is calculated while maintaining matching uniqueness constraints.

[0074] During the optimization process, to prevent duplicate matches between a single account record and multiple records, this embodiment introduces a one-to-one constraint mechanism. This mechanism, through Boolean flags or linear programming constraints, ensures that each account record appears only once in the matching set and is always paired with a unique account record. These constraints also effectively avoid redundant matches and circular references during the leveling path.

[0075] Based on the initial matching results, the system generates a matching score for each pair of account records based on the values ​​in the scoring tensor. This score, a quantitative expression of the credibility of the account reconciliation, is a key basis for the subsequent construction and optimization of the reconciliation path.

[0076] Continuing further, after matching records and generating their scores, the system constructs an account reconciliation path based on the known matching scores. Preferably, the system adopts a strategy that maximizes path weight, selecting the optimal combination of closed paths or chain structures from all matched record pairs, ensuring that the overall path structure has a global scoring advantage.

[0077] The generation of the rebalancing path can be achieved through dynamic programming or heuristic search, with the specific algorithm adapted to the implementation environment and accounting data structure. The resulting optimal path serves as the core execution target of the automatic accounting rebalancing engine and drives the subsequent accounting balancing process.

[0078] In summary, this embodiment achieves efficient and accurate automatic leveling of account records by combining the leveling score tensor with the sparse optimization algorithm, and ensures the rationality and global optimality of the matching results through path optimization technology, thereby improving the automation and accuracy of account leveling.

[0079] See also Figure 2 The present invention also provides a financial analysis system based on a multimodal large model, comprising: A modality recognition module, configured to acquire multimodal account data including text data, image data, and structured data, and perform modality type recognition on the multimodal account data; a feature extraction module, configured to extract features from the text data, image data, and structured data based on the identified modality types, and uniformly encode the extracted features into feature tensors in the same feature space; A feature fusion module is used to calculate corresponding confidence weights based on the quality of each modal data, and perform weighted fusion on the feature tensors according to the weights to obtain a fused semantic representation for subsequent account matching processing; A scoring construction module is used to construct a semantic similarity map between account records based on the fused semantic representation, and at the same time generate an amount residual tensor based on the amount information of each account record, and generate a leveling score tensor for account leveling based on this; a leveling matching module is used to use the leveling score tensor to determine the leveling matching relationship between each account record to complete the automatic account leveling processing.

[0080] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, so they will not be repeated here.

[0081] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The financial analysis method based on multimodal large model is characterized by: The following steps are involved: S1. Acquire multimodal account data including text data, image data, and structured data, and perform modality type identification on the multimodal account data; S2. Performing feature extraction on the text data, image data, and structured data based on the identified modality types, respectively, to convert the multimodal account data into a feature tensor in a unified feature space; S3. Calculating corresponding confidence weights based on the quality of each modal data, and performing weighted fusion on the feature tensors according to the weights to obtain a fused semantic representation for subsequent account matching processing; S4. Constructing a semantic similarity graph between the account records based on the fused semantic representation, generating an amount residual tensor based on the amount information of each account record, and generating a leveling score tensor for account leveling based on the amount residual tensor; S5. Determine the leveling matching relationship between each account record using the leveling score tensor to complete automatic account leveling processing.

2. The financial analysis method based on a multimodal large model according to claim 1, characterized in that: The step of modality type identification includes: Perform linguistic analysis on text data to identify it as a natural language document, financial report, or contract text; Perform image recognition on image data to confirm whether it is an invoice, receipt or bill; Perform format analysis on structured data to determine whether it is a table, data list or database record.

3. The financial analysis method based on a multimodal large model according to claim 1, characterized in that: The feature extraction step includes: Use natural language processing technology on text data to extract key information such as company name, transaction amount, and date; Apply optical character recognition technology to image data to extract digital, text and table information from images; Extract fields from structured data to identify and encode numerical information, row and column relationships, and header information in tables.

4. The financial analysis method based on a multimodal large model according to claim 1, characterized in that: The step of weightedly fusing the feature tensors according to the weights comprises: The weighting coefficient is calculated according to the quality assessment results of each modal data. Let the quality assessment results of each modal data be Q text , Q img , Q tab , where Q text , Q img , Q tab are the quality assessment values ​​for text data, image data, and table data respectively; Calculate the weighting coefficient α text , α img , α tab ,in: The characteristic tensor of each modal data is represented as T text , T img , T tab , where T text , T img , T tab are the feature tensors of text data, image data, and table data respectively; According to the weighting coefficient α text , α img , α tab Perform weighted fusion on the feature tensor to obtain the fused feature tensor F, where: F = α text ·T text +α img ·T img +α tab ·T tab .

5. The financial analysis method based on a multimodal large model according to claim 1, characterized in that: The steps for constructing the leveled score tensor include: Based on the fused semantic feature vectors, the semantic similarity between the account records is calculated to generate the semantic similarity matrix M, whose formula is: Among them, F i and F j represent the fusion feature representation of the i-th and j-th accounting records respectively, ||F i || and ||F j || represent F respectively i and F j The Euclidean norm of ; According to the amount information of the accounting record A i and A j , calculate the amount residual R(i,j), the formula is: Among them, A i and A j Represent the amounts of the i-th and j-th accounting records respectively; The leveling score tensor J is generated by the semantic similarity matrix M and the amount residual matrix R. Its formula is: J(i,j)=λ·M(i,j)-(1-λ)·R(i,j); Among them, λ is the balance coefficient, 0≤λ≤1, which is used to adjust the weights of semantic similarity and amount residual.

6. The financial analysis method based on a multimodal large model according to claim 5 is characterized in that: The step of generating the leveled score tensor further includes: The semantic similarity matrix M and the amount residual matrix R are standardized so that they are in the same dimensional range.

7. The financial analysis method based on a multimodal large model according to claim 1, characterized in that: The step of determining the leveling matching relationship includes: Solve the leveled score tensor using a sparse optimization algorithm; Based on the result of leveling the score tensor, the account record pairs are automatically matched and a matching score is generated for each pair of records.

8. The financial analysis method based on a multimodal large model according to claim 7, characterized in that: The step of determining the leveling matching relationship further includes: Constrain the matching results of accounting records to ensure that each accounting record is matched with only one other accounting record; Determine the optimal account reconciliation path based on the matching score.

9. A financial analysis system based on a multimodal large model, applied to the financial analysis method based on a multimodal large model as claimed in any one of claims 1 to 8, characterized in that: include: A modality recognition module, configured to acquire multimodal account data including text data, image data, and structured data, and perform modality type recognition on the multimodal account data; a feature extraction module, configured to extract features from the text data, image data, and structured data based on the identified modality types, and uniformly encode the extracted features into feature tensors in the same feature space; A feature fusion module is used to calculate corresponding confidence weights based on the quality of each modal data, and perform weighted fusion on the feature tensors according to the weights to obtain a fused semantic representation for subsequent account matching processing; A scoring construction module is used to construct a semantic similarity map between account records based on the fused semantic representation, and to generate an amount residual tensor based on the amount information of each account record, and to generate a leveling score tensor for account leveling based on the amount residual tensor; The leveling matching module is used to use the leveling scoring tensor to determine the leveling matching relationship between each account record to complete the automatic leveling process of the account.

Images