RAG-based voucher classification method, medium and equipment

Abstract

The invention relates to an RAG-based voucher classification method, a medium and equipment, and the method comprises the steps: receiving digital image data of a to-be-classified voucher, carrying out the multi-modal optical character recognition processing to generate a structured OCR result, extracting a text semantic feature vector and a visual layout feature vector based on the structured OCR result, carrying out the fusion of the text semantic feature vector and the visual layout feature vector to generate a multi-modal query vector, and carrying out the classification of the to-be-classified voucher. Similar samples and semantic similarity scores and category metadata thereof are obtained through approximate nearest neighbor retrieval, after an initial candidate category list is generated, key field values are extracted for each candidate category, evidence credibility scores are calculated, comprehensive confidence scores are generated by fusing the semantic similarity scores and the evidence credibility scores, reordering is conducted, and a candidate category list is obtained. And finally, selecting a classification decision path according to the score distribution, and outputting a classification result and an interpretability report. The accuracy and robustness of voucher classification are effectively improved, and complex voucher scenes with changeable formats and fuzzy semantics can be processed; and the interpretability and reliability of the classification decision are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, in particular to a RAG-based voucher classification method, medium and device. BACKGROUND

[0002] With the development of enterprise informatization and intelligent financial systems, the demand for electronic and structured processing of vouchers has surged. In scenarios such as financial reimbursement, tax audit, and travel management, there are numerous types of vouchers with complex formats, and manual review is inefficient, necessitating automated classification technology. OCR (Optical Character Recognition) technology is widely used for automatic extraction of text from bill images. Current mainstream OCR frameworks such as Tesseract, PaddleOCR, EasyOCR, and Google Cloud Vision API have continuously evolved natural language processing techniques in text classification. From early simple classification methods based on rules or keywords, to traditional machine learning methods based on Bag-of-Words (BoW) and TF-IDF (such as SVM and Naive Bayes), to deep learning models widely used in recent years, such as TextCNN, BiLSTM, Transformer, and BERT. However, in actual business scenarios, voucher data presents the following challenges: first, the text is short and the context information is limited; second, the structure is complex and the semantics are ambiguous, for example, "notice", "receipt", and "itinerary" are very similar in language; third, the formats of multiple types of vouchers are highly similar, making it difficult to accurately distinguish them based on keywords or semantic similarity. SUMMARY

[0003] In view of the above problems, the present application provides a RAG-based voucher classification method, medium and device, which integrates multiple technologies such as OCR recognition, text block segmentation, semantic embedding, vector retrieval, prompt word construction, and large language model reasoning, forming an automated, intelligent, and high-accuracy voucher classification process.

[0004] To achieve the above-mentioned purpose, the present application provides a RAG-based voucher classification method, comprising:

[0005] receiving digital image data of a voucher to be classified;

[0006] performing multi-modal optical character recognition on the digital image data to generate structured OCR results containing text content and its layout information;

[0007] based on the structured OCR results, extracting text semantic feature vectors through a semantic embedding model and visual layout feature vectors through an image feature extraction model;

[0008] performing feature fusion on the text semantic feature vectors and the visual layout feature vectors to generate multi-modal query vectors;

[0009] inputting the multi-modal query vector into a pre-constructed multi-modal knowledge vector library for approximate nearest neighbor search to obtain a search result, the search result including a plurality of similar samples ranked in the front, a semantic similarity score of each sample with the query vector, and category metadata associated with each sample;

[0010] generating an initial candidate category list according to the search result;

[0011] for each candidate category in the initial candidate category list, extracting a corresponding key field value from the structured OCR result according to a key field rule set corresponding to the candidate category;

[0012] calculating a matching degree of the extracted key field value and a corresponding category field mode in the category rule knowledge base to generate an evidence credibility score;

[0013] fusing the semantic similarity score and the evidence credibility score of each candidate category to generate a comprehensive confidence score, and reordering the candidate category list according to the comprehensive confidence score;

[0014] selecting a classification decision path according to a comprehensive confidence score distribution of the reordered candidate category list, including:

[0015] if the highest comprehensive confidence score meets a preset threshold condition, directly outputting a corresponding category as a classification result;

[0016] otherwise, dynamically constructing a comparison prompt word based on feature contradictions among the candidate categories, and submitting the comparison prompt word and the structured OCR result to a large language model for arbitration reasoning to obtain a final classification result;

[0017] outputting the classification result and an explainability report including the search samples, the key field evidence, and the decision path.

[0018] In some embodiments, generating an initial candidate category list according to the search result includes:

[0019] classifying and aggregating the plurality of similar samples returned by the search and ranked in the front according to corresponding category labels to obtain a plurality of classified categories;

[0020] calculating an initial confidence score of each classified category based on semantic similarity scores of all samples included in the category includes:

[0021] taking a maximum value of the semantic similarity scores of all similar samples in the same category;

[0022] alternatively, taking a weighted average value of the semantic similarity scores of all similar samples in the same category, the weight of the similar sample being determined according to an ordering position of the similar sample in the search result;

[0023] According to the initial confidence score, the sorted multiple categories are ranked, and the top N categories are selected to form an initial candidate category list.

[0024] In some embodiments, the semantic similarity score and the evidence confidence score of each candidate category are fused to generate a comprehensive confidence score, and the candidate category list is reordered according to the comprehensive confidence score, including:

[0025] The initial confidence score and the evidence confidence score of each candidate category in the initial candidate category list are obtained;

[0026] The initial confidence score and the evidence confidence score are fused by using a weighted summation algorithm to generate a comprehensive confidence score of the current candidate category, and the weight coefficient of the initial confidence score and the weight coefficient of the evidence confidence score are configured according to the classification scene requirements;

[0027] The above steps are repeated until the comprehensive confidence scores of all candidate categories are obtained;

[0028] According to the comprehensive confidence scores of all candidate categories, the initial candidate category list is reordered in descending order of the comprehensive confidence score, and a reordered candidate category list is generated.

[0029] In some embodiments, the matching degree of the extracted key field value and the corresponding category field mode in the category rule knowledge base is calculated to generate an evidence confidence score, including:

[0030] The field mode definition of the current candidate category is obtained from the category rule knowledge base, and the field mode definition includes the name, data type, format requirement and verification rule of each key field;

[0031] For each key field, the extracted field value is matched with the corresponding field mode definition to obtain a single field matching score, and the single field matching score calculation includes at least one of format compliance check, regular expression matching, numerical range verification, or semantic compliance judgment based on a pre-trained model;

[0032] Based on the matching scores of all key fields, a weighted average algorithm is used to calculate the overall evidence confidence score of the current candidate category, and the weight of each key field is configured according to its importance in category discrimination.

[0033] In some embodiments, the samples in the multi-modal knowledge vector library include text semantic vectors, visual layout vectors, category labels and key field rule sets;

[0034] In addition, the multi-modal knowledge vector library is configured to be constructed using a distributed vector database architecture;

[0035] The multi-modal query vector is input into the pre-constructed multi-modal knowledge vector library for approximate nearest neighbor search to obtain a search result, the search result including a plurality of similar samples ranked in the front, a semantic similarity score of each sample with the query vector, and category metadata associated with each sample, including:

[0036] A graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm is used to perform vector similarity search in the multi-modal knowledge vector library;

[0037] The similarity score of the multi-modal query vector and each sample vector in the multi-modal knowledge vector library is calculated, and the similarity score is obtained by a cosine similarity algorithm or an inner product similarity algorithm;

[0038] All search results are sorted in descending order according to the similarity score, and a plurality of samples ranked in the front are selected as similar samples;

[0039] Metadata associated with each similar sample is extracted from the multi-modal knowledge vector library, including category labels, key field rule set identifiers, and original feature information of sample source credentials.

[0040] In some embodiments, the text semantic feature vector and the visual layout feature vector are fused to generate a multi-modal query vector, including:

[0041] The text semantic feature vector and the visual layout feature vector are subjected to dimension alignment processing;

[0042] The processed text semantic feature vector and the visual layout feature vector are fused by using a weighted fusion algorithm;

[0043] The weighted fusion algorithm is configured to use an attention mechanism to adaptively weight the importance of the text and visual features, and to assign configurable weight coefficients to the text semantic feature vector and the visual layout feature vector, the weight coefficients being dynamically adjusted according to the credential type and the classification scenario;

[0044] The fused feature vector is subjected to normalization processing to generate a final multi-modal query vector.

[0045] In some embodiments, based on the structured OCR result, a text semantic feature vector is extracted by a semantic embedding model, and a visual layout feature vector is extracted by an image feature extraction model, including:

[0046] The text content in the structured OCR result is input into a pre-trained semantic embedding model, and the semantic embedding model is subjected to semantic encoding through a multi-layer Transformer structure of the semantic embedding model to output a fixed-dimension text semantic feature vector;

[0047] The digital image data is input into a pre-trained convolutional neural network model, feature extraction is performed through multiple convolutional layers and pooling layers of the convolutional neural network model, and a visual layout feature vector representing the overall layout structure of the certificate is output.

[0048] The semantic embedding model adopts a deep bidirectional language representation model based on a self-attention mechanism, and the image feature extraction model adopts a deep residual network or a visual Transformer architecture.

[0049] The extraction process of the text semantic feature vector further includes text cleaning and standardization preprocessing of the OCR recognition result, and the extraction process of the visual layout feature vector further includes size normalization and data enhancement processing of the input image.

[0050] In some embodiments, the digital image data is subjected to multi-modal optical character recognition processing to generate structured OCR results containing text content and layout information thereof, including:

[0051] A deep learning optical character recognition engine is used to perform text detection and recognition on the digital image data, extract the text content and its position coordinate information in the image, and obtain the recognition result.

[0052] The recognition result is subjected to structured processing through a layout analysis algorithm to detect and identify the layout information of text blocks, table regions, seal regions and logo regions.

[0053] The text content and its corresponding layout information are associated and integrated to generate structured OCR result data.

[0054] The structured OCR result is organized in JSON or XML format, containing text content, text position coordinates, font size, text block type and special region identification information.

[0055] The multi-modal optical character recognition processing further includes an image preprocessing step, including grayscale, binarization, noise removal and image enhancement operations on the input image to improve the character recognition accuracy.

[0056] In a second aspect, the present application further provides a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions being executed by a processor to implement the method of the first aspect.

[0057] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of the first aspect.

[0058] Differing from the prior art, the above technical scheme receives digital image data of a to-be-classified voucher, performs multi-modal optical character recognition processing to generate a structured OCR result, extracts a text semantic feature vector and a visual layout feature vector based on the structured OCR result and fuses them to generate a multi-modal query vector, acquires similar samples and their semantic similarity scores and category metadata through approximate nearest neighbor retrieval, extracts key field values and calculates evidence credibility scores for each candidate category after generating an initial candidate category list, fuses the semantic similarity scores and the evidence credibility scores to generate comprehensive confidence scores and performs reordering, and finally selects a classification decision path according to the score distribution and outputs a classification result and an explainability report. Through multi-modal feature fusion and approximate nearest neighbor retrieval, the accuracy and robustness of voucher classification are effectively improved, and complex voucher scenarios with variable formats and ambiguous semantics can be processed; through the fusion of the evidence credibility scores and the semantic similarity scores, the explainability and reliability of the classification decision are enhanced; and the finally output explainability report provides a complete technical basis for auditing and review, meeting the strict requirements of the finance, tax, and other fields for traceable classification results. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings are only intended to illustrate the principles, implementation manners, applications, characteristics and effects of the specific embodiments and other related contents, and cannot be considered as limitations to the present application.

[0060] In the drawings:

[0061] Figure 1 a schematic diagram of steps S101 to S111 described in the specific embodiment;

[0062] Figure 2 a schematic diagram of steps S201 to S203 described in the specific embodiment;

[0063] Figure 3 a schematic diagram of steps S301 to S304 described in the specific embodiment;

[0064] Figure 4 a schematic diagram of steps S401 to S403 described in the specific embodiment;

[0065] Figure 5 a flowchart of the specific embodiment. DETAILED DESCRIPTION

[0066] To explain possible application scenarios, technical principles, implementable specific schemes, and purposes and effects that can be achieved of the present application in detail, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described in the present text are only used to more clearly illustrate the technical scheme of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0067] Although OCR and natural language processing techniques have made some progress in the field of voucher text extraction and classification, the existing technology still has the following main defects and deficiencies: the existing voucher classification technology relies on keyword matching or shallow machine learning, which is difficult to cope with complex scenarios with variable formats, mixed terminologies and ambiguous semantics, resulting in low classification accuracy and being easily affected by format disturbances. Especially for vouchers such as "meeting notice", "receipt" and "itinerary" with similar formats but different semantics, misjudgment is easy to occur. The existing system generally completes model training based on a static training set, and cannot dynamically retrieve relevant knowledge documents according to user input content to assist classification decision-making. This makes the system have poor generalization ability and high error rate when facing boundary cases, new voucher categories or changes in user expression methods. Black box models based on deep neural networks often lack explainability and cannot answer "why it is judged as this type of voucher". For industries such as finance and auditing that require high compliance and transparency, this defect becomes an important obstacle to popularization and landing. Current systems often handle processes such as OCR, feature extraction, classification, and display separately, and there is a lack of unified protocols and standard interfaces between systems, making it difficult to build stable and efficient integrated processing flows and adapt to complex business flows and high-concurrency processing needs.

[0068] In summary, the existing technology still has obvious deficiencies in the intelligentization, automation, and semantic understanding ability of voucher classification, and there is an urgent need for a new method that integrates OCR recognition, semantic vector retrieval, language model reasoning, and other modules to build a voucher classification system with high accuracy, strong generalization ability, and good explainability to meet the urgent needs of intelligent financial processing in real-world scenarios.

[0069] For a better understanding of the present application, reference will be made to the following detailed description of the embodiments of the present application: Figure 1 The embodiment provides a voucher classification method based on RAG, which comprises the following steps:

[0070] S101, receiving digital image data of a voucher to be classified;

[0071] S102, performing multi-modal optical character recognition processing on the digital image data to generate a structured OCR result containing text content and layout information thereof;

[0072] S103, based on the structured OCR result, extracting a text semantic feature vector through a semantic embedding model and extracting a visual layout feature vector through an image feature extraction model;

[0073] S104, performing feature fusion on the text semantic feature vector and the visual layout feature vector to generate a multi-modal query vector;

[0074] S105, input the multi-modal query vector into the pre-constructed multi-modal knowledge vector library for approximate nearest neighbor search to obtain a search result, the search result including a plurality of similar samples ranked in the front, a semantic similarity score of each sample with the query vector, and category metadata associated with each sample;

[0075] S106, generate an initial candidate category list according to the search result;

[0076] S107, for each candidate category in the initial candidate category list, extract the corresponding key field value from the structured OCR result according to the key field rule set corresponding to the candidate category;

[0077] S108, calculate the matching degree of the extracted key field value and the corresponding category field mode in the category rule knowledge base to generate an evidence credibility score;

[0078] S109, fuse the semantic similarity score and the evidence credibility score of each candidate category to generate a comprehensive confidence score, and re-rank the candidate category list according to the comprehensive confidence score;

[0079] S110, select a classification decision path according to the comprehensive confidence score distribution of the re-ranked candidate category list, including:

[0080] If the highest comprehensive confidence score meets the preset threshold condition, the corresponding category is directly output as the classification result;

[0081] Otherwise, based on the feature contradiction points between the candidate categories, a comparison prompt word is dynamically constructed, and the comparison prompt word and the structured OCR result are submitted to a large language model for arbitration reasoning to obtain a final classification result;

[0082] S111, output the classification result and an explainability report including the search sample, the key field evidence, and the decision path.

[0083] In step S101, receiving digital image data of a document to be classified means obtaining a document image file or a PDF document uploaded by a user through a system interface. Through a standardized data input interface, multiple formats of document data access are supported, realizing unified access of multi-source heterogeneous document data.

[0084] In step S102, multi-modal optical character recognition processing of digital image data means using an improved OCR engine to extract text content and its layout information. Through deep learning OCR technology, not only the text content is recognized, but also the text position, table structure, seal area and other visual elements are detected to generate structured OCR results, obtaining complete document representation including semantic and layout information.

[0085] In step S103, extracting a feature vector based on a structured OCR result refers to processing text and visual information using a semantic embedding model and an image feature extraction model respectively. Deep semantic features of text are extracted through a pre-trained language model, and overall layout features of the certificate are extracted through a convolutional neural network. Dual-modal feature representations representing semantic content and visual structure of the certificate are obtained.

[0086] In step S104, feature fusion of the text semantic feature vector and the visual layout feature vector refers to generating a unified multi-modal query vector using feature concatenation or attention weighting. Text and visual information are represented in the same vector space through a feature fusion algorithm, generating a query vector containing multi-modal information, and effectively fusing text semantics and visual layout information.

[0087] In step S105, inputting the multi-modal query vector into a multi-modal knowledge vector library for retrieval refers to using an approximate nearest neighbor algorithm to search for similar samples in a pre-constructed vector library. Through efficient vector similarity calculation, the most similar historical samples and their metadata to the query certificate are quickly retrieved, realizing fast candidate generation based on multi-modal similarity.

[0088] In step S106, generating an initial candidate category list according to the retrieval result refers to counting and sorting the similar samples retrieved according to categories. By aggregating category information in the retrieval results, a preliminary classification candidate list and its confidence score are generated, realizing fast category filtering based on retrieval results.

[0089] In step S107, extracting key field values for candidate categories refers to extracting specific fields from the OCR result according to the rule set of each candidate category. Using named entity recognition and rule templates, the values of key discriminant fields for different categories are extracted, obtaining key field information that can be used for evidence verification.

[0090] In step S108, calculating the matching degree of the key field value and the field pattern refers to comparing and verifying the extracted field value with the pattern in the category rule knowledge base. Through format checking, regular matching, numerical verification, etc., the degree of conformity of the field value with the expected pattern is calculated, obtaining an evidence credibility score based on field verification, providing quantifiable verification evidence for classification decision.

[0091] In step S109, fusing the semantic similarity score and the evidence credibility score refers to using a weighted algorithm to integrate the retrieval similarity and the field verification result, and through a configurable weight coefficient, the content-based similarity judgment and the rule-based evidence verification are organically combined. A more reliable comprehensive confidence score is generated.

[0092] In step S110, selecting a decision path according to the comprehensive confidence score distribution means selecting different processing flows of direct output or LLM arbitration based on the score results, judging the classification confidence through a preset threshold, intelligently switching between efficient direct output and fine arbitration reasoning, realizing the optimal balance of classification efficiency and accuracy, and greatly improving the overall performance of the system.

[0093] In step S111, outputting the classification result and the explainability report means generating a structured output result containing the complete decision process. By recording the intermediate results and decision basis of the whole process, a classification report with traceability is generated. Complete audit clues and decision basis are provided to meet the strict requirements of financial, tax and other fields for the explainability of classification results.

[0094] The embodiment effectively improves the accuracy and robustness of document classification through multi-modal feature fusion and approximate nearest neighbor search, can handle complex document scenarios with variable formats and ambiguous semantics, enhances the explainability and reliability of classification decisions through the fusion mechanism of evidence credibility score and semantic similarity score, improves the system processing efficiency while ensuring classification accuracy, and finally provides complete technical basis for audit and review through the explainability report output, meeting the strict requirements of financial, tax and other fields for the traceability of classification results.

[0095] Please refer to Figure 2 In some embodiments, according to the search results, an initial candidate category list is generated, including:

[0096] S201, classifying and aggregating the top-ranked similar samples returned by the search according to their corresponding category labels to obtain multiple classified categories;

[0097] S202, calculating the initial confidence score of each category based on the semantic similarity scores of all samples included in the category, including:

[0098] Taking the maximum value of the semantic similarity scores of all similar samples in the same category;

[0099] Alternatively, taking the weighted average value of the semantic similarity scores of all similar samples in the same category, and the weight of the similar sample is determined according to the ranking position of the similar sample in the search results;

[0100] S203, sorting the multiple classified categories according to the initial confidence scores, and selecting the top N categories to form an initial candidate category list.

[0101] In step S201, the classification and aggregation of the top-ranked similar samples returned by the search according to the category label means grouping and counting the similar samples according to the category to which they belong. Through the key metadata of the category label, the search results are preliminarily clustered and analyzed, and the dispersed similar samples are reorganized according to the category dimension. The structured arrangement of the search results is achieved, and the repeated calculation of the similar samples of the same category is avoided.

[0102] In step S202, calculating the initial confidence score of each category based on the semantic similarity scores of all samples contained in the category means aggregating the sample-level similarity scores into a category-level confidence index by using a statistical method. Through mathematical statistical methods, the individual similarity information of multiple similar samples is fused into a comprehensive score representing the matching degree of the entire category. The conversion from the sample-level similarity to the category-level confidence is achieved.

[0103] Taking the maximum value or the weighted average value of the semantic similarity scores of all similar samples in the same category means using two different aggregation strategies to calculate the category confidence score. The maximum value strategy selects the most similar sample in the category as the representative, embodying the optimal matching idea; the weighted average value strategy considers the similarity of all samples and gives different weights according to the sample ranking position, embodying the overall matching idea. A flexible and configurable confidence calculation method is provided, the maximum value strategy is suitable for scenarios requiring strict matching, and the weighted average strategy is suitable for scenarios requiring comprehensive consideration, improving the adaptability and flexibility of the system.

[0104] In step S203, sorting the multiple categories after classification according to the initial confidence score and selecting the top N categories means arranging the categories in descending order according to the calculated confidence scores and selecting the top N categories. Through the sorting algorithm, the candidate categories are arranged from high to low according to the matching degree, and the size of the candidate set is controlled by the threshold N, balancing the recall rate and the accuracy, generating an initial candidate category list of controllable size and optimal quality, which ensures that important candidate categories are not missed and avoids the calculation redundancy caused by an excessively large candidate set, improving the overall processing efficiency.

[0105] The embodiment realizes the conversion from the original search results to the high-quality candidate category list through the progressive processing of three levels of classification aggregation, confidence calculation, and sorting selection. Through the provision of multiple confidence calculation strategies, the adaptability of the system in different business scenarios is enhanced; through the controllable candidate list generation mechanism, the calculation efficiency is optimized while ensuring the recall rate; the entire processing process provides high-quality input for subsequent evidence verification and decision arbitration.

[0106] Please refer to Figure 3In some embodiments, the semantic similarity scores and the evidence credibility scores of each candidate category are fused to generate a comprehensive confidence score, and the candidate category list is reordered according to the comprehensive confidence score, including:

[0107] S301, obtaining an initial confidence score and an evidence credibility score of each candidate category in an initial candidate category list;

[0108] S302, fusing the initial confidence score and the evidence credibility score by using a weighted summation algorithm to generate a comprehensive confidence score of the current candidate category, and the weight coefficient of the initial confidence score and the weight coefficient of the evidence credibility score are configured according to the classification scene requirement;

[0109] S303, repeating the above steps until the comprehensive confidence scores of all candidate categories are obtained;

[0110] S304, reordering the initial candidate category list according to the comprehensive confidence scores of all candidate categories in descending order of the comprehensive confidence score values to generate a reordered candidate category list.

[0111] In step S301, obtaining an initial confidence score and an evidence credibility score of each candidate category in an initial candidate category list means collecting two-dimensional score data from different processing modules. The initial confidence score based on semantic similarity and the evidence credibility score based on key field verification are obtained through a data interface, realizing unified collection of multi-dimensional score data and providing a complete data basis for comprehensive confidence calculation.

[0112] In step S302, fusing the initial confidence score and the evidence credibility score by using a weighted summation algorithm means linearly combining the two scores using a configurable weight coefficient. The relative importance of semantic similarity and evidence verification in the final decision is adjusted by the weight coefficient, and a comprehensive score that better reflects the actual business requirement is generated. A flexible score fusion mechanism is provided, and users can adjust the weight configuration according to different types of credentials and business scenarios, enhancing the adaptability and practicality of the system.

[0113] In step S303, repeating the above steps until the comprehensive confidence scores of all candidate categories are obtained means that the same score fusion calculation process is performed for each category in the initial candidate category list. Through batch processing, it is ensured that all candidate categories are subjected to unified score fusion calculation, ensuring consistency and integrity of the processing. Standardized processing of all candidate categories is realized, a unified score benchmark is provided, and the fairness and comparability of the sorting result are ensured.

[0114] In step S304, reordering according to the comprehensive confidence score of all candidate categories means arranging the candidate categories in descending order according to the comprehensive score. The candidate categories are reorganized from high to low in comprehensive confidence by an ordering algorithm, an ordered list reflecting the final confidence degree is generated, an optimized ordering result verified in multiple dimensions is produced, a more reliable and accurate basis is provided, and the accuracy and credibility of the final classification result are significantly improved.

[0115] The embodiment combines the initial confidence score based on content similarity and the evidence confidence score based on rule verification through a weighted fusion algorithm to generate a comprehensive confidence score that better reflects the actual business needs. Through a configurable weight coefficient mechanism, flexible adaptation to different business scenarios is achieved; through complete batch processing and global sorting, the comprehensiveness and consistency of candidate category evaluation are ensured.

[0116] Please refer to Figure 4 In some embodiments, the matching degree of the extracted key field value and the corresponding category field mode in the category rule knowledge base is calculated to generate an evidence confidence score, including:

[0117] S401, obtain the field mode definition of the current candidate category from the category rule knowledge base, the field mode definition including the name, data type, format requirement and verification rule of each key field;

[0118] S402, for each key field, calculate the matching degree between the extracted field value and the corresponding field mode definition to obtain the matching score of the single field, the matching score of the single field being calculated including at least one of format compliance check, regular expression matching, numerical range verification, or semantic compliance judgment based on a pre-trained model;

[0119] S403, based on the matching scores of all key fields, the overall evidence confidence score of the current candidate category is calculated by using a weighted average algorithm, and the weight of each key field is configured according to its importance in category discrimination.

[0120] In step S401, obtaining the field mode definition of the current candidate category from the category rule knowledge base means querying the pre-constructed structured rule base to obtain the complete field specification information of the specific credential category. Through the category identifier, the definition details of all key fields of the category are retrieved in the rule knowledge base, including field name, data type, format constraint and verification rule, etc. metadata. The complete rule benchmark is provided for subsequent field verification, ensuring the standardization and accuracy of the verification process.

[0121] In step S402, the matching degree calculation for each key field means that the extracted field value is checked for multi-dimensional compliance by using multiple verification techniques. Through multiple ways such as format checking, regular matching, range verification and semantic judgment, the matching degree of the field value and the expected mode is comprehensively evaluated to generate a quantitative matching score. Multi-level verification from simple existence checking to complex semantic compliance judgment is realized, and the fineness and accuracy of field verification are improved.

[0122] In step S403, the total evidence credibility score is calculated by using a weighted average algorithm, which means that different weights are assigned to the matching scores of each field according to the importance of the field and are fused by weighting. The relative importance of different key fields in category discrimination is reflected by the weight coefficient, and the matching scores of multiple fields are fused into a single score representing the overall evidence strength. The verification results of each field are considered, and the discrimination value of different fields is also reflected, so that the generated total evidence credibility score can more accurately reflect the actual matching degree of the candidate category and the current credential.

[0123] The embodiment realizes the complete conversion process from the original field value to the total evidence credibility score through the progressive processing of three levels of field mode definition obtained from the rule base, multi-dimensional field matching degree calculation and weighted fusion. Through the comprehensive use of multiple verification techniques, the accuracy of field-level verification is improved; through the weighted average algorithm, the field-level results are reasonably aggregated to the category-level evaluation. The evidence credibility score generated by the whole process has high reliability and interpretability, which significantly improves the accuracy and credibility of the classification system.

[0124] In some embodiments, the samples in the multi-modal knowledge vector library include text semantic vectors, visual layout vectors, category labels, and key field rule sets;

[0125] Furthermore, the multi-modal knowledge vector library is configured to be constructed using a distributed vector database architecture;

[0126] The multi-modal query vector is input into the pre-constructed multi-modal knowledge vector library for approximate nearest neighbor search to obtain a search result, which includes a number of similar samples ranked first, a semantic similarity score of each sample with the query vector, and category metadata associated with each sample, including:

[0127] A graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm is used to perform vector similarity search in the multi-modal knowledge vector library;

[0128] The similarity score between the multi-modal query vector and each sample vector in the multi-modal knowledge vector library is calculated, and the similarity score is obtained by using a cosine similarity algorithm or an inner product similarity algorithm;

[0129] According to the similarity scores, all the search results are sorted in descending order, and a number of samples with high ranks are selected as similar samples;

[0130] Metadata associated with each similar sample is extracted from the multi-modal knowledge vector library, including category labels, key field rule set identifiers, and original feature information of sample source credentials.

[0131] In this embodiment, the samples in the multi-modal knowledge vector library contain text semantic vectors, visual layout vectors, category labels, and key field rule sets, which means that each data record in the vector library contains text semantic features, visual layout features, category labels, and key field rule information. By uniformly storing multi-modal features and associated metadata, a complete knowledge representation system is constructed, which contains both content features and structured rule knowledge.

[0132] In this embodiment, the multi-modal knowledge vector library is configured to be built using a distributed vector database architecture, which means that a special vector database supporting distributed storage and parallel computing is used to deploy the knowledge base system. Through distributed architecture, massive vector data is stored and retrieved in parallel, and multi-node computing resources are used to improve retrieval performance and system scalability. The efficiency and throughput of large-scale vector retrieval are significantly improved, supporting multi-modal vector similarity retrieval and metadata association query, meeting the high concurrency processing needs in actual business scenarios.

[0133] In this embodiment, the vector similarity retrieval is performed in the multi-modal knowledge vector library using a graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm, which means that two efficient approximate algorithms are used to accelerate the similarity search process of high-dimensional vectors. By constructing navigation graphs or vector quantization, etc. approximate techniques, the computational complexity is greatly reduced under the premise of ensuring retrieval accuracy, and fast similar sample recall is achieved. The best balance between retrieval accuracy and computational efficiency is achieved, so that the system can quickly process high-dimensional multi-modal vector similarity retrieval tasks.

[0134] In this embodiment, the similarity scores of the multi-modal query vector and each sample vector in the multi-modal knowledge vector library are calculated, and the similarity scores are obtained by cosine similarity algorithm or inner product similarity algorithm, which means that the standard similarity measurement method in the vector space model is used to quantify the matching degree of the query vector and the sample vector in the library. The cosine value or dot product result between vectors is calculated by mathematical formula to obtain a numerical score representing the semantic similarity degree.

[0135] In the embodiment, all the retrieval results are ranked in descending order according to the similarity scores, and the top-ranked several samples are selected as similar samples, which means that the retrieval results are arranged in order from high to low according to the similarity scores, and the optimal result set is intercepted. Through the sorting algorithm, the top-ranked several samples most similar to the query vector are selected to form a high-quality candidate sample set. The fine selection of retrieval results is realized.

[0136] In the embodiment, the metadata associated with each similar sample is extracted from the multi-modal knowledge vector library, including the category label, the key field rule set identifier, and the original feature information of the sample source certificate, which means that the structured description information associated with the similar sample is obtained synchronously in the retrieval process. Through the metadata indexing mechanism of the vector database, the category definition, field rules and other key information associated with the similar vector are obtained at the same time. Complete context information is provided for subsequent category aggregation, evidence verification and decision reasoning, supporting the end-to-end classification reasoning process.

[0137] The embodiment realizes fast and accurate multi-modal vector retrieval and metadata association query by constructing a distributed vector database containing multi-modal features and rich metadata, using an efficient approximate nearest neighbor search algorithm and a standard similarity measurement method. The distributed vector database architecture ensures the high performance and high availability of the system; the graph-based approximate nearest neighbor search algorithm or the quantization-based approximate nearest neighbor search algorithm balances the retrieval accuracy and computational efficiency; and the efficiency and effectiveness of the retrieval link in the certificate classification system are significantly improved.

[0138] In some embodiments, the text semantic feature vector and the visual layout feature vector are fused to generate a multi-modal query vector, including:

[0139] The text semantic feature vector and the visual layout feature vector are subjected to dimension alignment processing.

[0140] The processed text semantic feature vector and the visual layout feature vector are fused using a weighted fusion algorithm.

[0141] The weighted fusion algorithm is configured to use an attention mechanism to adaptively weight the importance of the text and visual features, and assign configurable weight coefficients to the text semantic feature vector and the visual layout feature vector, which are dynamically adjusted according to the certificate type and classification scenario.

[0142] The fused feature vector is subjected to normalization processing to generate a final multi-modal query vector.

[0143] In this embodiment, the dimension alignment processing of the text semantic feature vector and the visual layout feature vector refers to adjusting the feature vectors of two different sources to the same dimension space through linear transformation or projection method. That is, the dimensions of the text and visual features are unified using a fully connected layer or a projection matrix to eliminate the dimension difference problem caused by different feature extraction models, ensuring the comparability and operability of different modal features in the mathematical space.

[0144] In this embodiment, the weighted fusion algorithm is used to fuse the processed text semantic feature vector and visual layout feature vector, which refers to using learnable weight parameters to linearly or nonlinearly combine the features of the two modalities. By adjusting the relative contribution of text and visual features in the final representation through weight coefficients, a unified feature representation that can retain both semantic information and visual information is generated. The effective integration of multi-modal information is achieved, and a more representative fusion feature than single modal feature is generated.

[0145] In this embodiment, the weighted fusion algorithm is configured to use an attention mechanism to adaptively weight the importance of text and visual features, which refers to dynamically learning the importance of different modal features in the current classification task through an attention calculation module. The attention weight generation network automatically calculates the weight coefficients of each modality according to the content of the input features, realizing content-aware feature weighting fusion. An intelligent feature importance evaluation mechanism is provided, which can automatically adjust the fusion strategy according to the characteristics of different credentials, improving the accuracy and adaptability of feature fusion.

[0146] In this embodiment, configurable weight coefficients are assigned to the text semantic feature vector and the visual layout feature vector, and the weight coefficients are dynamically adjusted according to the type of the credential and the classification scene, which refers to establishing a mapping relationship between the weight coefficients and the business scenarios to realize adaptive configuration of parameters. Through pre-set configuration rules or online learning mechanism, the fusion weights of each modal feature are dynamically adjusted according to different credential types (such as invoices, bills, notices, etc.) and classification scene requirements. The adaptability of the system in different business scenarios is enhanced, so that the feature fusion strategy can be flexibly optimized according to actual needs.

[0147] In this embodiment, the normalized feature vector is generated after the fused feature vector is normalized, which refers to converting the fused feature vector into a unit vector using L2 normalization or other standardization methods. Through mathematical transformation, the feature vector is mapped to the unit hypersphere, eliminating the influence of vector length on similarity calculation, ensuring the reliability and comparability of the similarity calculation result.

[0148] The embodiment realizes efficient fusion of text semantic features and visual layout features through multiple levels of fine operations such as dimension alignment, weighted fusion, attention mechanism, dynamic weight adjustment and normalization processing. The adaptive evaluation of feature importance is realized through the attention mechanism, the business adaptability is enhanced through dynamic weight configuration, and the stability of the retrieval is ensured through the normalization processing. The entire feature fusion process generates a high-quality multi-modal query vector, which significantly improves the accuracy and robustness of the credential classification system.

[0149] In some embodiments, based on the structured OCR result, a text semantic feature vector is extracted through a semantic embedding model, and a visual layout feature vector is extracted through an image feature extraction model, including:

[0150] The text content in the structured OCR result is input into a pre-trained semantic embedding model, and semantic encoding is performed through the multi-layer Transformer structure of the semantic embedding model, to output a fixed-dimension text semantic feature vector;

[0151] The digital image data is input into a pre-trained convolutional neural network model, and feature extraction is performed through multiple convolutional layers and pooling layers of the convolutional neural network model, to output a visual layout feature vector representing the overall layout structure of the credential;

[0152] The semantic embedding model adopts a deep bidirectional language representation model based on a self-attention mechanism, and the image feature extraction model adopts a deep residual network or a visual Transformer architecture;

[0153] The extraction process of the text semantic feature vector also includes text cleaning and standardization preprocessing of the OCR recognition result, and the extraction process of the visual layout feature vector also includes size normalization and data enhancement processing of the input image.

[0154] In the embodiment, inputting the text content in the structured OCR result into the pre-trained semantic embedding model for semantic encoding means that deep learning model is used for deep semantic understanding of the text information extracted by OCR. Through the self-attention mechanism in the multi-layer Transformer structure, the context dependency relationship and semantic association between words in the text are captured, and the variable-length text sequence is encoded into a fixed-dimension dense vector representation, obtaining a feature representation that can accurately reflect the semantic content of the text.

[0155] In this embodiment, inputting the digital image data into the pre-trained convolutional neural network model for feature extraction refers to using a deep convolutional network to learn visual layout features from the original image. Through the cascading operation of multiple convolutional layers and pooling layers, the local features of the image are gradually extracted and aggregated into global feature representations, and finally a high-dimensional feature vector representing the overall layout structure of the certificate is output. The visual information such as the layout structure, text arrangement, and seal position of the certificate image is captured.

[0156] In this embodiment, the semantic embedding model uses a deep bidirectional language representation model based on self-attention mechanism, which means using a pre-trained language model similar to BERT to process text semantic encoding tasks. Through the bidirectional Transformer architecture, the left and right context information of the text sequence is considered simultaneously, and the importance weight between words is dynamically calculated using the self-attention mechanism to generate text representations with rich semantic information. The accuracy and depth of text semantic understanding are improved, and it is particularly suitable for processing short texts and professional terms commonly found in certificates.

[0157] In this embodiment, the image feature extraction model uses a deep residual network or a visual Transformer architecture, which means using two advanced computer vision models to extract image features. The deep residual network solves the gradient vanishing problem of deep networks through skip connections, and the visual Transformer captures the global relationship between image blocks through self-attention mechanisms. Multiple high-performance visual feature extraction schemes are provided, and the appropriate model architecture can be flexibly selected according to different precision and efficiency requirements.

[0158] In this embodiment, the extraction process of the text semantic feature vector includes text cleaning and standardization preprocessing of the OCR recognition result, which means data purification processing of the original OCR output before semantic encoding. Through methods such as rule filtering, wrong word correction, and format standardization, noise and errors generated in the OCR recognition process are eliminated, and the quality of the input text is improved. The interference of noise data on the semantic model is reduced, and the accuracy and reliability of text feature extraction are improved.

[0159] In this embodiment, the extraction process of the visual layout feature vector includes size normalization and data enhancement processing of the input image, which means standardization and enhancement operations on image data before feature extraction. Through image scaling, padding, and enhancement preprocessing techniques, the input image meets the model requirements and improves the diversity of data. The stability and generalization ability of image feature extraction are improved, and the adaptability and robustness of the model to different size and quality certificate images are enhanced.

[0160] In this embodiment, by using a deep bidirectional language representation model based on self-attention mechanism and a deep residual network or visual Transformer architecture, efficient feature extraction of text semantic content and image layout structure is realized respectively. The quality of input data is improved through text cleaning and standardization preprocessing, and the stability of visual features is improved through image size normalization and data enhancement processing. The entire feature extraction process generates high-quality text semantic feature vectors and visual layout feature vectors, significantly improving the accuracy and robustness of the certificate classification system.

[0161] In some embodiments, the digital image data is subjected to multi-modal optical character recognition processing to generate structured OCR results containing text content and its layout information, including:

[0162] A deep learning optical character recognition engine is used to detect and recognize text in the digital image data, extract the text content and its position coordinate information in the image, and obtain the recognition result.

[0163] The recognition result is subjected to structured processing by a layout analysis algorithm to detect and identify the layout information of text blocks, table regions, seal regions and logo regions.

[0164] The text content and its corresponding layout information are associated and integrated to generate structured OCR result data.

[0165] The structured OCR result is organized in JSON or XML format, containing text content, text position coordinates, font size, text block type and special region identification information.

[0166] The multi-modal optical character recognition processing also includes an image preprocessing step, including grayscale, binarization, noise removal and image enhancement operations on the input image to improve the accuracy of character recognition.

[0167] In this embodiment, the use of a deep learning optical character recognition engine for text detection and recognition in digital image data refers to the use of OCR technology based on deep neural networks to automatically locate and recognize text content in images. Through convolutional neural networks for text region detection, and recurrent neural networks or Transformer architecture for sequence recognition, the final output is the text content and its precise position coordinates in the image, realizing high-precision text extraction and positioning.

[0168] In this embodiment, the structured processing of the recognition result by the layout analysis algorithm refers to using computer vision and machine learning algorithms to perform region segmentation and type identification on the OCR recognition result. Through image segmentation, region clustering, and classification model technologies, different types of layout elements such as text blocks, tables, seals, and logos are automatically detected and distinguished, and the bounding boxes and attribute information of each element are recorded. The complete structural information of the certificate image is obtained.

[0169] In this embodiment, the association and integration of the text content with its corresponding layout information refers to establishing a mapping relationship between the text content and the layout elements to form a structured data record. Through coordinate matching and spatial relationship analysis, the recognized text content is attributed to the corresponding layout area, and metadata information such as region type and font size is added. A complete structured representation with content semantics and spatial layout is generated.

[0170] In this embodiment, the structured OCR result is organized in JSON or XML format, which means using a standardized data exchange format to store and transmit the OCR processing result. Through a hierarchical data organization structure, text content, position coordinates, font attributes, region types, and other information are encapsulated and serialized in the form of key-value pairs. A machine-readable and easy-to-parse data format is provided, improving the interoperability and extensibility of the system.

[0171] In this embodiment, the multi-modal optical character recognition process includes an image preprocessing step, including grayscale, binarization, noise removal, and image enhancement operations on the input image, which means performing quality optimization processing on the original image before OCR recognition. Through image processing algorithms, image quality is improved, including color space conversion, threshold segmentation, filter denoising, contrast enhancement, and other techniques to improve image clarity and recognizability. The accuracy and robustness of subsequent OCR recognition are significantly improved, especially when dealing with low-quality, blurred, or unevenly lit certificate images.

[0172] In this embodiment, through the collaborative processing of deep learning OCR engine, layout analysis algorithm, structured data integration, standardized data format organization, and image preprocessing, a complete conversion process from original image to structured OCR result is realized. Through multi-modal optical character recognition processing, not only the text content is extracted, but also rich layout and structural information is captured, and the entire processing flow significantly improves the utilization efficiency and recognition accuracy of certificate image information, laying a solid data foundation for the RAG-based certificate classification system.

[0173] In a second aspect, the embodiment also provides a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions being executed by a processor to implement the method of the first aspect.

[0174] In a third aspect, the embodiments further provide an electronic device comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method in the first aspect.

[0175] For ease of understanding, please refer to Figure 5 The following examples are provided for specific illustration:

[0176] The present example provides a voucher classification system based on retrieval enhancement generation model and its construction method, which integrates OCR recognition, text block segmentation, semantic embedding, vector retrieval, prompt word construction and large language model reasoning, etc. Form a voucher classification process with automation, intelligence and high accuracy. Specifically, it includes the following technical steps and modules:

[0177] 1. Voucher data acquisition and OCR preprocessing module

[0178] This module aims to realize the image acquisition, OCR recognition and text preprocessing of multiple types of business vouchers, and provide high-quality and structured text input for subsequent semantic understanding and classification model. This module includes the following specific technical processes:

[0179] (1) Voucher image acquisition

[0180] This system supports multiple business source bill image acquisition methods, including: scanner input, mobile phone photograph upload, PDF file page segmentation, system interface reception (such as OA system, travel platform, etc.).

[0181] The types of vouchers collected cover but are not limited to: value-added tax ordinary invoice, electronic ticket, train ticket, ship ticket, accommodation invoice, meeting notice, hospital receipt, bank return slip, etc. A total of dozens of structured or semi-structured bill formats.

[0182] (2) OCR text recognition

[0183] This system optimally uses OCR engines supporting multiple languages and strong anti-interference ability, such as PaddleOCR, EasyOCR, Tencent Cloud OCR, etc. The OCR engine configuration supports automatic detection of image direction (rotation correction), multi-page PDF processing, and table structure restoration. After each voucher image is recognized, the text line content is output.

[0184] (3) Multi-language and layout adaptability strategy

[0185] For some foreign-related bills (such as English hotel bills, overseas tickets, etc.), the system supports mixed English and Chinese recognition; for vertical text (such as some handwritten bills, old-style documents), image direction detection and rotation correction are performed; if there is a recognition failure area, a backup OCR engine or a manual review interface can be used.

[0186] 2. Text block embedding and vector library construction module (detailed description)

[0187] This module aims to convert the text blocks extracted and segmented from the voucher image into a vector representation with semantic expression ability, and store it in the vector library together with the original voucher label, definition description and other meta-information, to build an efficient knowledge index system for voucher semantic matching and classification.

[0188] (1) Embedding model selection and loading

[0189] The system preferably uses a Chinese or Chinese-English mixed embedding model with strong semantic expression ability to ensure that the vector has high semantic similarity alignment ability.

[0190] (2) chunk vectorization process

[0191] For each text block (chunk) generated by the OCR module, the embedding encoding is performed according to the following steps:

[0192] Clean up spaces and special characters;

[0193] Control the maximum length not to exceed the upper limit supported by the model (such as 512 characters);

[0194] Batch input into the embedding model for vector encoding, output as a high-dimensional dense vector (such as 768-dimensional or 1024-dimensional);

[0195] Each vector maintains a binding relationship with the original chunk, forming a structured record.

[0196] In this example, the structured data record formed after the vectorization of a single text block (chunk) is stored in the vector index library to facilitate subsequent retrieval and matching. Each record contains text content, semantic vectors and related meta-information. Specifically, it includes the following parts:

[0197] Text content (chunk_text): This field stores the specific text segment identified and segmented from the original voucher image through OCR. For example, "This meeting will be held on July 1, 2023, at …".

[0198] Semantic vector (embedding): This field stores a high-dimensional numerical array (i.e., dense vector). This vector is obtained by calculating the "text content" through a deep learning embedding model, which can capture the deep semantic information of the text and is the core of implementing semantic similarity matching.

[0199] Category label: This field explicitly labels the category of the text block. For example, the label "meeting notice" indicates that this text comes from a meeting notice type of certificate.

[0200] Category prompt: This field stores a descriptive text of the category. This description defines the typical characteristics of the category certificate, which is used to dynamically build the prompt for the large language model to guide accurate classification. For example, "meeting notice type of certificate usually contains meeting name, topic, time, place and other natural language paragraphs".

[0201] By structurally binding text, vector, category label and various meta-information, the application constructs an information-rich knowledge vector library, laying a foundation for subsequent efficient, accurate and interpretable certificate classification.

[0202] (3) Vector index library construction and management

[0203] Vector engine selection: Milvus is distributed, scalable, supports attribute filtering and strong consistency recall;

[0204] Index construction process:

[0205] Write all vectors in batches to the vector library to establish index (Index);

[0206] Select appropriate approximate index structure: small-scale data: Flat, HNSW; medium and large-scale: IVF+PQ;

[0207] Configure vector library metadata field to support multi-dimensional retrieval (such as category filtering, confidence filtering, etc.);

[0208] Establish inverted relationship to support "category→vector" reverse query.

[0209] (4) Vector incremental update and management strategy

[0210] The system supports dynamic expansion: new credential categories or new data can be added to the vector library in real time; for low-quality chunks or low-confidence OCR output vectors, set the expiration flag or temporarily do not enter the library; periodically perform the index reconstruction task to optimize the Top-K retrieval effect; this step aims to solve the problem of declining retrieval performance caused by dynamic data growth. After the vector library continuously receives new data, the original index structure may become suboptimal, resulting in slower retrieval speed or reduced accuracy. Therefore, by periodically reconstructing a new index with the optimal structure based on the current full data in the library, data fragmentation can be eliminated, and the balance of the index can be restored, thereby ensuring that the system can maintain high speed and high accuracy in Top-K semantic retrieval. Support for automatically counting the distribution of samples in each category based on category labels for model training, balance judgment, and other downstream tasks.

[0211] (5) Top-K vector similarity retrieval (for subsequent modules)

[0212] In the classification stage, the system can call this vector library interface to perform Top-K nearest neighbor search on the text blocks to be classified, including:

[0213] Supporting Euclidean distance (L2), cosine similarity (Cosine), and dot product similarity (Dot Product);

[0214] Returns:

[0215] Matching chunk text;

[0216] The category to which the matching vector belongs;

[0217] Similarity score;

[0218] Associated prompt words;

[0219] Threshold filtering of low-quality candidates can be combined with confidence.

[0220] 3. User file processing module

[0221] This module is responsible for processing various types of credential images or PDF files uploaded by users through the system front end or interface, and obtaining semantic matching results between the classified credentials in the knowledge base through steps such as OCR recognition, text block segmentation, semantic embedding, and vector retrieval. This provides candidate categories for subsequent category judgment and prompt word construction. The entire module includes the following technical processes:

[0222] (1) User file access and format processing

[0223] Users can upload files through web interfaces, mobile apps, RPA interfaces, etc. The system supports the following file types: image files: JPEG, PNG, etc.; PDF files: support multiple pages, multiple languages, vector / scanning mixed structure; if it is a PDF document, the system automatically performs page processing and calls the OCR engine for recognition page by page; support single file or batch processing mode, automatically generate unique file ID, task tracking record.

[0224] (2) OCR recognition and text extraction

[0225] The system calls the trained OCR service (such as PaddleOCR deployment service) to perform text recognition on the uploaded image or PDF page by page; output structured text information, mainly text content. Image preprocessing strategies can be configured, such as image sharpness enhancement, rotation correction, color inversion, edge cropping, etc., to ensure optimal OCR effect.

[0226] (3) chunk segmentation and embedding encoding

[0227] The text extracted by OCR is sent to the embedding encoding model (such as BGE, E5, etc.) to generate high-dimensional semantic vectors; the data structure of the user uploaded file after OCR recognition, chunking and embedding encoding. This structure is the input query body for vector retrieval, which specifically includes:

[0228] Text content (chunk_text): the actual content of a text block extracted from the user uploaded credentials.

[0229] Semantic vector (embedding): the high-dimensional semantic vector corresponding to the text block content generated by the embedding model.

[0230] Unique identifier (chunk_id): a unique ID assigned to the text block, used for tracking in the processing flow, such as identifying that it comes from the first page of the nth text block of a certain upload task.

[0231] (4) Call vector retrieval service for semantic matching

[0232] The system uses the generated chunk vector as a query vector to call the vector retrieval engine deployed in the back-end for Top-N matching;

[0233] The retrieval target is all embedded text blocks in the knowledge base, and the retrieval method can be selected: Cosine Similarity (Cosine Similarity), Euclidean Distance (Euclidean Distance);

[0234] In this example, the vector retrieval service returns the original matching results, which are a list of N records found in the knowledge base that are semantically most similar to the user-uploaded text chunk. Each item in the list is an independent matching result, and specifically includes:

[0235] Matched text content (matched chunk text): The original content of the similar text chunk found in the knowledge base.

[0236] Category label (category label): The pre-assigned voucher category of the matching text chunk in the knowledge base.

[0237] Similarity score (similarity score): A quantitative value representing the semantic similarity between the user-uploaded text chunk and the matching text chunk. The higher the score, the more similar it is.

[0238] Category prompt (category prompt): Descriptive text associated with the matching category, used to build prompts for the large language model later.

[0239] (5) Candidate category generation and deduplication aggregation

[0240] Deduplicate the category label field in all matching results and retain the top-K high-frequency and high-category. The following strategies can be used for comprehensive sorting: single-category highest similarity sorting; category matching frequency weighted sorting; add a credibility threshold filter (such as the minimum matching score ≥ 0.6);

[0241] After deduplication and aggregation of the original matching results of vector retrieval, the final candidate category list is generated, which is a comprehensive refinement of the retrieval results and provides direct basis for the final classification decision, and specifically includes:

[0242] Top-K candidate categories (top_k_categories): A list of the most likely voucher category names obtained by comprehensive sorting (such as similarity, matching frequency weighting, etc.).

[0243] Category score (category_scores): A dictionary or mapping table recording each candidate category and its corresponding comprehensive score, which is an important basis for the system to judge credibility.

[0244] (6) Meta information binding (preparation for the next prompt construction)

[0245] Each Top-K category will bind its definition description, prompt word template information, which is used for prompt word generation module; the system supports automatic summarization and merging of prompt words from multiple matching sources according to the prompt field of the matching chunk; the original search record and Top-K statistical results are retained for subsequent audit and analysis.

[0246] 4. Prompt word construction and context enhancement module

[0247] To improve the accuracy and controllability of classification judgment, the system extracts corresponding prompt words from the pre-constructed prompt word template library based on Top-K candidate categories. Each voucher category is configured with a set of prompt word templates, including category definition, structural features, typical key field description, etc. The system splices multiple prompt words and voucher ocr results into a composite prompt, for example:

[0248] Main prompt word: You are a voucher classification expert proficient in image and text understanding.

[0249] Task goal: According to the title, seal, table, and text content in the picture, directly return the category. Only return the category name, no need to return other information.

[0250] Classification rule: {VOUCHERS_CLASSIFY_PROMPT}

[0251] Voucher content as follows: {OCR}

[0252] Description prompt words (VOUCHERS_CLASSIFY_PROMPT) for different categories:

[0253] Taxi receipt: After taking a taxi, the official receipt or invoice provided by the taxi company or driver. Usually the title indicates "Taxi Invoice", sometimes there are other "Passenger" "Machine Printed Invoice" and other content. Main content includes "invoice code, invoice number, date, pick-up and drop-off time, mileage, unit price, amount" and other information.

[0254] Ship ticket: Official ticket issued by shipping company, proving that the passenger has paid the fare and has the right to board the ship, usually containing ship number, departure and destination, date, cabin type, etc.

[0255] Splice the n category description prompt words returned by the main prompt word to construct a complete classification prompt word. Through this way of dynamically constructing prompt words, the number of input tokens of large models can be significantly reduced, the impact of redundant classification prompt word description can be reduced, and response speed and accuracy can be improved.

[0256] 5. Large language model inference and classification module

[0257] If the retrieved categories are left with only one unique category after deduplication aggregation, this category is directly output as the final result without submitting to large model inference. This module uses a language model to classify the user's OCR text combined with prompt words to generate the final unique category label.

[0258] 6. Output result and traceable classification record module

[0259] This module is used to record the detailed process and basis of this classification, which is convenient for later audit and traceability; the record includes: OCR text content, Top-K vector retrieval result, prompt word template content, large model response content, final determination category; support result archiving, audit tracking, error feedback correction and other business closed loop processing.

[0260] In summary, the present example proposes an automatic classification system for vouchers combining OCR recognition, semantic vector retrieval and pre-trained language model inference, with the following key innovations:

[0261] 1. Structured OCR recognition and semantic segmentation of multi-source invoices

[0262] Supports the collection and OCR recognition of multiple business voucher images such as invoices, train tickets, accommodation bills, and notification letters; converts the extracted text into "semantic chunks" through paragraph recognition and semantic segmentation algorithms, providing reasonable processing units for subsequent semantic embedding and retrieval;

[0263] 2. Constructing a domain knowledge vector library to achieve efficient semantic recall

[0264] Use pre-trained embedding models (such as BGE, Qwen-Embedding, etc.) to encode semantic vectors for chunks; construct a structured knowledge vector library containing voucher text, vector representation, category label and prompt words; use Milvus vector engine to achieve Top-K semantic similarity matching for newly uploaded voucher content.

[0265] 3. Constructing category prompt word templates based on matching results

[0266] Extract the Top-K matched voucher categories and their defining descriptions as prompt words; support multiple prompt words to build context and guide the language model to make classification judgments; prompt words can be customized according to different voucher types, such as "train ticket must contain train number / date / seat number", etc., to enhance model discrimination ability and controllability.

[0267] 4. RAG classification mechanism combining retrieval and generation

[0268] Build a RAG classification framework that combines "semantic retrieval + model inference";

[0269] The RAG classification mechanism constructed in this example is not a simple application of existing general RAG technology, but a creative reconstruction for the dual challenges of "high category density and high semantic confusion" and "high efficiency and low cost" in the context of financial document classification. Its core is a hybrid RAG architecture with "retrieval priority and reasoning backup", which embeds an intelligent dual-mode decision engine to achieve the best balance between processing efficiency and accuracy.

[0270] The following is the detailed processing flow of the RAG classification mechanism of this example:

[0271] Step one: Document vectorization and query preparation

[0272] After the user uploads the document image, the system first extracts the full text information through the OCR module. Then, the same embedding model as in the knowledge base construction is called to convert the extracted text content into a high-dimensional query vector.

[0273] Step two: Top-K semantic retrieval and metadata extraction

[0274] The system performs Top-K approximate nearest neighbor search in the pre-constructed vector knowledge base with the generated query vector. The key of this step is that the primary goal of retrieval is to obtain the metadata associated with the vector, especially the "category label". The retrieval service returns a list containing K most similar document samples, each containing the following key information: matching text block content, category label, semantic similarity score, and category definition description.

[0275] Step three: Candidate category aggregation and decision analysis

[0276] This is the first divergence point between this example and the general RAG. Instead of directly using the retrieved text content, the system performs aggregation and statistical analysis on the "category labels" of the K returned results. Category voting: count the frequency of each "category label" in the K results. Confidence calculation: combine the frequency of each category and its corresponding highest similarity score to calculate a comprehensive confidence score. Decision condition judgment: the system judges whether the aggregation result meets the "early exit" condition according to the preset threshold. For example, whether the "category with the highest confidence" is unique and its confidence exceeds 95%.

[0277] Step four: Dual-mode decision execution

[0278] Mode one: efficient "Retrieval-as-Classification" mode (trigger early exit);

[0279] Trigger condition: Step three's decision is "Yes", i.e., the search results are highly concentrated in a single category, representing a simple, unambiguous classification task.

[0280] Processing method: The system directly adopts the high-confidence category label as the final classification result and "early stops" here, terminating the subsequent process. The entire process completely avoids calling the large language model.

[0281] Data output: Directly output the classification result and record the decision path of this classification (e.g., based on Top-K search results).

[0282] Modality two: Precise "LLM-as-Arbitrator" mode (early stop not triggered);

[0283] Trigger condition: Step three's decision is "No", i.e., the category labels of the search results are divergent (such as "meeting notice" and "training notice" are recalled at the same time) or the confidence is generally low, representing a complex and ambiguous classification task.

[0284] Processing method:

[0285] Dynamic context enhancement: The system automatically switches to this mode and extracts all candidate categories (such as "meeting notice" and "training notice") and their corresponding "category definition descriptions" from the search results.

[0286] Constructing focused prompts: These descriptive texts are dynamically and specifically constructed into a context-rich prompt. This prompt explicitly requires the LLM to make a choice among the given candidate categories.

[0287] LLM inference: The dynamically constructed prompt is submitted to the large language model along with the original OCR text content of the credential for the final expert-level arbitration.

[0288] Data output: Analyze the LLM's returned results to obtain a unique final classification and record the complete decision path including search candidates, dynamic prompts, and LLM output.

[0289] Specialty of the RAG processing method in this example (difference from the general RAG)

[0290] The RAG mechanism in this example is fundamentally different from the general RAG architecture in terms of application target, process design, and model role:

[0291] Reconstruction of application target: From "content enhancement" to "metadata decision";

[0292] General RAG: Its core purpose of retrieval is to obtain text content related to the question and provide these contents as context to the LLM to generate a more content-rich and accurate answer. The retrieved text content is the main character.

[0293] This example: The core purpose of retrieval is to obtain metadata associated with the voucher - "category label". Its essence is to "borrow" the fuzzy matching ability of vector retrieval, "vote" and "consensus judgment" on the category label of the retrieval result, so as to make a direct classification decision. The retrieved text content is in a secondary position.

[0294] Innovation in process design: from "single path necessary process" to "double modal decision engine";

[0295] General RAG: No matter how simple or complex the task is, the complete path of "retrieval + LLM generation" must be taken. The call of LLM is the indispensable final link in its process.

[0296] This example: Creatively designed a "early stop" mechanism based on retrieval confidence, and built two processing modalities of "retrieval as classification" and "LLM arbitration". This enables the system to intelligently distinguish between simple and complex tasks, and use expensive LLM resources only for handling "difficult cases", thereby greatly improving the average processing efficiency and reducing the running cost when handling a large number of vouchers.

[0297] Role change of LLM: from "answer generator" to "expert arbitrator";

[0298] General RAG: LLM is the absolute core, responsible for understanding all the context retrieved and generating the final answer.

[0299] This example: The role of LLM changes from "the only decision maker" to "the expert who handles complex and ambiguous situations". It is only called when the system cannot draw a clear conclusion through retrieval, and its task is to arbitrate and choose among a few given candidate categories, rather than open-ended generation.

[0300] 5. Classification path is auditable and the system is highly scalable

[0301] The whole process records OCR text, retrieval results, candidate categories, prompt words, large model output and final classification, supporting traceability and explainable analysis; the system architecture supports flexible expansion of multiple voucher types, templates, vector libraries and model services; it can be widely applied to scenarios such as smart finance, travel reimbursement, tax risk control, medical audit, etc.

[0302] The present example can achieve the following technical effects in practical application:

[0303] 1. Significantly improve the accuracy and robustness of voucher classification

[0304] The semantic segmentation and vectorization processing is combined with the OCR text structure and semantic content, and no longer relies on shallow keyword matching; the semantic vector recall and large model language understanding ability are used to effectively identify bills with large structural changes and various expression methods; and the voucher classification accuracy is improved to more than 95%, and the robustness is significantly enhanced, and complex scenes such as fuzzy class boundaries, document mixed arrangement, and non-standard format can be effectively processed.

[0305] 2. Greatly improve system response efficiency and processing throughput

[0306] The processing logic of "semantic recall → early stop judgment → prompt word construction → large model reasoning" is adopted; the early stop mechanism of similarity threshold + class uniqueness judgment is introduced, which can directly hit the result in the case of high class concentration, reduce the number of large model calls, and greatly reduce the consumption of algorithm resources; it is suitable for concurrent processing in large-scale bill classification and financial audit scenarios.

[0307] 3. Improve the scalability and controllability of the classification system

[0308] All class prompt words, knowledge vectors, and templates are configurable items, and new classes and classification logic can be added without retraining; the system is suitable for bill classification needs in medical, transportation, financial, and education industries; it can be flexibly deployed on local servers, private clouds, or public cloud platforms, and is easy to integrate into existing business processes.

[0309] 4. Enhance the explainability and auditability of the classification process

[0310] The OCR content, Top-K search records, candidate classes, prompt word templates, and large model output results are preserved throughout the process; the system can provide "classification basis traceability", support manual review, error backtracking, and model tuning; and meet the strict requirements of financial audit, compliance supervision, and other classification results traceability.

[0311] 5. Realize the effective fusion of structured data and semantic models

[0312] The system first integrates structured OCR text information and natural language understanding models (LLM) in the voucher classification scenario; constructs a unified semantic vector space to provide a basis for subsequent question answering, auditing, and knowledge graph modules; and breaks through the complete intelligent chain of bills from image → text → semantic → category.

[0313] Compared with the prior art, the technical scheme described above receives digital image data of a to-be-classified voucher, performs multi-modal optical character recognition processing to generate a structured OCR result, extracts a text semantic feature vector and a visual layout feature vector based on the structured OCR result and fuses them to generate a multi-modal query vector, acquires similar samples and their semantic similarity scores and category metadata through approximate nearest neighbor retrieval, generates an initial candidate category list, extracts key field values and calculates evidence credibility scores for each candidate category, fuses the semantic similarity scores and the evidence credibility scores to generate a comprehensive confidence score and performs reordering, finally selects a classification decision path according to the score distribution and outputs a classification result and an explainability report. Through multi-modal feature fusion and approximate nearest neighbor retrieval, the accuracy and robustness of voucher classification are effectively improved, and complex voucher scenarios with variable formats and ambiguous semantics can be processed. Through the fusion of the evidence credibility scores and the semantic similarity scores, the explainability and reliability of the classification decision are enhanced. The final explainability report provides a complete technical basis for auditing and review, and meets the strict requirements of the financial, tax and other fields for traceable classification results.

[0314] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present application, they do not limit the patent protection scope of the present application. Any technical solution obtained by replacing or modifying the equivalent structure or equivalent process based on the essential concept of the present application, using the content described in the specification and drawings of the present application, and directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, etc., are all included in the patent protection scope of the present application.

Claims

1. A method for classifying RAG-based credentials, the method comprising: Comprising: receiving digital image data of a document to be classified; performing multi-modal optical character recognition on the digital image data to generate a structured OCR result containing text content and its layout information; extracting a text semantic feature vector through a semantic embedding model and a visual layout feature vector through an image feature extraction model based on the structured OCR result; fusing the text semantic feature vector and the visual layout feature vector to generate a multi-modal query vector; inputting the multi-modal query vector into a pre-constructed multi-modal knowledge vector library for approximate nearest neighbor search to obtain a search result, the search result including a number of similar samples ranked at the top, a semantic similarity score of each sample with the query vector, and category metadata associated with each sample; generating an initial candidate category list according to the search result; for each candidate category in the initial candidate category list, extracting corresponding key field values from the structured OCR result according to a key field rule set corresponding to the candidate category; calculating a matching degree of the extracted key field values with a corresponding category field pattern in a category rule knowledge base to generate an evidence credibility score; fusing the semantic similarity score and the evidence credibility score of each candidate category to generate a comprehensive confidence score, and reordering the candidate category list based on the comprehensive confidence score; selecting a classification decision path according to the comprehensive confidence score distribution of the reordered candidate category list, including: if the highest comprehensive confidence score meets a preset threshold condition, directly outputting the corresponding category as the classification result; otherwise, dynamically constructing a comparison prompt word based on feature contradictions among candidate categories, and submitting the comparison prompt word and the structured OCR result to a large language model for arbitration reasoning to obtain a final classification result; outputting the classification result and an explainability report containing search samples, key field evidence, and decision paths.

2. The RAG-based credential classification method of claim 1, wherein, Generating an initial candidate category list according to the search result, including: classifying and aggregating the top-ranked similar samples returned by the search according to their corresponding category labels to obtain multiple classified categories; calculating an initial confidence score for each classified category based on the semantic similarity scores of all samples included in the category, including: taking the maximum value of the semantic similarity scores of all similar samples in the same category; or, taking a weighted average of the semantic similarity scores of all similar samples in the same category, the weights of the similar samples being determined according to their ranking positions in the search result; sorting the multiple classified categories according to the initial confidence scores to select the top N categories to form the initial candidate category list.

3. The RAG-based credential categorization method of claim 1, wherein, Fusing the semantic similarity score and the evidence credibility score of each candidate category to generate a comprehensive confidence score, and reordering the candidate category list based on the comprehensive confidence score, including: obtaining the initial confidence score and the evidence credibility score of each candidate category in the initial candidate category list; fusing the initial confidence score and the evidence credibility score using a weighted summation algorithm to generate a comprehensive confidence score for the current candidate category, the weight coefficient of the initial confidence score and the weight coefficient of the evidence credibility score being configured according to classification scene requirements; Repeat the above steps until the comprehensive confidence scores of all candidate categories are obtained; According to the comprehensive confidence scores of all candidate categories, reorder the initial candidate category list in descending order of the scores of the comprehensive confidence scores to generate a reordered candidate category list.

4. The RAG-based credential categorization method of claim 1, wherein, Calculate the matching degree of the extracted key field value and the corresponding category field mode in the category rule knowledge base to generate an evidence credibility score, including: Obtain the field mode definition of the current candidate category from the category rule knowledge base, which includes the name, data type, format requirement and verification rule of each key field; For each key field, calculate the matching degree of the extracted field value and the corresponding field mode definition to obtain a single field matching score, which is obtained by at least one of the following methods: format compliance check, regular expression matching, numerical range verification, or semantic compliance judgment based on a pre-trained model; Based on the matching scores of all key fields, calculate the overall evidence credibility score of the current candidate category using a weighted average algorithm, and the weight of each key field is configured according to its importance in category discrimination.

5. The RAG-based credential categorization method of claim 1, wherein, The samples in the multi-modal knowledge vector library include text semantic vectors, visual layout vectors, category labels and key field rule sets; And the multi-modal knowledge vector library is configured to use a distributed vector database architecture; Input the multi-modal query vector into the pre-constructed multi-modal knowledge vector library for approximate nearest neighbor search to obtain search results, including the top-ranked similar samples, the semantic similarity score of each sample with the query vector, and the category metadata associated with each sample, including: Use a graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm to perform vector similarity search in the multi-modal knowledge vector library; Calculate the similarity score of the multi-modal query vector and each sample vector in the multi-modal knowledge vector library, which is obtained by cosine similarity algorithm or inner product similarity algorithm; Sort all search results in descending order according to the similarity score, and select the top-ranked samples as similar samples; Extract the metadata associated with each similar sample from the multi-modal knowledge vector library, including the category label, key field rule set identifier and sample source certificate original feature information.

6. The RAG-based credential categorization method of claim 1, wherein, Fuse the text semantic feature vector and the visual layout feature vector to generate a multi-modal query vector, including: Align the dimensions of the text semantic feature vector and the visual layout feature vector; Fuse the processed text semantic feature vector and visual layout feature vector using a weighted fusion algorithm; The weighted fusion algorithm is configured to use an attention mechanism to adaptively weight the importance of text and visual features, and assign configurable weight coefficients to the text semantic feature vector and the visual layout feature vector, which are dynamically adjusted according to the type of certificate and the classification scene; Normalize the fused feature vector to generate the final multi-modal query vector.

7. The RAG-based credential categorization method of claim 1, wherein, Based on the structured OCR result, a text semantic feature vector is extracted by a semantic embedding model, and a visual layout feature vector is extracted by an image feature extraction model, including: inputting the text content in the structured OCR result into a pre-trained semantic embedding model, performing semantic encoding through the multi-layer Transformer structure of the semantic embedding model, and outputting a fixed-dimensional text semantic feature vector; inputting the digital image data into a pre-trained convolutional neural network model, performing feature extraction through multiple convolutional layers and pooling layers of the convolutional neural network model, and outputting a visual layout feature vector representing the overall layout structure of the certificate; the semantic embedding model adopts a deep bidirectional language representation model based on a self-attention mechanism, and the image feature extraction model adopts a deep residual network or a visual Transformer architecture; the extraction process of the text semantic feature vector also includes text cleaning and standardization preprocessing of the OCR recognition result, and the extraction process of the visual layout feature vector also includes size normalization and data enhancement processing of the input image.

8. The RAG-based credential categorization method of claim 1, wherein, performing multi-modal optical character recognition processing on the digital image data to generate a structured OCR result containing text content and its layout information, including: using a deep learning optical character recognition engine to perform text detection and recognition on the digital image data, extracting the text content and its position coordinate information in the image, and obtaining a recognition result; performing structured processing on the recognition result through a layout analysis algorithm, detecting and identifying the layout information of text blocks, table regions, seal regions and Logo regions; integrating the text content and its corresponding layout information to generate structured OCR result data; the structured OCR result is organized in JSON or XML format, containing text content, text position coordinates, font size, text block type and special region identification information; the multi-modal optical character recognition processing also includes an image preprocessing step, including grayscale, binarization, noise removal and image enhancement operations on the input image to improve character recognition accuracy.

9. A computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1-8.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-8.

Images