A rag-based credential classification method, medium and device

Abstract

The present application relates to a kind of RAG-based certificate classification method, medium and equipment, by receiving the digital image data of to-be-classified certificate, carry out multimodal optical character recognition processing and generate structured OCR result, extract text semantic feature vector and visual layout feature vector based on structured OCR result and fusion and generate multimodal query vector, by approximate nearest neighbor search, obtain similar sample and its semantic similarity score and category metadata, generate initial candidate category list, for each candidate category, extract key field value and calculate evidence credibility score, fusion semantic similarity score and evidence credibility score generate comprehensive confidence score and reordering, finally, according to the distribution of score, select classification decision path and output classification result and explainability report.Effectively improve the accuracy and robustness of certificate classification, can handle complex certificate scene with variable format and semantic ambiguity;Enhance the explainability and reliability of classification decision.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more specifically to a method, medium, and device for classifying credentials based on RAG. Background Technology

[0002] With the development of enterprise informatization and intelligent financial systems, the demand for electronic and structured processing of invoices and vouchers has surged. In scenarios such as financial reimbursement, tax auditing, and travel management, vouchers are numerous and complex in format, making manual review inefficient and necessitating automated classification technology. OCR (Optical Character Recognition) technology is widely used for the automatic extraction of text from invoice images. Currently, mainstream OCR frameworks include Tesseract, PaddleOCR, EasyOCR, and Google Cloud Vision API. In text classification, natural language processing technology has continuously evolved, from early simple rule-based or keyword-based classification methods to traditional machine learning methods based on the Bag-of-Words (BoW) model and TF-IDF (such as SVM and Naive Bayes), and then to the widely used deep learning models in recent years, such as TextCNN, BiLSTM, Transformer, and BERT. However, in actual business scenarios, voucher data faces the following challenges: First, the text is short and the contextual information is limited; second, the structure is complex and the semantics are ambiguous, for example, documents such as "notice," "receipt," and "itinerary" are extremely similar in language; third, the formats of various vouchers are highly similar, making it difficult to accurately distinguish them based solely on keywords or semantic similarity. Summary of the Invention

[0003] In view of the above problems, the present invention provides a RAG-based credential classification method, medium and device, which integrates multiple technologies such as OCR recognition, text block segmentation, semantic embedding, vector retrieval, prompt word construction and large language model reasoning to form an automated, intelligent and highly accurate credential classification process.

[0004] To achieve the above objectives, the present invention provides a credential classification method based on RAG, comprising:

[0005] Receive digital image data of the vouchers to be classified;

[0006] Multimodal optical character recognition processing is performed on digital image data to generate structured OCR results containing text content and its layout information;

[0007] Based on the structured OCR results, text semantic feature vectors are extracted through a semantic embedding model, and visual layout feature vectors are extracted through an image feature extraction model.

[0008] The text semantic feature vector and the visual layout feature vector are fused to generate a multimodal query vector;

[0009] The multimodal query vector is input into a pre-built multimodal knowledge vector library for approximate nearest neighbor retrieval to obtain retrieval results. The retrieval results include several top-ranked similar samples, the semantic similarity score of each sample with the query vector, and the category metadata associated with each sample.

[0010] Based on the search results, an initial candidate category list is generated;

[0011] For each candidate category in the initial candidate category list, extract the corresponding key field values ​​from the structured OCR results based on its corresponding key field rule set;

[0012] Calculate the matching degree between the extracted key field values ​​and the corresponding category field patterns in the category rule knowledge base, and generate an evidence credibility score;

[0013] The semantic similarity scores and evidence credibility scores of each candidate category are combined to generate a comprehensive confidence score, and the candidate category list is reordered accordingly.

[0014] Based on the overall confidence score distribution of the reordered candidate category list, a classification decision path is selected, including:

[0015] If the highest overall confidence score meets the preset threshold condition, the corresponding category will be directly output as the classification result;

[0016] Otherwise, based on the feature contradictions between candidate categories, contrast prompts are dynamically constructed, and the contrast prompts and structured OCR results are submitted to the large language model for arbitration reasoning to obtain the final classification result;

[0017] Output classification results and an interpretable report including search samples, key field evidence, and decision path.

[0018] In some embodiments, an initial candidate category list is generated based on the search results, including:

[0019] The top-ranked similar samples returned by the retrieval are categorized and aggregated according to their corresponding category labels to obtain multiple categories.

[0020] Based on the semantic similarity scores of all samples included in each categorized class, the initial confidence score for the corresponding class is calculated as follows:

[0021] Take the maximum semantic similarity score of all similar samples in the same category;

[0022] Alternatively, take the weighted average of the semantic similarity scores of all similar samples in the same category, with the weight of the similar samples determined by their ranking position in the search results;

[0023] The multiple categories after classification are sorted according to the initial confidence scores, and the N categories with the highest rankings are selected to form an initial candidate category list.

[0024] In some embodiments, the semantic similarity scores and evidence credibility scores of each candidate category are fused to generate a comprehensive confidence score, and the candidate category list is reordered accordingly, including:

[0025] Obtain the initial confidence score and evidence credibility score for each candidate category in the initial candidate category list;

[0026] A weighted summation algorithm is used to fuse the initial confidence score and the evidence credibility score to generate a comprehensive confidence score for the current candidate category. The weight coefficients of the initial confidence score and the evidence credibility score are configured according to the classification scenario requirements.

[0027] Repeat the above steps until a comprehensive confidence score for all candidate categories is obtained;

[0028] Based on the overall confidence score of all candidate categories, the initial candidate category list is reordered in descending order of the overall confidence score, generating a reordered candidate category list.

[0029] In some embodiments, the matching degree between the extracted key field values ​​and the corresponding category field patterns in the category rule knowledge base is calculated to generate an evidence credibility score, including:

[0030] Retrieve the field pattern definition of the current candidate category from the category rule knowledge base. The field pattern definition includes the name, data type, format requirements and validation rules of each key field.

[0031] For each key field, the extracted field value is matched with the corresponding field pattern definition to obtain the matching score of a single field. The matching score of a single field is obtained by at least one of the following methods: format conformity check, regular expression matching, numerical range verification, or semantic conformity judgment based on a pre-trained model.

[0032] Based on the matching scores of all key fields, a weighted average algorithm is used to calculate the overall evidence credibility score of the current candidate category. The weight of each key field is configured according to its importance in category discrimination.

[0033] In some embodiments, the samples in the multimodal knowledge vector library include text semantic vectors, visual layout vectors, category labels, and key field rule sets;

[0034] Furthermore, the multimodal knowledge vector library is configured to be built using a distributed vector database architecture;

[0035] The multimodal query vector is input into a pre-built multimodal knowledge vector base for approximate nearest neighbor retrieval to obtain retrieval results. These results include several top-ranked similar samples, the semantic similarity score of each sample to the query vector, and the category metadata associated with each sample, including:

[0036] Vector similarity retrieval is performed in a multimodal knowledge vector base using either a graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm.

[0037] Calculate the similarity score between the multimodal query vector and each sample vector in the multimodal knowledge vector base. The similarity score is obtained by using the cosine similarity algorithm or the inner product similarity algorithm.

[0038] All search results are sorted in descending order based on similarity scores, and the top-ranked samples are selected as similar samples.

[0039] Metadata associated with each similar sample is extracted from the multimodal knowledge vector library. The metadata includes category labels, key field rule set identifiers, and original feature information of the sample source credentials.

[0040] In some embodiments, text semantic feature vectors and visual layout feature vectors are fused to generate multimodal query vectors, including:

[0041] Perform dimension alignment processing on the text semantic feature vector and the visual layout feature vector;

[0042] A weighted fusion algorithm is used to fuse the processed text semantic feature vector and visual layout feature vector;

[0043] The weighted fusion algorithm is configured to adaptively weight the importance of textual and visual features using an attention mechanism, assigning configurable weight coefficients to the textual semantic feature vector and the visual layout feature vector respectively, with the weight coefficients dynamically adjusted according to the voucher type and classification scenario;

[0044] The fused feature vectors are normalized to generate the final multimodal query vector.

[0045] In some embodiments, based on the structured OCR results, text semantic feature vectors are extracted using a semantic embedding model, and visual layout feature vectors are extracted using an image feature extraction model, including:

[0046] The text content in the structured OCR result is input into a pre-trained semantic embedding model. The semantic embedding model performs semantic encoding through a multi-layer Transformer structure and outputs a fixed-dimensional text semantic feature vector.

[0047] Digital image data is input into a pre-trained convolutional neural network model. Feature extraction is performed through multiple convolutional and pooling layers of the convolutional neural network model, and a visual layout feature vector representing the overall layout structure of the voucher is output.

[0048] The semantic embedding model adopts a deep bidirectional language representation model based on a self-attention mechanism, and the image feature extraction model adopts a deep residual network or a visual Transformer architecture.

[0049] The process of extracting text semantic feature vectors also includes text cleaning and standardization preprocessing of OCR recognition results, while the process of extracting visual layout feature vectors also includes size normalization and data augmentation processing of the input image.

[0050] In some embodiments, multimodal optical character recognition (OCR) processing is performed on digital image data to generate a structured OCR result containing text content and its layout information, including:

[0051] A deep learning optical character recognition engine is used to detect and recognize text in digital image data, extracting the text content and its position coordinates from the image to obtain the recognition result;

[0052] The recognition results are structured through layout analysis algorithms to detect and identify the layout information of text blocks, table areas, stamp areas and logo areas;

[0053] By associating and integrating the text content with its corresponding layout information, structured OCR result data is generated.

[0054] Structured OCR results are organized in JSON or XML format and include text content, text position coordinates, font size, text block type, and special area identification information;

[0055] Multimodal optical character recognition processing also includes image preprocessing steps, such as grayscale conversion, binarization, noise removal, and image enhancement of the input image, to improve character recognition accuracy.

[0056] In a second aspect, the present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first aspect.

[0057] In a third aspect, the present invention also provides an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0058] Unlike existing technologies, the above-mentioned technical solution receives digital image data of the vouchers to be classified, performs multimodal optical character recognition (OCR) processing to generate structured OCR results, extracts text semantic feature vectors and visual layout feature vectors based on the structured OCR results, and fuses them to generate a multimodal query vector. It then obtains similar samples and their semantic similarity scores and category metadata through approximate nearest neighbor retrieval, generates an initial candidate category list, extracts key field values ​​for each candidate category, calculates an evidence credibility score, fuses the semantic similarity score and evidence credibility score to generate a comprehensive confidence score, and re-ranks the samples. Finally, it selects a classification decision path based on the score distribution and outputs the classification results and an interpretability report. Through multimodal feature fusion and approximate nearest neighbor retrieval, the accuracy and robustness of voucher classification are effectively improved, enabling the handling of complex voucher scenarios with varying formats and ambiguous semantics. The fusion of evidence credibility score and semantic similarity score enhances the interpretability and reliability of the classification decision. The final interpretability report provides complete technical evidence for auditing and review, meeting the stringent requirements of finance, taxation, and other fields for traceability of classification results. Attached Figure Description

[0059] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0060] In the accompanying drawings of the instruction manual:

[0061] Figure 1 This is a schematic diagram of steps S101 to S111 as described in the specific implementation method;

[0062] Figure 2 This is a schematic diagram illustrating steps S201 to S203 as described in the specific implementation method.

[0063] Figure 3 This is a schematic diagram illustrating steps S301 to S304 of the specific implementation method.

[0064] Figure 4 This is a schematic diagram illustrating steps S401 to S403 as described in the specific implementation method.

[0065] Figure 5 This is a schematic diagram illustrating the process described in the specific implementation method. Detailed Implementation

[0066] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0067] While OCR and natural language processing technologies have made some progress in the field of voucher text extraction and classification, existing technologies still suffer from the following major shortcomings and deficiencies: Current voucher classification technologies rely on keyword matching or shallow machine learning, making it difficult to handle complex scenarios with varying formats, mixed terminology, and semantic ambiguity. This results in low classification accuracy and susceptibility to format perturbations. In particular, vouchers with similar formats but different semantics, such as "meeting notices," "receipts," and "itineraries," are prone to misclassification. Existing systems generally train models based on static training sets, failing to dynamically retrieve relevant knowledge documents based on user input to assist in classification decisions. This leads to poor generalization ability and increased error rates when facing boundary cases, new voucher categories, or changes in user expression. Black-box models based on deep neural networks often lack interpretability, unable to answer "why it was classified as this type of voucher." For industries with high compliance and transparency requirements, such as finance and auditing, this deficiency becomes a significant obstacle to widespread adoption. Current systems often separate OCR, feature extraction, classification, and display processes, lacking unified protocols and standard interfaces between systems. This makes it impossible to build a stable and efficient integrated processing flow, and difficult to adapt to complex business flows and high-concurrency processing requirements.

[0068] In summary, existing technologies still have significant shortcomings in terms of the intelligence, automation, and semantic understanding capabilities of voucher classification. There is an urgent need for a method that integrates OCR recognition, semantic vector retrieval, and language model reasoning to construct a voucher classification system with high accuracy, strong generalization ability, and good interpretability, so as to meet the urgent needs of intelligent financial processing in real-world scenarios.

[0069] Please see Figure 1 This embodiment provides a credential classification method based on RAG, including:

[0070] S101, Receive digital image data of the voucher to be classified;

[0071] S102. Perform multimodal optical character recognition processing on digital image data to generate structured OCR results containing text content and its layout information;

[0072] S103. Based on the structured OCR results, extract the text semantic feature vector through the semantic embedding model, and extract the visual layout feature vector through the image feature extraction model.

[0073] S104. Perform feature fusion between the text semantic feature vector and the visual layout feature vector to generate a multimodal query vector;

[0074] S105. Input the multimodal query vector into the pre-built multimodal knowledge vector base for approximate nearest neighbor retrieval to obtain retrieval results. The retrieval results include several top-ranked similar samples, the semantic similarity score of each sample with the query vector, and the category metadata associated with each sample.

[0075] S106. Generate an initial candidate category list based on the search results;

[0076] S107. For each candidate category in the initial candidate category list, extract the corresponding key field values ​​from the structured OCR results according to the key field rule set.

[0077] S108. Calculate the matching degree between the extracted key field values ​​and the corresponding category field patterns in the category rule knowledge base, and generate an evidence credibility score;

[0078] S109. Integrate the semantic similarity scores and evidence credibility scores of each candidate category to generate a comprehensive confidence score, and reorder the candidate category list accordingly.

[0079] S110. Based on the overall confidence score distribution of the reordered candidate category list, select the classification decision path, including:

[0080] If the highest overall confidence score meets the preset threshold condition, the corresponding category will be directly output as the classification result;

[0081] Otherwise, based on the feature contradictions between candidate categories, contrast prompts are dynamically constructed, and the contrast prompts and structured OCR results are submitted to the large language model for arbitration reasoning to obtain the final classification result;

[0082] S111 Output classification results and an interpretability report containing retrieval samples, key field evidence, and decision path.

[0083] In step S101, receiving the digital image data of the vouchers to be classified refers to obtaining the voucher image file or PDF document uploaded by the user through the system interface. Through a standardized data input interface, it supports the access of voucher data in multiple formats, realizing unified access to multi-source heterogeneous voucher data.

[0084] In step S102, multimodal optical character recognition processing of digital image data refers to using an improved OCR engine to simultaneously extract text content and its layout information. Through deep learning OCR technology, not only is text content recognized, but also visual elements such as text position, table structure, and seal areas are detected, generating structured OCR results and obtaining a complete credential representation containing semantic and layout information.

[0085] In step S103, extracting feature vectors based on structured OCR results involves processing textual and visual information using a semantic embedding model and an image feature extraction model, respectively. A pre-trained language model extracts deep semantic features of the text, while a convolutional neural network extracts the overall layout features of the voucher. This yields a dual-modal feature representation that characterizes both the semantic content and visual structure of the voucher.

[0086] In step S104, feature fusion of the text semantic feature vector and the visual layout feature vector refers to generating a unified multimodal query vector using feature concatenation or attention weighting. By using a feature fusion algorithm to represent textual and visual information in the same vector space, a query vector containing multimodal information is generated, achieving effective fusion of text semantics and visual layout information.

[0087] In step S105, inputting the multimodal query vector into the multimodal knowledge vector base for retrieval refers to using an approximate nearest neighbor algorithm to find similar samples in the pre-built vector base. Through efficient vector similarity calculation, the historical samples most similar to the query credentials and their metadata are quickly retrieved, realizing rapid candidate generation based on multimodal similarity.

[0088] In step S106, generating an initial candidate category list based on the search results refers to statistically analyzing and sorting the retrieved similar samples by category. By aggregating the category information in the search results, a preliminary candidate category list and its confidence score are generated, enabling rapid category filtering based on the search results.

[0089] In step S107, extracting key field values ​​for candidate categories refers to extracting specific fields from the OCR results based on the rule set for each candidate category. By utilizing named entity recognition and rule templates, the values ​​of key discriminant fields for different categories are extracted, obtaining key field information that can be used for evidence verification.

[0090] In step S108, calculating the matching degree between key field values ​​and field patterns involves comparing and verifying the extracted field values ​​with patterns in the category rule knowledge base. Through methods such as format checking, regular expression matching, and numerical verification, the degree of conformity between field values ​​and expected patterns is calculated, resulting in an evidence credibility score based on field verification. This provides quantifiable verification evidence for classification decisions.

[0091] In step S109, fusing semantic similarity score and evidence credibility score refers to using a weighted algorithm to combine retrieval similarity and field verification results. Through configurable weight coefficients, content-based similarity judgment and rule-based evidence verification are organically combined, generating a more reliable comprehensive confidence score.

[0092] In step S110, selecting the decision path based on the comprehensive confidence score distribution refers to selecting different processing flows such as direct output or LLM arbitration based on the score results. The classification confidence is judged by a preset threshold, and intelligent switching is performed between efficient direct output and refined arbitration reasoning. This achieves the optimal balance between classification efficiency and accuracy, and greatly improves the overall performance of the system.

[0093] In step S111, outputting the classification results and interpretability report refers to generating a structured output that includes the complete decision-making process. By recording the intermediate results and decision-making basis of the entire process, a traceable classification report is generated. This provides complete audit trails and decision-making basis, meeting the stringent requirements for interpretability of classification results in fields such as finance and taxation.

[0094] This embodiment effectively improves the accuracy and robustness of voucher classification through multimodal feature fusion and near nearest neighbor retrieval, enabling it to handle complex voucher scenarios with varying formats and ambiguous semantics. The fusion mechanism of evidence credibility score and semantic similarity score enhances the interpretability and reliability of classification decisions. While ensuring classification accuracy, it also improves system processing efficiency. The final interpretability report provides complete technical support for auditing and review, meeting the stringent requirements of finance, taxation, and other fields for traceability of classification results.

[0095] Please see Figure 2 In some embodiments, an initial candidate category list is generated based on the search results, including:

[0096] S201. For the top-ranked similar samples returned by the retrieval, classify and aggregate them according to their corresponding category labels to obtain multiple categories after classification.

[0097] S202. Based on the semantic similarity scores of all samples included in each category after classification, the initial confidence score for the corresponding category is calculated, including:

[0098] Take the maximum semantic similarity score of all similar samples in the same category;

[0099] Alternatively, take the weighted average of the semantic similarity scores of all similar samples in the same category, with the weight of the similar samples determined by their ranking position in the search results;

[0100] S203. Sort the multiple categories after classification according to the initial confidence scores, and select the N categories with the highest ranking to form an initial candidate category list.

[0101] In step S201, grouping and aggregating the top-ranked similar samples returned by search category labels means grouping and statistically analyzing the similar samples according to their respective credential categories. Using category labels as key metadata, preliminary cluster analysis is performed on the search results, reorganizing the scattered similar samples according to category dimensions. This achieves structured organization of the search results and avoids duplicate calculations of samples of the same category.

[0102] In step S202, calculating the initial confidence score for the corresponding category based on the semantic similarity scores of all samples included in each categorized category involves using statistical methods to aggregate sample-level similarity scores into a category-level confidence index. Through mathematical statistical methods, the individual similarity information of multiple similar samples is fused into a comprehensive score representing the overall matching degree of the category. This achieves the conversion from sample-level similarity to category-level confidence.

[0103] Taking the maximum or weighted average of the semantic similarity scores of all similar samples within the same category refers to employing two different aggregation strategies to calculate the category confidence score. The maximum value strategy selects the most similar sample in the category as the representative, reflecting the idea of ​​optimal matching; the weighted average strategy comprehensively considers the similarity of all samples and assigns different weights according to the sample's ranking position, reflecting the idea of ​​overall matching. It provides flexible and configurable confidence score calculation methods. The maximum value strategy is suitable for scenarios requiring strict matching, while the weighted average strategy is suitable for scenarios requiring comprehensive consideration, improving the system's adaptability and flexibility.

[0104] In step S203, sorting the multiple categories after classification based on the initial confidence scores and selecting the top N categories means arranging them in descending order according to the calculated category confidence scores and selecting the top N categories. By using a sorting algorithm to arrange candidate categories from high to low matching degree and controlling the size of the candidate set through a threshold N, a balance is achieved between recall and precision. This generates an initial candidate category list that is controllable in size and of optimal quality, ensuring that important candidate categories are not missed while avoiding computational redundancy caused by an excessively large candidate set, thus improving overall processing efficiency.

[0105] This embodiment achieves the transformation from raw search results to a high-quality candidate category list through a three-tiered progressive processing approach: category aggregation, confidence calculation, and ranking selection. By providing multiple confidence calculation strategies, the system's adaptability to different business scenarios is enhanced; the controllable candidate list generation mechanism optimizes computational efficiency while ensuring recall; and the entire process provides high-quality input for subsequent evidence verification and decision arbitration.

[0106] Please see Figure 3In some embodiments, the semantic similarity scores and evidence credibility scores of each candidate category are fused to generate a comprehensive confidence score, and the candidate category list is reordered accordingly, including:

[0107] S301. Obtain the initial confidence score and evidence credibility score for each candidate category in the initial candidate category list;

[0108] S302. The initial confidence score and the evidence credibility score are fused using a weighted summation algorithm to generate a comprehensive confidence score for the current candidate category. The weight coefficients of the initial confidence score and the evidence credibility score are configured according to the classification scenario requirements.

[0109] S303. Repeat the above steps until the overall confidence score for all candidate categories is obtained;

[0110] S304. Based on the overall confidence score of all candidate categories, reorder the initial candidate category list in descending order of the overall confidence score to generate a reordered candidate category list.

[0111] In step S301, obtaining the initial confidence score and evidence credibility score for each candidate category in the initial candidate category list refers to collecting scoring data from two dimensions from different processing modules. Obtaining the initial confidence score based on semantic similarity and the evidence credibility score based on key field verification through a data interface achieves unified collection of multi-dimensional scoring data, providing a complete data foundation for comprehensive confidence calculation.

[0112] In step S302, fusing the initial confidence score and the evidence credibility score using a weighted summation algorithm involves linearly combining the two scores using configurable weight coefficients. By adjusting the weight coefficients to regulate the relative importance of semantic similarity and evidence verification in the final decision, a more comprehensive score reflecting actual business needs is generated. A flexible and adjustable score fusion mechanism is provided, allowing users to adjust the weight configuration according to different credential types and business scenarios, enhancing the system's adaptability and practicality.

[0113] In step S303, repeating the above steps until the comprehensive confidence score of all candidate categories is obtained means performing the same score fusion calculation process on each category in the initial candidate category list. Batch processing ensures that all candidate categories undergo uniform score fusion calculation, guaranteeing consistency and completeness of the processing. This achieves standardized processing of all candidate categories, provides a unified scoring benchmark, and ensures the fairness and comparability of the ranking results.

[0114] In step S304, re-sorting based on the comprehensive confidence score of all candidate categories means arranging the candidate categories in descending order according to the comprehensive score. By reorganizing the candidate categories from high to low comprehensive confidence using a sorting algorithm, an ordered list reflecting the final confidence level is generated. This produces an optimized sorting result that has been validated across multiple dimensions, providing a more reliable and accurate basis and significantly improving the accuracy and credibility of the final classification result.

[0115] This embodiment organically combines an initial confidence score based on content similarity with an evidence credibility score based on rule verification using a weighted fusion algorithm to generate a comprehensive confidence score that better reflects actual business needs. A configurable weighting coefficient mechanism enables flexible adaptation to different business scenarios; complete batch processing and global sorting ensure the comprehensiveness and consistency of candidate category evaluation.

[0116] Please see Figure 4 In some embodiments, the matching degree between the extracted key field values ​​and the corresponding category field patterns in the category rule knowledge base is calculated to generate an evidence credibility score, including:

[0117] S401. Obtain the field pattern definition of the current candidate category from the category rule knowledge base. The field pattern definition includes the name, data type, format requirements and validation rules of each key field.

[0118] S402. For each key field, the extracted field value is matched with the corresponding field pattern definition to obtain the matching score of a single field. The matching score of a single field is obtained by at least one of the following methods: format conformity check, regular expression matching, numerical range verification, or semantic conformity judgment based on a pre-trained model.

[0119] S403. Based on the matching scores of all key fields, a weighted average algorithm is used to calculate the overall evidence credibility score of the current candidate category. The weight of each key field is configured according to its importance in category discrimination.

[0120] In step S401, retrieving the field pattern definition of the current candidate category from the category rule knowledge base involves querying a pre-built structured rule base to obtain complete field specification information for a specific credential category. A precise query is performed in the rule knowledge base using the category identifier to retrieve the definition details of all key fields for that category, including metadata such as field name, data type, format constraints, and validation rules. This provides a complete rule baseline for subsequent field validation, ensuring the standardization and accuracy of the validation process.

[0121] In step S402, calculating the matching degree for each key field involves using multiple validation techniques to perform multi-dimensional compliance checks on the extracted field values. Through methods such as format checking, regular expression matching, range validation, and semantic judgment, the degree of matching between the field values ​​and the expected pattern is comprehensively evaluated, generating a quantified matching score. This achieves multi-level validation, from simple existence checks to complex semantic compliance judgments, improving the granularity and accuracy of field validation.

[0122] In step S403, the weighted average algorithm for calculating the overall evidence credibility score involves assigning different weights to the matching scores of each field based on their importance and then weighting and merging them. The weight coefficients reflect the relative importance of different key fields in category discrimination, and the matching scores of multiple fields are merged into a single score representing the overall evidence strength. This approach considers both the verification results of each field and the discriminative value of different fields, resulting in an overall evidence credibility score that more accurately reflects the actual matching degree between the candidate category and the current credential.

[0123] This embodiment achieves a complete transformation from raw field values ​​to an overall evidence credibility score through a three-tiered progressive processing approach: retrieving field pattern definitions from the rule base, calculating multi-dimensional field matching scores, and weighted fusion. The comprehensive application of multiple verification techniques improves the accuracy of field-level verification; and the weighted average algorithm achieves a reasonable aggregation of field-level results to category-level evaluations. The evidence credibility score generated by the entire process possesses high reliability and interpretability, significantly improving the accuracy and credibility of the classification system.

[0124] In some embodiments, the samples in the multimodal knowledge vector library include text semantic vectors, visual layout vectors, category labels, and key field rule sets;

[0125] Furthermore, the multimodal knowledge vector library is configured to be built using a distributed vector database architecture;

[0126] The multimodal query vector is input into a pre-built multimodal knowledge vector base for approximate nearest neighbor retrieval to obtain retrieval results. These results include several top-ranked similar samples, the semantic similarity score of each sample to the query vector, and the category metadata associated with each sample, including:

[0127] Vector similarity retrieval is performed in a multimodal knowledge vector base using either a graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm.

[0128] Calculate the similarity score between the multimodal query vector and each sample vector in the multimodal knowledge vector base. The similarity score is obtained by using the cosine similarity algorithm or the inner product similarity algorithm.

[0129] All search results are sorted in descending order based on similarity scores, and the top-ranked samples are selected as similar samples.

[0130] Metadata associated with each similar sample is extracted from the multimodal knowledge vector library. The metadata includes category labels, key field rule set identifiers, and original feature information of the sample source credentials.

[0131] In this embodiment, the samples in the multimodal knowledge vector library include text semantic vectors, visual layout vectors, category labels, and key field rule sets. This means that each data record in the vector library simultaneously contains text semantic features, visual layout features, category identifiers, and key field rule information. By uniformly storing multimodal features and associated metadata, a complete knowledge representation system that includes both content features and structured rule knowledge is constructed.

[0132] In this embodiment, the multimodal knowledge vector base is configured to be built using a distributed vector database architecture, which means deploying the knowledge base system using a dedicated vector database that supports distributed storage and parallel computing. The distributed architecture enables sharded storage and parallel retrieval of massive vector data, leveraging multi-node computing resources to improve retrieval performance and system scalability. This significantly improves the efficiency and throughput of large-scale vector retrieval, supports multimodal vector similarity retrieval and metadata association queries, and meets the high-concurrency processing requirements of real-world business scenarios.

[0133] In this embodiment, either a graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm is used to perform vector similarity retrieval in a multimodal knowledge vector base. This means using two efficient approximation algorithms to accelerate the similarity search process for high-dimensional vectors. By constructing navigation graphs or using vector quantization and other approximation techniques, computational complexity is significantly reduced while maintaining retrieval accuracy, enabling rapid recall of similar samples. An optimal balance is achieved between retrieval accuracy and computational efficiency, allowing the system to quickly handle the similarity retrieval task of high-dimensional multimodal vectors.

[0134] In this embodiment, the similarity score between the multimodal query vector and each sample vector in the multimodal knowledge vector base is calculated. The similarity score is obtained through the cosine similarity algorithm or the inner product similarity algorithm, which refers to the standard similarity measurement method in the vector space model to quantify the degree of matching between the query vector and the sample vectors in the database. The cosine value of the angle between the vectors or the dot product result is calculated by mathematical formula to obtain a numerical score representing the semantic similarity.

[0135] In this embodiment, all search results are sorted in descending order based on similarity scores, and the top-ranked samples are selected as similar samples. This means arranging the search results in descending order of similarity scores and extracting the optimal result set. A sorting algorithm is used to select the top-ranked samples most similar to the query vector, forming a high-quality candidate sample set. This achieves refined filtering of search results.

[0136] In this embodiment, metadata associated with each similar sample is extracted from the multimodal knowledge vector database. This metadata includes category labels, key field rule set identifiers, and original feature information of the sample source credentials. It refers to the structured descriptive information associated with similar samples acquired synchronously during the retrieval process. Through the metadata indexing mechanism of the vector database, key information such as associated category definitions and field rules is obtained while returning similar vectors. This provides complete contextual information for subsequent category aggregation, evidence verification, and decision reasoning, supporting an end-to-end classification reasoning process.

[0137] This embodiment constructs a distributed vector database containing multimodal features and rich metadata, and employs an efficient approximate nearest neighbor search algorithm and standard similarity measurement methods to achieve fast and accurate multimodal vector retrieval and metadata association queries. The distributed vector database architecture ensures high performance and high availability of the system; the use of graph-based or quantization-based approximate nearest neighbor search algorithms balances retrieval accuracy and computational efficiency; and it significantly improves the efficiency and effectiveness of the retrieval process in the credential classification system.

[0138] In some embodiments, text semantic feature vectors and visual layout feature vectors are fused to generate multimodal query vectors, including:

[0139] Perform dimension alignment processing on the text semantic feature vector and the visual layout feature vector;

[0140] A weighted fusion algorithm is used to fuse the processed text semantic feature vector and visual layout feature vector;

[0141] The weighted fusion algorithm is configured to adaptively weight the importance of textual and visual features using an attention mechanism, assigning configurable weight coefficients to the textual semantic feature vector and the visual layout feature vector respectively, with the weight coefficients dynamically adjusted according to the voucher type and classification scenario;

[0142] The fused feature vectors are normalized to generate the final multimodal query vector.

[0143] In this embodiment, dimensional alignment of the text semantic feature vector and the visual layout feature vector refers to adjusting the two feature vectors from different sources to the same dimensional space through linear transformation or projection methods. That is, using fully connected layers or projection matrices to unify the dimensions of text and visual features eliminates the dimensionality differences caused by different feature extraction models, ensuring that features of different modalities are comparable and operable in mathematical space.

[0144] In this embodiment, a weighted fusion algorithm is used to fuse the processed text semantic feature vector and visual layout feature vector. This involves using learnable weight parameters to linearly or non-linearly combine the features of the two modalities. By adjusting the weight coefficients, the relative contributions of textual and visual features in the final representation are adjusted, generating a unified feature representation that simultaneously preserves semantic and visual information. This achieves effective integration of multimodal information and generates fused features with greater representational power than single-modal features.

[0145] In this embodiment, the weighted fusion algorithm is configured to adaptively weight the importance of textual and visual features using an attention mechanism. This means that the attention calculation module dynamically learns the importance of different modal features in the current classification task. An attention weight generation network is used to automatically calculate the weight coefficients of each modality based on the content of the input features, achieving content-aware weighted feature fusion. An intelligent feature importance evaluation mechanism is provided, which can automatically adjust the fusion strategy according to the characteristics of different credentials, improving the accuracy and adaptability of feature fusion.

[0146] In this embodiment, configurable weight coefficients are assigned to the text semantic feature vector and the visual layout feature vector, respectively. These weight coefficients are dynamically adjusted based on the voucher type and classification scenario, establishing a mapping relationship between the weight coefficients and the business scenario to achieve adaptive parameter configuration. Through preset configuration rules or online learning mechanisms, the fusion weights of each modality feature are dynamically adjusted according to different voucher types (such as invoices, receipts, and notification slips) and classification scenario requirements. This enhances the system's adaptability to different business scenarios, enabling the feature fusion strategy to be flexibly optimized according to actual needs.

[0147] In this embodiment, normalizing the fused feature vectors to generate the final multimodal query vectors involves using L2 normalization or other standardization methods to convert the fused feature vectors into unit vectors. By mapping the feature vectors onto a unit hypersphere through mathematical transformation, the influence of vector magnitude on similarity calculation is eliminated, ensuring the reliability and comparability of the similarity calculation results.

[0148] This embodiment achieves efficient fusion of textual semantic features and visual layout features through multiple layers of sophisticated operations, including dimension alignment, weighted fusion, attention mechanisms, dynamic weight adjustment, and normalization. The attention mechanism enables adaptive evaluation of feature importance, dynamic weight configuration enhances business adaptability, and normalization ensures retrieval stability. The entire feature fusion process generates high-quality multimodal query vectors, significantly improving the accuracy and robustness of the credential classification system.

[0149] In some embodiments, based on the structured OCR results, text semantic feature vectors are extracted using a semantic embedding model, and visual layout feature vectors are extracted using an image feature extraction model, including:

[0150] The text content in the structured OCR result is input into a pre-trained semantic embedding model. The semantic embedding model performs semantic encoding through a multi-layer Transformer structure and outputs a fixed-dimensional text semantic feature vector.

[0151] Digital image data is input into a pre-trained convolutional neural network model. Feature extraction is performed through multiple convolutional and pooling layers of the convolutional neural network model, and a visual layout feature vector representing the overall layout structure of the voucher is output.

[0152] The semantic embedding model adopts a deep bidirectional language representation model based on a self-attention mechanism, and the image feature extraction model adopts a deep residual network or a visual Transformer architecture.

[0153] The process of extracting text semantic feature vectors also includes text cleaning and standardization preprocessing of OCR recognition results, while the process of extracting visual layout feature vectors also includes size normalization and data augmentation processing of the input image.

[0154] In this embodiment, inputting the text content from the structured OCR result into a pre-trained semantic embedding model for semantic encoding refers to using a deep learning model to perform deep semantic understanding of the text information extracted by OCR. Through the self-attention mechanism in the multi-layer Transformer structure, the contextual dependencies and semantic associations between words in the text are captured, and the variable-length text sequence is encoded into a fixed-dimensional dense vector representation, thus obtaining a feature representation that accurately reflects the semantic content of the text.

[0155] In this embodiment, inputting digital image data into a pre-trained convolutional neural network model for feature extraction refers to using a deep convolutional network to learn visual layout features from the original image. Through cascaded operations of multiple convolutional and pooling layers, local features of the image are gradually extracted and aggregated into a global feature representation, ultimately outputting a high-dimensional feature vector representing the overall layout structure of the voucher. This captures visual information such as the voucher image's layout structure, text arrangement, and seal position.

[0156] In this embodiment, the semantic embedding model employs a deep bidirectional language representation model based on a self-attention mechanism. This refers to using a pre-trained language model similar to BERT to handle text semantic encoding tasks. By utilizing a bidirectional Transformer architecture, it simultaneously considers the left and right contextual information of the text sequence and dynamically calculates the importance weights between words using a self-attention mechanism, generating text representations with rich semantic information. This improves the accuracy and depth of text semantic understanding, making it particularly suitable for processing short texts and technical terms commonly found in vouchers.

[0157] In this embodiment, the image feature extraction model employs either a deep residual network or a visual Transformer architecture, meaning it uses two advanced computer vision models to extract image features. Deep residual networks address the vanishing gradient problem in deep networks through skip connections, while the visual Transformer captures global relationships between image patches through a self-attention mechanism. This provides a variety of high-performance visual feature extraction schemes, allowing for flexible selection of the appropriate model architecture based on different accuracy and efficiency requirements.

[0158] In this embodiment, the extraction process of text semantic feature vectors includes text cleaning and standardization preprocessing of the OCR recognition results. This refers to data purification processing of the original OCR output before semantic encoding. Through methods such as rule filtering, typo correction, and format standardization, noise and errors generated during OCR recognition are eliminated, improving the quality of the input text. This reduces the interference of noisy data on the semantic model and improves the accuracy and reliability of text feature extraction.

[0159] In this embodiment, the visual layout feature vector extraction process includes size normalization and data augmentation of the input image, which refers to standardizing and enhancing the image data before feature extraction. Preprocessing techniques such as image scaling, padding, and enhancement are used to make the input image meet the model requirements and increase data diversity. This improves the stability and generalization ability of image feature extraction, and enhances the model's adaptability and robustness to images of different sizes and quality.

[0160] In this embodiment, a deep bidirectional language representation model based on a self-attention mechanism and a deep residual network or visual Transformer architecture are employed to achieve efficient feature extraction of text semantic content and image layout structure, respectively. Text cleaning and standardization preprocessing improve the quality of the input data, while image size normalization and data augmentation enhance the stability of visual features. The entire feature extraction process generates high-quality text semantic feature vectors and visual layout feature vectors, significantly improving the accuracy and robustness of the credential classification system.

[0161] In some embodiments, multimodal optical character recognition (OCR) processing is performed on digital image data to generate a structured OCR result containing text content and its layout information, including:

[0162] A deep learning optical character recognition engine is used to detect and recognize text in digital image data, extracting the text content and its position coordinates from the image to obtain the recognition result;

[0163] The recognition results are structured through layout analysis algorithms to detect and identify the layout information of text blocks, table areas, stamp areas and logo areas;

[0164] By associating and integrating the text content with its corresponding layout information, structured OCR result data is generated.

[0165] Structured OCR results are organized in JSON or XML format and include text content, text position coordinates, font size, text block type, and special area identification information;

[0166] Multimodal optical character recognition processing also includes image preprocessing steps, such as grayscale conversion, binarization, noise removal, and image enhancement of the input image, to improve character recognition accuracy.

[0167] In this embodiment, the use of a deep learning optical character recognition engine for text detection and recognition of digital image data refers to the use of OCR technology based on deep neural networks to automatically locate and recognize text content in images. Text region detection is performed using a convolutional neural network, followed by sequence recognition using a recurrent neural network or Transformer architecture. The final output is the text content and its precise location coordinates within the image, achieving high-precision text extraction and localization.

[0168] In this embodiment, the structured processing of the recognition results through layout analysis algorithms refers to using computer vision and machine learning algorithms to perform region segmentation and type identification on the OCR recognition results. Through techniques such as image segmentation, region clustering, and classification models, different types of layout elements, such as text blocks, tables, seals, and logos, are automatically detected and distinguished, and the bounding boxes and attribute information of each element are recorded. This yields complete structural information of the voucher image.

[0169] In this embodiment, associating and integrating text content with its corresponding layout information means establishing a mapping relationship between text content and page elements to form a structured data record. Through coordinate matching and spatial relationship analysis, the identified text content is assigned to the corresponding page area, and metadata information such as area type and font size is added. This generates a complete structured representation that combines content semantics and spatial layout.

[0170] In this embodiment, the structured OCR results are organized using JSON or XML format, which means using a standardized data exchange format to store and transmit the OCR processing results. Through a hierarchical data organization structure, information such as text content, location coordinates, font attributes, and region type are encapsulated and serialized in key-value pairs. This provides a machine-readable and easily parsed data format, improving the system's interoperability and scalability.

[0171] In this embodiment, the multimodal optical character recognition (OCR) processing includes image preprocessing steps, such as grayscale conversion, binarization, noise removal, and image enhancement of the input image. This refers to quality optimization of the original image before OCR recognition. Image processing algorithms improve image quality, including color space conversion, threshold segmentation, filtering and noise reduction, and contrast enhancement, thereby enhancing image clarity and recognizability. This significantly improves the accuracy and robustness of subsequent OCR recognition, particularly when processing low-quality, blurry, or unevenly lit voucher images.

[0172] In this embodiment, a complete conversion process from the original image to a structured OCR result is achieved through the collaborative processing of multiple stages, including a deep learning OCR engine, layout analysis algorithm, structured data integration, standardized data format organization, and image preprocessing. Multimodal optical character recognition (OCR) processing not only extracts the text content but also captures rich layout and structural information. The entire processing significantly improves the utilization efficiency and recognition accuracy of voucher image information, laying a solid data foundation for a RAG-based voucher classification system.

[0173] In a second aspect, this embodiment also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0174] In a third aspect, this embodiment also provides an electronic device, including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0175] For easier understanding, please refer to Figure 5 The following examples are provided for further explanation:

[0176] This example provides a credential classification system based on a retrieval-enhanced generative model and its construction method. It integrates multiple technologies, including OCR recognition, text block segmentation, semantic embedding, vector retrieval, prompt word construction, and large language model inference, forming an automated, intelligent, and highly accurate credential classification process. Specifically, it includes the following technical steps and modules:

[0177] 1. Voucher Data Acquisition and OCR Preprocessing Module

[0178] This module aims to achieve image acquisition, OCR recognition, and text preprocessing for various types of business vouchers, providing high-quality, structured text input for subsequent semantic understanding and classification models. The module includes the following specific technical processes:

[0179] (1) Voucher Image Acquisition

[0180] This system supports various methods for acquiring invoice images from different business sources, including: scanner input, mobile phone photo upload, PDF file page splitting, and system interface reception (such as OA system, travel platform, etc.).

[0181] The types of vouchers collected cover, but are not limited to: general VAT invoices, e-tickets, train tickets, ship tickets, accommodation invoices, conference notices, hospital receipts, bank statements, etc., totaling dozens of structured or semi-structured document formats.

[0182] (2) OCR text recognition

[0183] This system preferably uses an OCR engine that supports multiple languages ​​and has strong anti-interference capabilities, such as PaddleOCR, EasyOCR, and Tencent Cloud OCR. The OCR engine configuration supports automatic image orientation detection (rotation correction), multi-page PDF processing, and table structure restoration. After each voucher image is recognized, the system outputs the text line content.

[0184] (3) Multilingual and typography adaptation strategies

[0185] For some foreign-related documents (such as hotel bills in English, international air tickets, etc.), the system supports mixed Chinese and English recognition; for vertical text (such as some handwritten documents and old-style documents), image orientation detection and rotation correction are performed; if a recognition failure area appears, it can roll back to use a backup OCR engine or trigger a manual review interface.

[0186] 2. Text Block Embedding and Vector Library Construction Module (Detailed Explanation)

[0187] This module aims to convert the extracted and segmented text chunks from the voucher image into a vector representation with semantic expressive capabilities, and store them together with the original voucher labels, definition descriptions, and other meta-information into a vector library to build an efficient knowledge indexing system for voucher semantic matching and classification.

[0188] (1) Embedded Model Selection and Loading

[0189] The system preferably uses Chinese or Chinese-English hybrid embedding models with strong semantic expressive capabilities to ensure that vectors have high semantic similarity alignment capabilities.

[0190] (2) Chunk Vectorization Process

[0191] For each text chunk generated by the OCR module, the following steps are performed for embedding encoding:

[0192] Clean up spaces and special characters;

[0193] The maximum length should not exceed the model's supported limit (e.g., 512 characters);

[0194] The vectors are fed into the embedding model in batches for vector encoding, and the output is a high-dimensional dense vector (such as 768-dimensional or 1024-dimensional).

[0195] Each vector maintains its binding relationship with the original chunk, forming a structured record.

[0196] In this example, a single text chunk is vectorized to form a structured data record, which is stored in a vector index for subsequent retrieval and matching. Each record contains text content, semantic vectors, and related metadata. Specifically, it includes the following parts:

[0197] Text content (chunk_text): This field stores the specific text fragments identified and segmented from the original voucher image using OCR. For example, "This meeting will be held on July 1, 2023, at…".

[0198] Semantic vector (embedding): This field stores a high-dimensional array of numbers (i.e., a dense vector). This vector is obtained by calculating the "text content" through a deep learning embedding model. It can capture the deep semantic information of the text and is the core of achieving semantic similarity matching.

[0199] Category label: This field explicitly indicates the category of the voucher to which the text block belongs. For example, the label "Meeting Notice" indicates that this text comes from a voucher of the meeting notice type.

[0200] Category Prompt: This field stores a descriptive text about the category to which the credential belongs. This description defines the typical characteristics of the credential for that category and is used to dynamically construct prompts submitted to the large language model to guide it in accurate classification. For example, "Meeting notification credentials typically include natural language paragraphs such as meeting name, agenda, time, and location."

[0201] By structurally binding text, vectors, category labels, and various metadata, this invention constructs an information-rich knowledge vector library, laying the foundation for subsequent efficient, accurate, and interpretable credential classification.

[0202] (3) Vector Index Library Construction and Management

[0203] Vector engine selection: Milvus, distributed, scalable, supports attribute filtering and strong consistency recall;

[0204] Index building process:

[0205] Write all vectors to the vector library in batches and create an index.

[0206] Choose an appropriate approximate index structure: Small-scale data: Flat, HNSW; Medium to large-scale data: IVF+PQ;

[0207] Configure the metadata fields of the vector library to support multi-dimensional retrieval (such as filtering by category, filtering by confidence level, etc.);

[0208] Establish an inverted index to support reverse lookups of "category → vector".

[0209] (4) Vector Incremental Update and Management Strategy

[0210] The system supports dynamic expansion: new voucher categories or new data can be added to the vector library in real time; low-quality chunks or OCR output vectors with low confidence can be flagged as expired or temporarily excluded from the library; a periodic index rebuild task is performed to optimize Top-K retrieval performance; this step aims to address the performance degradation caused by dynamic data growth. As the vector library continuously receives new data, the original index structure may become suboptimal, leading to slower retrieval speed or reduced accuracy. Therefore, by periodically rebuilding a new, optimally structured index based on the current full dataset in the library, data fragmentation can be eliminated, index balance can be restored, and the system can maintain high speed and high accuracy in Top-K semantic retrieval. The system also supports automatic statistical analysis of sample distribution across categories using category labels for downstream tasks such as model training and balance assessment.

[0211] (5) Top-K vector similarity retrieval (preparing for subsequent modules)

[0212] During the classification process, the system can call this vector library interface to perform Top-K nearest neighbor search on the text block to be classified, including:

[0213] Supports Euclidean distance (L2), cosine similarity, and dot product similarity.

[0214] return:

[0215] Matched chunk text;

[0216] Match the category to which the vector belongs;

[0217] Similarity score;

[0218] Related prompts;

[0219] Low-quality candidates can be filtered by setting a threshold based on confidence level.

[0220] 3. User-uploaded file processing module

[0221] This module processes various credential images or PDF files uploaded by users through the system front-end or interface. Through OCR recognition, text block segmentation, semantic embedding, and vector retrieval, it obtains semantic matching results between these images and categorized credentials in the knowledge base, providing candidate categories for subsequent category determination and prompt word construction. The entire module includes the following technical process:

[0222] (1) User file access and format processing

[0223] Users can upload files via web interface, mobile app, RPA interface, etc. The system supports the following file types: image files: JPEG, PNG, etc.; PDF files: support multi-page, multi-language, and vector / scanned mixed structure; if it is a PDF document, the system will automatically perform pagination and call the OCR engine for recognition page by page; it supports single file or batch processing mode and automatically generates unique file IDs and task tracking records.

[0224] (2) OCR recognition and text extraction

[0225] The system calls and trains a pre-defined OCR service (such as PaddleOCR deployment service) to perform text recognition on each page of the uploaded image or PDF; it outputs structured text information, mainly the text content. Configurable image preprocessing strategies, such as image sharpness enhancement, rotation correction, color inversion, and edge cropping, ensure optimal OCR results.

[0226] (3) Chunk segmentation and embedding encoding

[0227] The text extracted by OCR is fed into an embedding encoding model (such as BGE, E5, etc.) to generate a high-dimensional semantic vector; the data structure of the user-uploaded file after OCR recognition, segmentation, and embedding encoding. This structure is the input query body for vector retrieval, specifically including:

[0228] Chunk text: The actual content of a text block extracted from the user-uploaded credentials.

[0229] Semantic vector (embedding): A high-dimensional semantic vector generated by the embedding model corresponding to the content of the text block.

[0230] Unique Identifier (chunk_id): A unique ID assigned to this text block for tracking during the processing flow, such as identifying which page and which text block it comes from in a particular upload task.

[0231] (4) Call the vector retrieval service to perform semantic matching.

[0232] The system uses the generated chunk vector as the query vector and calls the vector retrieval engine deployed on the backend to perform Top-N matching;

[0233] The search target is all embedded text blocks in the knowledge base, and the search methods can be selected as: Cosine Similarity and Euclidean Distance.

[0234] In this example, the vector retrieval service returns the raw matching results, which is a list of N records found in the knowledge base that are semantically most similar to the user-uploaded text block. Each item in the list is an independent matching result, specifically including:

[0235] Matched_chunk_text: The original text content of similar text blocks matched in the knowledge base.

[0236] Category label: The credential category to which the matched text block is pre-assigned in the knowledge base.

[0237] Similarity score: A quantifiable value that represents the degree of semantic similarity between a user-uploaded text block and the matching text block. The higher the score, the more similar the text block.

[0238] Category prompt: Descriptive text associated with the matching category, used to build prompts for the large language model later.

[0239] (5) Candidate category generation and deduplication aggregation

[0240] The category label field of all matching results is deduplicated, retaining the top-K high-frequency, high-scoring categories; the following strategies can be used for comprehensive sorting: sorting by the highest similarity of a single category; sorting by the number of category matches; adding a confidence threshold filter (e.g., the lowest matching score ≥ 0.6);

[0241] After deduplication and aggregation of the original matching results from the vector retrieval, a final candidate category list is generated. This is a comprehensive refinement of the retrieval results, providing a direct basis for the final classification decision. Specifically, it includes:

[0242] Top-K candidate categories: A list of the most likely credential category names, ranked by a comprehensive factor (such as similarity, weighted by the number of matches).

[0243] Category scores: A dictionary or mapping table that records each candidate category and its corresponding comprehensive score. This score is an important basis for the system to judge credibility.

[0244] (6) Meta-information binding (preparing for the next step of prompt word construction)

[0245] Each Top-K category will be bound to its definition description and prompt word template information for the prompt word generation module; the system supports automatically summarizing and merging prompt words from multiple matching sources based on the prompt field of the matching chunk; and retains the original search records and Top-K statistical results for subsequent auditing and analysis.

[0246] 4. Prompt word construction and context enhancement module

[0247] To improve the accuracy and controllability of classification judgments, the system extracts corresponding prompts from a pre-built prompt word template library based on Top-K candidate categories. Each voucher category is configured with a set of prompt word templates, including category definition, structural features, and descriptions of typical key fields. The system concatenates multiple prompt words and voucher OCR results into a composite prompt, for example:

[0248] Main prompt: You are a voucher classification expert proficient in image and text understanding.

[0249] Task objective: Based on the title, stamp, table, and text content in the image, directly return the category. Only return the category name; no other information is required.

[0250] Classification rule: {VOUCHERS_CLASSIFY_PROMPT}

[0251] The voucher content is as follows: {OCR}

[0252] Different categories of descriptive prompts (VOUCHERS_CLASSIFY_PROMPT):

[0253] Taxi receipt: An official receipt or invoice provided by the taxi company or driver after a taxi ride. It is usually titled "Taxi-specific Invoice," but sometimes it may also include other terms such as "Passenger Transport" or "Machine-printed Invoice." Key information includes the invoice code, invoice number, date, pick-up and drop-off times, mileage, fare, and amount.

[0254] Ship ticket: A formal ticket issued by a shipping company to prove that a passenger has paid for the ticket and is entitled to travel on the ship. It usually includes information such as the ship number, departure and destination, date, and cabin type.

[0255] The main suggestion word is concatenated with descriptions of the n categories returned by the retrieval to construct a complete category suggestion word. This dynamic construction of suggestion words significantly reduces the number of input tokens for large models, minimizes the impact of redundant category suggestion word descriptions, and improves response speed and accuracy.

[0256] 5. Large Language Model Reasoning and Classification Module

[0257] If the retrieved categories, after deduplication and aggregation, leave only one category, then this category is directly output as the final result, without needing to be submitted to a larger model for inference. This module uses a language model to classify user OCR text combined with prompt words, generating the final unique category label.

[0258] 6. Output Results and Traceable Classification Record Module

[0259] This module records the detailed process and judgment criteria for this classification, facilitating later auditing and backtracking. The records include: OCR text content, Top-K vector retrieval results, prompt word template content, large model response content, and final classification. It supports closed-loop business processing such as result archiving, audit tracking, and error feedback correction.

[0260] In summary, this example proposes an automatic credential classification system that combines OCR recognition, semantic vector retrieval, and pre-trained language model inference, with the following key innovations:

[0261] 1. Structured OCR recognition and semantic segmentation of multi-source invoices

[0262] It supports the acquisition and OCR recognition of various business voucher images such as invoices, tickets, accommodation receipts, and notices; it converts extracted text into "semantic chunks" through paragraph recognition and semantic segmentation algorithms, providing a reasonably granular processing unit for subsequent semantic embedding and retrieval;

[0263] 2. Construct a domain knowledge vector library to achieve efficient semantic recall.

[0264] Semantic vector encoding of chunks is performed using pre-trained embedding models (such as BGE, Qwen-Embedding, etc.); a structured knowledge vector library containing voucher text, vector representation, category labels and prompt words is constructed; and Top-K semantic similarity matching of newly uploaded voucher content is achieved using the Milvus vector engine.

[0265] 3. Construct category suggestion word templates based on matching results

[0266] The Top-K matched credential categories and their defining descriptions are extracted as prompt words; multiple prompt words can be concatenated to construct context and guide the language model to make classification judgments; prompt words can be customized according to different credential types, such as "train tickets must include train number / date / seat number", to enhance the model's discrimination ability and controllability.

[0267] 4. A RAG classification mechanism that combines retrieval and generation.

[0268] Construct a RAG classification framework that integrates semantic retrieval and model reasoning;

[0269] The RAG classification mechanism constructed in this example is not a simple application of existing general RAG technology, but a creative reconstruction aimed at addressing the dual challenges of "high category density and high semantic confusion" and "high efficiency and low cost" in the financial document classification scenario. Its core is a hybrid RAG architecture of "retrieval priority and inference fallback," embedding an intelligent bimodal decision engine to achieve the optimal balance between processing efficiency and accuracy.

[0270] The following is a detailed processing flow of the RAG classification mechanism in this example:

[0271] Step 1: Voucher Vectorization and Query Preparation

[0272] After a user uploads an image of their credentials, the system first extracts the full text information using the OCR module. Then, it calls the same embedding model used during the knowledge base construction to convert the extracted text content, either as a whole or in chunks, into a high-dimensional query vector.

[0273] Step 2: Top-K Semantic Retrieval and Metadata Extraction

[0274] The system uses the generated query vector to perform a Top-K approximate nearest neighbor search in a pre-built vector knowledge base. The key to this step is that the primary goal of the retrieval is to obtain the metadata associated with the vector, particularly the "category label." The retrieval service returns a list containing K most similar credential samples, each containing the following key information: the content of the matching text block, the category label, the semantic similarity score, and a category definition description.

[0275] Step 3: Candidate Category Aggregation and Decision Analysis

[0276] This is the first divergence point between this example and the general RAG. The system does not directly use the retrieved text content, but instead aggregates and statistically analyzes the "category labels" of the returned K results. Category voting: The frequency of each "category label" in the K results is counted. Confidence calculation: A comprehensive confidence score is calculated by combining the frequency of each category and its corresponding highest similarity score. Decision condition judgment: The system determines whether the aggregated results meet the "early exit" condition based on a preset threshold. For example, it determines whether the "category with the highest confidence" is unique and whether its confidence exceeds 95%.

[0277] Step 4: Bimodal Decision Execution

[0278] Modality 1: Efficient "Retrieval-as-Classification" mode (triggers early stop);

[0279] Triggering condition: The decision judgment in step three is "yes", that is, the search results are highly concentrated in a single category, representing a simple and unambiguous classification task.

[0280] Handling method: The system directly accepts the high-confidence category label as the final classification result and "stops early" at this point, terminating subsequent processes. The entire process completely avoids calling the large language model.

[0281] Data output: Directly output the classification results and record the decision path of this classification (e.g., directly determine based on Top-K search results).

[0282] Modality 2: Precise “LLM-as-Arbitrator” mode (early stop not triggered);

[0283] Triggering condition: The decision judgment in step three is "no", that is, the category labels of the search results are inconsistent (such as recalling "meeting notice" and "training notice" at the same time) or the confidence level is generally low, which represents a complex and ambiguous classification task.

[0284] Handling method:

[0285] Dynamic context enhancement: The system automatically switches to this mode to extract all candidate categories (such as "meeting notice" and "training notice") and their corresponding "category definition descriptions" that appear in the search results.

[0286] Constructing a focus prompt: Dynamically and purposefully construct a context-rich prompt from the descriptive text. This prompt explicitly instructs the LLM to choose from several candidate categories.

[0287] LLM inference: The dynamically constructed prompts and the original OCR text of the credentials are submitted to the large language model for final expert arbitration.

[0288] Data output: Parse the results returned by LLM to obtain a unique final classification, and record the complete decision path including search candidates, dynamic prompts and LLM output.

[0289] The special characteristics of the RAG processing method in this example (differences from general RAG processing)

[0290] The RAG mechanism in this example differs fundamentally from the general RAG architecture, which is primarily for question answering, in terms of application goals, process design, and model roles.

[0291] Restructuring of application objectives: from "content enhancement" to "metadata decision-making";

[0292] General RAG: Its core objective is to retrieve textual content relevant to the question and provide this content as context to the LLM to generate a richer and more accurate answer. The retrieved textual content is the main focus.

[0293] In this example, the core objective of the retrieval is to obtain metadata associated with the credentials—"category tags." Essentially, it leverages the fuzzy matching capabilities of vector retrieval to "vote" and "consensus" on the category tags of the retrieval results, thereby directly making a classification decision. The retrieved text content is secondary.

[0294] Innovation in process design: from "single-path necessary process" to "dual-modal decision engine";

[0295] General RAG: Regardless of the task's complexity, it must complete the entire path of "retrieval + LLM generation". The LLM call is an indispensable final step in its process.

[0296] This example demonstrates a creatively designed "early stop" mechanism based on retrieval confidence, constructing two processing modes: "retrieval as classification" and "LLM arbitration." This enables the system to intelligently distinguish between simple and complex tasks, dedicating expensive LLM resources only to handling "difficult cases," thereby significantly improving average processing efficiency and reducing operating costs when processing massive amounts of vouchers.

[0297] The shift in the role of LLM: from "answer generator" to "expert arbitrator";

[0298] General RAG: LLM is the absolute core, responsible for understanding all the retrieved context and generating the final answer.

[0299] In this example, the role of LLM shifts from "the sole decision-maker" to "an expert in handling complex and ambiguous situations." It is only invoked when the system cannot reach a clear conclusion through retrieval, and its task is to arbitrate and choose among a few given candidate categories, rather than generating open-ended decisions.

[0300] 5. The classification path is auditable, and the system is highly scalable.

[0301] The entire process records OCR text, search results, candidate categories, prompts, large model output, and final classification, supporting traceability and interpretable analysis. The system architecture supports flexible expansion of multiple voucher types, templates, vector libraries, and model services. It can be widely applied to scenarios such as smart finance, travel expense reimbursement, tax risk control, and medical auditing.

[0302] This example can achieve the following technical effects in practical applications:

[0303] 1. Significantly improves the accuracy and robustness of voucher classification.

[0304] By combining OCR text structure and semantic content for semantic segmentation and vectorization, it no longer relies on shallow keyword matching. Utilizing semantic vector recall and large-scale model language understanding capabilities, it effectively identifies invoices with significant structural variations and diverse expressions. It can accurately handle complex real-world scenarios such as ambiguous category boundaries, mixed document layouts, and non-standard formats. The accuracy of voucher classification is improved to over 95%, with significantly enhanced robustness, effectively handling complex scenarios such as ambiguous category boundaries, mixed document layouts, and non-standard formats.

[0305] 2. Significantly improves system response efficiency and processing throughput.

[0306] It adopts a phased processing logic of "semantic recall → early stop judgment → prompt word construction → large model inference"; it introduces an early stop mechanism of similarity threshold + category uniqueness judgment, which can directly hit the result when the category is highly concentrated, reduce the number of large model calls, and significantly reduce the consumption of computing resources; it is suitable for concurrent processing in large-scale invoice classification and financial audit scenarios.

[0307] 3. Improve the scalability and controllability of the classification system.

[0308] All category prompts, knowledge vectors, and templates are configurable, allowing for the addition of new categories and adjustment of classification logic without retraining. The system is suitable for invoice classification needs in multiple industries such as healthcare, transportation, finance, and education. It can be flexibly deployed on local servers, private clouds, or public cloud platforms, facilitating integration into existing business processes.

[0309] 4. Enhance the interpretability and auditability of the classification process.

[0310] The entire process retains OCR content, Top-K search records, candidate categories, prompt word templates, and large model output results; the system can provide "classification basis traceability", supports manual review, error backtracking, and model optimization; and meets the strict requirements of financial auditing, compliance supervision, and other regulatory requirements for traceable classification results.

[0311] 5. Achieve effective integration of structured data and semantic models

[0312] The system is the first to integrate structured OCR text information with a natural language understanding model (LLM) in a voucher classification scenario; it constructs a unified semantic vector space to provide a foundation for subsequent modules such as question answering, review, and knowledge graphs; and it establishes a complete intelligent chain for vouchers from image to text to semantics to category.

[0313] Unlike existing technologies, the above-mentioned technical solution receives digital image data of the vouchers to be classified, performs multimodal optical character recognition (OCR) processing to generate structured OCR results, extracts text semantic feature vectors and visual layout feature vectors based on the structured OCR results, and fuses them to generate a multimodal query vector. It then obtains similar samples and their semantic similarity scores and category metadata through approximate nearest neighbor retrieval, generates an initial candidate category list, extracts key field values ​​for each candidate category, calculates an evidence credibility score, fuses the semantic similarity score and evidence credibility score to generate a comprehensive confidence score, and re-ranks the samples. Finally, it selects a classification decision path based on the score distribution and outputs the classification results and an interpretability report. Through multimodal feature fusion and approximate nearest neighbor retrieval, the accuracy and robustness of voucher classification are effectively improved, enabling the handling of complex voucher scenarios with varying formats and ambiguous semantics. The fusion of evidence credibility score and semantic similarity score enhances the interpretability and reliability of the classification decision. The final interpretability report provides complete technical evidence for auditing and review, meeting the stringent requirements of finance, taxation, and other fields for traceability of classification results.

[0314] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A credential classification method based on RAG, characterized in that, include: Receive digital image data of the vouchers to be classified; Multimodal optical character recognition processing is performed on digital image data to generate structured OCR results containing text content and its layout information; Based on the structured OCR results, text semantic feature vectors are extracted through a semantic embedding model, and visual layout feature vectors are extracted through an image feature extraction model. The text semantic feature vector and the visual layout feature vector are fused to generate a multimodal query vector; The multimodal query vector is input into a pre-built multimodal knowledge vector library for approximate nearest neighbor retrieval to obtain retrieval results. The retrieval results include several top-ranked similar samples, the semantic similarity score of each sample with the query vector, and the category metadata associated with each sample. Based on the search results, an initial candidate category list is generated; For each candidate category in the initial candidate category list, extract the corresponding key field values ​​from the structured OCR results based on its corresponding key field rule set; Calculate the matching degree between the extracted key field values ​​and the corresponding category field patterns in the category rule knowledge base, and generate an evidence credibility score; The semantic similarity scores and evidence credibility scores of each candidate category are combined to generate a comprehensive confidence score, and the candidate category list is reordered accordingly. Based on the overall confidence score distribution of the reordered candidate category list, a classification decision path is selected, including: If the highest overall confidence score meets the preset threshold condition, the corresponding category will be directly output as the classification result; Otherwise, based on the feature contradictions between candidate categories, contrast prompts are dynamically constructed, and the contrast prompts and structured OCR results are submitted to the large language model for arbitration reasoning to obtain the final classification result; Output classification results and an interpretable report including search samples, key field evidence, and decision path.

2. The RAG-based credential classification method according to claim 1, characterized in that, Based on the search results, an initial candidate category list is generated, including: The top-ranked similar samples returned by the retrieval are categorized and aggregated according to their corresponding category labels to obtain multiple categories. Based on the semantic similarity scores of all samples included in each categorized class, the initial confidence score for the corresponding class is calculated as follows: Take the maximum semantic similarity score of all similar samples in the same category; Alternatively, take the weighted average of the semantic similarity scores of all similar samples in the same category, where the weight of the similar samples is determined according to their ranking position in the search results; The multiple categories after classification are sorted according to the initial confidence scores, and the N categories with the highest rankings are selected to form the initial candidate category list.

3. The RAG-based credential classification method according to claim 1, characterized in that, The semantic similarity scores and evidence credibility scores of each candidate category are combined to generate a comprehensive confidence score, which is then used to reorder the candidate category list, including: Obtain the initial confidence score and evidence credibility score for each candidate category in the initial candidate category list; A weighted summation algorithm is used to fuse the initial confidence score and the evidence credibility score to generate a comprehensive confidence score for the current candidate category. The weight coefficients of the initial confidence score and the evidence credibility score are configured according to the classification scenario requirements. Repeat the above steps until a comprehensive confidence score for all candidate categories is obtained; Based on the overall confidence score of all candidate categories, the initial candidate category list is reordered in descending order of the overall confidence score, generating a reordered candidate category list.

4. The RAG-based credential classification method according to claim 1, characterized in that, Calculate the matching degree between the extracted key field values ​​and the corresponding category field patterns in the category rule knowledge base, and generate an evidence credibility score, including: Obtain the field pattern definition of the current candidate category from the category rule knowledge base. The field pattern definition includes the name, data type, format requirements and validation rules of each key field. For each key field, the extracted field value is matched with the corresponding field pattern definition to obtain the matching score of a single field. The matching score calculation of a single field includes at least one method, such as format conformity check, regular expression matching, numerical range verification, or semantic conformity judgment based on a pre-trained model. Based on the matching scores of all key fields, a weighted average algorithm is used to calculate the overall evidence credibility score of the current candidate category. The weight of each key field is configured according to its importance in category discrimination.

5. The RAG-based credential classification method according to claim 1, characterized in that, The samples in the multimodal knowledge vector library include text semantic vectors, visual layout vectors, category labels, and key field rule sets; Furthermore, the multimodal knowledge vector library is configured to be built using a distributed vector database architecture; The multimodal query vector is input into a pre-built multimodal knowledge vector base for approximate nearest neighbor retrieval to obtain retrieval results. These results include several top-ranked similar samples, the semantic similarity score of each sample to the query vector, and the category metadata associated with each sample, including: Vector similarity retrieval is performed in the multimodal knowledge vector base using either a graph-based approximate nearest neighbor search algorithm or a quantization-based approximate nearest neighbor search algorithm. Calculate the similarity score between the multimodal query vector and each sample vector in the multimodal knowledge vector base. The similarity score is obtained by using a cosine similarity algorithm or an inner product similarity algorithm. All search results are sorted in descending order based on similarity scores, and the top-ranked samples are selected as similar samples. Metadata associated with each similar sample is extracted from the multimodal knowledge vector library. The metadata includes category labels, key field rule set identifiers, and original feature information of sample source credentials.

6. The RAG-based credential classification method according to claim 1, characterized in that, The text semantic feature vector and the visual layout feature vector are fused to generate a multimodal query vector, including: Perform dimension alignment processing on the text semantic feature vector and the visual layout feature vector; A weighted fusion algorithm is used to fuse the processed text semantic feature vector and visual layout feature vector; The weighted fusion algorithm is configured to adaptively weight the importance of text and visual features using an attention mechanism, assigning configurable weight coefficients to the text semantic feature vector and the visual layout feature vector respectively, and the weight coefficients are dynamically adjusted according to the voucher type and classification scenario; The fused feature vectors are normalized to generate the final multimodal query vector.

7. The RAG-based credential classification method according to claim 1, characterized in that, Based on the structured OCR results, text semantic feature vectors are extracted using a semantic embedding model, and visual layout feature vectors are extracted using an image feature extraction model, including: The text content in the structured OCR result is input into a pre-trained semantic embedding model, and semantic encoding is performed through the multi-layer Transformer structure of the semantic embedding model to output a fixed-dimensional text semantic feature vector. The digital image data is input into a pre-trained convolutional neural network model, and features are extracted through multiple convolutional and pooling layers of the convolutional neural network model to output a visual layout feature vector representing the overall layout structure of the voucher. The semantic embedding model adopts a deep bidirectional language representation model based on a self-attention mechanism, and the image feature extraction model adopts a deep residual network or a visual Transformer architecture. The process of extracting the text semantic feature vector also includes text cleaning and standardization preprocessing of the OCR recognition results, and the process of extracting the visual layout feature vector also includes size normalization and data augmentation processing of the input image.

8. The RAG-based credential classification method according to claim 1, characterized in that, Multimodal optical character recognition (OCR) processing is performed on digital image data to generate structured OCR results containing text content and its layout information, including: A deep learning optical character recognition engine is used to perform text detection and recognition on the digital image data, extracting the text content and its position coordinates in the image to obtain the recognition result; The recognition results are structured through layout analysis algorithms to detect and identify the layout information of text blocks, table areas, stamp areas and logo areas; By associating and integrating the text content with its corresponding layout information, structured OCR result data is generated. The structured OCR results are organized in JSON or XML format and include text content, text position coordinates, font size, text block type, and special area identification information; The multimodal optical character recognition processing also includes an image preprocessing step, which includes grayscale conversion, binarization, noise removal, and image enhancement operations on the input image to improve the accuracy of character recognition.

9. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1 to 8.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1 to 8.

Images