Certificate intelligent analysis and scoring method and system based on unified semantic understanding

Abstract

The invention discloses an intelligent certificate analyzing and scoring method and system based on unified semantic understanding, and belongs to the technical field of artificial intelligence and information processing cross, and the method comprises five steps of multi-modal information unified representation, field enhanced semantic understanding, thinking chain reasoning and information completion, structured output and verification, and intelligent scoring. A three-layer architecture of a multi-modal information unified presentation layer, a large language model unified understanding and reasoning layer and an intelligent scoring and output layer is adopted. According to the certificate intelligent analyzing and scoring method and system based on unified semantic understanding, multi-dimensional intelligent scoring and full-process automatic processing are achieved, the error risk is reduced, various certificates can be efficiently processed, and the method and system are suitable for education evaluation, scientific achievement affirmation and other scenes.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and information processing, and in particular to a certificate intelligent parsing and scoring method and system based on unified semantic understanding. Background Technology

[0002] In fields such as educational evaluation, scientific research achievement recognition, and government qualification review, which rely on certificates as information credentials and value carriers, certificate digitization is a core component supporting business automation. Currently, mainstream certificate processing technologies fall into three main categories, but all suffer from significant technical bottlenecks and common shortcomings, making it difficult to meet the needs of complex scenarios. I. Mainstream Technologies and Bottlenecks Traditional OCR + rule template technology relies on regular expressions and fixed templates, which cannot adapt to the diversity of certificate formats and lacks semantic understanding capabilities.

[0003] Region recognition technology based on object detection: It uses models such as YOLO and Faster R-CNN to locate information regions, but it lacks the ability to generalize to unknown formats and cannot handle cross-regional semantic associations.

[0004] End-to-end information extraction model: requires a large amount of labeled data to train a dedicated NER model, has high update and maintenance costs, and is difficult to adapt to new certificate types.

[0005] II. Deficiencies of Existing Technology Insufficient natural language description recognition capability: Unable to parse common non-key-value pair natural language content in certificates; Lacks intelligent scoring functionality: It only focuses on information extraction and lacks the ability to conduct multi-dimensional value assessment based on certificate content; Insufficient utilization of multimodal information: using only text content and ignoring the collaborative analysis of visual information such as layout and formatting; The accuracy of issuing authority identification is low: it is difficult to handle variations and abbreviations of institution names, and there is no effective confidence assessment and manual review mechanism; Insufficient depth of understanding: Unable to perform deep semantic reasoning and hidden information mining on certificate content. Summary of the Invention

[0006] The purpose of this invention is to provide a certificate intelligent parsing and scoring method and system based on unified semantic understanding, which solves the problems of traditional OCR + rule template binding format without semantic understanding, poor generalization of target detection region recognition and lack of cross-region association, high cost of end-to-end extraction model annotation and lack of evaluation function, low accuracy of issuing authority recognition and insufficient multimodal utilization and shallow understanding depth in the existing technology.

[0007] To achieve the above objectives, this invention provides a certificate intelligent parsing and scoring method based on unified semantic understanding, comprising the following steps: S1. Unified representation of multimodal information: The input certificate image is preprocessed and enhanced OCR recognition is performed sequentially to extract the text content, spatial location and font style features in the image to obtain the multidimensional information of the certificate, and the extracted features are encoded into a formatted text sequence containing semantic information. S2, Domain-Enhanced Semantic Understanding: Dynamically load domain knowledge and business rules from certificate parsing from the domain knowledge base, dynamically assemble prompt words and inject domain knowledge into the formatted text sequence based on the multi-dimensional information of the certificate, and then input the combination of prompt words, domain knowledge and formatted text sequence into the large language model for end-to-end understanding; S3, Thought Chain Reasoning and Information Completion: Based on the results of end-to-end understanding, the large language model uses the thought chain mechanism to carry out multi-step reasoning, demonstrating the reasoning process from formatted text sequences to structured information: identifying the issuing authority, combining multi-dimensional information of the certificate with domain knowledge to complete information completion and semantic disambiguation, and establishing a semantic association network between various elements of the certificate. S4. Structured Output and Validation: Based on structured information, output structured data according to a predefined JSON Schema, evaluate the integrity, logical rationality and confidence of the data, and implement hierarchical processing based on the confidence evaluation results; S5. Intelligent scoring: Based on structured data, scoring elements are extracted, a judgment matrix is ​​constructed, and the relative weights of each dimension are calculated. After consistency verification and weight adjustment, the final scoring result is generated by weighted summation and dynamic adjustment based on the semantic association of cross-influencing factors.

[0008] Preferably, the preprocessing in step S1 includes noise reduction, contrast enhancement, and tilt correction of the certificate image; the enhanced OCR recognition adopts text recognition and layout analysis technology based on PaddleOCR.

[0009] Preferably, the identification of the issuing authority in step S3 specifically includes: Step 1: Preprocessing of the issuing authority name: Clean and standardize the text of the issuing authority recognized by OCR, removing irrelevant characters and standardizing the format; Step 2, Intelligent Similarity Matching: Based on the large language model, calculate the semantic similarity between the identified text and the pre-set domain knowledge database of institution names, and comprehensively consider string similarity, semantic similarity and contextual relevance to generate Top-K candidate matching results.

[0010] Preferably, the confidence assessment in step S4 is achieved by constructing a multi-dimensional confidence assessment model; the multi-dimensional confidence assessment model considers semantic similarity score, string edit distance and contextual consistency; Semantic similarity score: measures the degree of semantic matching between structured information and standard information in the domain knowledge base; Edit distance: Represents the number of character modifications between structured information and standard text; Multi-dimensional information matching score: Combine the multi-dimensional information of the certificate to evaluate the logical rationality score of the structured information; The confidence level is calculated using the following formula: in, , , These are weighting coefficients, obtained through machine learning optimization; Indicates confidence level; Indicates semantic similarity score; Indicates the string edit distance; This represents the score for matching multi-dimensional information.

[0011] Preferably, the grading process in step S4: High confidence level: If the confidence level is ≥0.8, the recognition result is used directly and the process automatically proceeds to the next step. Medium confidence level: 0.5 ≤ confidence level < 0.8, the method suggests a result, and after user confirmation, proceeds to the next step; Low confidence level: Confidence level < 0.5, triggering manual review process, awaiting manual confirmation.

[0012] Preferably, the manual review process includes: Step 1: Provide a user-friendly review interface that displays the recognition results, candidate list, and confidence level information; Step 2: Reviewers select candidate results or manually enter the correct name; Step 3: Feedback on the review results, used to optimize subsequent recognition performance.

[0013] Preferably, the extraction and preprocessing of scoring elements in step S5 are achieved by extracting core elements, calculating the multi-dimensional weights of core elements, and identifying auxiliary elements based on the structured information obtained in step S3; the core elements include the issuing authority level, project level, and award level; the auxiliary elements include team size and member ranking.

[0014] Preferably, the steps for multi-dimensional weight calculation specifically include: Step 1: Construct a judgment matrix by comparing the relative importance of the five rating elements pairwise; Step 2: Combine the reasoning engine to supplement the semantic association and importance logic of the five rating elements. Based on the judgment matrix constructed in Step 1, calculate the maximum eigenvalue of the judgment matrix and its corresponding eigenvector. After normalizing the eigenvector, obtain the relative weights of the five rating elements. Step 3: Perform consistency checks and weight adjustments.

[0015] Preferably, the scoring calculation in step S5 specifically includes: Step 1: Perform a weighted summation based on the weights generated from the five rating elements; Step 2: Consider the semantic relationships of cross-influencing factors and make dynamic adjustments; Step 3: Generate the final score result.

[0016] This invention also provides a system for intelligent certificate parsing and scoring based on unified semantic understanding. It adopts a three-layer architecture based on unified understanding of a large language model, with each layer communicating through well-defined interfaces. Specifically, it includes: The unified representation layer for multimodal information consists of an image preprocessing module, an OCR enhancement and recognition module, and an information formatting module. The image preprocessing module is used for noise reduction, contrast enhancement, and tilt correction of certificate images. The OCR enhancement and recognition module is used for text recognition and layout analysis based on PaddleOCR. The information formatting module is used to encode the OCR results and layout information into a structured text sequence. The unified understanding and reasoning layer of the large language model consists of a dynamic prompt word assembly engine, a domain knowledge injection module, a thought chain reasoning engine, and a structured output constraint module. The dynamic prompt word assembly engine is used to dynamically assemble method prompt words based on certificate type and parsing task; the domain knowledge injection module is used to embed domain knowledge and business rules for certificate parsing; the thought chain reasoning engine is used to guide the large language model to perform multi-step reasoning and semantic understanding; the structured output constraint module is used to ensure that the output conforms to the predefined JSON Schema format; The intelligent scoring and output layer consists of a multi-dimensional weight mapping library, a scoring execution engine, and a manual review interface. The multi-dimensional weight mapping library is used to store the weight relationships of each scoring dimension. The scoring execution engine is used to perform multi-dimensional intelligent scoring based on the parsing results. The manual review interface is used for manual review and confirmation of low-confidence results.

[0017] Therefore, the present invention employs the above-mentioned certificate intelligent parsing and scoring method and system based on unified semantic understanding, which has the following beneficial effects: (1) Breakthrough in understanding depth: Based on the unified understanding framework of the large language model, it realizes deep understanding of certificate semantics and mining of implicit information, solving the information extraction blind spot of traditional technology; (2) Significantly improved accuracy in identifying issuing authorities: Through semantic understanding and confidence assessment, the problem of identifying variants and abbreviations of institution names is solved, and the final accuracy of key information is ensured by combining a manual review mechanism; (3) Improved scientific rigor of intelligent scoring: Through multi-dimensional comprehensive evaluation and reasoning, the subjectivity and bias of manual scoring are avoided, and the scoring results are more objective and fair. (4) Significantly improved processing efficiency: The entire process is automated, from certificate image recognition to intelligent scoring output, achieving one-stop processing and greatly improving work efficiency; (5) Enhanced method reliability: Through confidence assessment and manual review mechanism, low-quality identification results are effectively identified and processed, reducing business risks caused by incorrect identification.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is a system architecture diagram of a certificate intelligent parsing and scoring method and system based on unified semantic understanding, as proposed in this invention. Figure 2 This is a data flow diagram for the unified understanding and reasoning of the large language model in this invention; Figure 3 This is a data flow diagram of the intelligent scoring system of the present invention; Figure 4 This is a data flow diagram for identifying the issuing authority in this invention. Detailed Implementation

[0020] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] Example 1 I. Preparations before the experiment (a) Determining the certificate sample: a problem-solving certificate for natural language description The content information is as follows: The certificate is marked with "Certificate No.: 2024-ZJ-01"; The main content states: "The Chongqing Municipal Major Decision-Making Consultation Research Project 'Research on the Construction of Elderly Canteens in Rural Areas of Chongqing' (Additional Project, General Project, No.: 2024ZJ01), presided over by Comrade Liu Bei, has completed its research tasks, meets the conditions for exemption from evaluation and completion, and is hereby approved for completion. This certificate is hereby issued." The note below states "Main researchers in the research group: Guan Yu, Zhang Fei, Zhao Yun, Ma Chao"; A red circular official seal is affixed to the lower right corner, with the text "Chongqing Municipal People's Government Development Research Center" and the date "August 19, 2024".

[0022] (II) Installation and Configuration of Local Testing Environment 1. Basic Dependency Installation Programming language and version: Python 3.8.8. This version has the best compatibility with core dependency libraries such as PaddleOCR 2.7.0, avoiding dependency conflicts caused by version incompatibility; Core library installation: Install critical dependencies using the pip command, as follows: Execute the command `pip install paddlepaddle paddleocr==2.7.0` to deploy. Figure 1 The OCR enhancement recognition module, with PaddleOCR and its deep learning framework dependencies installed, is used for text recognition and layout analysis of certificate images.

[0023] Execute the command `pip install fastapi uvicorn python-multipart` to set up... Figure 1 The Web interface of the unified representation layer for multimodal information enables the uploading of certificate files and the return of recognition results.

[0024] Execute the command `pip install pdf2image` to achieve the following: Figure 1 The OCR enhancement recognition module supports PDF format certificates, which are used to convert PDF certificate files into image format for PaddleOCR recognition.

[0025] For Windows systems, you need to download poppler version 23.11.0, extract it, and add its bin directory path to the system environment variable PATH to ensure that pdf2image can call poppler to perform PDF to image conversion operations normally.

[0026] 2. Preparation of large language model: The deepseek large language model is selected as the core model for unified semantic understanding and reasoning. It can be called through the official API interface or deployed locally to realize deep semantic understanding, entity extraction and intelligent scoring reasoning of certificate content.

[0027] II. System Deployment and Service Startup (I) OCR service code writing and deployment, implementation Figure 1 The core function of the unified representation layer for multimodal information Create a file named services.py and write the OCR service code to implement the upload and recognition functions for certificate images (JPEG, PNG, BMP, etc.) and PDF files. The specific code is as follows (including comments for key functions):

[0028] (II) Service Launch Navigate to the directory containing services.py in the command line and execute the command `python services.py` to start the OCR service. Figure 1 All three modules of the unified representation layer for multimodal information (image preprocessing, OCR enhancement and recognition, and information formatting) are ready. After successful startup, the console will output the service address and API documentation address, which can be used to test file upload and recognition functions.

[0029] III. Implementation of the intelligent certificate resolution process, such as Figures 1-4 As shown (a) Certificate Image Acquisition and Upload 1. Obtain the certificate file: Obtain the image file or PDF file of the project completion certificate, ensuring that the file is clear and free of obvious stains or tilting; 2. Upload files using the OCR service: Upload the certificate file by calling the http: / / localhost:8000 / ocr / recognize interface through Postman, curl, or a custom front-end page. For example, using Postman: Select POST as the request method and enter the API address; In the "Body" tab, select "form-data", set the key to "file", the type to "File", and select the certificate file to be uploaded; Click the "Send" button to send the request and wait for the service to return the recognition result.

[0030] (II) Acquisition and Processing of OCR Recognition Results 1. Obtaining Recognition Results: After successful processing, the service will return recognition results in JSON format. For this completion certificate, the core recognition results returned are as follows:

[0031] 2. Organize and Recognize Text: Concatenate the text fragments in the "texts" list according to the logical order of the certificate content to form a complete natural language text, correcting line breaks caused by OCR recognition. The organized text is as follows: Certificate No.: 2024-ZJ-01 The Chongqing Municipal Major Decision-Making Consultation Research Project "Research on the Construction of Elderly Canteens in Rural Areas of Chongqing" (Additional Project, General Project, No.: 2024ZJ01), presided over by Comrade Liu Bei, has completed its research tasks and meets the conditions for exemption from evaluation and completion. It is hereby approved for project completion.

[0032] This certificate is hereby issued.

[0033] Key researchers in the research group: Guan Yu, Zhang Fei, Zhao Yun, Ma Chao Government Damin Government of Chongqing Development Research Center August 19, 2024 (III) Domain knowledge injection and prompt word setting 1. Build a domain knowledge base: Organize the domain knowledge required for certificate resolution, the specific content of which is as follows:

[0034] 2. Set domain expert prompts: Based on the domain knowledge base, write prompts to clarify the task objectives (entity extraction, score calculation) and output format (JSON) of the large language model. The prompt words are as follows: "You are a certificate parsing assistant. Please strictly follow my requirements to output JSON data and do not output any other unnecessary information." The JSON output format requirements are as follows:

[0035] (iv) Entity extraction and reasoning of large language models 1. Call the deepseek large language model: Through the deepseek model's API interface or local deployment interface, pass the processed certificate text and domain expert prompts to the model and request the model to perform entity extraction and intelligent scoring reasoning.

[0036] 2. Obtain Model Output Results: After the model processing is complete, it returns a JSON result in a preset format. The extraction and scoring results of the completion certificate are as follows:

[0037] (v) Result verification and manual review 1. Automatic Validation: The system automatically validates the model output results according to preset rules, including: Format validation: Checks whether the JSON structure conforms to the preset schema; Logical verification: Verify whether the score calculation is correct and whether the number of team members matches the number of members in the member list.

[0038] 2. Manual Review Triggered: If the confidence level of the issuing authority in the model output is "medium" or "low", or if the automatic verification detects logical anomalies, the manual review process will be triggered. Reviewers will use the review interface provided by the system to view the original certificate image, OCR-recognized text, and model output results, and correct any abnormal fields (the corrected results will be fed back to the system as the final parsing result).

[0039] Therefore, this invention adopts the above-mentioned certificate intelligent parsing and scoring method and system based on unified semantic understanding. Through a three-layer architecture design and four core technological innovations, it constructs a full-process intelligent processing system "from certificate image input to structured scoring output", which completely solves the core pain points of existing technologies in semantic understanding, multimodal utilization, issuing authority identification and intelligent scoring.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A certificate intelligent parsing and scoring method based on unified semantic understanding, characterized in that, Includes the following steps: S1. Unified representation of multimodal information: The input certificate image is preprocessed and enhanced OCR recognition is performed sequentially to extract the text content, spatial location and font style features in the image to obtain the multidimensional information of the certificate, and the extracted features are encoded into a formatted text sequence containing semantic information. S2, Domain-Enhanced Semantic Understanding: Dynamically load domain knowledge and business rules from certificate parsing from the domain knowledge base, dynamically assemble prompt words and inject domain knowledge into the formatted text sequence based on the multi-dimensional information of the certificate, and then input the combination of prompt words, domain knowledge and formatted text sequence into the large language model for end-to-end understanding; S3, Thought Chain Reasoning and Information Completion: Based on the results of end-to-end understanding, the large language model uses the thought chain mechanism to carry out multi-step reasoning, demonstrating the reasoning process from formatted text sequences to structured information: identifying the issuing authority, combining multi-dimensional information of the certificate with domain knowledge to complete information completion and semantic disambiguation, and establishing a semantic association network between various elements of the certificate. S4. Structured Output and Validation: Based on structured information, output structured data according to a predefined JSON Schema, evaluate the integrity, logical rationality and confidence of the data, and implement hierarchical processing based on the confidence evaluation results; S5. Intelligent Scoring: Based on structured data, scoring elements are extracted, a judgment matrix is ​​constructed, and the relative weights of each dimension are calculated. After consistency verification and weight adjustment, the final scoring result is generated by weighted summation and dynamic adjustment based on the semantic association of cross-influencing factors.

2. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 1, characterized in that, The preprocessing in step S1 includes noise reduction, contrast enhancement, and tilt correction of the certificate image; enhanced OCR recognition uses text recognition and layout analysis technology based on PaddleOCR.

3. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 1, characterized in that, The identification of the issuing authority in step S3 specifically includes: Step 1: Preprocessing of the issuing authority name: Clean and standardize the text of the issuing authority recognized by OCR, removing irrelevant characters and standardizing the format; Step 2, Intelligent Similarity Matching: Based on the large language model, calculate the semantic similarity between the identified text and the pre-set domain knowledge database of institution names. Taking into account string similarity, semantic similarity and contextual relevance, generate Top-K candidate matching results.

4. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 1, characterized in that, The confidence assessment in step S4 is achieved by constructing a multi-dimensional confidence assessment model; the multi-dimensional confidence assessment model considers semantic similarity score, string edit distance, and contextual consistency; Semantic similarity score: measures the degree of semantic matching between structured information and standard information in the domain knowledge base; Edit distance: Represents the number of character modifications between structured information and standard text; Multi-dimensional information matching score: Combine the multi-dimensional information of the certificate to evaluate the logical rationality score of the structured information; The confidence level is calculated using the following formula: in, , , These are weighting coefficients, obtained through machine learning optimization; Indicates confidence level; Indicates semantic similarity score; Indicates the string edit distance; This represents the score for matching multi-dimensional information.

5. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 4, characterized in that, The hierarchical processing in step S4: High confidence level: If the confidence level is ≥0.8, the recognition result is used directly and the process automatically proceeds to the next step. Medium confidence level: 0.5 ≤ confidence level < 0.8, the method suggests a result, and the user confirms before proceeding to the next step; Low confidence level: Confidence level < 0.5, triggering manual review process, awaiting manual confirmation.

6. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 5, characterized in that, The manual review process specifically includes: Step 1: Provide a user-friendly review interface that displays the recognition results, candidate list, and confidence level information; Step 2: Reviewers select candidate results or manually enter the correct name; Step 3: Feedback on the review results, used to optimize subsequent recognition performance.

7. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 1, characterized in that, The extraction and preprocessing of scoring elements in step S5 are based on the structured information obtained in step S3, and are achieved by extracting core elements, calculating the multi-dimensional weights of core elements, and identifying auxiliary elements. Core elements include the issuing authority level, project level, and award level; auxiliary elements include team size and member ranking.

8. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 7, characterized in that, The specific steps for multi-dimensional weight calculation include: Step 1: Construct a judgment matrix by comparing the relative importance of the five rating elements pairwise; Step 2: Combine the reasoning engine to supplement the semantic association and importance logic of the five rating elements. Based on the judgment matrix constructed in Step 1, calculate the maximum eigenvalue of the judgment matrix and its corresponding eigenvector. After normalizing the eigenvector, obtain the relative weights of the five rating elements. Step 3: Perform consistency checks and weight adjustments.

9. The certificate intelligent parsing and scoring method based on unified semantic understanding according to claim 8, characterized in that, The scoring calculation in step S5 specifically includes: Step 1: Perform a weighted summation based on the weights generated from the five rating elements; Step 2: Consider the semantic relationships of cross-influencing factors and make dynamic adjustments; Step 3: Generate the final score result.

10. A system for executing the certificate intelligent parsing and scoring method based on unified semantic understanding as described in claims 1-9, characterized in that, A three-tier architecture based on a unified understanding of a large language model is adopted, with each layer communicating through well-defined interfaces, specifically including: The unified representation layer for multimodal information consists of an image preprocessing module, an OCR enhancement and recognition module, and an information formatting module. The image preprocessing module is used for noise reduction, contrast enhancement, and tilt correction of certificate images. The OCR enhancement and recognition module is used for text recognition and layout analysis based on PaddleOCR. The information formatting module is used to encode the OCR results and layout information into a structured text sequence. The unified understanding and reasoning layer of the large language model consists of a dynamic prompt word assembly engine, a domain knowledge injection module, a thought chain reasoning engine, and a structured output constraint module. The dynamic prompt word assembly engine is used to dynamically assemble method prompt words based on certificate type and parsing task; the domain knowledge injection module is used to embed domain knowledge and business rules for certificate parsing; the thought chain reasoning engine is used to guide the large language model to perform multi-step reasoning and semantic understanding; the structured output constraint module is used to ensure that the output conforms to the predefined JSON Schema format; The intelligent scoring and output layer consists of a multi-dimensional weight mapping library, a scoring execution engine, and a manual review interface. The multi-dimensional weight mapping library is used to store the weight relationships of each scoring dimension. The scoring execution engine is used to perform multi-dimensional intelligent scoring based on the parsing results. The manual review interface is used for manual review and confirmation of low-confidence results.

Images