Template pre-checking based power contract section classification and multi-modal recognition method

Abstract

The application discloses a power contract chapter classification and multi-modal recognition method based on template pre-checking, comprising: acquiring a power contract image sequence and performing clarity evaluation; inputting the image sequence into a layout analysis model to extract chapter titles and organize into a contract directory structure tree, and performing item-by-item comparison on the number of chapters, chapter texts and sequences with a standard directory structure, and terminating if the comparison is inconsistent; according to the page numbers of the chapter titles in the directory structure tree that passes the check, the chapter page interval is demarcated, and a semantic label is assigned to each page; according to the page type matching recognition engine combination, information extraction is performed, the recognition result is summarized to perform compliance checking, and the result is pushed to a business system interface. The application can intercept invalid contracts through the pre-checking mechanism, reduce the consumption of computing resources, and the accuracy rate of key field recognition can reach more than 95%.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision and natural language processing, and in particular relates to a method for classifying and recognizing chapters of power contracts based on template pre-verification. Background Technology

[0002] In electricity marketing, power supply contracts serve as core legal documents, and key pages such as the signature page, cover page, and property boundary diagram must be strictly archived and their content ensured to be compliant. Traditional contract processing relies on manual verification, which is inefficient and prone to errors. In recent years, the industry has attempted to introduce artificial intelligence technologies such as Optical Character Recognition (OCR), Large Language Modeling (LLM), and object detection to achieve automated recognition, but existing solutions still have significant limitations:

[0003] On the one hand, most systems adopt a sequential process of "identification first, verification later." For example, CN202510939373.3 discloses a contract information extraction method based on dynamic template matching and semantic similarity comparison. Although it can adapt to certain format changes, its template matching occurs after field identification, making it impossible to intercept obviously invalid contracts (such as non-standard templates, missing pages, or mixed irrelevant documents) before identification. This leads to a waste of subsequent computing resources and makes it difficult to trace back errors. Similarly, although CN202510745850.2 introduces LLM for field reconstruction, it does not solve the problem of pre-judgment of the legality of the overall contract structure.

[0004] On the other hand, existing technologies for recognizing seals and signatures on signature pages often separate seal detection from text recognition. For example, CN202210887226.2 only extracts the text in the seal area using OCR, without considering the significant impact of the arc-shaped seal layout commonly found in power contracts on recognition accuracy; at the same time, it lacks an automatic verification mechanism for the key compliance item of "whether the seal user is consistent with the contract signatory," still requiring manual review.

[0005] Furthermore, current solutions generally treat page classification, field extraction, and compliance verification as independent modules, lacking unified collaborative control logic, making it difficult to achieve high-precision and robust end-to-end processing in complex and non-standardized power contract scenarios.

[0006] Therefore, there is an urgent need for a new technical solution that can verify the legality of contract templates before identification and deeply integrate layout analysis, multimodal recognition and business rule verification to solve the fundamental defects of existing methods in terms of efficiency, accuracy and compliance. Summary of the Invention

[0007] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a method for classifying and identifying chapters of power contracts based on template pre-verification.

[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0009] A method for classifying and recognizing chapters of power contracts based on template pre-verification includes the following steps:

[0010] Step S1, File Acquisition and Preprocessing: Acquire the original PDF or image file of the power contract, convert the PDF file into an image sequence page by page according to the preset resolution, and evaluate the clarity quality of each image, removing blurry images;

[0011] Step S2, Contract Template Consistency Pre-verification: Input the image sequence of the entire contract into the layout analysis model, extract the main title and chapter title of each page, organize the extracted title sequence into a contract directory structure tree in page order, and compare it item by item with the standard directory structure pre-stored in the contract template library. The comparison dimensions include the number of chapters, the text of the chapter titles, and the order of the chapters. If any comparison is inconsistent, the process is terminated and the error code "Contract template is incorrect" is returned. If all comparisons are consistent, proceed to step S3.

[0012] Step S3, Intelligent Page Classification: Based on the contract directory structure tree and the order of contract images, all images are divided into chapters, and semantic tags are assigned to each page of images;

[0013] Step S4, Multi-model Collaborative Recognition: Based on the page type corresponding to the semantic tags assigned in Step S3, select a combination of recognition engines that matches the page type for information extraction, specifically including:

[0014] For the cover page, use a visual language big model to extract contract metadata fields;

[0015] For the main text page, after extracting the full text using an OCR model, custom fields are extracted using regular expression matching.

[0016] For the signature page, a comprehensive recognition is performed using layout analysis model, handwriting detection model, visual language model, and seal recognition model.

[0017] For the property boundary diagram page, a multimodal model is used to verify the authenticity of the image and count the number of stamps.

[0018] For physical image data pages, a multimodal model is used to verify the image authenticity;

[0019] Step S5, Results Summary and Post-processing: Summarize the recognition results of each model in Step S4, perform compliance verification, and push the verified structured recognition results to the business system interface.

[0020] Furthermore, in step S2, the contract directory structure tree is represented as follows:

[0021] ;

[0022] in Page number, For title text, ∈{0,1,2} represents the hierarchy, where 0 represents the cover title, 1 represents the chapter title, and 2 represents the section title.

[0023] Furthermore, in step S2, the standard directory structure shall include at least the following eight core chapters: contract cover, Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, signature page, power supply wiring and property boundary diagram, and on-site physical image data of power supply wiring and property boundary; in the item-by-item comparison, only missing attachments are allowed, but the contract cover, Chapter 1 and signature page must exist and the title text and order must be completely consistent.

[0024] Furthermore, in step S2, before comparing the title text, a step of standardizing the extracted title is included. The standardization preprocessing includes: automatically cleaning line breaks, redundant spaces and irrelevant punctuation; automatically merging the same chapter title that was split due to line breaks in the layout; uniformly performing full-width and half-width conversion, standardizing the chapter number format and removing invalid modifier characters.

[0025] Furthermore, in step S3, the semantic tags include: contract cover, Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, signature page, power supply wiring diagram and property boundary point diagram, and physical image data of power supply wiring and property boundary point; the rule for dividing the chapters is: when the title of Chapter N appears on page a and the title of Chapter N+1 appears on page b, then pages a to b-1 belong to the content of Chapter N.

[0026] Furthermore, step S4, specifically the identification of the signature page, includes the following sub-steps:

[0027] S41: First, use a visual language model combined with a handwritten character detection model to identify the presence of circular seals, square seals, and handwritten characters in the signature page image;

[0028] S42: Use the layout analysis model to locate the circular and square stamp areas and crop out individual stamp images;

[0029] S43: Perform polar coordinate transformation on each detected stamp area to unfold the circular text into a horizontal linear sequence;

[0030] S44: Input the transformed image into the seal recognition model and output the company name and clarity score;

[0031] S45: Use the ResNet-18 handwriting / printing classifier to classify the signature area. When the handwriting probability is greater than 0.7, it is judged as a handwritten signature.

[0032] S45 extracts the signing date by combining keyword positioning with a large visual language model.

[0033] Furthermore, the compliance verification in step S5 includes at least the following three items:

[0034] Main body consistency verification: Calculate the edit distance between the electricity user name extracted from the cover and the electricity user name extracted from the seal recognition on the signature page. ,when If the number of users is ≤2 or the name of the electricity user on the cover is included in the name of the electricity user on the seal, the subject consistency check is deemed to have passed.

[0035] Verification of the number of stamps: The signature page must contain at least one stamp from the power supplier and at least one stamp from the power consumer;

[0036] Seal clarity verification: The recognition confidence of each seal is greater than or equal to 0.85.

[0037] Furthermore, in step S5, the step of pushing the structured recognition result to the business system interface includes:

[0038] The structured recognition results are converted into JSON format data aligned with the fields of the electricity marketing business system;

[0039] Call the business system API to retrieve the user profile and historical contract data corresponding to the contract;

[0040] Perform field-level difference comparisons, and trigger a risk warning when a key field deviation is detected to exceed a preset threshold;

[0041] The front-end interface highlights the differences and generates a difference report that includes field names, old values, new values, and suggested actions.

[0042] Furthermore, the sharpness quality assessment in step S1 uses the Laplacian operator to calculate the sharpness variance:

[0043] ;

[0044] in For the input image, Total number of pixels This is the Laplace response value. The mean of the Laplace response; when If the value is less than 100, mark the page as blurry.

[0045] Furthermore, the seal recognition model is a CRNN seal recognition model, which uses ResNet-34 as the backbone for visual feature extraction and extracts the texture and shape features of the seal image through convolutional layers; it uses bidirectional LSTM sequence modeling in the CRNN decoder to capture the contextual dependencies between text characters; and it uses the CTC loss function for sequence alignment training, eliminating the need for precise character-level annotation.

[0046] Compared with existing technologies, the template-based pre-verification method for classifying and multimodal identifying chapters of power contracts described in this invention has the following advantages:

[0047] This invention innovatively introduces a contract directory structure consistency verification mechanism at the very beginning of the identification process, performing subsequent identification only on contracts that conform to the standard template; effectively avoiding systemic errors such as field misalignment and content mispronunciation caused by template misuse, missing chapters, or title tampering; directly intercepting and alerting documents with "incorrect contract templates", reducing invalid computing resource consumption by more than 85%, while ensuring the compliance and authority of data entry in the Marketing 2.0 system.

[0048] This invention designs a dedicated identification strategy for the five core pages of an electricity contract (cover, first chapter, signature page, property boundary diagram, and physical image data); the accuracy rate of metadata extraction from the cover is 98.5%, and the accuracy rate of electricity information identification in the first chapter is 96.2%; the signature page achieves full automation of the process of determining the existence of a seal, distinguishing the form of the signature (stamped / handwritten), identifying the signatory, and extracting the signing time; the property boundary page supports authenticity verification and accurate statistics of the number of seals, meeting the strict requirements of property ownership audit.

[0049] This invention constructs a multimodal collaborative recognition framework, which improves the average recognition accuracy of key fields to over 95% in complex real-world business scenarios such as fuzzy scanning, handwritten annotations, overlapping stamps, and non-standard typesetting. This is nearly 20 percentage points higher than the traditional single OCR solution (which is usually below 80%).

[0050] This invention innovatively implements an automatic comparison logic of "signature page stamp content vs. cover main information". The stamp is deemed valid only when the name of the electricity user / power supplier's stamp completely matches (or contains) the cover. Combining stamp clarity scoring and quantity statistics, it provides a quantitative basis for contract validity review. It effectively prevents legal risks such as misuse of official seals and mismatch of subject matter, and improves the efficiency of contract compliance review by 300%.

[0051] The identification results of this invention are output in a standardized JSON format, with field names highly aligned with the Marketing 2.0 system; it automatically triggers field-level comparison and risk warning mechanisms (such as electricity capacity deviation exceeding the 10% threshold, address inconsistency, etc.); it realizes a closed-loop business process of "identification-verification-warning-correction", and shortens the processing time of abnormal contracts from 3 days to real time.

[0052] The contract template update of this invention only requires maintaining the standard directory structure and field mapping relationship in the knowledge base, without retraining the model or writing hard-coded rules; it supports a unified processing framework for four types of contracts, namely high-pressure, low-pressure, temporary, and wholesale, with a model reuse rate of over 80%; it has good generalization ability and business scalability, and the access cycle for new contract types is shortened from 2 weeks to 2 days. Attached Figure Description

[0053] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0054] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0055] Figure 2 This is a schematic diagram illustrating the extraction of the contract cover name according to the present invention;

[0056] Figure 3 This is a schematic diagram illustrating the extraction of chapter titles and subsection titles from the first chapter of this invention;

[0057] Figure 4 This is a schematic diagram illustrating the extraction of the signature page title according to the present invention;

[0058] Figure 5 This is a schematic diagram illustrating the extraction of contract cover page content according to the present invention;

[0059] Figure 6 This is a schematic diagram illustrating the extraction of electricity address and electricity category according to the present invention;

[0060] Figure 7 This is a schematic diagram illustrating the extraction power capacity of the present invention;

[0061] Figure 8 This is a schematic diagram illustrating the extraction of the signature page of the electricity user and electricity supplier's stamps, handwritten words, signature seals, and signing date in this invention.

[0062] Figure 9 This is a schematic diagram illustrating the determination of whether a photograph is authentic and the extraction of the number of stamps in this invention. Detailed Implementation

[0063] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0064] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0065] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0066] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0067] refer to Figure 1 This invention provides a method for classifying and recognizing chapters of power contracts based on template pre-verification (the term "multimodal" in this invention refers to heterogeneous information forms coexisting in power contracts, such as printed text, handwritten signatures, curved seal text, and property boundary diagrams). This method uses template consistency verification as the entry condition and achieves efficient, compliant, and low-false-positive intelligent recognition and verification through a five-stage pipeline architecture of "pre-verification → page classification → collaborative recognition → compliance comparison → system integration." The specific steps are as follows:

[0068] Step S1, File Acquisition and Preprocessing:

[0069] Obtain the original PDF or image files of the power contracts from the business system or offline, and convert them page by page into a JPG image sequence (page_01.jpg, page_02.jpg, …, page_n.jpg) at 300 DPI resolution using the pdf2image tool. Calculate the sharpness variance for each image using the Laplacian operator.

[0070] ;

[0071] in For the input image, Total number of pixels This is the Laplace response value. The mean of the Laplace response; if If the value is less than 100, it is marked as blurry and a re-acquisition is prompted to ensure the quality of subsequent recognition input.

[0072] Step S2, Contract Template Consistency Pre-verification:

[0073] The image sequence of the entire contract is input into the layout analysis model (based on the YOLOv11n architecture, with a total of 2.6M parameters, trained on 24,000 pages of contract images, with single-page inference ≤80ms). The main title and first- and second-level chapter titles of each page are extracted, and the extracted title sequence is organized into a contract directory structure tree in page order:

[0074] ;

[0075] in Page number, For title text, ∈{0,1,2} represents the hierarchy, where 0 represents the cover title, 1 represents the chapter title, and 2 represents the section title.

[0076] The contract templates are compared item by item with the standard directory structure pre-stored in the contract template library. The comparison dimensions include: number of chapters, chapter title text, and chapter order. Among them, the eight core chapters, namely the contract cover, Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, signature page, power supply wiring and property boundary diagram, and on-site physical images of power supply wiring and property boundary, must be consistent.

[0077] In response to the problem of non-standard identification or manual filling, this application also sets up a step of standardizing the extracted title before comparing the title text: (1) Title preprocessing: automatically cleans line breaks, redundant spaces, and irrelevant punctuation, and automatically merges the same chapter title that was split due to line breaks in the layout; (2) Standardized matching: uniformly converts the preprocessed title to full-width and half-width characters, standardizes the chapter number format, removes invalid modifier characters, and uses fixed rules for accurate comparison.

[0078] This invention introduces a template consistency verification step based on the contract directory structure at the very beginning of the identification process. It extracts chapter titles using a layout analysis model and constructs a contract directory structure tree, which is then rigorously compared with a pre-stored standard template (such as the "High Voltage Power Supply Contract Standard Template V3.2"). Only when the titles and order of core chapters (cover, Chapter 1, signature page, etc.) completely match are subsequent identification processes allowed; otherwise, processing is immediately terminated and marked as "incorrect contract template." This mechanism fundamentally avoids field misalignment and information mis-extraction caused by template misuse, missing chapters, or human tampering, effectively intercepting invalid or abnormal contracts, reducing the system's false alarm rate by over 90%, and simultaneously reducing unnecessary computational resource consumption by over 85%, ensuring that the data input into the Marketing 2.0 system is highly compliant and authoritative.

[0079] Step S3, Intelligent Page Categorization:

[0080] Obtain the page number positions corresponding to each chapter title in the verified contract directory structure tree from step S2, and define the page range of each chapter based on the page numbers of adjacent chapter titles. If the title of Chapter N appears on page a and the title of Chapter N+1 appears on page b, then pages a to b-1 are all assigned to Chapter N. This process is repeated for all chapter titles to form a mapping relationship between each chapter and its contained pages. Based on this mapping relationship, assign semantic tags to each page image. Semantic tags include: contract cover, Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, signature page, power supply wiring and property boundary diagram, and physical image data of power supply wiring and property boundary. Record the assigned page semantic tags and corresponding page numbers as a page-tag mapping table for use by the recognition engine combination for each page type in step S4.

[0081] Step S4, Multi-model collaborative recognition:

[0082] Based on the page type corresponding to the semantic tags of each page in the page-tag mapping relationship, a combination of recognition engines matching the page type is selected for information extraction, specifically including:

[0083] For the cover page, the Visual Language Model (VLM) is used to design system prompts to guide the model to output structured JSON for specified fields, extracting information such as the electricity user, power supplier, user number, contract name, and contract number, centered on a newline.

[0084] For the main text pages (Chapters 1 to 5), after extracting the full text using an OCR model, custom fields such as electricity address, electricity type, and electricity capacity are extracted through regular expression matching. During testing and iteration, the regular expression-related extraction rules are optimized, such as refining the extraction of elements by chapter, summary, and bullet points. If the matching fails, LLM-assisted reasoning is invoked, prompting the user to extract the corresponding fields when the entire chapter text is input.

[0085] For the signature page, the Visual Language Model (VLM) combined with a handwriting detection model is used to identify the presence of circular seals, square seals, and handwriting. Then, a layout analysis model is used to locate the areas of the circular and square seals and crop out individual seal images. Next, a seal recognition model is used to process the curved text, outputting the company name and a clarity score. Simultaneously, a ResNet-18-tuned handwriting / print classifier is used as input to the cropped signature area, outputting the handwriting probability P. If P > 0.7, it is determined to be a handwritten signature. Nearby text is located using the keyword "signing date," and the signing date is extracted using the Visual Language Model.

[0086] For the property boundary diagram page, a multimodal model (based on GLM-4.1V-Thinking-Flash) is called to verify the authenticity of the image (to determine whether it is a real scene photo, not a screenshot or printout) and to count the number of stamps.

[0087] For physical image data pages, a multimodal model (based on GLM-4.1V-Thinking-Flash) is called to verify the authenticity of the image (to determine whether it is a real scene photo).

[0088] To address the limitations of OCR in handling line breaks, column breaks, and non-fixed format text, this invention innovatively introduces a Visual Language Model (VLM) as an end-to-end recognition model. The system inputs the original image as context into the VLM and guides it to reconstruct logically complete fields using carefully designed prompts (e.g., merging "Electricity User:\nShanxi XX Company" into a complete entity). This "VLM end-to-end understanding output" model endows the system with powerful semantic reasoning and fault tolerance capabilities, significantly improving the completeness and accuracy of extracting key metadata (such as contract number, signing date, and names of both parties) on standardized but varied pages, especially on covers and signature pages.

[0089] Step S5, Results Summary and Post-processing:

[0090] The identification results from each model are aggregated to construct a structured contract object, and compliance verification is performed based on specific business needs. Compliance verification includes:

[0091] (1) Main body consistency verification: compare the "electricity user" information on the cover. The content of the electricity user's seal on the signature page Edit distance ( ):

[0092] ;

[0093] in Edit distance indicates the distance between the strings. Transform into The minimum number of editing operations (insert, delete, replace) required. ≤2 or If so, then it passes.

[0094] (2) Verification of the number of stamps: The signature page must contain at least one stamp of the power supplier and one stamp of the power consumer.

[0095] (3) Seal clarity verification: the recognition confidence level of each seal. ≥0.85, otherwise marked as "blurred stamp".

[0096] (4) Field-level difference comparison: The structured identification results are compared with the user profile and historical contract data corresponding to the contract in the power marketing business system. When the power capacity deviation exceeds the 10% threshold or the power address is inconsistent, a risk warning is triggered.

[0097] If a conflict is found in a key field, a risk warning is triggered. The difference and modification suggestions are highlighted on the front end, and an Excel report containing the field name, old value, new value, and suggested operation is generated.

[0098] The present invention will be illustrated below with specific examples.

[0099] 1. Obtain a paper copy of the "High Voltage Power Supply Contract", scan it using a high-speed scanner with a resolution of no less than 300 DPI, and generate a PDF file; rename it to "Shanxi_1234567890.pdf" according to the naming convention; use the pdf2image tool to convert each page of the PDF into a JPG image and save them as page_01.jpg, page_02.jpg, ..., page_n.jpg; perform a quality assessment on each image.

[0100] 2. The layout analysis model (based on the YOLOv11n architecture, with a total of 2.6M parameters, trained on 24,000 pages of contract images, with single-page inference ≤80ms) is invoked to extract text blocks and their hierarchical structure from each page image. The input is a single-page JPG image (640×640), and the model outputs the coordinates of the detection boxes and hierarchical labels (0=cover title, 1=chapter title, 2=section title), which, combined with OCR text, form structured JSON. The model automatically extracts the main title (e.g., "High Voltage Power Supply Contract," corresponding to level=0) and first / second-level chapter titles (e.g., "Chapter 1 Basic Information on Power Supply," corresponding to level=1, "Section 1 Power Address," corresponding to level=2) from each page. The extracted titles are filtered by level while maintaining page number order to construct the contract directory structure tree T. Load the standard template from the contract template knowledge base and perform a step-by-step comparison. Only missing attachments are allowed, but the cover, Chapter 1, and signature page must be present and their titles and order must be completely consistent. If there are any discrepancies (e.g., a missing "signature page" or "Chapter 1" being altered to "Electricity Terms"), immediately terminate the process, return the error code ERR_TEMPLATE_MISMATCH, and record the contract ID for manual review. Figure 2-4 As shown.

[0101] 3. Intelligent page categorization

[0102] Based on the verified (T), assign semantic tags to each page:

[0103] page_01.jpg → “Cover”

[0104] page_02.jpg → "Chapter One"

[0105] page_k.jpg (containing the words "Signature and Seal of Both Parties") → "Signature Page"

[0106] Pages containing the phrase "Property Boundary Point Diagram" → "Power Supply Wiring Property Boundary Point Diagram"

[0107] Pages containing real-life photos but without tables → "Physical Image Data of Power Supply Wiring Property Boundary Points"

[0108] 4. Multi-model collaborative recognition

[0109] (1) Cover page recognition

[0110] Using the Visual Language Model (VLM), the system prompt is designed as follows:

[0111] You are a professional assistant for extracting electricity contract information. Please carefully read the following contract image and accurately identify and extract the following 5 fields:

[0112] Electricity user (i.e., the full name of the user unit)

[0113] Power supplier (i.e., the full name of the power supply company)

[0114] User ID (usually a 13-digit number)

[0115] Contract title (e.g., "High Voltage Power Supply Contract")

[0116] Contract number (usually begins with a letter and includes the year and serial number)

[0117] Require:

[0118] ① Answer based solely on the content of the image; do not fabricate information.

[0119] ② If a field does not appear in the graph, please return null.

[0120] ③ The output must be in a strictly valid JSON format, containing the value and confidence level (0.0~1.0) of each field.

[0121] ④ Confidence level explanation:

[0122] Text is clearly legible → ≥0.95

[0123] Text partially obscured but can be inferred → 0.7~0.9

[0124] Ambiguity exists → ≤0.6

[0125] Please output the JSON directly, without any explanation or additional text.

[0126] The Visual Language Model (VLM) outputs structured results, such as:

[0127] {“Electricity User”:“Shanxi XX Co., Ltd.”,“confidence”:0.98}

[0128] like Figure 5 As shown.

[0129] (2) Chapter 1 Identification

[0130] After extracting the full text using OCR, key fields are matched using regular expressions:

[0131] ;

[0132] ;

[0133] ;

[0134] in, The full text of the first chapter extracted by OCR; This is a regular expression matching function; = indicates the character class that can be matched; This indicates that it can match zero or more whitespace characters; This indicates that it can match one or more arbitrary characters; A zero-width positive lookahead assertion indicates a match to a period. To match numbers (integers or decimals), the decimal point is optional. The option to leave blank followed by the unit kVA is used to confirm that the matched value is a capacity value.

[0135] If the match fails, LLM-assisted reasoning is invoked, and the entire chapter text is input, with prompts to extract the three items mentioned above.

[0136] like Figure 6 , 7 As shown.

[0137] (3) Signature page recognition

[0138] The signature page recognition employs a multi-model collaborative strategy:

[0139] First, a Visual Language Model (VLM) combined with a handwritten character detection model is used to identify the presence of circular seals, square seals, and handwritten characters. Then, a layout analysis model is used to locate the regions of the circular and square seals and crop out individual seal images. The loss function used for locating the seal regions is:

[0140] ;

[0141] in Weights are assigned to the coordinate prediction loss to balance the contribution of bounding box coordinate regression. For the first The center point of each predicted bounding box Coordinate values; For the first The center point of a real-world labeled bounding box coordinate; For the first The center point of each predicted bounding box Coordinate values; For the first The center point of a real-world labeled bounding box coordinate; The target confidence loss weight is used to balance the loss weights with and without the target. To predict the intersection-union ratio (IoU) between the bounding box and the ground truth bounding box; This represents the ideal intersection-union ratio corresponding to the true annotation.

[0142] For each detected stamp area Perform polar coordinate transformation preprocessing, where, These are the original image coordinates of the pixels. The width of the bounding rectangle of the seal, and the total horizontal pixel length; The height of the outer rectangle of the stamp, and the total length in pixels in the vertical direction; and Both determine the size of the cut stamp image, along with the center point. Position the stamp on the contract page image, then expand the circular text into a horizontal linear sequence:

[0143] ;

[0144] in, The coordinates of the seal's center are This represents the radial distance from the pixel to the center of the stamp. The angle of the pixel relative to the center of the seal; the transformed image is fed into the CRNN (Convolutional Recurrent Neural Network) seal recognition model to output the company name; the specific optimizations of this model include: (1) polar coordinate transformation preprocessing for the arc-shaped seal text, which unfolds the circular text into a horizontal linear sequence to reduce the recognition difficulty; (2) using ResNet-34 (Residual Network) as the view

[0145] (2) Extract the main features of the image and extract the texture and shape features of the seal image through the convolutional layer; (3) Model the bidirectional LSTM sequence in the CRNN decoder to capture the contextual dependencies between characters; (4) Use the CTC (Connectionist Temporal Classification) loss function for sequence alignment training, without the need for precise character-level annotation.

[0146] (4) Using a ResNet-18 fine-tuned handwritten / printed signature classifier, input the cropped signature area and output the probability P. If P > 0.7, it is determined to be a handwritten signature. The calculation formula is:

[0147]

[0148] Feature vectors extracted for the ResNet-18 backbone network. and These are the weight matrix and bias term of the fully connected layer, respectively.

[0149] (5) Signing date extraction: The signing date is extracted by locating nearby text using the keyword "signing date" and combining it with the Visual Language Model (VLM). The following prompt words are designed:

[0150] You are a professional contract information extraction assistant. Please carefully observe the following contract image, accurately identify and extract the value of the "signing date" field.

[0151] Note:

[0152] The "signing date" is usually located after the label "Signing Date:", and may be handwritten or printed.

[0153] If it is handwritten, reasonably infer the year, month and day according to the stroke structure and context (such as formats like "2025.6.5", "June 5, 2025", etc.).

[0154] The output must be a date string in standard format, for example: "June 5, 2025" or "2025-06-05".

[0155] If the specific date cannot be determined, please return "unknown". Please output only one date string, do not add any explanations or extra content.

[0156] As Figure 8 shown in the figure.

[0157] (6) Schematic diagram of power supply wiring property demarcation point

[0158] Authenticity verification: Call the multimodal model (based on GLM-4.1V-Thinking-Flash) to determine whether it is a real-scene photo (not a screenshot / printed copy).

[0159] Seal statistics: Run the seal detection model on the verified image to count the number of seals , and perform content recognition on each seal.

[0160] As Figure 9 shown in the figure.

[0161] 5. Result aggregation and compliance verification

[0162] Aggregate all fields, construct a structured contract object (C), and perform key consistency verification.

[0163] 6. Convert C into a JSON format aligned with the fields of the Marketing 2.0 system; call the Marketing 2.0 API to obtain the historical archive (H) of the corresponding account number; perform difference comparison, if an early warning is triggered, highlight the difference field on the front end, and generate an Excel report including field name, old value, new value, and suggested operation.

[0164] The model training and deployment details of the present invention are as follows:

[0165] Dataset Construction: Collect 10,000 scanned copies of real electricity contracts and divide them into training / validation / test sets in an 8:1:1 ratio. Add polygonal annotations (text, tables, and stamp areas) to each page.

[0166] Seal recognition model training:

[0167] Backbone network: ResNet-34 + CRNN decoder.

[0168] Loss function: CTC Loss:

[0169] ;

[0170] in, It is used to calculate the alignment loss between the predicted sequence and the target sequence without requiring precise character-level annotation; The input feature sequence is the sequence of feature maps obtained after the seal image is extracted by the CNN (ResNet-34) backbone network; The target is labeled with a sequence of characters, which is the sequence of characters corresponding to the actual text content on the seal. The character sequence output by the model at each time step. Length of input feature sequence The number of time steps is the same; B is to align a path. Consecutive repeating characters in the text are merged into one, and all whitespace characters are removed; This represents the set of inverse mappings of the folding function B, i.e., all sequences that, after folding, yield the target labeled sequence. Alignment path A set; To in a given input sequence Under these conditions, the model outputs an aligned path. The probability of the path, which is equal to the probability of the path. Output the product of the probabilities of the corresponding character at each time step.

[0171] Optimizer: AdamW (Adaptive Moment Estimation W Optimizer), Learning Rate batchsize=32.

[0172] LoRA (Low-Rank Adaptation) Fine-tuning: In the LLM information extraction module, only the low-rank adaptation matrix is ​​updated, the original weights are frozen, and memory consumption is reduced by more than 60%.

[0173] Deployment environment: Ubuntu 20.04 + Python 3.9 + PyTorch 2.0 + ONNX Runtime (Open Neural Network Exchange Runtime), average processing time per contract ≤ 8 seconds.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for classifying and recognizing chapters of power contracts based on template pre-verification, characterized in that: Includes the following steps: Step S1, File Acquisition and Preprocessing: Acquire the original PDF or image file of the power contract, convert the PDF file into an image sequence page by page according to the preset resolution, and evaluate the clarity quality of each image, removing blurry images; Step S2, Contract Template Consistency Pre-verification: Input the image sequence of the entire contract into the layout analysis model, extract the main title and chapter title of each page, organize the extracted title sequence into a contract directory structure tree in page order, and compare it item by item with the standard directory structure pre-stored in the contract template library. The comparison dimensions include the number of chapters, the text of the chapter titles, and the order of the chapters. If any of the comparisons are inconsistent, the process is terminated and the error code "Contract template is incorrect" is returned. If all comparisons match, proceed to step S3; Step S3, Intelligent Page Classification: Based on the contract directory structure tree and the order of contract images, all images are divided into chapters, and semantic tags are assigned to each page of images; Step S4, Multi-model Collaborative Recognition: Based on the page type corresponding to the semantic tags assigned in Step S3, select a combination of recognition engines that matches the page type for information extraction, specifically including: For the cover page, use a visual language big model to extract contract metadata fields; For the main text page, after extracting the full text using an OCR model, custom fields are extracted using regular expression matching. For the signature page, a comprehensive recognition is performed using layout analysis model, handwriting detection model, visual language model, and seal recognition model. For the property boundary diagram page, a multimodal model is used to verify the authenticity of the image and count the number of stamps. For physical image data pages, a multimodal model is used to verify the image authenticity; Step S5, Results Summary and Post-processing: Summarize the recognition results of each model in Step S4, perform compliance verification, and push the verified structured recognition results to the business system interface.

2. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: In step S2, the contract directory structure tree is represented as follows: ； in Page number, For title text, ∈{0,1,2} represents the hierarchy, where 0 represents the cover title, 1 represents the chapter title, and 2 represents the section title.

3. The method for power contract chapter classification and multimodal recognition based on template pre-verification according to claim 1, characterized in that: In step S2, the standard directory structure shall include at least the following eight core chapters: contract cover, Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, signature page, power supply wiring and property boundary diagram, and on-site physical image data of power supply wiring and property boundary. In the item-by-item comparison, only missing attachments are allowed, but the contract cover, Chapter 1 and signature page must be present and the title text and order must be completely consistent.

4. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: In step S2, before comparing the title text, a step of standardizing the extracted title is included. The standardization preprocessing includes: automatically cleaning line breaks, redundant spaces and irrelevant punctuation; automatically merging the same chapter title that was split due to line breaks in the layout; uniformly converting full-width and half-width characters, standardizing the chapter number format and removing invalid modifier characters.

5. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: In step S3, the semantic tags include: contract cover, Chapter 1, Chapter 2, Chapter 3, Chapter 4, Chapter 5, signature page, power supply wiring diagram and property boundary point diagram, and physical image data of power supply wiring and property boundary point; the rule for dividing the chapters is: when the title of Chapter N appears on page a and the title of Chapter N+1 appears on page b, then pages a to b-1 belong to the content of Chapter N.

6. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: Step S4, specifically the identification of the signature page, includes the following sub-steps: S41: First, use a visual language model combined with a handwritten character detection model to identify the presence of circular seals, square seals, and handwritten characters in the signature page image; S42: Use the layout analysis model to locate the circular and square stamp areas and crop out individual stamp images; S43: Perform polar coordinate transformation on each detected stamp area to unfold the circular text into a horizontal linear sequence; S44: Input the transformed image into the seal recognition model and output the company name and clarity score; S45: Use the ResNet-18 handwriting / printing classifier to classify the signature area. When the handwriting probability is greater than 0.7, it is judged as a handwritten signature. S45 extracts the signing date by combining keyword positioning with a large visual language model.

7. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: The compliance verification in step S5 includes at least the following three items: Main body consistency verification: Calculate the edit distance d between the electricity user name extracted from the cover and the electricity user name extracted from the seal recognition on the signature page. When d≤2 or the electricity user name on the cover is included in the electricity user name on the seal, the main body consistency verification is deemed to pass. Verification of the number of stamps: The signature page must contain at least one stamp from the power supplier and at least one stamp from the power consumer; Seal clarity verification: The recognition confidence of each seal is greater than or equal to 0.

85.

8. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: In step S5, the step of pushing the structured recognition result to the business system interface includes: The structured recognition results are converted into JSON format data aligned with the fields of the electricity marketing business system; Call the business system API to retrieve the user profile and historical contract data corresponding to the contract; Perform field-level difference comparisons, and trigger a risk warning when a key field deviation is detected to exceed a preset threshold; The front-end interface highlights the differences and generates a difference report that includes field names, old values, new values, and suggested actions.

9. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 1, characterized in that: The sharpness quality assessment in step S1 uses the Laplace operator to calculate the sharpness variance: ； in For the input image, Total number of pixels This is the Laplace response value. The mean of the Laplace response; when If the value is less than 100, mark the page as blurry.

10. The method for classifying and multimodal identifying chapters of power contracts based on template pre-verification according to claim 6, characterized in that: The seal recognition model is a CRNN seal recognition model, which uses ResNet-34 as the backbone for visual feature extraction and extracts the texture and shape features of the seal image through convolutional layers. The bidirectional LSTM sequence modeling in the CRNN decoder captures the contextual dependencies between text characters; the CTC loss function is used for sequence alignment training, eliminating the need for precise character-level annotation.

Images