Contract text recognition method, system and equipment based on region division Transform and medium

Abstract

The invention relates to the technical field of contract text recognition, in particular to a contract text recognition method, system and device based on region division Transform and a medium. According to the method, a contract picture is divided into a handwritten font region and a printed font region by constructing a dual-channel Transform module in combination with a region division technology, and each region is processed through an independent channel, so that accurate extraction of features is realized, mutual interference of the two fonts in the recognition process is reduced, and the recognition efficiency is improved. According to the method, the global text positioning fonts and the printing fonts are input into the two parallel Transform structures respectively, each channel is configured with different network structures and parameters to adapt to the characteristics of different fonts, and the overall performance and accuracy of the OCR system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of contract text recognition, and in particular to a contract text recognition method, system, device and medium based on region partitioning Transformer. Background Art

[0002] Optical character recognition (OCR) is a technology that converts text in an image into machine-readable text. The development of this technology has gone through several important stages, from early template matching to modern deep learning methods. OCR optical character recognition technology also plays an important role in the processing of contract images, especially in automated and digital document management systems. Contract images usually contain a large amount of text information, which requires high-precision text recognition technology to convert into an editable and searchable format. In contract management, OCR technology can automatically identify and extract key information in contract documents, such as date, name, amount, etc., to support processes such as contract review, archiving, and retrieval. This not only improves efficiency, but also reduces errors in manual input. Through these technologies, OCR processing of contract images not only improves the automation of document processing, but also provides support for all aspects of contract management, including rapid review, effective archiving, and convenient retrieval of contracts.

[0003] In the OCR processing of contract images, the mixed use of handwritten fonts and printed fonts often leads to a decrease in recognition accuracy. This is because traditional OCR systems often have difficulty distinguishing the characteristics of different fonts when processing mixed text types, which interfere with each other. Although current OCR technology has made great progress in multi-font and multi-language recognition, it still faces challenges when processing documents with complex backgrounds and multiple font styles. Especially in formal documents such as contracts, the mixed use of handwritten annotations and printed text is particularly common, which places higher demands on the accuracy and flexibility of OCR systems. For example, Paddle OCR is currently the most widely used image character recognition algorithm in China. However, through analysis, it was found that the recognition results of Paddle OCR had obvious problems in the positioning and recognition of fonts at the junction of handwritten fonts and printed fonts, resulting in a significant decrease in the final recognition accuracy.

[0004] The Chinese invention patent with the patent publication number "CN109409373A" uses traditional image processing methods to collect TTC font files, extract character feature vectors from them to form a font feature vector database, and the user uploads the image form information of a single character and the corresponding character text information. Using modern cutting technology and character shape features, the image is specifically grayed and binarized, and then the character features are extracted and feature vectors are generated. The font recognition is calculated with the data of the character text information input by the user for all characters in the database, and finally the font set with the highest recognition is output through sorting. Compared with the prior art, this invention mainly solves the problems of poor accuracy and inflexible human-computer interaction in the prior art when identifying fonts, and increases the accuracy and flexibility of the current computer-based font recognition.

[0005] The Chinese invention patent with the patent publication number "CN112862024A" proposes a text recognition system, which collects image sample sets, including: data source labels; according to the data source labels, the image sample sets are input into the text recognition network for training to obtain the text recognition model, and the handwritten font recognition model and the printed font recognition model are obtained to more accurately recognize the image to be recognized, and the text recognition results are updated by extracting semantic labels to improve the recognition accuracy. The Chinese invention patent with the patent publication number "CN113920520A" proposes an image text recognition method, the basic process of which is to obtain the image to be recognized; correct the image to be recognized to obtain a corrected image, wherein the text information in the corrected image is arranged horizontally; perform font recognition on the text information in the corrected image to obtain the recognition result, and determine the text library corresponding to the font used by the current text information according to the recognition result; continuously recognize the text information in the corrected image according to the text library corresponding to the current font to obtain a text recognition document. Although the above two patents improve recognition accuracy through different font recognition models and rules, they do not clearly distinguish and process handwritten fonts and printed fonts separately, which may not be sufficient to deal with the problem of mutual interference between fonts in complex contract text recognition scenarios. Summary of the invention

[0006] In view of the problem that the handwritten fonts and printed fonts in the contract image interfere with each other during recognition in the above-mentioned recognition method, the present invention proposes a contract text recognition method, system, device and medium based on region partitioning Transformer; the method divides the contract image into a handwritten font area and a printed font area by constructing a dual-channel Transformer module in combination with region partitioning technology, and each area is processed through an independent channel, thereby achieving accurate feature extraction and reducing mutual interference between the two fonts during the recognition process; the global text positioning font and the printed font are respectively input into two parallel Transformer structures, and each channel is configured with different network structures and parameters to adapt to the characteristics of different fonts, thereby improving the overall performance and accuracy of the OCR system.

[0007] The specific implementation contents of the present invention are as follows:

[0008] A contract text recognition method based on region partitioning Transformer specifically includes the following steps:

[0009] Step S1: preprocessing the acquired contract image;

[0010] Step S2: construct a dual-channel Transformer module based on the contract text type extracted from the preprocessed contract image to divide the contract image into regions;

[0011] Step S3: Calling the CTC algorithm sequence modeling to generate the contract feature image after the text is divided into regions, and obtaining a formatted contract text sequence;

[0012] Step S4: According to the constructed multi-task loss function, the constructed dual-channel Transformer module is trained in segments to obtain the final formatted contract text sequence.

[0013] In order to better implement the present invention, further, the step S2 specifically includes the following steps:

[0014] Step S21: ResNet50 is used as the backbone feature extraction network to extract multi-scale features from the preprocessed contract image;

[0015] Step S22: introducing a feature pyramid network FPN to generate a contract feature image containing multi-level semantic information from coarse to fine based on multi-scale features;

[0016] Step S23: calling the Fast R-CNN network and the PRN network to locate and classify the contract feature image to obtain the global positioning text and the local handwritten text;

[0017] Step S24: construct a dual-channel Transformer structure according to the text type, extract and process the text in the handwritten font area and the printed font area respectively, and obtain a handwritten sequence and a global sequence.

[0018] In order to better implement the present invention, further, the specific operation of step S3 is: calling the CTC algorithm to model the handwritten sequence and the global sequence to obtain a fused sequence, removing repeated characters and blank labels in the fused sequence according to the introduced set blank labels, and generating a clear formatted contract text sequence.

[0019] In order to better implement the present invention, further, the step S4 specifically includes the following steps:

[0020] Step S41: constructing a text localization stage loss function Ldet according to the loss function LCRNN of the Fast R-CNN network and the loss function LRPN of the PRN network;

[0021] Step S42: constructing a CTC loss function LCTC according to the probability after decoding by the CTC algorithm;

[0022] Step S43: constructing a text recognition stage loss function LC-CTC according to the set weight coefficient and CTC loss function LCTC;

[0023] Step S44: Obtain a multi-task loss function L according to the text localization stage loss function Ldet and the text recognition stage loss function LC-CTC;

[0024] Step S45: According to the constructed multi-task loss function, the backbone feature extraction network, the feature pyramid network FPN, the Fast R-CNN network and the PRN network are segmented and trained, and the constructed dual-channel Transformer module is constructed to obtain the final formatted contract text sequence.

[0025] In order to better implement the present invention, further, the step S1 specifically includes the following steps:

[0026] Step S11: converting the acquired contract image into a grayscale contract image;

[0027] Step S12: calling bilateral filtering to obtain a filtered grayscale contract image;

[0028] Step S13: calling the sober operator to obtain an enhanced grayscale contract image.

[0029] In order to better implement the present invention, further, the step S12 specifically includes the following steps:

[0030] Step S121: Calculate the spatial domain weight according to the spatial distance between the pixel point p and the pixel point q in the grayscale contract image;

[0031] Step S122: Calculate the intensity domain weight according to the intensity difference between the pixel point p and the pixel point q of the grayscale contract image;

[0032] Step S123: multiplying the spatial domain weight and the intensity domain weight to obtain a final weight value;

[0033] Step S124: performing weighted summation on the pixel values ​​in the region according to the final weight value to obtain a grayscale contract image after bilateral filtering.

[0034] In order to better implement the present invention, further, the step S13 specifically includes the following steps:

[0035] Step S131: calling a histogram equalization method to enhance the contrast of the filtered grayscale contract image to obtain a contrast-enhanced grayscale contract image;

[0036] Step S132: calling the edge detection method of the sober operator, applying the horizontal kernel and the vertical kernel to each pixel of the enhanced grayscale contract image to obtain the edge direction;

[0037] Step S133: Obtaining a grayscale contract image after edge enhancement according to the edge direction.

[0038] Based on the above-mentioned contract text recognition method based on region partitioning Transformer, in order to better realize the present invention, further, a contract text recognition system based on region partitioning Transformer is proposed, which is used to execute the above-mentioned contract text recognition method based on region partitioning Transformer; it includes a preprocessing unit, a dual-channel unit, and an output unit;

[0039] The preprocessing unit is used to preprocess the acquired contract image;

[0040] The dual-channel unit is used to construct a dual-channel Transformer module structure and divide the contract image into regions according to the contract text type extracted from the preprocessed contract image;

[0041] The output unit is used to call the CTC algorithm sequence modeling text segmentation contract feature image to obtain a formatted contract text sequence.

[0042] Based on the above-mentioned region partitioning Transformer-based contract text recognition method, in order to better realize the present invention, further, an electronic device is proposed, including a memory and a processor; a computer program is stored on the memory; when the computer program is executed on the processor, the above-mentioned region partitioning Transformer-based contract text recognition method is implemented.

[0043] Based on the above-mentioned region partitioning Transformer-based contract text recognition method, in order to better realize the present invention, further, a computer-readable storage medium is proposed, on which computer instructions are stored; when the computer instructions are executed on the above-mentioned electronic device, the above-mentioned region partitioning Transformer-based contract text recognition method is implemented.

[0044] The present invention has the following beneficial effects:

[0045] (1) The present invention clearly divides the contract image into a handwritten font area and a printed font area, and configures a dedicated processing channel for each area, so that feature extraction can be performed more accurately based on the characteristics of each area; feature interference caused by mixed font types is reduced, and overall recognition accuracy is improved; when processing complex documents such as contract images, not only the recognition accuracy and efficiency are improved, but also the adaptability and robustness of the model are enhanced, making it an effective tool for processing difficult OCR tasks.

[0046] (2) The present invention targets different types of fonts, such as handwritten and printed fonts, which have different visual features and structural complexities. Through independent channel processing, the model can more flexibly adjust network parameters and structures to adapt to the characteristics of various fonts, thereby enhancing the recognition ability of irregular fonts.

[0047] (3) The partition processing of the present invention allows the model to process data in different regions in parallel, which improves the processing speed in practical applications. In addition, targeted feature extraction reduces the interference of irrelevant data and improves the efficiency and quality of feature extraction.

[0048] (4) After feature extraction, the present invention can design different feature fusion strategies according to actual application requirements. This flexibility allows the model to optimize the recognition results according to specific scenarios while maintaining high recognition accuracy, such as adjusting the sensitivity to specific legal terms in legal document recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the structure of the contract font recognition model based on region partitioning Transformer provided by the present invention.

[0050] Figure 2This is a schematic diagram of the backbone feature extraction network structure provided by the present invention.

[0051] Figure 3 This is a schematic diagram of the structure of the image positioning and classification module provided by the present invention.

[0052] Figure 4 Schematic diagram of the encoding module-region partitioning dual-channel Transformer composition provided by the present invention. DETAILED DESCRIPTION

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0055] Embodiment 1:

[0056] This embodiment proposes a contract text recognition method based on region partitioning Transformer, which specifically includes the following steps:

[0057] Step S1: pre-processing the acquired contract image.

[0058] The step S1 specifically includes the following steps:

[0059] Step S11: converting the acquired contract image into a grayscale contract image;

[0060] Step S12: calling bilateral filtering to obtain a filtered grayscale contract image;

[0061] Step S13: calling the sober operator to obtain an enhanced grayscale contract image.

[0062] Furthermore, the step S12 specifically includes the following steps:

[0063] Step S121: Calculate the spatial domain weight according to the spatial distance between the pixel point p and the pixel point q in the grayscale contract image;

[0064] Step S122: Calculate the intensity domain weight according to the intensity difference between the pixel point p and the pixel point q of the grayscale contract image;

[0065] Step S123: multiplying the spatial domain weight and the intensity domain weight to obtain a final weight value;

[0066] Step S124: performing weighted summation on the pixel values ​​in the region according to the final weight value to obtain a grayscale contract image after bilateral filtering.

[0067] Furthermore, the step S13 specifically includes the following steps:

[0068] Step S131: calling a histogram equalization method to enhance the contrast of the filtered grayscale contract image to obtain a contrast-enhanced grayscale contract image;

[0069] Step S132: calling the edge detection method of the sober operator, applying the horizontal kernel and the vertical kernel to each pixel of the enhanced grayscale contract image to obtain the edge direction;

[0070] Step S133: Obtaining a grayscale contract image after edge enhancement according to the edge direction.

[0071] Step S2: According to the contract text type extracted from the preprocessed contract image, a dual-channel Transformer module is constructed to divide the contract image into regions.

[0072] The step S2 specifically includes the following steps:

[0073] Step S21: ResNet50 is used as the backbone feature extraction network to extract multi-scale features from the preprocessed contract image;

[0074] Step S22: introducing a feature pyramid network FPN to generate a contract feature image containing multi-level semantic information from coarse to fine based on multi-scale features;

[0075] Step S23: calling the Fast R-CNN network and the PRN network to locate and classify the contract feature image to obtain the global positioning text and the local handwritten text;

[0076] Step S24: construct a dual-channel Transformer structure according to the text type, extract and process the text in the handwritten font area and the printed font area respectively, and obtain a handwritten sequence and a global sequence.

[0077] Step S3: Call the CTC algorithm sequence modeling to generate the contract feature image after the text is divided into regions, and obtain a formatted contract text sequence.

[0078] The specific operation of step S3 is: calling the CTC algorithm to model the handwritten sequence and the global sequence to obtain a fused sequence, removing repeated characters and blank labels in the fused sequence according to the introduced set blank labels, and generating a clear formatted contract text sequence.

[0079] Step S4: According to the constructed multi-task loss function, the constructed dual-channel Transformer module is trained in segments to obtain the final formatted contract text sequence.

[0080] The step S4 specifically comprises the following steps:

[0081] Step S41: constructing a text localization stage loss function Ldet according to the loss function LCRNN of the Fast R-CNN network and the loss function LRPN of the PRN network;

[0082] Step S42: constructing a CTC loss function LCTC according to the probability after decoding by the CTC algorithm;

[0083] Step S43: constructing a text recognition stage loss function LC-CTC according to the set weight coefficient and CTC loss function LCTC;

[0084] Step S44: Obtain a multi-task loss function L according to the text localization stage loss function Ldet and the text recognition stage loss function LC-CTC;

[0085] Step S45: According to the constructed multi-task loss function, the backbone feature extraction network, the feature pyramid network FPN, the Fast R-CNN network and the PRN network are segmented and trained, and the constructed dual-channel Transformer module is constructed to obtain the final formatted contract text sequence.

[0086] Working principle: This embodiment designs a dual-channel Transformer module based on region division, which can extract features from the printed font area and the handwritten font area in the contract image respectively, which can effectively solve the problem of reduced model recognition accuracy caused by the mixed use of handwritten annotations and printed texts, and improve the overall performance and practicality of the system. The model of this embodiment has the following main purposes:

[0087] 1. Improve recognition accuracy: By dividing the area, different types of text (printed fonts and handwritten fonts) are processed separately to reduce the interference between different fonts, thereby improving the overall text recognition accuracy.

[0088] 2. Optimize processing speed: The dual-channel model can process feature extraction from different regions in parallel, which can effectively improve the processing speed compared to sequential processing.

[0089] 3. Enhance the generalization ability of the model: Process different types of text in a targeted manner to make the model more adaptable and robust when facing a variety of text types.

[0090] In order to solve the problem of mutual interference between handwritten fonts and printed fonts during character recognition of contract images, this embodiment innovatively proposes a Transformer method based on region partitioning and designs a dual-channel Transformer module for extracting features from the printed font area and handwritten font area of ​​the contract image respectively, thereby solving the problem of mutual interference between the two during recognition. At the same time, it improves the accuracy of contract text recognition, enhances the model's adaptability to processing different fonts, and provides an innovative solution for complex text recognition scenarios.

[0091] Embodiment 2:

[0092] This embodiment is based on the above embodiment 1. Figure 1 , Figure 2 , Figure 3 , Figure 4 As shown, a specific embodiment is described in detail.

[0093] Step S1: Image preprocessing.

[0094] Image standardization for contract images is the first step in the processing process, which aims to convert images of different sources and formats into a unified standard format so that subsequent processing steps can be performed more efficiently and accurately. The specific image standardization process includes the following key steps:

[0095] (1) Image grayscale processing: Converting color images into grayscale images can not only reduce the consumption of computing resources, but also the color information is often redundant for text recognition.

[0096] (2) Noise Removal: Bilateral filtering is a nonlinear filtering technique that considers both spatial proximity and pixel value similarity. It is used for image denoising and smoothing while maintaining sharp edges. It reduces image noise while avoiding edge blurring caused by traditional filtering methods such as Gaussian filtering. The core idea of ​​bilateral filtering is that for each pixel, its filtering output depends not only on its spatial neighboring pixels, but also on the similarity of pixel values ​​within the neighborhood.

[0097] When filtering a pixel, not only the pixel values ​​in its neighborhood are considered, but also the difference between these pixels and the current pixel in color space, usually brightness. The value of each pixel is determined by the weighted average of the pixels around it, and the weight is determined by two factors: spatial proximity and pixel similarity. Multiply the spatial domain weight and the intensity domain weight to get the final weight value. Use the final weight value to weighted sum the pixel values ​​in the neighborhood to get the new value of the current pixel. Divide the result of the weighted sum by the sum of all weights to ensure that the sum of the pixel values ​​remains unchanged. Repeat the above steps until all pixels in the image have been processed.

[0098] The output of bilateral filtering is I′ P The value at pixel p can be calculated using the following formula:

[0099]

[0100] Where: I′ P is the intensity value of the original image at pixel p; S is the neighborhood (filter window) centered on p; q is the pixel point in the neighborhood; is a spatial Gaussian weight function that depends on the spatial distance between pixels p and q; is a range (intensity) Gaussian weighting function that depends on the intensity difference between pixels p and q;

[0101] W p is a normalization term that ensures that the sum of the weights is 1 and is defined as:

[0102]

[0103] (3) Image enhancement: The filtering process will remove high-frequency noise in the image, but it may also blur the details of the image and make the image look too smooth or dim. Therefore, it is necessary to use image enhancement technology after filtering to restore some lost high-frequency information and improve the clarity and contrast of the image. This embodiment processes the image from two aspects: increasing the overall contrast of the image and enhancing the edges. First, the overall contrast between the text and the background in the image is enhanced by the histogram equalization method, and then the edge of the text is clarified by using the edge detection technology of the sober operator.

[0104] The Sober operator consists of a horizontal kernel G x and a vertical core G y . G x and G yApplied to each pixel of the image, for each pixel p in the image, its gradient strength can be calculated by the following formula (2). The direction of the gradient can also be calculated by formula (3), which represents the direction of the edge. According to the comprehensive gradient strength calculated by the Sober operator, nonlinear mapping is performed on the pixels with larger gradient strength, such as power law function, to adjust the brightness of the pixel, making the edge area more obvious.

[0105]

[0106] Among them, G x and G y are the gradient values ​​of the point in the horizontal and vertical directions respectively.

[0107]

[0108] Among them, θ can be used to determine the direction of the edge.

[0109] By adjusting the brightness and contrast of the image through the above preprocessing steps, the contrast between the text and the background in the image can be enhanced, making the text more prominent, which can significantly improve the accuracy and efficiency of the OCR system in recognizing the text in the contract image. These steps ensure that the image has better quality and a more suitable format for recognition before entering the OCR recognition stage.

[0110] Step S2: Figure 2 As shown in the figure, ResNet50 is used as the backbone network to extract multi-scale feature information of the input image, the feature pyramid network FPN is introduced to generate a feature map containing multi-level semantic information from coarse to fine, FastR-CNN and PRN are used to locate and classify text, and a dual-channel Transformer structure is used for text recognition.

[0111] (1) In the OCR system, ResNet is used to extract abstract features of the input image. ResNet effectively solves the gradient vanishing problem in deep network training through its residual learning framework, which enables the network to learn deeper feature representations. Especially when processing contract images with rich details and complex structures, ResNet can capture the key information in the image, providing a solid foundation for subsequent text recognition.

[0112] (2) In order to further improve the efficiency and accuracy of feature extraction, the feature pyramid network FPN is introduced, such as Figure 2As shown. The design of FPN allows the network to process features of different scales simultaneously, which is particularly important for the diversity of text sizes in natural images. In contract images, text may appear in different sizes and orientations. FPN, through its multi-scale feature fusion strategy, is able to generate feature maps containing multi-level semantic information from coarse to fine. This strategy not only enhances the model's ability to recognize text of different sizes, but also improves the model's accuracy for text localization and classification tasks. ResNet50 is used as the backbone network, and its architecture with a depth of 50 achieves a good balance between computational efficiency and feature extraction capabilities. Combined with FPN, our system is able to build a powerful feature extraction framework that can not only extract multi-scale features of images, but also fuse features of different resolutions through a top-down architecture, thereby achieving higher accuracy at marginal cost.

[0113] (3)PRN is a network specifically designed to generate high-quality region proposals. It plays a vital role in target detection tasks. Its structure is as follows: Figure 3 As shown in the figure. PRN outputs a series of rectangular candidate boxes that may contain the target object by sliding a window on the feature map and extracting features using a convolutional layer. Compared with traditional region proposal methods, the advantage of PRN is that it can use predefined anchor boxes of multiple scales and aspect ratios, which can cover potential objects of different positions and sizes in the image. For each anchor box, PRN not only predicts the probability of whether the candidate box contains an object, but also predicts the coordinates of adjusting the anchor position to better match the real object. In particular, in view of the scale characteristics of the anchor boxes in the contract text, we adjust PRN to limit the aspect ratio of the recommended boxes it generates to be greater than or equal to 1:1 to meet the specific needs of the contract text. Fast R-CNN is an efficient object detection algorithm that further classifies and regresses the region proposals generated by PRN. Fast R-CNN solves the problems of speed and training complexity of the previous R-CNN series of methods by introducing a region of interest pooling layer that can convert candidate regions of different sizes and ratios into fixed-size feature maps, thereby simplifying the subsequent processing flow. These fixed-size feature maps are then fed into a series of fully connected layers, which ultimately output the object category and precise bounding box coordinates for each region proposal. For the task scenario of contract recognition, by combining PRN and Fast R-CNN, our system can effectively classify and locate handwritten and printed fonts. The located text boxes will be fed into the text recognition module for further text feature extraction.

[0114] (4) In the text recognition stage, a dual-channel Transformer structure was innovatively proposed. Figure 4As shown in the recognition module, the two channels perform specialized processing and comprehensive text analysis on handwritten fonts respectively. Since handwritten fonts have unique styles and variations, such as the continuity and irregularity of strokes, this specialized processing can capture handwritten features more accurately, thereby significantly improving the recognition accuracy and efficiency of handwritten fonts. The processing of all text information includes the processing of printed fonts and handwritten fonts that are not specifically located. The purpose of this channel is to ensure comprehensive coverage of all text information, avoid omissions, and process the handwritten font parts that may not be completely and accurately located by the first channel. After dual-channel processing, the system needs to effectively fuse the processing results of the two channels. The key to this step is how to integrate the information obtained from the two different processing flows to enhance the accuracy and reliability of the final text recognition results. The strategy adopted in this embodiment is that when the results of the two channels conflict in certain areas, priority rules can be set or additional discriminators can be used to determine the final output. The dual-channel Transformer structure is specifically designed for the processing of important documents such as contracts and legal documents, in which handwritten annotations often contain key information. Through this structure, the efficiency and accuracy of processing such documents can be greatly improved.

[0115] Step S3: The CTC algorithm recognizes and obtains a formatted text sequence.

[0116] The task of the fusion and post-processing module is to integrate and optimize the information obtained in the previous steps. This module not only needs to merge the results of the dual-channel Transform structure output, but also perform final error correction and format adjustment. CTC is a deep learning algorithm for sequence modeling. We use CTC to format the text output, such as removing unnecessary spaces and lines, and adjusting the text layout to meet the output requirements. CTC handles the problem of inconsistent lengths between the input sequence and the target sequence by introducing a special blank label (usually represented as "-" or "blank"). Under the framework of CTC, the model outputs a probability distribution sequence with the same length as the input data, and outputs a probability distribution at each time step, indicating the predicted probability of each character. The output of the model is converted into the final text sequence through a decoding process. This decoding process includes removing consecutive repeated characters and blank labels, retaining only single characters, so that a clear text sequence can be obtained from the original output of the model.

[0117] Step S4: Model training.

[0118] Input data: Model input data includes contract images, contract text, and the location and category information of the text. Contract image: This is the direct input of the model. Contract text: Provides the actual text content of each text block for training the model for text recognition. Text location and category: Includes the coordinates of the text box and the category of the text, such as handwritten or printed. The above four types of information are necessary requirements for training the model. The data set used for training and testing the model in this embodiment comes from 1,879 electricity trading contracts and label information of a power sales company of the State Grid. 1,240 of them are used to train the network, and the remaining 639 are used for verification and testing.

[0119] Data processing: The contract image data is uniformly formatted into input dimension size after the above-mentioned "image preprocessing" step. ResNet50 and FPN receive the preprocessed image as input, fuse the feature maps of different levels, and output feature-enhanced multi-scale feature maps. PRN slides the window on each feature map, uses the convolution layer to extract features, and generates candidate regions, while Fast R-CNN classifies these candidate regions and accurately regresses the bounding boxes. The dual-channel Transformer structure extracts and processes features of the handwritten fonts and comprehensive text in the candidate regions respectively. The handwritten font channel focuses on capturing handwritten features, while the comprehensive text channel processes all text information. The fusion module integrates the output results of the two channels and resolves conflicts by setting priorities and using discriminators. The CTC algorithm models the sequence, removes repeated characters and blank labels, and generates a clear text sequence.

[0120] Loss function: The model of this embodiment includes multiple tasks. This embodiment naturally defines a multi-task loss function: As shown in formula (4), it is the loss function L of the text localization stage det .

[0121] L det =L rpn +L RCNN (4)

[0122] Where L rpn and L RCNN is the loss function of PRN and Fast R-CNN.

[0123] CTC Loss means that for a certain feature sequence (f1, f2, ... ft), the probability that the result is equal to the label after CTC decoding is y', then the probability that the CTC decoding result is not the label is (1-y'); it is not difficult to find that the CTC Loss value and y' have the following relationship:

[0124] L CTC = -log(y′) (5)

[0125] Regarding the loss function of the text recognition stage, we use the same loss function C-CTCLoss as Paddle OCR. C-CTC Loss is the abbreviation of CTC Loss + Center Loss. Center Loss comes from the paper "A Discriminative Feature Learning Approach for Deep Face Recognition" and was first used in face recognition tasks to increase the distance between classes and reduce the distance within classes. It is an earlier and more commonly used algorithm in the field of Metric Learning. The final text recognition stage loss function is:

[0126] L C-CTC =L CTC +λ*L Center (6)

[0127]

[0128] where x i is the feature vector of the i-th sample, It is the yth i The center vector of each category, m is the number of samples, and λ is the weight coefficient used to balance the two loss terms. Therefore, the final loss function of this model is:

[0129] L=L det +L C-CTC (8)

[0130] Model training: Phased training is a deep learning model training strategy that breaks down the entire training process into multiple consecutive stages, each of which focuses on optimizing different parts of the model or solving specific subtasks. In the initial stage, the model's feature extraction network is first trained independently to learn to extract useful features from the input data; the intermediate stage is trained for specific tasks such as text localization and text recognition to further refine the model's understanding and application of features; after all subtasks are trained, the learning results of each stage are combined through joint fine-tuning to optimize the parameters of the entire network to improve the overall performance of the model on all tasks. This strategy allows the model to be adjusted and optimized step by step, making it easier to monitor and balance the learning progress of different tasks.

[0131] Working principle: In this embodiment, the dual-channel Transformer module designed to solve the problem of mutual interference between handwritten fonts and printed fonts in contract images uses region division technology to divide contract images into handwritten font areas and printed font areas. Each area is processed through an independent channel to achieve accurate feature extraction and reduce mutual interference between the two fonts during the recognition process. Use object detection to identify the handwritten font area and the printed font area in the contract image. This step is critical because accurate region division directly affects the effect of subsequent feature extraction and recognition accuracy. The global text positioning font and printed font are input into two parallel Transformer structures respectively, and each channel is configured with different network structures and parameters to adapt to the characteristics of different fonts. For example, the handwritten font channel may require a more complex network to handle the irregularity and diversity of the font, while the printed font channel may focus more on accurately capturing the features of the standard font. Each channel uses a Transformer network to extract features of the corresponding area. These networks can enhance the model's ability to understand and extract text features through self-attention mechanisms, especially under complex backgrounds or different lighting conditions.

[0132] Through this dual-channel Transformer method based on region partitioning, the model can more effectively handle font recognition problems in contract images, improving the overall performance and accuracy of the OCR system. This method is particularly suitable for scenarios such as contracts and legal documents, where handwritten annotations and printed text often appear mixed and extremely high recognition accuracy is required. This design not only improves recognition accuracy, but also enhances the model's adaptability to different font processing, providing an innovative solution for complex text recognition scenarios.

[0133] This embodiment can more accurately extract features based on their respective characteristics by clearly dividing the contract image into a handwritten font area and a printed font area and configuring a dedicated processing channel for each area. This method reduces feature interference caused by mixed font types, thereby improving overall recognition accuracy.

[0134] Different types of fonts, such as handwritten and printed fonts, have different visual characteristics and structural complexity. By processing independent channels, the model can more flexibly adjust network parameters and structures to adapt to the characteristics of various fonts and enhance the recognition ability of irregular fonts.

[0135] Partition processing allows the model to process data in different regions in parallel, which may improve the processing speed in practical applications. In addition, targeted feature extraction reduces the interference of irrelevant data and improves the efficiency and quality of feature extraction.

[0136] After feature extraction, different feature fusion strategies can be designed according to actual application requirements. This flexibility allows the model to optimize the recognition results according to specific scenarios while maintaining high recognition accuracy, such as adjusting the sensitivity to specific legal terms in legal document recognition.

[0137] Through the above analysis, we can see that the dual-channel Transformer module based on region division not only improves the recognition accuracy and efficiency when processing complex documents such as contract images, but also enhances the adaptability and robustness of the model, making it an effective tool for handling difficult OCR tasks.

[0138] The other parts of this embodiment are the same as those of the above-mentioned embodiment 1, and thus will not be described in detail.

[0139] Embodiment 3:

[0140] Based on any one of the above-mentioned embodiments 1-2, this embodiment proposes a contract text recognition system based on region partitioning Transformer, which is used to execute the above-mentioned contract text recognition method based on region partitioning Transformer; it includes a preprocessing unit, a dual-channel unit, and an output unit;

[0141] The preprocessing unit is used to preprocess the acquired contract image;

[0142] The dual-channel unit is used to construct a dual-channel Transformer module structure and divide the contract image into regions according to the contract text type extracted from the preprocessed contract image;

[0143] The output unit is used to call the CTC algorithm sequence modeling text segmentation contract feature image to obtain a formatted contract text sequence.

[0144] This embodiment also proposes an electronic device, including a memory and a processor; a computer program is stored on the memory; when the computer program is executed on the processor, the above-mentioned contract text recognition method based on region partitioning Transformer is implemented.

[0145] This embodiment also proposes a computer-readable storage medium, on which computer instructions are stored; when the computer instructions are executed on the above-mentioned electronic device, the above-mentioned contract text recognition method based on region partitioning Transformer is implemented.

[0146] The other parts of this embodiment are the same as any one of the above-mentioned embodiments 1-2, so they will not be repeated here.

[0147] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A contract text recognition method based on region partitioning Transformer, characterized in that: The specific steps include: Step S1: preprocessing the acquired contract image; Step S2: construct a dual-channel Transformer module based on the contract text type extracted from the preprocessed contract image to divide the contract image into regions; Step S3: Calling the CTC algorithm sequence modeling to generate the contract feature image after the text is divided into regions, and obtaining a formatted contract text sequence; Step S4: According to the constructed multi-task loss function, the constructed dual-channel Transformer module is trained in segments to obtain the final formatted contract text sequence.

2. According to claim 1, a method for contract text recognition based on region partitioning Transformer is characterized in that: The step S2 specifically includes the following steps: Step S21: ResNet50 is used as the backbone feature extraction network to extract multi-scale features from the preprocessed contract image; Step S22: introducing a feature pyramid network FPN to generate a contract feature image containing multi-level semantic information from coarse to fine based on multi-scale features; Step S23: calling the Fast R-CNN network and the PRN network to locate and classify the contract feature image, and obtain the global positioning text and the local handwritten text; Step S24: construct a dual-channel Transformer structure according to the text type, extract and process the text in the handwritten font area and the printed font area respectively, and obtain a handwritten sequence and a global sequence.

3. According to claim 2, a method for contract text recognition based on region partitioning Transformer is characterized in that: The specific operation of step S3 is: calling the CTC algorithm to model the handwritten sequence and the global sequence to obtain a fused sequence, removing repeated characters and blank labels in the fused sequence according to the introduced set blank labels, and generating a clear formatted contract text sequence.

4. According to claim 2, a method for contract text recognition based on region partitioning Transformer is characterized in that: The step S4 specifically comprises the following steps: Step S41: According to the loss function L of the Fast R-CNN network CRNN , the loss function L of the PRN network RPN , construct the text localization stage loss function L det ; Step S42: Construct the CTC loss function L according to the probability after decoding by the CTC algorithm CTC ; Step S43: According to the set weight coefficient, CTC loss function L CTC , construct the text recognition stage loss function L C-CTC ; Step S44: Based on the text positioning stage loss function L det , text recognition stage loss function L C-CTC , get the multi-task loss function L; Step S45: According to the constructed multi-task loss function, the backbone feature extraction network, the feature pyramid network FPN, the Fast R-CNN network and the PRN network are segmented and trained, and the constructed dual-channel Transformer module is constructed to obtain the final formatted contract text sequence.

5. According to claim 1, a method for contract text recognition based on region partitioning Transformer is characterized in that: The step S1 specifically includes the following steps: Step S11: converting the acquired contract image into a grayscale contract image; Step S12: calling bilateral filtering to obtain a filtered grayscale contract image; Step S13: calling the sober operator to obtain an enhanced grayscale contract image.

6. According to claim 5, a method for contract text recognition based on region partitioning Transformer is characterized in that: The step S12 specifically includes the following steps: Step S121: Calculate the spatial domain weight according to the spatial distance between the pixel point p and the pixel point q in the grayscale contract image; Step S122: Calculate the intensity domain weight according to the intensity difference between the pixel point p and the pixel point q of the grayscale contract image; Step S123: multiplying the spatial domain weight and the intensity domain weight to obtain a final weight value; Step S124: performing weighted summation on the pixel values ​​in the region according to the final weight value to obtain a grayscale contract image after bilateral filtering.

7. According to claim 5, a method for contract text recognition based on region partitioning Transformer is characterized in that: The step S13 specifically includes the following steps: Step S131: calling a histogram equalization method to enhance the contrast of the filtered grayscale contract image to obtain a contrast-enhanced grayscale contract image; Step S132: calling the edge detection method of the sober operator, applying the horizontal kernel and the vertical kernel to each pixel of the enhanced grayscale contract image to obtain the edge direction; Step S133: Obtaining a grayscale contract image after edge enhancement according to the edge direction.

8. A contract text recognition system based on region partitioning Transformer, used to execute the contract text recognition method based on region partitioning Transformer as claimed in claim 1; characterized in that: It includes a pre-processing unit, a dual-channel unit, and an output unit; The preprocessing unit is used to preprocess the acquired contract image; The dual-channel unit is used to construct a dual-channel Transformer module structure and divide the contract image into regions according to the contract text type extracted from the preprocessed contract image; The output unit is used to call the CTC algorithm sequence modeling text segmentation contract feature image to obtain a formatted contract text sequence.

9. An electronic device, characterized in that: It includes a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, the contract text recognition method based on region partitioning Transformer as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions; when the computer instructions are executed on the electronic device as described in claim 9, the contract text recognition method based on region partitioning Transformer as described in any one of claims 1-7 is implemented.

Images