Contract data identification and extraction method and system

Abstract

The invention provides a contract data identification and extraction method and system. A contract image data set is divided into a training set and a test set after the contract image data set is constructed; performing text line processing on the contract image by adopting rearranged CRNN feature serialization; constructing a character recognition model and a digital recognition model based on an FPN network; features extracted by the character recognition model and the digital recognition model are combined into a feature vector through weighted fusion, and after a CNN-RECR model is constructed, a training set is input into the model for training; and respectively inputting the training set and the test set into the trained model, and performing contract data identification and extraction. According to the method, the feature vector is converted into a multi-dimensional form by applying the residual structure unit so as to obtain the distribution condition of the statement structure, the character recognition model and the digital recognition model are fused and then introduced into the Hash layer to be combined so as to obtain the CNN-RECR model, and the high-dimensional feature is mapped into the low-dimensional binary code, so that the storage requirement can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and system for identifying and extracting contract data. Background Art

[0002] In all walks of life today, there are a large number of contract documents, and contracts often need to be called and reviewed. There are also many inconveniences in archiving and searching. Even if the contract is stored in the form of an image, it still takes a lot of time and manpower to retrieve it. Digitizing the contract makes it easier to back up and retrieve detailed information. In order to ensure that the information in the contract is consistent with the information in the relevant database, and thus to ensure the legality and accuracy of the transaction, accurate and efficient identification and extraction of contract information is an important prerequisite.

[0003] Contract image data recognition and extraction refers to the process of using computer technology to retrieve contract images similar to user queries from contract image databases. Existing technologies usually rely on manually designed feature extraction and traditional similarity measurement methods. Although these methods can effectively extract underlying features such as color and texture, they require domain expertise and may not be able to capture complex image information, thus lacking the ability to describe the deep semantics of images.

[0004] With the rise of deep learning technology in the field of computer vision, especially the successful application of architectures such as convolutional neural networks, contract image data recognition and extraction has made significant progress. Neural networks can learn hierarchical feature representations from contract images, making it possible to extract features from different levels. Compared with traditional methods, deep learning methods can more accurately capture high-level semantic information in images, such as important elements such as contract terms and signature positions. However, when processing the content of contract texts, existing technologies have problems such as large quantity, high complexity, and multi-threading. When recognizing tilted texts, the character features in the extracted feature vectors will be deformed. At the same time, there are problems such as weak feature extraction representation capabilities and difficulty in training deep neural networks. Summary of the invention

[0005] Purpose of the invention: The first purpose of the present invention is to provide a contract data identification and extraction method with strong feature extraction representation capability and convenient data storage. The second purpose is to provide a contract data identification and extraction system.

[0006] Technical solution: A method for identifying and extracting contract data, comprising the following steps:

[0007] (1) Preprocessing the contract images to construct a contract image dataset, and dividing the contract image dataset into a training set and a test set according to a set ratio;

[0008] (2) Using rearranged CRNN feature serialization to perform text line processing on contract images;

[0009] (3) Construct text recognition model and number recognition model based on FPN network;

[0010] (4) Based on the CNN model, the text recognition model and the number recognition model are integrated to construct a CNN-RECR model. The features extracted by the text recognition model and the number recognition model are combined into a feature vector using weighted fusion. The feature vector is input into a fully connected layer. The output dimension of the fully connected layer matches the number of categories of the final classification task. The training set is input into the CNN-RECR model for training.

[0011] (5) The training set and test set are respectively input into the trained CNN-RECR model to perform contract data recognition and extraction.

[0012] Specifically, in step (1), the preprocessing includes: adjusting the size of the contract image to 244×244 pixels, and then performing normalization and Gaussian blurring to remove noise in the contract image.

[0013] Specifically, step (2) includes: serializing the features in CRNN Rearrange to And its value is subjected to N(0,1) standard normal distribution.

[0014] Specifically, in step (3), when constructing the text recognition model, a residual structure unit is added to the FPN network to perform the conversion of the one-dimensional convolution layer and convert the feature vector into a multi-dimensional form.

[0015] Specifically, in step (3), when constructing the digital recognition model, the contract image is mapped to the corresponding digital label.

[0016] Specifically, in step (4), during the training process of the CNN-RECR model, a hash layer is introduced to replace the Softmax classification layer, a hash code is generated in the output, and a symbolic function is used to calculate the binary code:

[0017] b(x)=sgn(h(x))

[0018] Where: b(x) is the binary code, h(x) is the hash code, and sgn is the sign function.

[0019] Specifically, in step (4), the CNN-RECR model is trained using a comprehensive loss function, and the comprehensive loss function is as follows:

[0020] L TOTAL =αL CTC +βL LOC +γLCLASS

[0021] Where: L TOTAL is the comprehensive loss function, L CTC is the CTC loss function, L LOC is the positioning loss function, L CLASS is the classification loss function, and α, β, and γ are the loss weights.

[0022] Preferably, the above method further comprises the following steps:

[0023] (6) Calculate the accuracy of the test set data recognition and extraction results in step (5). Use the F1 score to measure the accuracy of the data recognition and extraction results.

[0024] The present invention also provides a contract data identification and extraction system, comprising:

[0025] Dataset generation module: used to construct a contract image dataset after preprocessing the contract image, and divide the contract image dataset into a training set and a test set according to a set ratio;

[0026] Text line processing module: used to perform text line processing on contract images using rearranged CRNN feature serialization;

[0027] Model generation module: used to build text recognition model and number recognition model based on FPN network;

[0028] Model training module: used to fuse the text recognition model and the number recognition model based on the CNN model to build a CNN-RECR model. The features extracted by the text recognition model and the number recognition model are combined into a feature vector using weighted fusion, and the feature vector is input into a fully connected layer. The output dimension of the fully connected layer matches the number of categories of the final classification task. The training set is input into the CNN-RECR model for training.

[0029] Model testing module: used to input the training set and test set into the trained CNN-RECR model respectively to perform contract data recognition and extraction.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following significant effects:

[0031] 1. Apply the residual structure unit to perform the conversion of the one-dimensional convolution layer and convert the feature vector into a multi-dimensional form to obtain the distribution of the sentence structure. Design the text recognition model and the number recognition model for fusion and introduce the hash layer to obtain the CNN-RECR model. By mapping the high-dimensional features into low-dimensional binary codes, it helps to reduce storage requirements.

[0032] 2. The rearranged CRNN feature serialization is used to perform text line processing on the contract image, so that the position of the character features can be determined when facing rotation and tilt.

[0033] 3. Compared with the various models commonly used in the prior art, the present invention has a higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the method of the present invention.

[0035] Figure 2 This is a schematic diagram of feature serialization of Example 1 of the present invention.

[0036] Figure 3 It is a schematic diagram of a residual structure unit according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0037] A preferred embodiment of the present invention is further described below in conjunction with the accompanying drawings.

[0038] Example 1

[0039] See also Figure 1 As shown, this embodiment provides a contract data identification and extraction method, comprising the following steps:

[0040] (1) After preprocessing the contract images, a contract image dataset is constructed, and the contract image dataset is divided into a training set and a test set according to a set ratio.

[0041] In this embodiment, the preprocessing includes: adjusting the size of the contract image to 244×244 pixels, and then performing a normalization operation and a Gaussian blur method to remove noise in the contract image. The training set and the test set are divided in a 1:1 ratio.

[0042] (2) Rearranged CRNN feature serialization is used to perform text line processing on contract images.

[0043] Please refer to Figure 2 As shown in the figure, when the tilted text is recognized, the character features in the extracted feature vector will be deformed. In the traditional CRNN model, due to the temporal nature of CRNN, the recognition is incomplete, and the feature f is averaged and pooled along the h dimension to obtain the sequence feature. Where c1, h, w are the unified channel number, height, and width of the input image. The present invention serializes the features and arranges them into a new order: And redistribute its values ​​and make it N(0,1) standard normal distribution, so that the position of character features can be accurately identified when facing rotation and tilt.

[0044] (3) Construct text recognition model and number recognition model based on FPN network.

[0045] In this embodiment, an improvement is made based on the FPN network, and a feature extraction layer is added to form a CDTD-Net detection algorithm for extracting and fusing features of different scales. The feature extraction layer can detect and recognize smaller handwritten text.

[0046] When building a text recognition model, add a residual structure unit to perform the conversion of the one-dimensional convolutional layer, convert the feature vector into a multi-dimensional form, and obtain the distribution of the sentence structure. Please refer to Figure 3 As shown in the figure, in the residual structure unit, x is the input, which represents the input feature; F(x) is the convolution layer, which represents the feature vector obtained after the convolution operation on the input; relu is the activation function, which represents a feature map obtained. After adding shortcut connections to the feature map formula, the optimization difficulty will be reduced, and the learning identity map will be transformed into the difference between the feature map and the input data. By introducing residual connections, deep networks can be better trained at a deep level.

[0047] When building a digital recognition model, the contract image is mapped to the corresponding digital label. Specifically, the numbers in the real estate transaction contract are identified and extracted, and the image is mapped to the corresponding digital label to realize the automation of digital recognition, and the image size and length and width are modified and predicted.

[0048] (4) Construct the CNN-RECR model (Real estate transaction contract information detection and recognition method based on improved convolutional neural network, contract data item identification and extraction method based on optimized convolutional neural network). This model is based on the convolutional neural network CNN, and the text recognition model and the number recognition model are fused to construct the CNN-RECR model. Specifically, the features extracted by the text recognition model and the number recognition model are first merged into a feature vector using weighted fusion, and the feature vector is input into a fully connected layer. The output dimension of the fully connected layer matches the number of categories of the final classification task. After the initial convolution layer, the feature map size is (112,112,64), while in the residual block, the feature map size becomes (56,56,128), (28,28,256) and (14,14,512). The feature map P5 is adjusted to 256 channels through 1×1 convolution; then P5 is upsampled by 2 times to obtain M5, and the feature map is connected to P4 for additive fusion. The fused feature map maintains 256 channels. After upsampling and addition operations, all feature maps undergo 3×3 convolution and finally global average pooling. The output feature vector dimension is (1,256), which is suitable for subsequent classification tasks.

[0049] The training set is input into the CNN-RECR model for training. During the training phase, a hash layer is introduced to replace the original Softmax classification layer. A hash code h(x) is generated in the output of the model. The symbolic function is used to calculate the final binary code for fast retrieval and matching, thereby improving processing efficiency:

[0050] b(x)=sgn(h(x))

[0051] Where: b(x) is the binary code, h(x) is the hash code, and sgn is the sign function.

[0052] The CNN-RECR model is trained based on the comprehensive loss function connecting the backbone network and the text recognition head. The comprehensive loss function is defined as follows:

[0053] L TOTAL =αL CTC +βL LOC +γL CLASS

[0054] Where, L CTC is the CTC loss function, L LOC is the positioning loss function, L CLASSis the classification loss function, α, β, and γ are loss weights, which are used to adjust the impact of different loss items on the overall loss.

[0055] (5) The training set and test set are respectively input into the trained CNN-RECR model to perform contract data recognition and extraction.

[0056] (6) Calculate the accuracy of the test set data recognition and extraction results in step (5).

[0057] In this embodiment, the PyTorch framework is used, the hardware for training and testing is AMD Ryzen5 4500UCPU, the number of iterations is set to 3000, the number of batches is 100, the impulse is set to 0.8, the learning rate is set to 0.001, and the SGD optimizer is used for optimization.

[0058] Calculate the F1 score as the evaluation criterion for recognition accuracy.

[0059]

[0060] Where precision is the accuracy rate, which refers to the proportion of samples predicted to be correct among the samples judged as 1 by the classifier; recall is the recall rate / recall rate, which refers to the proportion of samples predicted to be positive examples among the total positive examples.

[0061] In order to prove the actual effect of the model of the present invention, under the same training and testing conditions, three commonly used models in the field of text recognition, YOLOv3-CRNN, PSENet-CRNN and Tesseract, were used for comparative experiments. The results are shown in Table 1 below:

[0062] Table 1

[0063]

[0064] It can be seen from Table 1 that compared with the YOLOv3-CRNN, PSENet-CRNN and Tesseract models, the CNN-RECR model proposed in the present invention has a higher accuracy rate for image text recognition and extraction, while also reducing the computational complexity and storage overhead.

[0065] Example 2

[0066] This embodiment provides a contract data identification and extraction system, including:

[0067] Dataset generation module: used to construct a contract image dataset after preprocessing the contract image, and divide the contract image dataset into a training set and a test set according to a set ratio;

[0068] Text line processing module: used to perform text line processing on contract images using rearranged CRNN feature serialization;

[0069] Model generation module: used to build text recognition model and number recognition model based on FPN network;

[0070] Model training module: used to fuse the text recognition model and the number recognition model based on the CNN model to build a CNN-RECR model. The features extracted by the text recognition model and the number recognition model are combined into a feature vector using weighted fusion, and the feature vector is input into a fully connected layer. The output dimension of the fully connected layer matches the number of categories of the final classification task. The training set is input into the CNN-RECR model for training.

[0071] Model testing module: used to input the training set and test set into the trained CNN-RECR model respectively to perform contract data recognition and extraction.

Claims

1. A contract data identification and extraction method, characterized in that: The following steps are involved: (1) Preprocessing the contract images to construct a contract image dataset, and dividing the contract image dataset into a training set and a test set according to a set ratio; (2) Using rearranged CRNN feature serialization to perform text line processing on contract images; (3) Construct text recognition model and number recognition model based on FPN network; (4) Based on the CNN model, the text recognition model and the number recognition model are integrated to construct a CNN-RECR model. The features extracted by the text recognition model and the number recognition model are combined into a feature vector using weighted fusion. The feature vector is input into a fully connected layer. The output dimension of the fully connected layer matches the number of categories of the final classification task. The training set is input into the CNN-RECR model for training. (5) The training set and test set are respectively input into the trained CNN-RECR model to perform contract data recognition and extraction.

2. The contract data identification and extraction method according to claim 1, characterized in that: In the step (1), the preprocessing includes: adjusting the size of the contract image to 244×244 pixels, and then performing normalization and Gaussian blurring to remove noise in the contract image.

3. The contract data identification and extraction method according to claim 1, characterized in that: The step (2) includes: serializing the features in the CRNN Rearrange to And its values ​​are subjected to standard normal distribution.

4. The contract data identification and extraction method according to claim 1, characterized in that: In the step (3), when constructing the text recognition model, a residual structure unit is added to the FPN network to perform the conversion of the one-dimensional convolution layer and convert the feature vector into a multi-dimensional form.

5. The contract data identification and extraction method according to claim 1, characterized in that: In the step (3), when constructing the digital recognition model, the contract image is mapped to the corresponding digital label.

6. The contract data identification and extraction method according to claim 1, characterized in that: In step (4), during the training process of the CNN-RECR model, a hash layer is introduced to replace the Softmax classification layer, a hash code is generated in the output, and a symbolic function is used to calculate the binary code: b(x)=sgn(h(x)) Where: b(x) is the binary code, h(x) is the hash code, and sgn is the sign function.

7. The contract data identification and extraction method according to claim 1, characterized in that: In step (4), the CNN-RECR model is trained using a comprehensive loss function, and the comprehensive loss function is as follows: L TOTAL =αL CTC +βL LOC +γL CLASS Where: L TOTAL is the comprehensive loss function, L CTC is the CTC loss function, L LOC is the positioning loss function, L CLASS is the classification loss function, and α, β, and γ are the loss weights.

8. The contract data identification and extraction method according to claim 1, characterized in that: The following steps are also included: (6) Calculate the accuracy of the test set data recognition and extraction results in step (5).

9. The contract data identification and extraction method according to claim 8, characterized in that: In step (6), the F1 score is used to measure the accuracy of the data recognition and extraction results.

10. A contract data identification and extraction system, characterized in that: include: Dataset generation module: used to construct a contract image dataset after preprocessing the contract image, and divide the contract image dataset into a training set and a test set according to a set ratio; Text line processing module: used to perform text line processing on contract images using rearranged CRNN feature serialization; Model generation module: used to build text recognition model and number recognition model based on FPN network; Model training module: used to fuse the text recognition model and the number recognition model based on the CNN model to build a CNN-RECR model. The features extracted by the text recognition model and the number recognition model are combined into a feature vector using weighted fusion, and the feature vector is input into a fully connected layer. The output dimension of the fully connected layer matches the number of categories of the final classification task. The training set is input into the CNN-RECR model for training. Model testing module: used to input the training set and test set into the trained CNN-RECR model respectively to perform contract data recognition and extraction.

Images