OCR-Based Bill Information Recognition Method, Device, Equipment and Storage Medium

Abstract

The present application relates to the technical field of character recognition, and discloses a method, device, equipment and storage medium for recognizing bill information based on OCR. The method includes: obtaining a plurality of initial sample bill images and performing information annotation to obtain a plurality of target sample bill images; performing character recognition training through an initial OCR model to obtain a plurality of sample bill text data and a target OCR model; performing bill classification training through a plurality of LayoutLMv3 models to obtain a bill information classification model; obtaining a first bill image to be recognized and performing preprocessing to obtain a second bill image to be recognized; inputting the second bill image to be recognized into the target OCR model for character recognition to obtain target bill text data; inputting the target bill text data into the bill information classification model for bill information classification to obtain a bill information classification result. The present application adopts OCR technology and LayoutLMv3 model, which improves the accuracy of bill information recognition and classification.

Description

Technical Field

[0001] This application relates to the field of character recognition technology, and in particular to an OCR-based bill information recognition method, device, equipment and storage medium. Background Art

[0002] Traditional OCR technology only ensures that the characters in the picture can be highly restored to text during the recognition of document pdf or pictures, and does not guarantee the meaning of the text and the meaning of each line and each table. However, during the process of document recognition, not only the coordinates of the characters in the picture need to be recognized, but also the business meaning represented by the characters. Therefore, it is required that the document model needs to label coordinate information and business meaning information during the training and annotation process.

[0003] The deficiencies of the prior art include: it can only recognize characters, without format and tables, and the recognized character information cannot be used for business consumption; traditional OCR can only recognize characters and cannot understand field business information. Summary of the Invention

[0004] This application provides an OCR-based bill information recognition method, device, equipment and storage medium, and further adopts OCR technology and LayoutLMv3 model to improve the accuracy of bill information recognition and classification.

[0005] In the first aspect of this application, an OCR-based bill information recognition method is provided. The OCR-based bill information recognition method includes:

[0006] Obtain multiple initial sample bill images, and respectively perform field information splitting and annotation on the multiple initial sample bill images through the label-studio data annotation tool to obtain multiple target sample bill images;

[0007] Input the multiple target sample bill images into a preset initial OCR model for character recognition training to obtain multiple sample bill text data and a target OCR model;

[0008] Input the multiple sample bill text data into multiple preset LayoutLMv3 models for bill classification training to obtain a bill information classification model;

[0009] Obtain a first bill image to be recognized, and preprocess the first bill image to be recognized to obtain a second bill image to be recognized;

[0010] Input the second bill image to be recognized into the target OCR model for character recognition to obtain target bill text data;

[0011] Input the target bill text data into the bill information classification model for bill information classification to obtain a bill information classification result.

[0012] The second aspect of this application provides an OCR-based bill information recognition device, and the OCR-based bill information recognition device includes:

[0013] An acquisition module, configured to acquire a plurality of initial sample bill images, and respectively perform field information splitting and annotation on the plurality of initial sample bill images through a label-studio data annotation tool to obtain a plurality of target sample bill images;

[0014] An identification training module, configured to input the plurality of target sample bill images into a preset initial OCR model for character recognition training to obtain a plurality of sample bill text data and a target OCR model;

[0015] A classification training module, configured to input the plurality of sample bill text data into a plurality of preset LayoutLMv3 models for bill classification training to obtain a bill information classification model;

[0016] A preprocessing module, configured to acquire a first bill image to be recognized, and preprocess the first bill image to be recognized to obtain a second bill image to be recognized;

[0017] A character recognition module, configured to input the second bill image to be recognized into the target OCR model for character recognition to obtain target bill text data;

[0018] An information classification module, configured to input the target bill text data into the bill information classification model for bill information classification to obtain a bill information classification result.

[0019] The third aspect of this application provides a computer device, including: a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory to enable the computer device to execute the above-mentioned OCR-based bill information recognition method.

[0020] The fourth aspect of this application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when it runs on a computer, it enables the computer to execute the above-mentioned OCR-based bill information recognition method.

[0021] In the technical solution provided by this application, the bill images are acquired and processed in an automated manner, greatly reducing the demand for human resources and the possibility of human errors, which is particularly important in high-load and high-frequency bill processing scenarios. By combining an advanced OCR model and a LayoutLMv3 model, not only can the text in the image be recognized, but also the specific business meaning and context relationship of the text in the bill can be understood. This multi-modal recognition method can accurately classify and analyze complex bill layouts, such as tables, headings, and paragraphs, thereby improving the accuracy and reliability of data extraction. With multiple datasets and multi-model fusion strategies, this not only enhances the generalization ability of the model but also ensures high recognition accuracy on bill images of different types and qualities. Brief Description of the Drawings

[0022] Figure 1 It is a schematic diagram of an embodiment of the OCR-based bill information recognition method in an embodiment of this application;

[0023] Figure 2 It is a schematic diagram of an embodiment of the OCR-based bill information recognition device in an embodiment of this application. Detailed Embodiments

[0024] The embodiments of this application provide an OCR-based bill information recognition method, device, equipment, and storage medium, and further adopt OCR technology and LayoutLMv3 model to improve the accuracy of bill information recognition and classification.

[0025] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of this application and the above drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order different from that shown or described here. In addition, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or equipment that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or equipment.

[0026] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to Figure 1 An embodiment of the OCR-based bill information recognition method in an embodiment of this application includes:

[0027] Step 101: Obtain multiple initial sample bill images, and use the label-studio data annotation tool to perform field information splitting and annotation on the multiple initial sample bill images respectively, so as to obtain multiple target sample bill images;

[0028] It can be understood that the execution subject of this application can be an OCR-based bill information recognition device, or a terminal or a server. Specifically, it is not limited here. In this embodiment of the application, the server is taken as the execution subject for illustration.

[0029] Specifically, obtain multiple initial sample bill images, which can come from different sources, including scanned paper bills, screenshots of electronic invoices, etc. To ensure the comprehensiveness and diversity of model training, divide the initial sample bill images into a fusion training set, a statistical data set, and a fine-tuning data set according to a preset ratio. The fusion training set is used for the preliminary training of the model, the statistical data set is used to evaluate the model performance, and the fine-tuning data set is used to optimize and adjust the model in the later stage of training. Call multiple target models in the label-studio data annotation tool. Label-studio is a flexible data annotation platform that supports various types of data annotation tasks. Input the fusion training set into the target model, and each target model predicts the data set labels for the input image to generate an initial label prediction result. Since different models may produce different label prediction results when processing the same data set, in order to improve the accuracy of annotation, fuse the initial label prediction results. The fusion strategy can include a simple voting mechanism, weighted average, or a confidence-based comprehensive method. Through these methods, an annotation model with a target fusion strategy is obtained. Perform field information splitting and annotation through the annotation model with the target fusion strategy, identify and locate the key information (such as date, amount, invoice number, etc.) in the bill image, and output it in a structured form. During this process, apply a preset fuzzy supervised learning strategy. The fuzzy supervised learning strategy can alleviate the problems of data noise and annotation deviation to a certain extent by introducing fuzzy logic and uncertainty processing mechanisms, thereby improving the accuracy and robustness of annotation. The fuzzy supervised learning strategy can process information with unclear boundaries or difficult to clearly classify during the annotation process, and perform reasoning and decision-making through fuzzy sets and fuzzy rules to generate more reliable annotation results. Finally, obtain multiple target sample bill images.

[0030] Obtain multiple preset candidate fusion strategies, which include weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy. Respectively, perform fusion processing on the initial label prediction results corresponding to each target model according to the weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy to obtain the target label fusion results corresponding to each candidate fusion strategy. The weighted average fusion strategy obtains a weighted average label result through weighted calculation of the prediction results of each model; the linear regression fusion strategy trains a linear regression model and performs a linear combination according to the prediction results of each model to obtain a regression fusion label result; the multi-layer perceptron fusion strategy trains a deep neural network, inputs the prediction results of each model into the network, and obtains a deep learning fusion label result after multiple non-linear transformations. According to the target label fusion results corresponding to each candidate fusion strategy, perform fusion strategy training on multiple candidate fusion strategies. By using the labeled data set, train the weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy respectively, and adjust their parameters to optimize the fusion effect of each strategy to obtain multiple trained fusion strategies. Select the target fusion strategy from multiple trained fusion strategies. The selection of the target fusion strategy can be based on multiple metrics, such as the accuracy of the fusion effect, the complexity of the model, and the computational efficiency. After selecting the target fusion strategy, calculate the confidence interval of the strategy to evaluate the credibility and stability of its fusion results. The calculation of the confidence interval can be estimated through statistical methods, such as cross-validation, to estimate the performance of the target fusion strategy on different data sets, so as to ensure its high reliability. Finally, according to multiple target models and the selected target fusion strategy, generate an annotation model with the target fusion strategy. This annotation model can more accurately annotate the field information of the bill image by fusing the prediction results of multiple target models and combining the best fusion strategy. Among them, by using an open-source annotation platform (label-studio), a powerful open-source data annotation tool that supports multiple media formats, such as pictures, audio, and video. According to the type of document, mark different field positions and table positions in different types to ensure that different styles under each type are fully annotated. Different business scenarios require different samples to be annotated, and the coordinate positions and business meanings of the samples are not completely systematic, so a large number of samples are required during the annotation process. For complex regions, they need to be split for annotation and finally presented to the UI side in the form of algorithmic merging in the application system.

[0031] Step 102: Input multiple target sample bill images into a preset initial OCR model for character recognition training to obtain multiple sample bill text data and a target OCR model;

[0032] Specifically, create a set of bill keywords, including multiple preset keywords, which are extracted from important fields commonly found in various bills, such as date, amount, invoice number, etc. Preprocess multiple target sample bill images. Grayscale these images. Grayscaling is the conversion of a color image into a grayscale image, making the image contain only brightness information, thus simplifying subsequent processing steps. Perform binarization on the grayscale images, converting the pixel values in the images into images with only black and white values, making the text and background in the images more distinct and facilitating subsequent character recognition. Remove noise and correct the angle for multiple binarized sample bill images. Noise removal refers to removing the miscellaneous points and interference information in the images to improve the clarity of the images; angle correction is to rotate and adjust the images to make them present a standard horizontal or vertical state, thus ensuring the correct recognition of the text. After the processing steps, multiple standardized sample bill images are obtained. Perform line segmentation and character segmentation on the standardized sample bill images. Line segmentation means separating each line of text in the image, while character segmentation further separates each character in each line individually. Through line segmentation and character segmentation, the text information in the images is processed more precisely. Perform text recognition on multiple segmented sample bill images, and identify the text information in the images through OCR technology to obtain the initial bill text data. According to the initial bill text data and in combination with the preset set of bill keywords, determine the bill keywords for each target sample bill image to obtain multiple sample bill text data. The determination of keywords can be achieved through matching and extraction methods, that is, searching for and matching the preset keywords in the recognized text information and extracting these keywords as the key information of the bill. Train the initial OCR model with multiple sample bill text data. During the training process, continuously adjust the model parameters and optimize the algorithm to enable the model to more accurately recognize and extract the text information in the bills. After multiple rounds of training and verification, the target OCR model is finally obtained.

[0033] Preprocessing the initial bill text data is to standardize the initial bill text data to obtain standard bill text data. This includes removing noise characters in the text, unifying character formats, and processing special characters, etc. Calculate the keyword similarity between the standard bill text data and the bill keyword set. Use a text similarity algorithm to calculate the similarity between each keyword in the standard bill text data and the keywords in the preset keyword set. There are multiple methods for similarity calculation, including cosine similarity, Jaccard similarity, or deep learning-based text embedding methods (such as BERT), etc. These methods can effectively measure the similarity between two text segments, thereby identifying potential keywords in the standard bill text data. Analyze the calculated target similarity corresponding to each keyword, and compare these similarities with the preset similarity threshold. When the target similarity is greater than the preset similarity threshold, it indicates that the keyword has a high matching degree and credibility in the standard bill text data, so it is used as the bill keyword of the target sample bill image. Through the method of similarity comparison and threshold judgment, important keywords in the standard bill text data are effectively screened out, ensuring that the extracted keywords have high accuracy and relevance. Integrate the extracted bill keywords to obtain multiple sample bill text data. Remove duplicate and redundant keywords, and merge and unify similar or related keywords to generate a more concise and accurate bill keyword set. During the integration process, adopt multiple strategies, such as a merging strategy based on word meaning, a screening strategy based on word frequency, etc., to ensure that the integrated keyword set has high representativeness and effectiveness.

[0034] Step 103: Input multiple sample bill text data into multiple preset LayoutLMv3 models for bill classification training to obtain a bill information classification model;

[0035] It should be noted that multiple pre-set LayoutLMv3 models are obtained. The models include an encoder, a multi-modal Transformer network, a decoder, and a policy decision network. The multiple sample bill text data are matched with the multiple LayoutLMv3 models to find the most suitable LayoutLMv3 model for each sample bill text data, ensuring that each sample data can be optimally processed and classified. Each sample bill text data is respectively input into the corresponding LayoutLMv3 model, and the vector transformation of each sample bill text data is performed through the first embedding layer in the encoder to convert the text data into the corresponding first sample encoding vector. The position encoding of the first sample encoding vector is performed through the first position encoding layer in the encoder to obtain the second sample encoding vector. The purpose of position encoding is to introduce position information so that the model can understand the relative relationship between words or characters at different positions in the text. The multi-layer feature processing of the second sample encoding vector is performed through the first multi-layer convolutional layer in the encoder to obtain the sample convolutional encoding feature vector. The multi-layer convolutional processing can extract the deep features of the text data and provide rich feature information for the subsequent attention mechanism. The sample convolutional encoding feature vector is input into the multi-modal Transformer network for attention feature enhancement to obtain the sample attention feature vector. The multi-modal Transformer network can perform weighted processing on the important parts in the text data through the attention mechanism, highlight the key features, ignore the secondary information, and improve the accuracy of classification. The multi-layer convolutional processing of the sample attention feature vector is performed through the second multi-layer convolutional layer in the decoder to obtain the sample convolutional feature decoding vector. The multi-layer convolutional processing can further optimize the feature vector and extract more representative features. The position decoding of the sample convolutional feature decoding vector is performed through the second position encoding layer in the decoder to obtain the first sample decoding vector. The purpose of position decoding is to restore the position information in the encoding process so that the model can accurately restore the position information of the original text. The vector transformation of the first sample decoding vector is performed through the second embedding layer in the decoder to obtain the corresponding second sample decoding vector. The decoded feature vector is converted into a format suitable for classification for subsequent classification processing. The bill classification of the second sample decoding vector is performed through multiple decision trees in the policy decision network to obtain the sample bill classification information of each LayoutLMv3 model. The policy decision network uses the decision tree algorithm to make classification decisions according to the different features of the feature vector to achieve accurate classification of the bill. To evaluate and optimize the classification results, the loss calculation of the sample bill classification information is performed through the pre-set cross-entropy function to obtain the loss value of each LayoutLMv3 model. The cross-entropy loss function can measure the gap between the prediction result and the actual result and provide a basis for model optimization.According to the calculated loss values, gradient descent updates are performed on each LayoutLMv3 model respectively to obtain multiple first bill classification models. Gradient descent update is to continuously adjust the model parameters through an optimization algorithm to minimize the loss function, thereby improving the classification accuracy of the model. To further enhance the performance and generalization ability of the model, reinforcement learning training and model integration are carried out on multiple first bill classification models. Reinforcement learning optimizes the decision-making ability of the model through a reward mechanism, while model integration generates a more robust and accurate bill information classification model by combining the prediction results of multiple models. In this embodiment, a highly efficient and accurate bill information classification model is finally obtained. This model can not only accurately identify and classify the text information in the bill, but also improve the processing ability of complex bill text data through multi-modal processing and deep learning algorithms, thereby achieving efficient and accurate identification and classification of bill information.

[0036] Perform network decision execution strategy analysis on the policy decision networks in multiple first bill classification models respectively. By analyzing the execution process and decision path of each policy decision network, obtain the initial network decision execution strategy of each network. The initial strategy reflects the decision-making behavior and strategy selection of the model in the current state. Conduct execution strategy learning on the policy decision networks. By analyzing and summarizing existing decision-making strategies, construct a target execution strategy search space. The search space covers all possible strategy combinations and execution paths, providing a wide range of choices for optimizing strategies. Based on the target execution strategy search space, calculate the reward feedback data for the initial network decision execution strategy. By introducing a reward mechanism, evaluate and feedback the execution results of each strategy to obtain the reward feedback parameters. The reward feedback parameters reflect the execution effects and advantages and disadvantages of each strategy in different situations, providing a quantitative basis for the update and optimization of strategies. Update the strategies of the policy decision networks in each first bill classification model based on the reward feedback parameters, and generate the corresponding target network decision execution strategies for the policy decision networks in each first bill classification model. Strategy update is an iterative optimization process. By continuously adjusting and optimizing strategy parameters, the policy decision network can make better decisions in different situations and improve the overall performance of the model. After the strategy update is completed, optimize the decision-making strategies of multiple first bill classification models according to the target network decision execution strategies to obtain multiple second bill classification models. By integrating and adjusting the decision-making strategies of each model, the overall model can be more efficient and accurate in processing bill classification tasks. Calculate the model weight data of each second bill classification model respectively. The model weight data reflects the performance and contributions of each model on different tasks and datasets. By calculating the weight data, measure and compare the relative importance and advantages and disadvantages of each model. According to the model weight data, perform model integration on multiple second bill classification models to generate the final bill information classification model. Model integration is to comprehensively and fuse the prediction results of multiple models through weighted average, voting mechanism or other integration methods, so as to improve the generalization ability and prediction accuracy of the overall model. In this way, not only can the advantages and specialties of each model be fully utilized, but also the errors and biases of a single model can be reduced through the integration strategy, improving the overall performance.

[0037] Step 104: Obtain the first to-be-recognized bill image, and preprocess the first to-be-recognized bill image to obtain the second to-be-recognized bill image;

[0038] Specifically, obtain the first bill image to be recognized, which may include scanned copies or photos of various bills. Since these original images may have various noises, distortions, and quality problems, perform grayscale processing on the images. Grayscale processing is to convert a color image into a grayscale image, so that each pixel only contains brightness information without color information, simplifying subsequent processing and reducing computational complexity. Perform binarization processing to convert the grayscale image into an image with only two pixel values of black and white. By setting a threshold, pixels with brightness higher than the threshold are converted to white, and pixels lower than the threshold are converted to black. This helps to highlight the text information in the image, making the contrast between the text and the background more obvious and facilitating subsequent character recognition. Perform noise removal. Use image processing techniques such as median filtering and mean filtering to remove random noises and miscellaneous points in the image and smooth the image. Perform geometric correction of the image. Due to possible tilting and distortion during scanning or shooting, the text and structure in the image may be deformed, affecting the recognition effect. Through image rotation and geometric transformation, the image is restored to a standard horizontal or vertical state. Use the Hough transform to detect straight lines in the image, calculate the tilting angle, and perform rotation correction. At the same time, use morphological processing techniques such as dilation, erosion, opening operation, and closing operation to perform morphological correction on the image, remove small interfering elements, and enhance the main structure in the image. After completing the geometric correction, perform line segmentation and character segmentation on the image. Line segmentation is to split each line of text in the image so that each line of text becomes an independent image area; character segmentation is to split each character in each line of text separately, providing a finer-grained processing unit for subsequent character recognition. Line segmentation and character segmentation can use techniques such as projection analysis method and connected component analysis method. By analyzing the pixel distribution and connected areas in the image, accurately locate and segment the text lines and characters. Perform edge detection and contour extraction on the preprocessed image to enhance the text contour in the image and make it clearer. Edge detection can use classic algorithms such as the Canny operator and the Sobel operator to detect the edge pixels in the image; contour extraction is to track and extract the contours of the text and image areas based on the edge detection results. After these preprocessing steps, the original first bill image to be recognized is converted into a second bill image to be recognized with higher quality and more prominent features.

[0039] Step 105: Input the second bill image to be recognized into the target OCR model for text recognition to obtain the target bill text data;

[0040] Specifically, the second to-be-recognized bill image is input into the target OCR model for text recognition. The target OCR model is a data-driven model trained with a large amount of data, such as an OCR model based on convolutional neural networks and recurrent neural networks. Before inputting the image into the OCR model, it is necessary to ensure that the input format of the model is consistent with the image data format. This usually includes adjusting the size, resolution, and color channels of the image, etc. For example, some OCR models may require input images of a fixed size, or grayscale images instead of color images. The second to-be-recognized bill image is adjusted and standardized accordingly to match the input requirements of the model. The working principle of the OCR model usually includes several main stages: feature extraction, character detection, character classification, and text generation. In the feature extraction stage, the model extracts basic features in the image through convolutional layers, such as edges, corners, textures, etc. These features are the basis for character detection and classification. In the character detection stage, the model identifies the regions in the image that may contain characters. By locating these regions, candidate regions are provided for subsequent character classification. Character classification is the core function of the OCR model. The model converts the pixel information in the detected character regions into specific character categories through neural networks or other classification algorithms. For example, a deep learning model classifies each character region through multiple layers of neural networks and outputs the corresponding character labels. After character classification, the model combines all the recognized characters to generate complete text data. Such as through character merging, misspelling correction, format adjustment, etc., to ensure that the generated text data is consistent with the text in the original bill. For example, the model may merge characters with segmentation errors, or correct the spelling and grammar of the recognized text through a language model. The target bill text data is further processed to meet specific application requirements. For example, for key information such as the amount, date, invoice number, etc. in the bill, they are extracted through regular expressions or specific rules and subjected to structured processing. This not only helps to improve the readability and usability of the data, but also facilitates subsequent data analysis and processing. To improve the accuracy of OCR recognition, multiple technologies and strategies are combined. For example, more training data is generated through data augmentation techniques to enhance the generalization ability of the model; through ensemble learning methods, the recognition results of multiple models are combined, and the optimal output is selected; or through post-processing steps, such as context-based misspelling correction and semantic analysis, to further optimize the recognition results. By continuously monitoring and evaluating the performance of the OCR model and constantly adjusting and optimizing the model parameters, better recognition effects can be obtained in different application scenarios.

[0041] Step 106: Input the target bill text data into the bill information classification model for bill information classification to obtain the bill information classification result.

[0042] Specifically, preprocess the target bill text data, including removing irrelevant characters, normalizing the text format, handling missing values, and unifying the encoding format, etc. Extract features from the preprocessed bill text data to convert the text data into a numerical representation that can be understood and processed by machine learning models. Common feature extraction methods include the Bag-of-Words model, TF-IDF (Term Frequency-Inverse Document Frequency), and word vector models (such as Word2Vec, GloVe, BERT, etc.). By converting the words in the text into feature vectors, the text data can be used as the input of the classification model. For example, the TF-IDF method converts the text into a vector representation by calculating the frequency of words in the document and the inverse frequency in all documents, thereby highlighting the weights of important words. Pre-trained language models such as BERT can capture the context semantic information in the text through deep learning methods and generate richer feature representations. Input the extracted feature vectors into the preset bill information classification model for classification to obtain the bill information classification result. In the classification model, the model calculates based on the input feature vectors and outputs one or more class labels. For example, for invoice data, the classification model may output multiple classification results such as invoice type, date, amount, etc.

[0043] In the embodiments of this application, by automatically acquiring and processing bill images, the demand for human resources and the possibility of human errors are significantly reduced, which is particularly important in high-load and high-frequency bill processing scenarios. By combining an advanced OCR model and the LayoutLMv3 model, not only can the text in the image be recognized, but also the specific business meaning and context relationship of the text in the bill can be understood. This multi-modal recognition method can accurately classify and analyze complex bill layouts, such as tables, headings, and paragraphs, etc., thereby improving the accuracy and reliability of data extraction. With multiple datasets and multi-model fusion strategies, this not only enhances the generalization ability of the model but also ensures high recognition accuracy on bill images of different types and qualities.

[0044] In a specific embodiment, the process of executing step 101 may specifically include the following steps:

[0045] (1) Obtain multiple initial sample bill images and divide the multiple initial sample bill images into datasets according to a preset ratio to obtain a fusion training set, a statistical dataset, and a fine-tuning dataset;

[0046] (2) Invoke multiple target models in the label-studio data annotation tool and input the fusion training set into the multiple target models;

[0047] (3) Respectively perform dataset label prediction on the fused training set through multiple target models to obtain the initial label prediction results corresponding to each target model;

[0048] (4) Fuse the initial label prediction results corresponding to each target model to obtain an annotation model with a target fusion strategy;

[0049] (5) Perform field information splitting and annotation through the annotation model with a target fusion strategy, and obtain multiple target sample bill images according to the preset fuzzy supervised learning strategy.

[0050] Specifically, obtain multiple initial sample bill images, which can be collected from various sources, such as scanned paper bills, screenshots of electronic invoices, etc. The samples should cover different types and formats of bills to ensure that the model can generalize to various actual application scenarios. For example, assume there are 1000 sample bill images, and divide these images into a fusion training set, a statistical data set, and a fine-tuning data set according to a preset ratio. A common division ratio can be 70% for training, 20% for statistics, and 10% for fine-tuning. Call multiple target models in the label-studio data annotation tool. Label-studio is a flexible open-source data annotation tool that supports multiple annotation tasks and model integration. Pre-configure multiple target models in label-studio, and each model can be a different machine learning or deep learning model, and these models have their own advantages in different annotation tasks. For example, configure a rule-based model to identify fixed-format fields in bills, such as dates and amounts; at the same time, configure a deep learning model to identify text fields in complex layouts. Input the fusion training set into these target models, and each model predicts the data set labels for the images in the training set. After each model completes the label prediction, obtain the initial label prediction results corresponding to each target model. For example, assume a rule-based model predicts the position of the date field in the image, and a deep learning model predicts the position of the text field, and fuse these prediction results. The fusion strategy can adopt weighted average, voting mechanism or other integration methods to integrate the prediction results of multiple models to improve the accuracy and robustness of the prediction. By fusing the initial label prediction results corresponding to each target model, obtain an annotation model with a target fusion strategy. Use this model to split and annotate the field information. The annotation model locates and annotates different fields in each bill image according to the fused prediction results. For example, assume a bill contains multiple fields such as date, amount, invoice number, etc., and the annotation model will accurately label the positions and contents of these fields. To further improve the accuracy of the annotation, introduce a preset fuzzy supervised learning strategy. The fuzzy supervised learning strategy improves the performance of the model when dealing with real-world data by handling the uncertainty and ambiguity in the data. This strategy uses fuzzy logic and fuzzy set theory to optimize the annotation results, so that the model can still maintain a high accuracy when facing fuzzy or uncertain annotation data. In this embodiment, finally obtain multiple target sample bill images, and these images have been subject to high-quality splitting and annotation of field information.

[0051] In a specific embodiment, the process of performing steps to fuse the initial label prediction results corresponding to each target model to obtain an annotation model with a target fusion strategy may specifically include the following steps:

[0052] (1) Obtain multiple preset candidate fusion strategies, where the multiple candidate fusion strategies include: weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy;

[0053] (2) Respectively perform fusion processing on the initial label prediction results corresponding to each target model according to the weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy to obtain the target label fusion results corresponding to each candidate fusion strategy;

[0054] (3) According to the target label fusion results corresponding to each candidate fusion strategy, perform fusion strategy training on the multiple candidate fusion strategies to obtain multiple trained fusion strategies;

[0055] (4) Select a target fusion strategy from the multiple trained fusion strategies, calculate the confidence interval of the target fusion strategy, and generate an annotation model with the target fusion strategy according to the multiple target models.

[0056] Specifically, obtain multiple preset candidate fusion strategies, which include weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy. The weighted average fusion strategy assigns different weights to the prediction results of each target model, and then sums the weighted results to obtain the final fusion result. The linear regression fusion strategy constructs a linear regression model, takes the prediction results of each target model as input, and the actual label as output, and determines the optimal weights of each input through training. The multi-layer perceptron fusion strategy is a neural network-based method that constructs a multi-layer perceptron (MLP) to perform non-linear combination on the prediction results of each target model, thereby achieving more complex fusion. Respectively perform fusion processing on the initial label prediction results corresponding to each target model according to the weighted average fusion strategy, linear regression fusion strategy, and multi-layer perceptron fusion strategy. For example, assume there are three target models A, B, and C, which respectively perform label prediction on the same sample, and the obtained prediction results are . For the weighted average fusion strategy, assign a weight to each model, and then calculate the weighted average . For the linear regression fusion strategy, construct a linear regression model, take as input, and determine the optimal weight parameters through training data to obtain the fusion result . For the multi-layer perceptron fusion strategy, construct a multi-layer neural network, take as input, and through multi-layer non-linear transformation, obtain the final fusion result 。Perform fusion strategy training on candidate fusion strategies. Optimize the parameters of each fusion strategy through actual data so that it can exhibit high accuracy and robustness in different situations. For example, for the weighted average fusion strategy, adjust the weights of each model through cross-validation to achieve the best performance on the validation set. For the linear regression fusion strategy, determine the optimal regression coefficients by minimizing the mean square error. For the multi-layer perceptron fusion strategy, optimize the parameters of the neural network through the backpropagation algorithm and gradient descent to maximize the classification accuracy on the training set. After the fusion strategy training, multiple trained fusion strategies are obtained. Select the target fusion strategy from them and calculate the confidence interval of the target fusion strategy. The selection of the target fusion strategy can be based on multiple metrics, such as accuracy, precision, recall, F1 value, etc. The calculation of the confidence interval is to evaluate the stability and reliability of the target fusion strategy on different data sets. For example, through the bootstrap method or cross-validation method, calculate the performance fluctuation range of the target fusion strategy in multiple repeated experiments to determine its confidence interval. According to the selected target fusion strategy and confidence interval, generate an annotation model with the target fusion strategy. Apply the target fusion strategy to the actual annotation task, fuse the prediction results of each target model, and obtain the final annotation result.

[0057] In a specific embodiment, the process of executing step 102 may specifically include the following steps:

[0058] (1) Create a bill keyword set, where the bill keyword set includes multiple preset keywords;

[0059] (2) Perform grayscale and binarization processing on multiple target sample bill images to obtain multiple binarized sample bill images;

[0060] (3) Remove noise and correct the angles of multiple binarized sample bill images to obtain multiple standardized sample bill images;

[0061] (4) Perform row segmentation and character segmentation on multiple standardized sample bill images to obtain multiple segmented sample bill images, and perform character recognition on multiple segmented sample bill images to obtain initial bill text data;

[0062] (5) Determine the bill keywords of each target sample bill image according to the initial bill text data to obtain multiple sample bill text data;

[0063] (6) Train the initial OCR model with multiple sample bill text data to obtain the target OCR model.

[0064] Specifically, create a set of invoice keywords, define the keyword fields that need to be recognized in the invoice, including date, amount, invoice number, customer name, etc. Perform grayscale processing on multiple target sample invoice images to convert the color images into grayscale images. Grayscale processing can simplify the complexity of the images, retain the brightness information, and facilitate subsequent image processing steps. Perform binarization processing on the grayscale images to convert the pixel values in the images into two values of black and white, highlighting the text part in the images, making the contrast between the text and the background more obvious, and being conducive to improving the accuracy of text recognition. Perform noise removal and angle correction on the binarized sample invoice images. Noise removal refers to removing the redundant interference information in the images, such as noise points and impurities, which is achieved by using techniques such as median filtering or Gaussian filtering. Angle correction is to correct the tilt or distortion in the images to ensure that the text in the images is presented in a standard horizontal or vertical state. Detect the straight lines in the images through the Hough transform and perform rotation correction. After the noise removal and angle correction processing, multiple standardized sample invoice images are obtained. Perform line segmentation and character segmentation on the standardized sample invoice images. Line segmentation is to split each line of text in the images so that each line of text becomes an independent image area. Character segmentation is to split each character in each line of text separately so that each character becomes an independent image unit. Perform text recognition on the segmented sample invoice images to obtain the initial invoice text data. Text recognition can use OCR (Optical Character Recognition) technology. By recognizing the text in the images, it is converted into editable text data. Determine the invoice keywords of each target sample invoice image according to the initial invoice text data. By analyzing the recognized text data, match the preset keyword set, extract the key information in each invoice, and obtain multiple sample invoice text data. Train the initial OCR model with the sample invoice text data. Continuously optimize the model parameters through the labeled data to improve the recognition accuracy of the model.

[0065] In a specific embodiment, the process of performing the step of determining the invoice keywords of each target sample invoice image according to the initial invoice text data and obtaining multiple sample invoice text data may specifically include the following steps:

[0066] (1) Preprocess the initial invoice text data to obtain the standard invoice text data corresponding to the initial invoice text data;

[0067] (2) Calculate the keyword similarity between the standard invoice text data and the invoice keyword set to obtain the target similarity corresponding to each keyword in the standard invoice text data;

[0068] (3)Compare the target similarity corresponding to each keyword with the preset similarity threshold, and use the keyword corresponding to the target similarity greater than the similarity threshold as the bill keyword for each target sample bill image;

[0069] (4)Integrate the bill keywords to obtain multiple sample bill text data.

[0070] Specifically, preprocess the initial bill text data to obtain the standard bill text data corresponding to the initial bill text data. Ensure that the text data format is unified, eliminate noise and redundant information, and make it easier for subsequent processing. The specific preprocessing steps include removing extra spaces, line breaks, and special characters, unifying the character encoding format, and formatting keyword fields such as dates and amounts. Calculate the keyword similarity between the standard bill text data and the preset bill keyword set. The bill keyword set contains various common and important fields, such as "date", "amount", "invoice number", "customer name", etc. To calculate the similarity, use multiple text similarity algorithms, such as cosine similarity, Jaccard similarity, or deep learning-based embedding models (such as Word2Vec, BERT, etc.). These methods can effectively measure the similarity between two text segments. For example, using cosine similarity, represent the text as a vector, and calculate the cosine value of the angle between the two vectors to represent the similarity. Analyze the target similarity corresponding to each keyword and compare these similarities with the preset similarity threshold. The similarity threshold is a preset value used to determine whether the similarity is high enough to identify a certain field as a keyword. Assume that the preset similarity threshold is 0.8. If the similarity of a certain field is greater than 0.8, the system considers that the field matches successfully. In this way, filter out the key information in the standard bill text data, and use this information as the bill keyword for each target sample bill image. Integrate the identified bill keywords to obtain multiple sample bill text data. Remove duplicate and redundant keywords, and merge and unify similar or related keywords. Optimize the keywords through methods such as a semantic-based merging strategy and a frequency-based screening strategy.

[0071] In a specific embodiment, the process of executing step 103 may specifically include the following steps:

[0072] (1)Obtain multiple preset LayoutLMv3 models, where each LayoutLMv3 model includes: an encoder, a multi-modal Transformer network, a decoder, and a policy decision network;

[0073] (2)Match the corresponding relationships between multiple sample bill text data and multiple LayoutLMv3 models respectively to obtain the LayoutLMv3 model corresponding to each sample bill text data;

[0074] (3)Input each sample bill text data into the corresponding LayoutLMv3 model respectively, and perform vector conversion on each sample bill text data through the first embedding layer in the encoder to obtain the corresponding first sample encoding vector;

[0075] (4)Perform position encoding on the first sample encoding vector through the first position encoding layer in the encoder to obtain the second sample encoding vector;

[0076] (5)Perform vector multi-layer feature processing on the second sample encoding vector through the first multi-layer convolutional layer in the encoder to obtain the sample convolutional encoding feature vector;

[0077] (6)Input the sample convolutional encoding feature vector into the multi-modal Transformer network for attention feature enhancement to obtain the sample attention feature vector;

[0078] (7)Perform multi-layer convolutional processing on the sample attention feature vector through the second multi-layer convolutional layer in the decoder to obtain the sample convolutional feature decoding vector;

[0079] (8)Perform position decoding on the sample convolutional feature decoding vector through the second position encoding layer in the decoder to obtain the first sample decoding vector;

[0080] (9)Perform vector conversion on the first sample decoding vector through the second embedding layer in the decoder to obtain the corresponding second sample decoding vector;

[0081] (10)Classify the bills for the second sample decoding vector through multiple decision trees in the policy decision network to obtain the sample bill classification information of each LayoutLMv3 model;

[0082] (11)Calculate the loss for the sample bill classification information through the preset cross-entropy function to obtain the loss value of each LayoutLMv3 model;

[0083] (12)Perform gradient descent update on each LayoutLMv3 model respectively according to the loss value to obtain multiple first bill classification models;

[0084] (13)Perform reinforcement learning training and model integration on multiple first bill classification models to generate a bill information classification model.

[0085] Specifically, obtain multiple pre - set LayoutLMv3 models. Each model contains an encoder, a multi - modal Transformer network, a decoder, and a policy decision network. Match the multiple sample bill text data with the LayoutLMv3 models to find the most suitable LayoutLMv3 model for each sample bill text data, thus ensuring that the data can be correctly processed and classified. Input each sample bill text data into the corresponding LayoutLMv3 model respectively. Each model performs vector transformation on the input bill text data through the first embedding layer in the encoder to obtain the first sample encoding vector. The embedding layer converts the text data into a high - dimensional vector representation so that it can be further processed by the model. These vectors are position - encoded through the first position encoding layer in the encoder to add position information and obtain the second sample encoding vector. Position encoding helps the model understand the positional relationship of each element in the text and thus better capture the text structure. The second sample encoding vector undergoes vector multi - layer feature processing through the first multi - layer convolutional layer in the encoder to extract local features in the text data and enhance the model's expressive ability. After multi - layer convolutional processing, a sample convolutional encoding feature vector is obtained. The feature vector is input into the multi - modal Transformer network for attention feature enhancement. The multi - modal Transformer network can focus on the importance of different parts of the text through the attention mechanism, enhance the capture of key information, and obtain the sample attention feature vector. The sample attention feature vector undergoes multi - layer convolutional processing through the second multi - layer convolutional layer in the decoder to extract and optimize features and obtain the sample convolutional feature decoding vector. The decoding vector undergoes position decoding through the second position encoding layer in the decoder to restore position information and obtain the first sample decoding vector. Then, the first sample decoding vector is vector - transformed through the second embedding layer in the decoder to obtain the corresponding second sample decoding vector. The second sample decoding vector is classified for bills through multiple decision trees in the policy decision network. The decision trees make classification decisions based on different features of the feature vector and output the sample bill classification information of each LayoutLMv3 model. To evaluate the performance of the model, the loss of the sample bill classification information is calculated through a pre - set cross - entropy function to obtain the loss value of each LayoutLMv3 model. The cross - entropy loss function can measure the gap between the prediction result and the actual result and guide the optimization of the model. According to the calculated loss value, each LayoutLMv3 model is updated by gradient descent respectively. By adjusting the model parameters, the loss function is minimized to improve the classification accuracy and robustness of the model. After multiple iterations and updates, multiple first - stage bill classification models are obtained. The multiple first - stage bill classification models are trained through reinforcement learning. In the reinforcement learning training, by introducing a reward mechanism, the decisions of the model are optimized. Through continuous experimentation and adjustment, the decision - making ability of the model in different situations is strengthened to improve the classification accuracy.Through model integration, the advantages of multiple first bill classification models are combined to generate a final bill information classification model. The integration method can include voting mechanisms, weighted averages, etc. By synthesizing the prediction results of multiple models, the performance of the overall model is improved.

[0086] In a specific embodiment, the process of performing reinforcement learning training and model integration on multiple first bill classification models to generate a bill information classification model may specifically include the following steps:

[0087] (1) Analyze the network decision execution strategies of the policy decision networks in multiple first bill classification models respectively to obtain initial network decision execution strategies;

[0088] (2) Perform execution strategy learning on the policy decision networks to obtain a target execution strategy search space;

[0089] (3) Calculate reward feedback data for the initial network decision execution strategies based on the target execution strategy search space to obtain reward feedback parameters;

[0090] (4) Update the policies of the policy decision networks in each first bill classification model based on the reward feedback parameters to generate target network decision execution strategies corresponding to the policy decision networks in each first bill classification model;

[0091] (5) Optimize the decision strategies of multiple first bill classification models according to the target network decision execution strategies to obtain multiple second bill classification models;

[0092] (6) Calculate the model weight data of each second bill classification model respectively, and perform model integration on multiple second bill classification models according to the model weight data to obtain a bill information classification model.

[0093] Specifically, conduct a network decision execution strategy analysis on the policy decision networks in multiple first bill classification models to understand their decision-making behaviors in different situations. This includes analyzing decision trees or other decision algorithms to determine their decision paths and logics, and obtaining the initial network decision execution strategy. Conduct an execution strategy learning on the policy decision networks to construct a target execution strategy search space. The search space covers all possible policy combinations and execution paths. By analyzing the initial network decision execution strategy, identify the key decision points and possible policy combinations, and define a search space that includes these policies. For example, in a policy decision network based on a decision tree, combine the decision logics of different branches and nodes to form a multi-dimensional policy search space. Based on the target execution strategy search space, calculate the reward feedback data for the initial network decision execution strategy. By introducing a reward mechanism, evaluate and feedback the execution results of each policy. Design a reward function to evaluate the performance of the policy in different situations, such as classification accuracy, processing time, and resource consumption. By calculating the reward feedback parameters, quantify the execution effect of each policy to provide a basis for policy update. Based on the reward feedback parameters, update the policies of the policy decision networks in each first bill classification model to generate the corresponding target network decision execution strategies for the policy decision networks in each first bill classification model. Policy update is an iterative optimization process. By continuously adjusting the policy parameters, maximize the value of the reward function to improve the overall performance of the decision network. Use gradient descent or other optimization algorithms to update and adjust the policy parameters. For example, in a policy decision network based on a neural network, the network weights can be updated through the backpropagation algorithm and gradient descent to optimize the policy execution effect. Optimize the decision strategies of multiple first bill classification models according to the target network decision execution strategies to obtain multiple second bill classification models. Apply the optimized policies to the decision-making process of each model, re-evaluate and adjust the classification logics and parameters of the model to improve its overall performance. For example, in a bill classification model based on a random forest, optimize the overall classification effect by adjusting the splitting rules and node decisions of each tree. Calculate the model weight data of each second bill classification model respectively. The model weight data reflects the performance and contributions of each model on different tasks and datasets. By calculating the weight data, measure and compare the relative importance and advantages and disadvantages of each model. Through cross-validation and performance evaluation, calculate indicators such as accuracy, precision, and recall of each model on different datasets, and assign different weights to each model according to these indicators. Based on the calculated model weight data, conduct model integration on multiple second bill classification models to generate the final bill information classification model. Model integration is to comprehensively and fuse the prediction results of multiple models through weighted average, voting mechanism, or other integration methods to improve the generalization ability and prediction accuracy of the overall model.For example, the weighted average method is adopted to perform weighted summation on the prediction results according to the weights of each model to obtain the final classification result; or a voting mechanism is adopted to select the prediction that is consistent among the majority of models as the final result by counting the prediction results of multiple models.

[0094] The above describes the OCR-based bill information recognition method in the embodiments of the present application. Next, the OCR-based bill information recognition device in the embodiments of the present application will be described. Please refer to Figure 2 An embodiment of the OCR-based bill information recognition device in the embodiments of the present application includes:

[0095] An acquisition module 201, configured to acquire multiple initial sample bill images, and perform field information splitting and annotation on the multiple initial sample bill images respectively through a label-studio data annotation tool to obtain multiple target sample bill images;

[0096] An identification training module 202, configured to input the multiple target sample bill images into a preset initial OCR model for character recognition training to obtain multiple sample bill text data and a target OCR model;

[0097] A classification training module 203, configured to input the multiple sample bill text data into a preset multiple LayoutLMv3 models for bill classification training to obtain a bill information classification model;

[0098] A preprocessing module 204, configured to acquire a first bill image to be recognized, and preprocess the first bill image to be recognized to obtain a second bill image to be recognized;

[0099] A character recognition module 205, configured to input the second bill image to be recognized into the target OCR model for character recognition to obtain target bill text data;

[0100] An information classification module 206, configured to input the target bill text data into the bill information classification model for bill information classification to obtain a bill information classification result.

[0101] Through the collaborative cooperation of the above-mentioned various components, the acquisition and processing of bill images are carried out in an automated manner, significantly reducing the demand for human resources and the possibility of human errors, which is particularly important in high-load and high-frequency bill processing scenarios. By combining an advanced OCR model and the LayoutLMv3 model, not only can the text in the image be recognized, but also the specific business meaning and context relationship of the text in the bill can be understood. This multi-modal recognition method can accurately classify and analyze complex bill layouts, such as tables, headings, and paragraphs, thus improving the accuracy and reliability of data extraction. With multiple datasets and multi-model fusion strategies, this not only enhances the generalization ability of the model but also ensures high recognition accuracy on bill images of different types and qualities.

[0102] This application also provides a computer device, which includes a memory and a processor. Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor is caused to execute the steps of the OCR-based bill information recognition method in the above-mentioned various embodiments.

[0103] This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer, the computer is caused to execute the steps of the OCR-based bill information recognition method.

[0104] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0106] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A bill information recognition method based on OCR, characterized in that: The OCR-based bill information recognition method includes: Acquire multiple initial sample bill images, and use the label-studio data annotation tool to split and annotate the field information of the multiple initial sample bill images to obtain multiple target sample bill images; specifically include: acquiring multiple initial sample bill images, and dividing the data sets of the multiple initial sample bill images according to a preset ratio to obtain a fusion training set, a statistical data set and a fine-tuning data set; calling multiple target models in the label-studio data annotation tool, and inputting the fusion training set into the multiple target models; performing data set label prediction on the fusion training set through the multiple target models to obtain the initial label prediction result corresponding to each target model; obtaining multiple preset candidate fusion strategies, wherein the multiple candidate fusion strategies include: weighted average fusion strategy, linear regression Fusion strategy and multi-layer perceptron fusion strategy; respectively fuse the initial label prediction results corresponding to each target model according to the weighted average fusion strategy, the linear regression fusion strategy and the multi-layer perceptron fusion strategy to obtain the target label fusion result corresponding to each candidate fusion strategy; according to the target label fusion result corresponding to each candidate fusion strategy, perform fusion strategy training on the multiple candidate fusion strategies to obtain multiple trained fusion strategies; select a target fusion strategy from the multiple trained fusion strategies, calculate the confidence interval of the target fusion strategy, and generate a labeling model with the target fusion strategy based on the multiple target models; perform field information splitting and labeling through the labeling model with the target fusion strategy, and obtain multiple target sample bill images according to the preset fuzzy supervised learning strategy; Input the multiple target sample bill images into a preset initial OCR model for character recognition training to obtain multiple sample bill text data and a target OCR model; specifically including: creating a bill keyword set, wherein the bill keyword set includes multiple preset keywords; graying and binarizing the multiple target sample bill images to obtain multiple binarized sample bill images; performing noise removal and angle correction on the multiple binarized sample bill images to obtain multiple standardized sample bill images; performing line segmentation and character segmentation on the multiple standardized sample bill images to obtain multiple segmented sample bill images, and performing text recognition on the multiple segmented sample bill images to obtain initial bill text data; Preprocessing the initial bill text data to obtain standard bill text data corresponding to the initial bill text data; performing keyword similarity calculation on the standard bill text data and the bill keyword set to obtain a target similarity corresponding to each keyword in the standard bill text data; comparing the target similarity corresponding to each keyword with a preset similarity threshold, and using the keyword corresponding to the target similarity when the target similarity is greater than the similarity threshold as the bill keyword of each target sample bill image; performing keyword integration on the bill keywords to obtain multiple sample bill text data; and training the initial OCR model using the multiple sample bill text data to obtain a target OCR model; Input the multiple sample bill text data into the preset multiple LayoutLMv3 models for bill classification training to obtain a bill information classification model; specifically including: obtaining multiple preset LayoutLMv3 models, wherein each LayoutLMv3 model includes: an encoder, a multimodal Transformer network, a decoder and a policy decision network; respectively matching the multiple sample bill text data with the multiple LayoutLMv3 models to obtain a LayoutLMv3 model corresponding to each sample bill text data; respectively input each sample bill text data into the corresponding LayoutLMv3 model, and through the first embedding in the encoder The input layer performs vector conversion on each sample bill text data to obtain the corresponding first sample encoding vector; the first position encoding layer in the encoder performs position encoding on the first sample encoding vector to obtain the second sample encoding vector; the first multi-layer convolution layer in the encoder performs vector multi-layer feature processing on the second sample encoding vector to obtain a sample convolution encoding feature vector; the sample convolution encoding feature vector is input into the multimodal Transformer network for attention feature enhancement to obtain a sample attention feature vector; the second multi-layer convolution layer in the decoder performs multi-layer convolution processing on the sample attention feature vector to obtain a sample convolution feature decoding vector; the decoder performs multi-layer convolution processing on the sample attention feature vector to obtain a sample convolution feature decoding vector; the decoder performs multi-layer convolution on the sample attention feature vector to obtain a sample convolution feature decoding vector. The second position encoding layer performs position decoding on the sample convolution feature decoding vector to obtain a first sample decoding vector; the first sample decoding vector is vector-converted by the second embedding layer in the decoder to obtain a corresponding second sample decoding vector; the second sample decoding vector is classified into bills by multiple decision trees in the strategy decision network to obtain sample bill classification information of each LayoutLMv3 model; the sample bill classification information is calculated by a preset cross entropy function to obtain the loss value of each LayoutLMv3 model; according to the loss value, each LayoutLMv3 model is updated by gradient descent to obtain multiple first bill classification models; performing network decision execution strategy analysis on the strategy decision networks in the plurality of first bill classification models to obtain an initial network decision execution strategy; performing execution strategy learning on the strategy decision networks to obtain a target execution strategy search space; calculating reward feedback data on the initial network decision execution strategy based on the target execution strategy search space to obtain a reward feedback parameter; performing strategy update on the strategy decision network in each first bill classification model based on the reward feedback parameter to generate a target network decision execution strategy corresponding to the strategy decision network in each first bill classification model; performing decision strategy optimization on the plurality of first bill classification models according to the target network decision execution strategy to obtain a plurality of second bill classification models;Calculating model weight data of each second bill classification model respectively, and integrating the plurality of second bill classification models according to the model weight data to obtain a bill information classification model; Acquire a first bill image to be identified, and preprocess the first bill image to be identified to obtain a second bill image to be identified; Inputting the second bill image to be recognized into the target OCR model for text recognition to obtain target bill text data; The target bill text data is input into the bill information classification model to perform bill information classification to obtain a bill information classification result.

2. A bill information recognition device based on OCR, characterized in that: The OCR-based bill information recognition device includes: An acquisition module is used to acquire multiple initial sample bill images, and perform field information splitting and annotation on the multiple initial sample bill images respectively through the label-studio data annotation tool to obtain multiple target sample bill images; specifically including: acquiring multiple initial sample bill images, and dividing the data sets of the multiple initial sample bill images according to a preset ratio to obtain a fusion training set, a statistical data set and a fine-tuning data set; calling multiple target models in the label-studio data annotation tool, and inputting the fusion training set into the multiple target models; performing data set label prediction on the fusion training set respectively through the multiple target models to obtain the initial label prediction result corresponding to each target model; acquiring multiple preset candidate fusion strategies, wherein the multiple candidate fusion strategies include: weighted average fusion strategy, Linear regression fusion strategy and multi-layer perceptron fusion strategy; fusing the initial label prediction results corresponding to each target model according to the weighted average fusion strategy, the linear regression fusion strategy and the multi-layer perceptron fusion strategy, respectively, to obtain the target label fusion result corresponding to each candidate fusion strategy; performing fusion strategy training on the multiple candidate fusion strategies according to the target label fusion result corresponding to each candidate fusion strategy, to obtain multiple trained fusion strategies; selecting a target fusion strategy from the multiple trained fusion strategies, calculating the confidence interval of the target fusion strategy, and generating a labeling model with the target fusion strategy based on the multiple target models; performing field information splitting and labeling through the labeling model with the target fusion strategy, and obtaining multiple target sample bill images according to a preset fuzzy supervised learning strategy; The recognition training module is used to input the multiple target sample bill images into a preset initial OCR model for character recognition training to obtain multiple sample bill text data and a target OCR model; specifically includes: creating a bill keyword set, wherein the bill keyword set includes multiple preset keywords; graying and binarizing the multiple target sample bill images to obtain multiple binarized sample bill images; performing noise removal and angle correction on the multiple binarized sample bill images to obtain multiple standardized sample bill images; performing line segmentation and character segmentation on the multiple standardized sample bill images to obtain multiple segmented sample bill images, and performing text recognition on the multiple segmented sample bill images to obtain initial bill text data; preprocessing the initial bill text data to obtain standard bill text data corresponding to the initial bill text data; performing keyword similarity calculation on the standard bill text data and the bill keyword set to obtain a target similarity corresponding to each keyword in the standard bill text data; comparing the target similarity corresponding to each keyword with a preset similarity threshold, and using the keyword corresponding to the target similarity when the target similarity is greater than the similarity threshold as the bill keyword of each target sample bill image; performing keyword integration on the bill keywords to obtain multiple sample bill text data; training the initial OCR model using the multiple sample bill text data to obtain a target OCR model; The classification training module is used to input the multiple sample bill text data into the preset multiple LayoutLMv3 models for bill classification training to obtain a bill information classification model; specifically comprising: obtaining multiple preset LayoutLMv3 models, wherein each LayoutLMv3 model includes: an encoder, a multimodal Transformer network, a decoder, and a strategy decision network; respectively matching the multiple sample bill text data with the multiple LayoutLMv3 models to obtain a LayoutLMv3 model corresponding to each sample bill text data; respectively inputting each sample bill text data into the corresponding LayoutLMv3 model, and The first embedding layer in the encoder performs vector conversion on each sample bill text data to obtain the corresponding first sample encoding vector; the first position encoding layer in the encoder performs position encoding on the first sample encoding vector to obtain the second sample encoding vector; the first multi-layer convolution layer in the encoder performs vector multi-layer feature processing on the second sample encoding vector to obtain a sample convolution encoding feature vector; the sample convolution encoding feature vector is input into the multimodal Transformer network for attention feature enhancement to obtain a sample attention feature vector; the second multi-layer convolution layer in the decoder performs multi-layer convolution processing on the sample attention feature vector to obtain a sample convolution feature decoding vector; The second position encoding layer in the decoder performs position decoding on the sample convolution feature decoding vector to obtain a first sample decoding vector; the first sample decoding vector is vector-converted by the second embedding layer in the decoder to obtain a corresponding second sample decoding vector; the second sample decoding vector is classified into bills by multiple decision trees in the policy decision network to obtain sample bill classification information of each LayoutLMv3 model; the sample bill classification information is calculated by a preset cross entropy function to obtain the loss value of each LayoutLMv3 model; according to the loss value, each LayoutLMv3 model is updated by gradient descent to obtain multiple first bill classification models ; Perform network decision execution strategy analysis on the strategy decision networks in the multiple first bill classification models respectively to obtain an initial network decision execution strategy; perform execution strategy learning on the strategy decision network to obtain a target execution strategy search space; calculate reward feedback data for the initial network decision execution strategy based on the target execution strategy search space to obtain a reward feedback parameter; perform strategy update on the strategy decision network in each first bill classification model based on the reward feedback parameter to generate a target network decision execution strategy corresponding to the strategy decision network in each first bill classification model; perform decision strategy optimization on the multiple first bill classification models according to the target network decision execution strategy to obtain multiple second bill classification models;Calculating model weight data of each second bill classification model respectively, and integrating the plurality of second bill classification models according to the model weight data to obtain a bill information classification model; a preprocessing module, configured to obtain a first bill image to be identified, and preprocess the first bill image to be identified to obtain a second bill image to be identified; A text recognition module, configured to input the second bill image to be recognized into the target OCR model for text recognition to obtain target bill text data; The information classification module is used to input the target bill text data into the bill information classification model to perform bill information classification and obtain a bill information classification result.

3. A computer device, characterized in that: The computer device includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the OCR-based bill information recognition method according to claim 1.

4. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the OCR-based bill information recognition method according to claim 1 is implemented.

Images