A kind of bill identification method, device, equipment and storage medium

Abstract

The application discloses a kind of bill identification method, device, equipment and storage medium, it is related to computer technical field, including: using OCR technology to the different types of text area in bill image is identified, obtain multiple bill text blocks;The coordinates of each bill text block and confidence are generated to obtain text block coordinates and text block confidence, and whether the confidence of text block is greater than the first confidence threshold is judged, if all confidence is greater than threshold, then the type and confidence of the target bill are predicted to obtain multiple bill types and multiple type confidence;Whether the maximum value in type confidence is greater than or equal to the second confidence threshold is judged, if yes, then the mask template and prompt word corresponding to the maximum value are determined;Through multimodal big model and using mask template, the core field area in bill image is strengthened, and based on prompt word, the key information in the bill image after strengthening is extracted to obtain bill identification result.The application can improve the precision and efficiency of bill identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a bill identification method and device, equipment and a storage medium. BACKGROUND

[0002] With the vigorous development of different industries (such as the automobile finance industry), the market (for example, the automobile aftermarket, such as the second-hand car, vehicle accessories, insurance, and other financial businesses) activity types are more and more, and under the same type of business, there are a large number of electronic and paper bills in different formats, and there are many malicious modification cases, and it is difficult to extract and check the content. At present, the mainstream bill automatic identification method includes a traditional OCR (Optical Character Recognition) + rule engine method, a multi-modal large model identification method, and a field-specific model + lightweight rule method.

[0003] Among them, the OCR + rule engine method extracts the amount, date and other text fields in the bill through the OCR technology, and then checks according to the pre-set fixed rules such as format regular, numerical range, which is commonly used for simple invoice processing, but has poor adaptability when facing complex bills and different version format bills. Once the rules are changed, they need to be redeveloped, and the robustness is insufficient; the multi-modal large model identification method combines image features (bill layout, seal position) and text features (field content semantics), and realizes extraction and checking through joint analysis of large models, which is suitable for complex insurance policies and other scenarios, but model inference requires high-performance hardware support, and the power consumption is large. The power consumption is large, and the adaptation to different types of bills requires a large amount of training power, and the feature extraction accuracy decreases for fuzzy and wrinkled low-quality images, and the image itself cannot be checked; and the field-specific model + lightweight rule method needs to develop a special model for different fields, and match simplified rules (such as checking only the key field logic), so as to balance accuracy and efficiency, but the model is only suitable for known bills in a specific field, and has weak cross-type generalization ability. New bill types need to re-tune the model, the adaptation cost is high, and the small model has insufficient ability in identification and checking. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a bill identification method, device, equipment and storage medium, which can improve the accuracy of bill identification and does not require human intervention, thereby improving the efficiency of bill identification and saving labor costs. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a bill identification method applied to a server, comprising:

[0006] receiving a current bill image uploaded by a client, and identifying different types of text regions in the current bill image by using an OCR technology to obtain a plurality of bill text blocks;

[0007] generate the coordinates and the confidence of each of the text blocks of the document respectively, to obtain text block coordinates and text block confidence, and determine whether the text block confidence is greater than a first confidence threshold value respectively;

[0008] If all the text block confidence is greater than the first confidence threshold value, the type of the target document in the current document image and the confidence of each type are predicted to obtain multiple document types and multiple type confidence;

[0009] determine whether the maximum value in the multiple type confidence is greater than or equal to a second confidence threshold value, if the maximum value is greater than or equal to the second confidence threshold value, determine the target mask template and the target prompt word corresponding to the target document type; the target document type is the document type corresponding to the maximum value;

[0010] input the current document image, the target mask template and the target prompt word into a multi-modal large model to strengthen the core field area in the current document image using the target mask template to obtain a strengthened document image, and extract key information in the strengthened document image based on the form field in the target prompt word to obtain a document recognition result.

[0011] Optionally, the use of OCR technology to identify different types of text regions in the current document image to obtain multiple document text blocks, comprising:

[0012] detect the physical contour of the target document in the current document image, and calculate the completeness of the physical contour to obtain the physical contour completeness;

[0013] use OCR technology to identify the proportion of text regions in the target document to obtain a text region proportion, and calculate the coverage rate of the text region based on the text region proportion to obtain a text region coverage rate;

[0014] determine whether the physical contour completeness is greater than a first threshold value, and whether the text region coverage rate is greater than a second threshold value;

[0015] If the physical contour completeness is greater than the first threshold value, and the text region coverage rate is greater than the second threshold value, use a pre-created large language model to identify different types of text regions in the current document image to obtain multiple document text blocks; the large language model is a model based on the neural network structure of EAST, YOLO and CRNN.

[0016] Optionally, the use of a pre-created large language model to identify different types of text regions in the current document image to obtain multiple document text blocks, comprising:

[0017] extract edges of the target document by a Canny operator to obtain actual document edges;

[0018] detect actual edge lines corresponding to the actual document edges by a Hough transform, and fit a contour of the target document based on the actual edge lines to obtain a fitted contour;

[0019] calculate a deviation between a physical contour of the target document and the fitted contour to obtain a deviation value, and determine whether the deviation value exceeds a preset threshold;

[0020] if the deviation value does not exceed the preset threshold, calculate an inclination angle of the actual edge lines by the Hough transform, and determine whether the inclination angle is 0;

[0021] if the inclination angle is not 0, correct the actual edge lines in the current document image by a bilinear interpolation algorithm to obtain a corrected document image;

[0022] identify different types of text regions in the corrected document image by using a pre-created large language model to obtain a plurality of document text blocks.

[0023] Optionally, the identifying different types of text regions in the corrected document image by using the pre-created large language model to obtain the plurality of document text blocks comprises:

[0024] perform quality enhancement on the corrected document image by using a non-uniform illumination image enhancement algorithm with a fusion feature attention mechanism to obtain an enhanced document image;

[0025] divide the enhanced document image to obtain a text region, a non-text region, and an adjacent region between the text region and the non-text region;

[0026] respectively perform denoising processing on the text region, the non-text region, and the adjacent region by using a first Gaussian filter kernel, a second Gaussian filter kernel, and a third Gaussian filter kernel to obtain a denoised document image; wherein a filter kernel size of the second Gaussian filter kernel is greater than a filter kernel size of the third Gaussian filter kernel, and the filter kernel size of the third Gaussian filter kernel is greater than a filter kernel size of the first Gaussian filter kernel;

[0027] identify different types of text regions in the denoised document image by using the pre-created large language model to obtain the plurality of document text blocks.

[0028] Optionally, a calculation formula of the physical contour completeness is:

[0029] ;

[0030] wherein, is the physical contour completeness, , are the angle completeness weight and the side completeness weight, respectively, is the number of detected right-angle corners, and is less than or equal to 4; is the theoretical distance between any two of the four corners, is the actual detected length of each side.

[0031] Optionally, the calculating the coverage rate of the text region based on the text region proportion comprises:

[0032] aggregating the plurality of text regions in the target document into K regions using a K-neighbor algorithm to obtain K aggregated regions;

[0033] counting the total area of the target document and the area of each of the aggregated regions to obtain a total document area and aggregated region areas;

[0034] counting the sum of all the aggregated region areas to obtain a total region area, and calculating the ratio of the total region area to the total document area to obtain the text region coverage rate.

[0035] Optionally, the strengthening the core field region in the current document image using the target mask template to obtain a strengthened document image, and extracting key information in the strengthened document image based on the form field in the target prompt word to obtain a document recognition result, comprises:

[0036] performing a pixel-by-pixel weighted operation on the current document image and the first mask region and the second mask region in the target mask template to obtain a strengthened document image; the mask value of the first mask region is greater than the mask value of the second mask region;

[0037] encoding the strengthened document image using a computer vision encoder to obtain document encoding features, and encoding the form field in the target prompt word using a text feature encoder to obtain prompt word encoding features;

[0038] performing multi-modal fusion on the document encoding features and the prompt word encoding features through a cross-attention mechanism to obtain fused encoding features;

[0039] performing decoding operation on the fused encoding features to obtain decoded encoding features, and extracting key field information containing the form field and the corresponding key field information in the strengthened document image from the decoded encoding features to obtain a document recognition result.

[0040] In a second aspect, the present application discloses a bill recognition device applied to a server, comprising:

[0041] A receiving and recognizing module is configured to receive a current bill image uploaded by a client, and recognize different types of text regions in the current bill image by using an OCR technology to obtain a plurality of bill text blocks.

[0042] A generating module is configured to generate coordinates and confidence of each bill text block respectively to obtain text block coordinates and text block confidence.

[0043] A first judging module is configured to judge whether the text block confidence is greater than a first confidence threshold respectively.

[0044] A predicting module is configured to predict a type of a target bill in the current bill image and confidence of each type if all the text block confidences are greater than the first confidence threshold to obtain a plurality of bill types and a plurality of type confidences.

[0045] A second judging module is configured to judge whether a maximum value in the plurality of type confidences is greater than or equal to a second confidence threshold.

[0046] A determining module is configured to determine a target mask template and a target prompt word corresponding to a target bill type if the maximum value is greater than or equal to the second confidence threshold; the target bill type is a bill type corresponding to the maximum value.

[0047] A bill recognition module is configured to input the current bill image, the target mask template and the target prompt word into a multi-modal large model to strengthen a core field region in the current bill image by using the target mask template to obtain a strengthened bill image, and extract key information in the strengthened bill image based on a form field in the target prompt word to obtain a bill recognition result.

[0048] In a third aspect, the present application discloses an electronic device comprising a processor and a memory; wherein the processor implements the bill recognition method as described above when executing the computer program saved in the memory.

[0049] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the bill recognition method as described above.

[0050] It can be seen that the application is applied to a server. After receiving a current bill image uploaded by a client, the OCR technology is used to identify different types of text regions in the current bill image, to obtain a plurality of bill text blocks. Then, the coordinates and confidence of each bill text block are generated, to obtain text block coordinates and text block confidence. Whether the text block confidence is greater than a first confidence threshold is judged respectively. If all the text block confidences are greater than the first confidence threshold, the type of a target bill in the current bill image and the confidence of each type are predicted, to obtain a plurality of bill types and a plurality of type confidences. Further, whether the maximum value in the plurality of type confidences is greater than or equal to a second confidence threshold is judged. If the maximum value is greater than or equal to the second confidence threshold, a target mask template and a target prompt word corresponding to the target bill type are determined. The target bill type is the bill type corresponding to the maximum value. Finally, the current bill image, the target mask template, and the target prompt word are input into a multi-modal large model, to use the target mask template to strengthen the core field region in the current bill image, to obtain a strengthened bill image, and to extract key information in the strengthened bill image based on the form field in the target prompt word, to obtain a bill recognition result. Through the double confidence judgment mechanism, the confidence of the text region (i.e. the text block) identified by the OCR technology is first verified, and after the confidence verification passes, the confidence of the predicted bill type is verified. Then, after the confidence verification passes, the maximum type confidence is determined, and the corresponding mask template and prompt word are determined based on the maximum type confidence, and are input into the multi-modal large model, to use the mask template to strengthen the core field region in the bill image, and to extract the key information in the strengthened bill image based on the prompt word. Through the above method, i.e. the fusion of the double confidence judgment mechanism and the multi-modal large model, the accuracy of bill recognition can be improved, and manual intervention is not required, thereby improving the efficiency of bill recognition and saving labor costs. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0052] Figure 1 A bill recognition method flowchart disclosed by the present application;

[0053] Figure 2 A specific top corner distance and edge length calculation schematic diagram disclosed by the present application;

[0054] Figure 3 A specific text area aggregation result diagram disclosed by the present application is shown in the following figure:

[0055] Figure 4 A specific invoice identification method flowchart disclosed by the present application is shown in the following figure:

[0056] Figure 5 A specific mask template diagram disclosed by the present application is shown in the following figure:

[0057] Figure 6 A specific invoice key information extraction diagram disclosed by the present application is shown in the following figure:

[0058] Figure 7 A specific texture detection diagram disclosed by the present application is shown in the following figure:

[0059] Figure 8 A specific invoice identification device structure diagram disclosed by the present application is shown in the following figure:

[0060] Figure 9 A specific electronic device structure diagram disclosed by the present application is shown in the following figure: DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0062] The embodiments of the present application disclose an invoice identification method, applied to a server, as shown in the following figure: Figure 1 The method comprises the following steps:

[0063] Step S11: receiving a current invoice image uploaded by a client, and identifying different types of text areas in the current invoice image by using an OCR technology to obtain a plurality of invoice text blocks.

[0064] It should be pointed out that the invoice identification scheme proposed by the present application is applied to a server. When a current invoice image uploaded by a client (such as a receipt image photographed by a camera, a tablet computer or a smart phone and uploaded through an APP, a mini program, etc.) is received, first, different types of text areas (such as common content blocks in an invoice, such as text, tables, handwritten signatures, etc.) in the current invoice image are identified by using an OCR technology to obtain a plurality of invoice text blocks.

[0065] Specifically, the utilizing the OCR technology to recognize the different types of text regions in the current bill image to obtain a plurality of bill text blocks can include: detecting a physical contour of a target bill in the current bill image, and calculating a completeness of the physical contour to obtain a physical contour completeness; utilizing the OCR technology to recognize a proportion of the text region in the target bill to obtain a text region proportion, and calculating a coverage rate of the text region based on the text region proportion to obtain a text region coverage rate; determining whether the physical contour completeness is greater than a first threshold value and whether the text region coverage rate is greater than a second threshold value; if the physical contour completeness is greater than the first threshold value and the text region coverage rate is greater than the second threshold value, utilizing a pre-created large language model to recognize the different types of text regions in the current bill image to obtain a plurality of bill text blocks; and the large language model is a model based on a neural network structure of EAST, YOLO and CRNN. In the embodiment, in order to improve the accuracy of recognition, the received bill image can also be subjected to integrity detection. Specifically, a lightweight YOLO model (You Only Look Once, a real-time target detection model) can be used to detect the physical contour (such as the frame, the corner, etc.) of the bill in the current bill image, and then the completeness of the physical contour is calculated to obtain the physical contour completeness, at the same time, an OCR pre-scanning operation is started to recognize the proportion of the text region in the bill image to obtain the text region proportion (to exclude pure blank or large area shielding), and then the coverage rate of the text region is calculated based on the text region proportion to obtain the text region coverage rate; further, it is determined whether the physical contour completeness and the text region coverage rate are both greater than the corresponding preset threshold value, only when both meet the threshold condition, the next step is entered, and for the incomplete bill image, an error prompt can be returned to require the client to re-shoot and upload a new bill image. If the physical contour completeness and the text region coverage rate are both greater than the corresponding threshold value, a large language model (Large Language Model, LLM) based on a neural network structure of EAST (Efficient and Accurate Scene Text Detector, an efficient and accurate scene text detector based on a fully convolutional network) + YOLO + CRNN (Convolutional Recurrent Neural Network, a convolutional recurrent neural network) is utilized to perform lightweight OCR recognition on the current bill image, for example, to frame the text, table, handwritten signature and other text regions in the bill image within 20 milliseconds to obtain a plurality of bill text blocks.

[0066] The formula for calculating the physical contour completeness is:

[0067] ;

[0068] wherein, is the physical contour integrity, , is the angle integrity weight and the side integrity weight, respectively, and by default can be set to , both of which are set to 0.5, that is, the angle and side integrity weights are the same; is the number of detected right-angle corners, and is less than or equal to 4; is the theoretical distance between any two of the four corners, is the actual detected length of each side.

[0069] Specifically, as shown in Figure 2 , four corners are detected, which are J1, J2, J3, and J4, respectively, and are D0, D1, D2, and D3, respectively; are L0, L1, L2, and L3, respectively. Among them, D0 is the theoretically calculated distance from J1 to J2, D1 is the theoretically calculated distance from J4 to J1, D2 is the theoretically calculated distance from J2 to J3, and D3 is the theoretically calculated distance from J3 to J4. L0 is the actual detected distance between J1 and J2 by image recognition, L1 is the actual detected distance between J4 and J1 by image recognition, L2 is the actual detected distance between J2 and J3 by image recognition, and L3 is the actual detected distance between J3 and J4 by image recognition.

[0070] In this embodiment, the calculation of the coverage rate of the text area based on the text area proportion can specifically include: using a K-neighbor algorithm to aggregate a plurality of text areas in the target document into K areas to obtain K aggregated areas; counting the total area of the target document and the area of each of the aggregated areas to obtain the total area of the document and the aggregated area; counting the sum of all the aggregated areas to obtain the total area of the regions, and calculating the ratio of the total area of the regions to the total area of the document to obtain the text area coverage rate. For example, as shown in Figure 3 , a clustering algorithm such as a K-neighbor (KNN, K-Nearest Neighbor) algorithm is used to aggregate a plurality of text areas in the target document into three (i.e., K=3, which can be set to 3-5 by default, and the larger the value, the smaller the corresponding text coverage rate threshold) areas to obtain three aggregated areas A i , then the total area of the target document is counted, and the area of each of the aggregated areas A i is counted to obtain the total area of the document and the aggregated area ; then, the total area of the regions is obtained by summing up the areas of all the regions, and the ratio of the total area of the regions to the total area of the document is calculated to obtain the coverage rate of the text regions , and the specific calculation formula is:

[0071] .

[0072] In a specific embodiment, the use of a pre-created large language model to identify different types of text regions in the current invoice image to obtain multiple invoice text blocks can specifically include: extracting the edge of the target invoice through a Canny operator to obtain an actual invoice edge; detecting the actual edge line corresponding to the actual invoice edge through a Hough transform, and fitting the contour of the target invoice based on the actual edge line to obtain a fitted contour; calculating the deviation of the physical contour of the target invoice and the fitted contour to obtain a deviation value, and determining whether the deviation value exceeds a preset threshold; if the deviation value does not exceed the preset threshold, calculating the inclination angle of the actual edge line through the Hough transform, and determining whether the inclination angle is 0; if the inclination angle is not 0, correcting the actual edge line in the current invoice image through a bilinear interpolation algorithm to obtain a corrected invoice image; using a pre-created large language model to identify different types of text regions in the corrected invoice image to obtain multiple invoice text blocks. In this embodiment, in order to improve the accuracy of recognition, the received invoice image can also be pre-processed. Specifically, the actual invoice edge of the target invoice is first extracted through a Canny operator (an edge detection operator), and then the straight line corresponding to the actual invoice edge, the actual edge line, is detected using a Hough transform, and the rectangular contour of the target invoice (i.e., the fitted ideal quadrilateral contour) is fitted based on the actual edge line to obtain a fitted contour. Then, the deviation between the physical contour of the target invoice and the fitted contour is calculated to obtain a deviation value (such as the bending degree of each side, the displacement of the vertex, etc.), and it is determined whether the deviation value exceeds a preset threshold. If the deviation value exceeds the preset threshold (such as single-side bending degree > 15°), it is determined that the target invoice has a curved distortion (such as a curled paper service sheet), and a pop-up window can be prompted to "please flatten the invoice and retake". If the deviation value does not exceed the preset threshold, the inclination angle of the recognized actual edge line is calculated through the Hough transform, and it is determined whether the inclination angle is 0. If the inclination angle is not 0, it indicates that the target invoice is inclined, and the edge line in the invoice image can be corrected through a bilinear interpolation (Bilinear Interpolation) algorithm to obtain a corrected invoice image. This method can adapt to complex invoice recognition scenarios, such as a second-hand car receipt, an insurance policy, and an additional product receipt taken at any angle. Finally, a pre-created large language model is used to identify different types of text regions in the above corrected invoice image to obtain multiple invoice text blocks.

[0073] In this embodiment, the pre-created large language model is used to identify different types of text regions in the corrected document image to obtain a plurality of document text blocks. Specifically, a non-uniform illumination image enhancement algorithm (UIIEA) with a fusion feature attention mechanism is used to enhance the quality of the corrected document image to obtain an enhanced document image. The enhanced document image is divided into a text region, a non-text region, and an adjacent region between the text region and the non-text region. A first Gaussian filter kernel, a second Gaussian filter kernel, and a third Gaussian filter kernel are used to denoise the text region, the non-text region, and the adjacent region, respectively, to obtain a denoised document image. The filter kernel size of the second Gaussian filter kernel is larger than that of the third Gaussian filter kernel, and the filter kernel size of the third Gaussian filter kernel is larger than that of the first Gaussian filter kernel. A pre-created large language model is used to identify different types of text regions in the denoised document image to obtain a plurality of document text blocks. In this embodiment, after the current document image is corrected, it can be further enhanced in quality and denoised. Specifically, a non-uniform illumination image enhancement algorithm (UIIEA) with a fusion feature attention mechanism is used to enhance the quality of the corrected document image to optimize uneven illumination, overexposure, and automatically balance light and dark areas. This method can enhance the quality of the image without distinguishing the type of the document, thereby obtaining an enhanced document image. Then, the enhanced document image is divided into three regions: a text region, a non-text region, and an adjacent region between the text region and the non-text region. A dynamic filter kernel is used to dynamically adjust the size of the filter kernel according to the pre-OCR text density. The preset filter kernel rule is that the text region uses a 3x3 Gaussian filter kernel, the non-text region uses a 5x5 Gaussian filter kernel, and the convolution operation is performed on the image in the region. Specifically, for the text region, a 3x3 Gaussian filter kernel is used to slide with a 1-pixel step, and each window can take the surrounding 9 pixels to calculate the center pixel value to preserve the stroke edge details. For the non-text region, a 5x5 Gaussian filter kernel is also used to slide, and 25 surrounding pixels are taken to calculate the weighted value to strengthen noise removal. When filtering the adjacent region between the text region and the non-text region, a 4x4 Gaussian filter kernel can be used to avoid boundary pixel brightness mutation. The entire denoising process automatically matches the region type to achieve accurate denoising, which can reduce background noise and preserve text details. After the above denoising operation, a pre-created large language model can be used to identify different types of text regions in the denoised document image.

[0074] Through the above pre-processing and integrity detection of the bill image, such as integrity detection by double indicators (physical profile integrity + text area coverage), effective bill images are screened out, and combined with tilt correction, curved distortion detection, UIIEA light optimization and dynamic denoising, it can be adapted to many categories of bills such as used car tickets, insurance policies, etc., and solves the problem of random shooting angle and poor image quality.

[0075] Step S12: Generate the coordinates and confidence of each bill text block respectively, obtain the text block coordinates and text block confidence, and determine whether the text block confidence is greater than the first confidence threshold respectively.

[0076] In this embodiment, after identifying the bill text block in the current bill image, the pixel coordinates of each bill text block can be generated synchronously to obtain the text block coordinates, and then a large language model (100M parameter lightweight model) is used to process the segmented bill text block, for example, converting the text area to pure text, converting the table area to a general JSON (JavaScript Object Notation, JavaScript object notation) format text, thereby outputting structured data based on "text content-coordinates" (i.e. structured OCR information), and synchronously labeling the text block confidence. In addition, the above structured data (i.e. structured OCR information) can also be subjected to semantic error correction, for example, correcting character confusion such as "1 / I", "0 / O", "B / 8", filtering redundant text such as "invalid remarks", "advertisement column", correcting sentence errors based on general grammar logic, etc., to obtain corrected OCR information. Through semantic error correction, character confusion can be corrected and redundant text can be filtered, thereby ensuring the accuracy of text information recognition.

[0077] Step S13: If all the text block confidences are greater than the first confidence threshold, predict the type of the target bill in the current bill image and the confidence of each type to obtain multiple bill types and multiple type confidences.

[0078] In this embodiment, it is first determined whether each text block confidence is greater than the preset confidence threshold. If all confidences are greater than the confidence threshold, the above corrected OCR information is subjected to multi-classification recognition by using the trained deep learning model to predict the type of the target bill and the confidence of each type (i.e. type confidence).

[0079] In a specific embodiment, the trained deep learning model is a model obtained by training a DistilBERT model (Distilled Bidirectional Encoder Representations from Transformers) using a relevant bill image dataset of the automobile finance aftermarket.

[0080] Step S14: determining whether a maximum value in the plurality of type confidences is greater than or equal to a second confidence threshold value, and if the maximum value is greater than or equal to the second confidence threshold value, determining a target mask template and a target prompt corresponding to a target bill type; the target bill type is a bill type corresponding to the maximum value.

[0081] In this embodiment, the maximum value in all type confidences can be determined first, and then it is determined whether the maximum value is greater than or equal to a second confidence threshold value (such as 0.8). If the maximum value is less than the second confidence threshold value (i.e., the maximum confidence < 0.8), the top 3 high-probability type confidences and corresponding bill types can be displayed in a pop-up window, and the feature descriptions of each bill type are carried. After being confirmed by the business personnel, the subsequent link is entered. If the maximum value is greater than or equal to the second confidence threshold value (i.e., the maximum confidence ≥ 0.8), the bill type corresponding to the maximum value is directly taken as the target bill type, and then an image mask template corresponding to the target bill type is selected from a preset image mask library, and a target prompt corresponding to the target bill type is selected from a preset prompt template library. For example, the insurance bill prompt is to extract the policy number, premium amount (distinguish between compulsory insurance and commercial insurance), insurance premium, insurance start and end date, and insured person's ID number. Through the judgment of the second confidence threshold value, type misjudgment can be avoided.

[0082] Step S15: inputting the current bill image, the target mask template, and the target prompt into a multi-modal large model to strengthen the core field area in the current bill image using the target mask template, obtaining a strengthened bill image, and extracting key information in the strengthened bill image based on the form fields in the target prompt, to obtain a bill recognition result.

[0083] In this embodiment, the received current bill image, the determined target mask template and the target prompt word are input into a multimodal large language model (MLLMs). After the multimodal large language model receives the above information, the core field region in the current bill image is first strengthened (i.e., the core field region is focused) by using different mask regions in the target mask template to obtain a strengthened bill image, and then the key information in the strengthened bill image is extracted based on the form fields (such as the frame number, the commercial insurance premium, and other key fields) in the target prompt word, so as to obtain a bill recognition result.

[0084] It can be seen that, by using the double confidence judgment mechanism, the text regions (i.e., text blocks) recognized by the OCR technology are first subjected to confidence verification, and after the confidence verification is passed, the confidence of the predicted bill type is verified, and then after the confidence verification is passed, the maximum type confidence is determined, and the corresponding mask template and prompt word are determined based on the maximum type confidence and input into the multimodal large language model to strengthen the core field region in the bill image by using the mask template, and extract the key information in the strengthened bill image based on the prompt word. Through the above manner, i.e., the fusion of the double confidence judgment mechanism and the multimodal large language model, the accuracy of bill recognition can be improved, and manual intervention is not required, thereby improving the efficiency of bill recognition and saving labor costs.

[0085] The embodiment of the present application discloses a specific bill recognition method, which is applied to a server, as shown in Figure 4 The method comprises the following steps:

[0086] Step S21: receiving a current bill image uploaded by a client, and using an OCR technology to recognize different types of text regions in the current bill image to obtain a plurality of bill text blocks.

[0087] Step S22: generating the coordinates and the confidence of each bill text block to obtain text block coordinates and text block confidence, and respectively judging whether the text block confidence is greater than a first confidence threshold.

[0088] Step S23: if all the text block confidences are greater than the first confidence threshold, predicting the type of the target bill in the current bill image and the confidence of each type to obtain a plurality of bill types and a plurality of type confidences.

[0089] Step S24: determining whether a maximum value in the plurality of type confidences is greater than or equal to a second confidence threshold value, and if the maximum value is greater than or equal to the second confidence threshold value, determining a target mask template and a target prompt corresponding to a target document type; the target document type is a document type corresponding to the maximum value.

[0090] Step S25: inputting the current document image, the target mask template and the target prompt into a multi-modal large model to perform pixel-by-pixel weighting operation on the current document image and first and second mask regions in the target mask template, to obtain a strengthened document image; a mask value of the first mask region is greater than a mask value of the second mask region.

[0091] In this embodiment, as shown in Figure 5 After the multi-modal large model receives the document image, the mask template and the prompt, it can perform pixel-by-pixel weighting operation on the document image and two mask regions in the target mask template in Figure 5 , to obtain a strengthened document image; wherein the mask regions can be pre-divided by humans according to the document type, the mask value of the region where the important information is located (dark part) can be set to 1 pixel, and the mask value of the region where the secondary information is located (light part) can be set to 0.6-0.8 pixels. The weighting process for the region where the important information is located (dark part) is: the mask value of the important region is 1 pixel -> weighted pixel value = original pixel value x 1 (the important region pixel information can be completely preserved without attenuation); the weighting process for the region where the secondary information is located (light part) is: the mask value of the secondary region is 0.8 pixel -> weighted pixel value = original pixel value x 0.8 (the secondary region pixel information is moderately attenuated to reduce interference).

[0092] Step S26: encoding the strengthened document image using a computer vision encoder to obtain document encoding features, and encoding the form fields in the target prompt using a text feature encoder to obtain prompt encoding features.

[0093] In this embodiment, as shown in Figure 6 There are two branches in the multi-modal large model (i.e. multi-modal key information extraction model), one connected with a computer vision encoder (Vison Transformer, VIT) and the other connected with a text feature encoding module. The Vison Transformer is used to encode the strengthened document image to obtain document encoding features, and the text feature encoding module is used to encode the prompt to obtain prompt encoding features. That is, the visual branch focuses on the mask region (such as the chassis number position of the used car ticket), and the text branch is used to encode the text features of the prompt.

[0094] Step S27: multi-modal fusion of the invoice encoding features and the prompt encoding features through a cross-attention mechanism to obtain fused encoding features.

[0095] In this embodiment, as shown in FIG. 6, the multi-modal fusion of the invoice encoding features and the prompt encoding features can be performed through a cross-attention module to obtain fused encoding features. Figure 6

[0096] Step S28: decoding operation on the fused encoding features to obtain decoded encoding features, and extracting key-value pairs containing the table field and the key field information in the enhanced invoice image corresponding to the table field from the decoded encoding features to obtain an invoice recognition result.

[0097] In this embodiment, as shown in FIG. 6, the multi-modal fusion of the invoice encoding features and the prompt encoding features can be performed through a cross-attention module to obtain fused encoding features. Figure 6

[0098] In this way, the accuracy of key information extraction can be improved: the important areas are strengthened and the secondary areas are weakened through mask weighting, the model is prevented from being disturbed by irrelevant visual information, and the visual branch is accurately focused on the core field area, thereby improving the accuracy of key-value pair recognition. On the other hand, the newly added invoice type can be dynamically adapted: when a new invoice type (such as "car maintenance receipt") is added, the structure of the multi-modal model does not need to be modified, only the important / secondary areas of the invoice need to be manually labeled, a new exclusive mask template is generated and preloaded into the mask library, and the corresponding exclusive prompt is matched, so that the key information extraction of the new invoice can be quickly supported, and the adaptation period can be shortened to 1-2 days.

[0099] Further, the format of the key table field in the invoice recognition result can be verified, and the key information in the invoice recognition result can be verified according to the invoice type and the corresponding rules. For example, the insurance period start and end date logic (start period < end period), the premium amount and the premium amount matching range (such as the compulsory insurance premium meeting the official unified standard), the matching of the second-hand car transaction price tax total and the loan application amount, and the consistency of the insurance premium and the total amount of the installment repayment plan, and when the rules are not met, an abnormality can be marked.

[0100] ​​In addition, for high-risk form fields (such as frame number, premium, amount, etc.), double tamper detection can also be carried out: first, the consistency of the semantic error-corrected OCR information and the extraction result of the multi-modal model (i.e., the document recognition result) is compared, and a warning is generated when the difference exceeds 5%; second, the pixel texture of the field region is analyzed, as shown in Figure 7 As shown in FIG. 9, a texture detection model based on MobileNetV3 can be used, which includes a global pooling module (focusing on global texture) and an attention module (focusing on local texture), and the model parameters are optimized using Focal Loss (a loss function for handling class imbalance problems) + Dice Loss (a loss function commonly used in image segmentation tasks). Through the model, it can be judged whether there is tampering behavior, and if suspected tampering behavior is found, an artificial review process is triggered. By checking the key information with format logic (such as insurance period, amount matching) and double tamper detection (text consistency + texture model detection), the needs of automobile finance risk control can be met.

[0101] In addition, after verification, the structured key-value pairs (i.e., document recognition results) can also be pushed to the automobile finance core system, which archives them and associates the original document image, structured OCR information, and key information key-value pairs when archiving. Specifically, an index can be established according to "customer ID-vehicle VIN code (Vehicle Identification Number)-document type" to support compliance tracing.

[0102] The more specific processing procedures of steps S21 to S24 can be referred to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0103] As can be seen, the embodiments of the present application perform pixel-by-pixel weighted operation on different mask regions in the current document image by using a multi-modal large model to obtain a strengthened document image, encode the strengthened document image using a computer vision encoder to obtain document encoding features, and encode the form fields in the target prompt word using a text feature encoder to obtain prompt word encoding features for multi-modal fusion. The fused encoding features are decoded, and the key-value pairs containing form fields and key fields information are extracted from the decoded encoding features to obtain the document recognition result. By multi-modal fusion of the encoded prompt word encoding features and document encoding features, it is beneficial to accurately extract core key-value pairs in the field of automobile finance, thereby reducing the field extraction error.

[0104] Correspondingly, the embodiments of the present application also disclose a document recognition device applied to a server, as shown in FIG. 10, which comprises: Figure 8 ​

[0105] The receiving and recognizing module 11 is configured to receive a current invoice image uploaded by a client, and recognize different types of text regions in the current invoice image by using an OCR technology, to obtain a plurality of invoice text blocks.

[0106] The generating module 12 is configured to generate coordinates and confidence of each of the invoice text blocks respectively, to obtain text block coordinates and text block confidence.

[0107] The first judging module 13 is configured to judge whether the text block confidence is greater than a first confidence threshold value respectively.

[0108] The predicting module 14 is configured to predict a type of a target invoice and a confidence of each type in the current invoice image if all the text block confidences are greater than the first confidence threshold value, to obtain a plurality of invoice types and a plurality of type confidences.

[0109] The second judging module 15 is configured to judge whether a maximum value in the plurality of type confidences is greater than or equal to a second confidence threshold value.

[0110] The determining module 16 is configured to determine a target mask template and a target prompt word corresponding to a target invoice type if the maximum value is greater than or equal to the second confidence threshold value; the target invoice type is a type of invoice corresponding to the maximum value.

[0111] The invoice recognizing module 17 is configured to input the current invoice image, the target mask template and the target prompt word into a multi-modal large model, to strengthen a core field region in the current invoice image by using the target mask template, to obtain a strengthened invoice image, and to extract key information in the strengthened invoice image based on a form field in the target prompt word, to obtain an invoice recognition result.

[0112] The specific working process of each module can refer to the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0113] It can be seen that, in the embodiment of the application, through the double confidence judgment mechanism, the confidence of the text area (i.e., the text block) recognized by the OCR technology is first verified, and the confidence of the predicted type of the document is verified after the confidence verification passes. Then, after the confidence verification passes, the maximum type confidence is determined, and the corresponding mask template and prompt word are determined based on the maximum type confidence, and are input into the multi-modal large model to strengthen the core field area in the document image using the mask template, and extract the key information in the strengthened document image based on the prompt word. Through the above-mentioned manner, i.e., the fusion of the double confidence judgment mechanism and the multi-modal large model, not only the accuracy of the document recognition can be improved, but also the efficiency of the document recognition is improved without human intervention, thereby saving the labor cost.

[0114] Further, the embodiment of the application also discloses an electronic device, Figure 9 is an electronic device 20 structure diagram shown according to an exemplary embodiment, the contents in the figure cannot be considered as any limitation on the use range of the application.

[0115] Figure 9 A structure diagram of an electronic device 20 is provided in the embodiment of the application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the related steps in the document recognition method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0116] In the embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the application, which is not limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0117] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0118] The operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs capable of performing other specific work in addition to the computer program capable of performing the invoice identification method disclosed by the electronic device 20 in any of the foregoing embodiments.

[0119] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the invoice identification method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0120] Further, the present application also discloses a computer program product comprising computer programs / instructions, which are executed by a processor to implement the steps of the invoice identification method disclosed above.

[0121] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0122] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0123] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0124] Finally, it needs to be pointed out that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0125] The above provides a kind of bill identification method, device, equipment and storage medium provided in the present application in detail, the principle and implementation mode of the present application are described in this paper by applying specific examples, the above example is only for helping to understand the method of the present application and its core idea;Meanwhile, for the person skilled in the art, according to the idea of the present application, there will be changes in specific implementation mode and application range, and the above-mentioned content of the specification should not be understood as the limitation of the present application.

Claims

1. A document recognition method, characterized in that, Applied to the server side, including: Receive the current document image uploaded by the client, and use OCR technology to recognize different types of text regions in the current document image to obtain multiple document text blocks; The coordinates and confidence scores of each document text block are generated respectively. The text block coordinates and text block confidence scores are obtained, and it is determined whether the text block confidence scores are greater than the first confidence score threshold. If the confidence scores of all the text blocks are greater than the first confidence threshold, then the type of the target document in the current document image and the confidence scores of each type are predicted to obtain multiple document types and multiple type confidence scores. Determine whether the maximum value among multiple confidence scores of the stated type is greater than or equal to a second confidence threshold. If the maximum value is greater than or equal to the second confidence threshold, then determine the target mask template and target prompt word corresponding to the target document type; the target document type is the document type corresponding to the maximum value. The current document image, the target mask template, and the target prompt word are input into a multimodal large model to enhance the core field area in the current document image using the target mask template, resulting in an enhanced document image. Key information in the enhanced document image is then extracted based on the form field in the target prompt word to obtain the document recognition result. The step of enhancing the core field region in the current document image using the target mask template to obtain the enhanced document image includes: performing a pixel-by-pixel weighted operation on the current document image and the first mask region and the second mask region in the target mask template to obtain the enhanced document image; the mask value of the first mask region is greater than the mask value of the second mask region.

2. The document identification method according to claim 1, characterized in that, The OCR technology is used to recognize different types of text regions in the current document image, resulting in multiple document text blocks, including: Detect the physical outline of the target document in the current document image, and calculate the completeness of the physical outline to obtain the physical outline completeness. The text region percentage is obtained by using OCR technology to identify the proportion of text regions in the target document, and the text region coverage rate is calculated based on the text region percentage. Determine whether the physical outline integrity is greater than a first threshold and whether the text region coverage is greater than a second threshold; If the physical contour completeness is greater than the first threshold and the text region coverage is greater than the second threshold, then a pre-created large language model is used to identify different types of text regions in the current document image to obtain multiple document text blocks; the large language model is a model based on the neural network structure of EAST, YOLO and CRNN.

3. The document identification method according to claim 2, characterized in that, The process involves using a pre-created large language model to identify different types of text regions in the current document image, resulting in multiple document text blocks, including: The edges of the target document are extracted using the Canny operator to obtain the actual document edges; The actual edge lines corresponding to the actual document edge are detected by Hough transform, and the contour of the target document is fitted based on the actual edge lines to obtain the fitted contour. Calculate the deviation between the physical contour of the target document and the fitted contour, obtain the deviation value, and determine whether the deviation value exceeds a preset threshold. If the deviation value does not exceed the preset threshold, the tilt angle of the actual edge line is calculated by Hough transform, and it is determined whether the tilt angle is 0. If the tilt angle is not 0, the actual edge lines in the current document image are corrected using a bilinear interpolation algorithm to obtain a corrected document image. By using a pre-created large language model, different types of text regions in the corrected document image are identified, resulting in multiple document text blocks.

4. The document identification method according to claim 3, characterized in that, The process involves using a pre-created large language model to identify different types of text regions in the corrected document image, resulting in multiple document text blocks, including: The image quality of the corrected document image is enhanced by a non-uniform illumination image enhancement algorithm that utilizes a fusion feature attention mechanism to obtain an enhanced document image. The enhanced document image is divided into text regions, non-text regions, and adjacent regions between the text regions and the non-text regions; The text region, the non-text region, and the adjacent region are denoised using a first Gaussian filter kernel, a second Gaussian filter kernel, and a third Gaussian filter kernel, respectively, to obtain a denoised document image; wherein the size of the second Gaussian filter kernel is larger than the size of the third Gaussian filter kernel, and the size of the third Gaussian filter kernel is larger than the size of the first Gaussian filter kernel. By using a pre-created large language model, different types of text regions in the denoised document image are identified, resulting in multiple document text blocks.

5. The document identification method according to claim 2, characterized in that, The formula for calculating the physical contour integrity is: ； In the formula, The physical contour integrity. , These are the corner integrity weight and the edge integrity weight, respectively. The number of right-angled corners detected, and Less than or equal to 4; This represents the theoretical distance between any two of the four vertices. This represents the actual length of each detected edge.

6. The document identification method according to claim 2, characterized in that, The calculation of the text region coverage rate based on the text region proportion, to obtain the text region coverage rate, includes: The K-nearest neighbor algorithm is used to aggregate multiple text regions in the target document into K regions, resulting in K aggregated regions. The total area of ​​the target document and the area of ​​each aggregated region are calculated to obtain the total document area and the area of ​​the aggregated region. The total area of ​​the region is obtained by summing the areas of all the aggregated regions, and the ratio of the total area of ​​the region to the total area of ​​the document is calculated to obtain the text region coverage rate.

7. The document identification method according to any one of claims 1 to 6, characterized in that, The step of extracting key information from the enhanced document image based on the form fields in the target prompt words to obtain the document recognition result includes: The enhanced document image is encoded using a computer vision encoder to obtain document encoding features, and the form fields in the target prompt words are encoded using a text feature encoder to obtain prompt word encoding features. The document coding features and the prompt word coding features are fused using a cross-attention mechanism to obtain the fused coding features. The fused encoded features are decoded to obtain decoded encoded features. Key-value pairs containing the form fields and the corresponding key field information in the enhanced document image are extracted from the decoded encoded features to obtain the document recognition result.

8. A document recognition device, characterized in that, Applied to the server side, including: The receiving and recognition module is used to receive the current document image uploaded by the client and use OCR technology to recognize different types of text regions in the current document image to obtain multiple document text blocks; The generation module is used to generate the coordinates and confidence scores of each document text block, thereby obtaining the text block coordinates and text block confidence scores. The first judgment module is used to determine whether the confidence level of the text block is greater than the first confidence threshold. The prediction module is used to predict the type of the target document in the current document image and the confidence of each type if the confidence of all the text blocks is greater than the first confidence threshold, so as to obtain multiple document types and multiple type confidence. The second judgment module is used to determine whether the maximum value among the multiple confidence levels of the types is greater than or equal to the second confidence threshold. The determination module is used to determine the target mask template and target prompt word corresponding to the target document type if the maximum value is greater than or equal to the second confidence threshold; the target document type is the document type corresponding to the maximum value. The document recognition module is used to input the current document image, the target mask template, and the target prompt word into a multimodal large model, so as to enhance the core field area in the current document image using the target mask template to obtain an enhanced document image, and extract key information in the enhanced document image based on the form field in the target prompt word to obtain the document recognition result; The document recognition module is specifically used to perform a pixel-by-pixel weighted operation on the current document image and the first mask region and the second mask region in the target mask template to obtain an enhanced document image; the mask value of the first mask region is greater than the mask value of the second mask region.

9. An electronic device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the document identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the document identification method as described in any one of claims 1 to 7.

Images