A receipt identification method, apparatus, device and medium

Abstract

This application discloses a receipt recognition method, apparatus, device, and medium, relating to the field of receipt content recognition technology. The method includes: acquiring a receipt image through a preset acquisition method, and recognizing the receipt image using a preset optical character recognition model to determine the text content and corresponding location information in the receipt image; matching the receipt image against a preset multimodal large-scale model prompt word library to obtain a first prompt word, and inputting the first prompt word, text content, and location information into the preset multimodal large-scale model to obtain key information key-value pairs; matching the receipt image against a preset language large-scale model prompt word library to obtain a second prompt word, and inputting the second prompt word, text content, location information, and key information key-value pairs into the preset language large-scale model to obtain target key information key-value pairs. The technical solution of this application can improve the accuracy of key field content recognition for non-standard receipts.

Description

Technical Field

[0001] This invention relates to the field of document content recognition technology, and in particular to a receipt recognition method, apparatus, device, and medium. Background Technology

[0002] Receipts are crucial documents for auto finance loan disbursements. With the increasing popularity of used car transactions, there is a growing demand for used car financing loans. However, used car dealerships are generally smaller in scale and have lower levels of standardized management compared to traditional 4S stores, resulting in mostly handwritten receipts with significant format variations. Currently, technologies for automated recognition of non-standard used car receipts mainly include: using traditional OCR (Optical Character Recognition) recognition, standardizing OCR recognition after promoting standardized loan receipts, transferring OCR-recognized content to large-scale language models for recognition, and using multimodal large-scale models for end-to-end image content recognition.

[0003] However, the above methods have their own problems: traditional OCR recognition and standardized OCR recognition after standardizing loan receipts are both prone to errors in receipt recognition; while recognition based on large-scale language models or multimodal large models suffers from high failure rates if the size is too small, and large sizes consume too much storage resources, making it unaffordable for small and medium-sized enterprises. Therefore, providing a solution to these technical problems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a receipt recognition method, apparatus, device, and medium that can significantly improve the accuracy of recognizing key fields of non-standard receipts using a large model with a relatively small size, achieving an accuracy exceeding that of a single large model. The specific solution is as follows:

[0005] Firstly, this application discloses a receipt identification method, including:

[0006] A receipt image is acquired using a preset acquisition method, and the receipt image is then recognized using a preset optical character recognition model to determine the text content in the receipt image and the positional information corresponding to the text content; an attention mechanism module is embedded in the preset optical character recognition model.

[0007] The receipt image is matched against a pre-set multimodal large model prompt word library to obtain a first prompt word. The first prompt word, the text content, and the location information are then input into a pre-set multimodal large model to obtain key information key-value pairs. The pre-set multimodal large model embeds a visual transformer and a cross-attention mechanism module.

[0008] The receipt image is matched against a pre-set language model prompt word library to obtain a second prompt word. The second prompt word, the text content, the location information, and the key information key-value pair are then input into the pre-set language model to obtain the target key information key-value pair.

[0009] Optionally, acquiring the receipt image through a preset acquisition method includes:

[0010] The receipt is photographed from any angle using a terminal device with a shooting function to obtain an initial image, and the initial image is then recognized by a target detection model based on the YOLO algorithm to generate a corresponding recognition result.

[0011] Based on the recognition results, the shooting angle of the initial image is automatically detected by Hough transform, and the shooting angle is rotated and corrected to an upright posture to obtain the receipt image.

[0012] Optionally, the process of acquiring the receipt image through a preset acquisition method further includes:

[0013] When the image quality of the receipt image is detected to be inconsistent with the preset shooting lighting standard, the image is enhanced by using a limited contrast adaptive histogram equalization algorithm.

[0014] Optionally, the step of using a preset optical character recognition model to recognize the receipt image to determine the text content in the receipt image and the location information corresponding to the text content includes:

[0015] The receipt image is identified using optical character recognition technology to obtain a single character in the receipt image and the first position information corresponding to the single character;

[0016] The receipt image is subjected to connected component analysis to obtain multiple connected components, and target connected components with an interval of less than a preset interval are determined from the multiple connected components.

[0017] Based on the single character corresponding to the target connected component and the first position information, the single character and the first position information are merged respectively to obtain the text content in the receipt image and the second position information corresponding to the text content.

[0018] Optionally, the step of inputting the second prompt word, the text content, the location information, and the key information key-value pairs into a preset language model to obtain the target key information key-value pairs includes:

[0019] The second prompt word, the text content, the location information, and the key information key-value pair are input into a preset language model. Based on the second prompt word, the key information key-value pair is updated using the text content according to the location information to obtain the target key information key-value pair.

[0020] Optionally, after inputting the second prompt word, the text content, the location information, and the key information key-value pairs into a preset language model to obtain the target key information key-value pairs, the method further includes:

[0021] The key-value pairs of the target key information are matched and verified sequentially based on key information of geometric distance and key information of positional relationship to generate matching and verification results;

[0022] Based on the matching verification results, the target key information key-value pairs are corrected using a pre-set standardized key information name field thesaurus to obtain corrected key information key-value pairs, and the corrected key information key-value pairs are then matched and verified.

[0023] Optionally, the step of performing matching verification on the target key information key-value pairs based sequentially on geometric distance key information and positional relationship key information to generate a matching verification result includes:

[0024] Determine the centroids corresponding to the keys and values ​​of the target key information key-value pairs, and determine whether the matching degree between the key and the value is less than a preset threshold based on the centroids;

[0025] If the matching degree between the key and the value is less than the preset threshold, a first prompt message is generated so that manual confirmation can be performed based on the first prompt message;

[0026] If the matching degree between the key and the value is not less than the preset threshold, then the angle between the centroid connection line and the image width is obtained, and it is determined whether the angle exceeds the preset angle range; wherein, the centroid connection line is the line connecting the centroid corresponding to the key and the centroid corresponding to the value; the image width is the width of the contour rectangle where the centroid is located, determined based on optical character recognition.

[0027] If the included angle exceeds the preset angle range, a second prompt message is generated so that manual confirmation can be performed based on the second prompt message;

[0028] If the included angle does not exceed the preset angle range, then the step of correcting the target key information key-value pairs using a preset standardized key information name field thesaurus is triggered.

[0029] Secondly, this application discloses a receipt recognition device, comprising:

[0030] The text recognition module is used to acquire a receipt image through a preset acquisition method and to recognize the receipt image using a preset optical character recognition model to determine the text content in the receipt image and the position information corresponding to the text content; the preset optical character recognition model embeds an attention mechanism module;

[0031] The spatial relationship recognition module is used to match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content, and the location information into the preset multimodal large model to obtain key information key-value pairs; the preset multimodal large model embeds a visual transformer and a cross-attention mechanism module;

[0032] The semantic matching and correction module is used to match the receipt image with information in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain the target key information key-value pair.

[0033] Thirdly, this application discloses an electronic device including a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the receipt recognition method as described above.

[0034] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the receipt recognition method as described above.

[0035] This application provides a receipt recognition method, comprising: acquiring a receipt image through a preset acquisition method, and recognizing the receipt image using a preset optical character recognition model to determine the text content in the receipt image and the location information corresponding to the text content; the preset optical character recognition model embeds an attention mechanism module; matching the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and inputting the first prompt word, the text content, and the location information into the preset multimodal large model to obtain key information key-value pairs; the preset multimodal large model embeds a visual transformer and a cross-attention mechanism module; matching the receipt image with information in a preset language large model prompt word library to obtain a second prompt word, and inputting the second prompt word, the text content, the location information, and the key information key-value pairs into the preset language large model to obtain target key information key-value pairs.

[0036] The beneficial technical effects of this application are as follows: By comprehensively utilizing the accurate text recognition capabilities of OCR, the spatial understanding and relationship matching capabilities of a multimodal large-scale model, and the general language understanding and matching correction capabilities of a language large-scale model, it is possible to achieve the accuracy required for receipt content recognition using three smaller-sized models. This significantly reduces the video memory and GPU resources consumed by recognition, improving interface response speed. Since text recognition, spatial relationship recognition, and semantic matching correction are implemented by different models, during the training phase, training can be performed separately to address the shortcomings of each stage. Only the training data for the corresponding stage needs to be prepared, greatly reducing the required training data, the computing power consumed during training, and the video memory occupied, thus significantly improving the speed of the entire training process. As a result, the accuracy of key field content recognition on non-standard receipts is significantly improved, enhancing the overall efficiency of used car loan disbursement.

[0037] In addition, the receipt recognition device, equipment and storage medium provided in this application correspond to the above-mentioned receipt recognition method and have the same effect. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a flowchart of a receipt recognition method disclosed in this application;

[0040] Figure 2This is a schematic diagram of an OCR model structure that incorporates an attention mechanism, as disclosed in this application.

[0041] Figure 3 This is a schematic diagram of a pre-defined multimodal large model structure disclosed in this application;

[0042] Figure 4 This is a schematic diagram of a pre-defined language large model structure disclosed in this application;

[0043] Figure 5 This is a schematic diagram illustrating centroid identification of key-value pairs of target information disclosed in this application.

[0044] Figure 6 This is a schematic diagram illustrating an exemplary preset angle range disclosed in this application;

[0045] Figure 7 This is a schematic diagram of the structure of a receipt recognition device disclosed in this application;

[0046] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] Currently, the main technologies for automatically recognizing non-standard receipts for used cars include the following methods, each with its own problems:

[0049] One approach is to use traditional OCR: pre-set as many templates as possible, and transfer incorrect recognition results to human operators. However, there are a few invoices with minor differences, and the pre-set templates cannot detect obvious errors, resulting in incorrectly recognized content entering the system.

[0050] Secondly, there's the issue of promoting standardized loan receipts followed by standardized OCR recognition. The drawback is that used car dealers are often small and subject to frequent changes, making promotion slow and difficult to achieve widespread coverage. Furthermore, the skill levels of used car dealers vary greatly, leading to instances where ordinary receipts are uploaded as standardized ones, causing recognition errors.

[0051] Thirdly, after recognizing the content using traditional OCR, the recognition process is transferred to a large-scale language model: leveraging the language model's understanding capabilities to extract key information. The drawback is that smaller models (e.g., 7B, 13B) still have insufficient understanding of spatial location, resulting in a higher failure rate for semantic association between information in different lines or with slight intervals. If strong spatial understanding and field relationship matching capabilities are desired, the model size needs to be 60B or even 130B or larger. Larger models significantly reduce the response speed during inference and increase memory usage.

[0052] Fourth, end-to-end image content recognition is achieved using multimodal large models: leveraging the spatial relationship understanding, information extraction, and text recognition capabilities of multimodal large models, target data key-value pairs can be directly extracted from documents. The drawback is that because a single model encompasses OCR, language understanding, and spatial understanding capabilities, a model size of 72 bytes or larger is required to achieve good overall results. However, the model's ability to recognize single text elements (such as handwritten text) is limited. If fine-tuning is desired to improve handwritten text recognition or add customized recognition strategies, fine-tuning models larger than 72 bytes consumes significant GPU (Graphics Processing Unit) resources, making it difficult for small and medium-sized enterprises to afford.

[0053] To this end, this application provides a receipt recognition scheme that can significantly improve the accuracy of recognizing key fields of non-standard receipts by using a large model with a smaller size, achieving the accuracy of a single large model.

[0054] This invention discloses a receipt recognition method, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0055] Step S11: Acquire a receipt image using a preset acquisition method, and use a preset optical character recognition model to recognize the receipt image in order to determine the text content in the receipt image and the location information corresponding to the text content.

[0056] In this embodiment, for receipt materials requiring identification, the corresponding receipt image is acquired through a preset acquisition method. This receipt image can be captured using any device with a shooting function, such as a camera, tablet, or smartphone, and can be rotated from various angles. After capturing the image, it can be uploaded to the server via an app, mini-program, or similar means.

[0057] After acquiring the receipt image, a pre-defined optical character recognition (OCR) model is used to identify the text content and corresponding location information in the receipt image. This pre-defined OCR model is constructed using a combination of traditional OCR recognition and an attention mechanism. Introducing the attention mechanism allows the OCR model to focus on the parts of the image containing important information, reducing noise interference.

[0058] Specifically, the process of using a preset optical character recognition model to recognize the receipt image in order to determine the text content in the receipt image and the location information corresponding to the text content includes the following steps:

[0059] Step 1: Recognize the receipt image using optical character recognition technology to obtain a single character in the receipt image and the first position information corresponding to the single character;

[0060] Step 2: Perform connected component analysis on the receipt image to obtain multiple connected components, and determine the target connected components from the multiple connected components whose interval between adjacent connected components is less than a preset interval;

[0061] Step 3: Based on the single character corresponding to the target connected component and the first position information, merge the single character and the first position information respectively to obtain the text content in the receipt image and the second position information corresponding to the text content.

[0062] First, OCR is used to recognize individual characters in the receipt image, simultaneously obtaining their corresponding location information. Then, connected component analysis is used to merge neighboring characters to obtain word and sentence fragments, while also merging their location information to obtain the final OCR result. That is, the text content and its corresponding location information. It is evident that the values ​​obtained in this step are various scattered text fragments, thus demonstrating strong text recognition capabilities.

[0063] like Figure 2 The diagram shown is an exemplary embodiment of an OCR model structure incorporating an attention mechanism provided in this application. After inputting a receipt image, the image sequentially passes through a convolutional neural network layer, a sequence encoding layer, a multi-head attention mechanism layer, a decoding layer, a fully connected layer, and a SoftMax layer to obtain the final recognized text output.

[0064] Step S12: Match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content, and the location information into a preset multimodal large model to obtain key information key-value pairs.

[0065] In this embodiment, a pre-built multimodal large model prompt word library is used to match corresponding prompt words based on the receipt type of the receipt image—a used car receipt. The prompt words contain key information relevant to the automated decision-making process for this type of receipt. Extracting information based on the receipt type improves recognition accuracy.

[0066] Furthermore, the text content and location information obtained in step S11, along with the matched prompt words, are simultaneously input into a preset multimodal large model. The multimodal large model then outputs the extracted key-value pairs. Each key-value pair consists of a key information name and a field content. For example, customer name - A, product name - B, product details - C, with a clear relationship.

[0067] It should be noted that the preset multimodal large model embeds a VisionTransformer (VIT) model and a cross-attention mechanism module. The VisionTransformer acts as a visual data encoder in the multimodal large model, achieving effective fusion of images with other modal data such as text by applying the Transformer architecture to image processing. The introduction of the cross-attention mechanism further enhances the ability to extract key information.

[0068] like Figure 3 The diagram shown is an exemplary structural schematic of a preset multimodal large model provided in an embodiment of this application. Based on the image acquisition and OCR stages, according to the input receipt type—used car receipt—the corresponding prompt word is matched in the preset multimodal large model prompt word library. The receipt image and prompt word are simultaneously input into the multimodal large model. A visual transformer model is used, and a cross-attention mechanism is introduced to further enhance the ability to extract key information. After encoding and decoding by other internal modules, the extracted key-value pairs (KV) consist of the key information name and field content.

[0069] Step S13: Match the receipt image with information in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information, and the key information key-value pair into the preset language model to obtain the target key information key-value pair.

[0070] In this embodiment, a pre-built language large model prompt word library is used to match the corresponding prompt word based on the receipt type of the receipt image—a used car receipt. The prompt word contains key information for this type of receipt and instructions for merging OCR results with multimodal results.

[0071] Furthermore, based on the key information key-value pairs obtained from the multimodal model, the OCR-recognized data is used as the precise text. A pre-set language model is used to integrate text location information and text semantics to accurately replace key-value pairs, thereby obtaining target key information key-value pairs and achieving accurate recognition and content understanding of complex formats.

[0072] Specifically, the second prompt word, the text content, the location information, and the key information key-value pair are input into a preset language model. Based on the second prompt word, the key information key-value pair is updated using the text content according to the location information to obtain the target key information key-value pair.

[0073] In this embodiment, the text content and location information obtained in step S11, the extraction result (key information key-value pair K-V) output by the preset multimodal large model in step S12, and the second prompt word matched in this step are simultaneously input into the preset language large model. Using the more accurate text recognition results from OCR, the content of the key information key-value pair is updated to obtain the target key information key-value pair K-V'.

[0074] like Figure 4 The diagram shown is an exemplary structural schematic of a preset language big model provided in an embodiment of this application. Based on the OCR results obtained from the image acquisition and OCR stages, the key information key-value pairs extracted by the multimodal big model, and the prompt words matched in the preset language big model prompt word library, are simultaneously input into the language big model. After processing by the corresponding processing modules (embedding module, Transformer module, etc.), the target key information key-value pairs K-V' with content correction are output.

[0075] As can be seen, based on the aforementioned steps, by comprehensively utilizing the precise text recognition capabilities of OCR, the spatial understanding and relationship matching capabilities of the multimodal large model, and the general language understanding and matching correction capabilities of the language large model, the accuracy of a single large model exceeding 70B is achieved with three smaller models (e.g., 0.1B+7B+7B), reaching 97.4%. This significantly reduces the memory and GPU resources required for recognition and improves interface response speed. Furthermore, experiments have shown that this invention can significantly improve the accuracy of recognizing key fields in non-standard receipts, thereby increasing the overall efficiency of used car loan disbursement and shortening the average loan disbursement time by 25 minutes.

[0076] This application provides a receipt recognition method, comprising: acquiring a receipt image through a preset acquisition method, and recognizing the receipt image using a preset optical character recognition model to determine the text content in the receipt image and the location information corresponding to the text content; the preset optical character recognition model embeds an attention mechanism module; matching the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and inputting the first prompt word, the text content, and the location information into the preset multimodal large model to obtain key information key-value pairs; the preset multimodal large model embeds a visual transformer and a cross-attention mechanism module; matching the receipt image with information in a preset language large model prompt word library to obtain a second prompt word, and inputting the second prompt word, the text content, the location information, and the key information key-value pairs into the preset language large model to obtain target key information key-value pairs.

[0077] The beneficial technical effects of this application are as follows: By comprehensively utilizing the accurate text recognition capabilities of OCR, the spatial understanding and relationship matching capabilities of a multimodal large-scale model, and the general language understanding and matching correction capabilities of a language large-scale model, it is possible to achieve the accuracy required for receipt content recognition using three smaller-sized models. This significantly reduces the video memory and GPU resources consumed by recognition, improving interface response speed. Since text recognition, spatial relationship recognition, and semantic matching correction are implemented by different models, during the training phase, training can be performed separately to address the shortcomings of each stage. Only the training data for the corresponding stage needs to be prepared, greatly reducing the required training data, the computing power consumed during training, and the video memory occupied, thus significantly improving the speed of the entire training process. As a result, the accuracy of key field content recognition on non-standard receipts is significantly improved, enhancing the overall efficiency of used car loan disbursement.

[0078] Based on the above embodiments, this embodiment will specifically describe S11 in the above embodiments. Specifically, to ensure correct reading of the receipt image and good image quality, the process of acquiring the receipt image through a preset acquisition method may include the following steps:

[0079] Step 1: Use a terminal device with a shooting function to take a picture of the receipt from any angle to obtain an initial image, and use a target detection model based on the YOLO algorithm to identify the initial image to generate the corresponding recognition result;

[0080] Step 2: Based on the recognition results, the shooting angle of the initial image is automatically detected by Hough transform, and the shooting angle is rotated and corrected to an upright posture to obtain the receipt image.

[0081] Step 3: When the image quality of the receipt image does not meet the preset shooting lighting standard, the image is enhanced using a limited contrast adaptive histogram equalization algorithm.

[0082] In this embodiment, a terminal device with shooting capabilities, such as a camera, tablet computer, or smartphone, is used to capture the receipt from any angle to obtain an initial image. That is, the user is allowed to operate the image from any angle during shooting, accepting rotation of the image from various angles.

[0083] After acquiring the initial image, the YOLO (You Only Look Once) object detection model is used to identify the image. Corner and border detection ensures image clarity and integrity. For incomplete images, an error message is returned, requiring a re-capture and upload.

[0084] For a complete initial image, the server automatically detects horizontal or vertical lines in the image using Hough transform to determine the image's shooting angle. Based on the shooting angle, the initial image is rotated and corrected to restore it to an upright position, resulting in the final receipt image and ensuring its correct reading.

[0085] It is worth noting that for cases where the image quality of the receipt image does not meet the preset shooting lighting standards, such as when the flash is turned on during shooting due to dim indoor lighting, resulting in uneven image lighting, the Contrast Limited Adaptive Histogram Equalization algorithm is used to improve image quality.

[0086] Based on the above embodiments, in one feasible implementation, the process of correcting the target key information key-value pairs to complete the entire document recognition process is carried out. Specifically, after inputting the second prompt word, the text content, the location information, and the key information key-value pairs into a preset language model to obtain the target key information key-value pairs, the following steps may also be included:

[0087] The key-value pairs of the target key information are matched and verified sequentially based on key information of geometric distance and key information of positional relationship to generate matching and verification results;

[0088] Based on the matching verification results, the target key information key-value pairs are corrected using a pre-set standardized key information name field thesaurus to obtain corrected key information key-value pairs, and the corrected key information key-value pairs are then matched and verified.

[0089] In this embodiment, firstly, a matching verification is performed based on key geometric distance information. Positions are obtained using OCR recognition, and the centroids of the corresponding positions are calculated. If the matching degree is less than a threshold, manual confirmation is prompted; otherwise, the process continues into the subsequent automatic workflow.

[0090] Specifically, the centroids corresponding to the keys and values ​​of the target key information key-value pairs are determined, and the matching degree between the key and the value is judged based on the centroids to see if it is less than a preset threshold; if the matching degree between the key and the value is less than the preset threshold, a first prompt message is generated so that manual confirmation can be performed based on the first prompt message.

[0091] like Figure 5 The diagram illustrates an exemplary method for obtaining location information based on OCR and calculating the centroid of each key value corresponding to that location. The formula for calculating the matching degree is as follows: ;

[0092] The Euclidean distance between K and V is calculated using the location information of the OCR recognition results corresponding to the key information name and content. Since the outline of the location information currently recognized by OCR is generally rectangular, the distance is obtained by calculating the centroid of the two.

[0093] It is an image overall resolution normalization parameter, which solves the problem that the higher the image resolution, the greater the pixel distance between two identical fields on the same document, resulting in a larger geometric distance. ;in, for Figure 5 The angle between the centroid line and the image width; h is the pixel value of the image height; w is the pixel value of the image width.

[0094] Secondly, matching and verification are performed based on key information about positional relationships, including the angle between the centroid line and the wider side. Set the angle range according to the common ticket type, and set the preset angle range to... ,like Figure 6 As shown, if the value is outside the range, a manual confirmation prompt will be displayed.

[0095] Specifically, if the matching degree between the key and the value is not less than the preset threshold, the process continues to the subsequent automatic process. The angle between the centroid line and the image width is obtained, and it is determined whether the angle exceeds a preset angle range; wherein, the centroid line is the line connecting the centroids corresponding to the key and the value; the image width is the width of the contour rectangle containing the centroids as determined based on optical character recognition; if the angle exceeds the preset angle range, a second prompt message is generated for manual confirmation based on the second prompt message.

[0096] Furthermore, if the included angle does not exceed the preset angle range, the step of correcting the target key information key-value pair using a preset standardized key information name field thesaurus is triggered. In this embodiment, when the included angle between the centroid line and the wide side conforms to the set angle range for a conventional document type, the key information in the target key information key-value pair K-V' is matched and replaced using a preset standardized key information name field thesaurus to obtain the corrected K'-V' key-value pair. Finally, for a few core contents of the corrected key information key-value pair K'-V', a one-to-one precise verification is performed with the existing system through the backend API interface (Application Program Interface). If the verification fails, manual processing is prompted; if successful, the entire document recognition process is completed, and the information is output to the subsequent processing flow.

[0097] Accordingly, this application also discloses a receipt recognition device, see [link to relevant documentation]. Figure 7 As shown, the device includes:

[0098] The text recognition module 11 is used to acquire a receipt image through a preset acquisition method and to recognize the receipt image using a preset optical character recognition model to determine the text content in the receipt image and the position information corresponding to the text content; the preset optical character recognition model embeds an attention mechanism module.

[0099] The spatial relationship recognition module 12 is used to match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content and the location information into the preset multimodal large model to obtain key information key-value pairs; the preset multimodal large model embeds a visual transformer and a cross-attention mechanism module;

[0100] The semantic matching and correction module 13 is used to perform information matching on the receipt image in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain the target key information key-value pair.

[0101] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0102] Therefore, the above-described scheme in this embodiment acquires a receipt image through a preset acquisition method and uses a preset optical character recognition model to recognize the receipt image to determine the text content and the corresponding location information in the receipt image. An attention mechanism module is embedded in the preset optical character recognition model. The receipt image is matched against a preset multimodal large-scale model prompt word library to obtain a first prompt word. The first prompt word, the text content, and the location information are then input into the preset multimodal large-scale model to obtain key information key-value pairs. A visual transformer and a cross-attention mechanism module are embedded in the preset multimodal large-scale model. The receipt image is matched against a preset language large-scale model prompt word library to obtain a second prompt word. The second prompt word, the text content, the location information, and the key information key-value pairs are then input into the preset language large-scale model to obtain target key information key-value pairs.

[0103] The beneficial technical effects of this application are as follows: By comprehensively utilizing the accurate text recognition capabilities of OCR, the spatial understanding and relationship matching capabilities of a multimodal large-scale model, and the general language understanding and matching correction capabilities of a language large-scale model, it is possible to achieve the accuracy required for receipt content recognition using three smaller-sized models. This significantly reduces the video memory and GPU resources consumed by recognition, improving interface response speed. Since text recognition, spatial relationship recognition, and semantic matching correction are implemented by different models, during the training phase, training can be performed separately to address the shortcomings of each stage. Only the training data for the corresponding stage needs to be prepared, greatly reducing the required training data, the computing power consumed during training, and the video memory occupied, thus significantly improving the speed of the entire training process. As a result, the accuracy of key field content recognition on non-standard receipts is significantly improved, enhancing the overall efficiency of used car loan disbursement.

[0104] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0105] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the receipt recognition method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0106] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0107] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0108] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the receipt recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0109] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. The computer program, when executed by a processor, implements the aforementioned receipt recognition method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0111] The steps of the receipt recognition method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0112] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The above provides a detailed description of a receipt identification method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A receipt recognition method, characterized in that, include: The receipt image is acquired by a preset acquisition method, and the receipt image is identified by a preset optical character recognition model to determine the text content in the receipt image and the location information corresponding to the text content. The preset optical character recognition model embeds an attention mechanism module; The receipt image is matched with information in a preset multimodal large model prompt word library to obtain a first prompt word. The first prompt word, the text content and the location information are then input into a preset multimodal large model to obtain key information key-value pairs. The preset multimodal large model embeds a visual transformer and a cross-attention mechanism module; The receipt image is matched with information in a preset language model prompt word library to obtain a second prompt word. The second prompt word, the text content, the location information, and the key information key-value pair are then input into the preset language model to obtain the target key information key-value pair. The second prompt word includes key information from the receipt image and instructions for merging OCR results and multimodal results; the step of inputting the second prompt word, the text content, the location information, and the key information key-value pairs into a preset language model to obtain the target key information key-value pairs includes: The second prompt word, the text content, the location information, and the key information key-value pair are input into a preset language model. Based on the second prompt word, the key information key-value pair is updated using the text content according to the location information to obtain the target key information key-value pair.

2. The receipt recognition method according to claim 1, characterized in that, The process of acquiring the receipt image through a preset acquisition method includes: The receipt is photographed from any angle using a terminal device with a shooting function to obtain an initial image, and the initial image is then recognized by a target detection model based on the YOLO algorithm to generate a corresponding recognition result. Based on the recognition results, the shooting angle of the initial image is automatically detected by Hough transform, and the shooting angle is rotated and corrected to an upright posture to obtain the receipt image.

3. The receipt recognition method according to claim 1, characterized in that, The process of acquiring the receipt image through a preset acquisition method also includes: When the image quality of the receipt image is detected to be inconsistent with the preset shooting lighting standard, the image is enhanced by using a limited contrast adaptive histogram equalization algorithm.

4. The receipt recognition method according to claim 1, characterized in that, The step of using a preset optical character recognition model to recognize the receipt image to determine the text content in the receipt image and the location information corresponding to the text content includes: The receipt image is identified using optical character recognition technology to obtain a single character in the receipt image and the first position information corresponding to the single character; The receipt image is subjected to connected component analysis to obtain multiple connected components, and target connected components with an interval of less than a preset interval are determined from the multiple connected components. Based on the single character corresponding to the target connected component and the first position information, the single character and the first position information are merged respectively to obtain the text content in the receipt image and the second position information corresponding to the text content.

5. The receipt identification method according to any one of claims 1 to 4, characterized in that, After inputting the second prompt word, the text content, the location information, and the key information key-value pair into a preset language model to obtain the target key information key-value pair, the method further includes: The key-value pairs of the target key information are matched and verified sequentially based on key information of geometric distance and key information of positional relationship to generate matching and verification results; Based on the matching verification results, the target key information key-value pairs are corrected using a pre-set standardized key information name field thesaurus to obtain corrected key information key-value pairs, and the corrected key information key-value pairs are then matched and verified.

6. The receipt recognition method according to claim 5, characterized in that, The matching and verification of the target key information key-value pairs based on geometric distance key information and positional relationship key information in sequence to generate matching and verification results includes: Determine the centroids corresponding to the keys and values ​​of the target key information key-value pairs, and determine whether the matching degree between the key and the value is less than a preset threshold based on the centroids; If the matching degree between the key and the value is less than the preset threshold, a first prompt message is generated so that manual confirmation can be performed based on the first prompt message; If the matching degree between the key and the value is not less than the preset threshold, then the angle between the centroid connection line and the image width is obtained, and it is determined whether the angle exceeds the preset angle range; wherein, the centroid connection line is the line connecting the centroid corresponding to the key and the centroid corresponding to the value; the image width is the width of the contour rectangle where the centroid is located, determined based on optical character recognition. If the included angle exceeds the preset angle range, a second prompt message is generated so that manual confirmation can be performed based on the second prompt message; If the included angle does not exceed the preset angle range, then the step of correcting the target key information key-value pairs using a preset standardized key information name field thesaurus is triggered.

7. A receipt recognition device, characterized in that, include: The text recognition module is used to acquire a receipt image through a preset acquisition method and to recognize the receipt image using a preset optical character recognition model in order to determine the text content in the receipt image and the location information corresponding to the text content. The preset optical character recognition model embeds an attention mechanism module; The spatial relationship recognition module is used to match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content, and the location information into the preset multimodal large model to obtain key information key-value pairs; the preset multimodal large model embeds a visual transformer and a cross-attention mechanism module; The semantic matching and correction module is used to match the receipt image with information in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain the target key information key-value pair; The second prompt includes key information from the receipt image and instructions for merging OCR results with multimodal results; the semantic matching correction module is specifically used for: The second prompt word, the text content, the location information, and the key information key-value pair are input into a preset language model. Based on the second prompt word, the key information key-value pair is updated using the text content according to the location information to obtain the target key information key-value pair.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the receipt recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein the computer program, when executed by a processor, implements the receipt recognition method as described in any one of claims 1 to 6.

Images