Receipt identification method, apparatus and device, and medium

Abstract

The invention discloses a receipt identification method and device, equipment and a medium, and relates to the technical field of bill content identification. The method comprises the following steps: acquiring a receipt image through a preset acquisition mode, and identifying the receipt image by using a preset optical character identification model to determine text content and corresponding position information in the receipt image; performing information matching on the receipt image in a preset multi-mode large model prompt word bank to obtain a first prompt word, and inputting the first prompt word, the text content and the position information into a preset multi-mode large model to obtain a key information key value pair; and performing information matching on the receipt image in a preset language large model prompt word library to obtain a second prompt word, and inputting the second prompt word, the text content, the position information and the key information key value pair into a preset language large model to obtain a target key information key value pair. Through the technical scheme of the invention, the accuracy of identifying the key field content of the non-standard receipt can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bill content recognition, and in particular to a receipt recognition method, device, equipment and medium. Background Art

[0002] Receipts are key materials for auto finance loans. With the popularity of used car transactions, there is an increasing demand for used car finance loans. However, used car dealers are smaller than traditional 4S stores, with a lower degree of management standardization. Receipts are mostly handwritten and have large format differences. At present, the technologies for automatic recognition of some non-standard receipts for used cars mainly include: using traditional OCR (Optical Character Recognition) recognition, promoting standardized loan receipts and then performing standardized OCR recognition, transferring OCR recognition content to large-scale language models for recognition, and using multimodal large models for end-to-end image content recognition.

[0003] However, the above methods have corresponding problems: the traditional OCR recognition and the standardized OCR recognition after the standardized loan receipt are both prone to receipt recognition errors; and when using a large-scale language model or a multimodal large model for recognition, if a smaller size is used, the recognition failure rate will also be high, and if a larger size is used, it will consume a lot of storage resources and make it difficult for small and medium-sized enterprises to afford it. Therefore, how to provide a solution to the above technical problems is a problem that technicians in this field currently need to solve. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a receipt recognition method, device, equipment and medium, which can greatly improve the accuracy of non-standard receipt key field content recognition through a small-sized large model, and achieve the effect of exceeding the accuracy of a single large-sized large model. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a receipt recognition method, comprising:

[0006] Acquire a receipt image by a preset acquisition method, and recognize the receipt image by using a preset optical character recognition model to determine text content in the receipt image and position information corresponding to the text content; the preset optical character recognition model is embedded with an attention mechanism module;

[0007] Matching the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and inputting the first prompt word, the text content and the location information into a preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross attention mechanism module;

[0008] The receipt image is matched with information in a preset language model prompt word library to obtain a second prompt word, and the second prompt word, the text content, the location information and the key information key-value pair are input into the preset language model to obtain a target key information key-value pair.

[0009] Optionally, acquiring the receipt image by a preset acquisition method includes:

[0010] Using a terminal device with a photographing function to photograph the receipt at any angle to obtain an initial image, and using a target detection model based on the YOLO algorithm to recognize the initial image to generate a corresponding recognition result;

[0011] Based on the recognition result, the shooting angle of the initial image is automatically detected through Hough transformation, and the shooting angle is rotated and corrected to an upright posture to obtain the receipt image.

[0012] Optionally, the process of acquiring the receipt image by a preset acquisition method further includes:

[0013] When it is detected that the image quality of the receipt image does not meet the preset shooting illumination standard, the receipt image is enhanced by using a contrast-limited adaptive histogram equalization algorithm.

[0014] Optionally, the using a preset optical character recognition model to recognize the receipt image to determine text content in the receipt image and position information corresponding to the text content includes:

[0015] Recognize the receipt image by optical character recognition technology to obtain a single character in the receipt image and first position information corresponding to the single character;

[0016] Performing a connected domain analysis process on the receipt image to obtain a plurality of connected domains, and determining a target connected domain from the plurality of connected domains, wherein the interval between adjacent connected domains is less than a preset interval;

[0017] According to the single character and the first position information corresponding to the target connected domain, the single character and the first position information are merged respectively to obtain the text content in the receipt image and the second position information corresponding to the text content.

[0018] Optionally, the step of inputting the second prompt word, the text content, the location information, and the key information key-value pair into a preset language macro model to obtain a target key information key-value pair includes:

[0019] The second prompt word, the text content, the location information and the key information key-value pair are input into a preset language model, and based on the second prompt word, the key information key value is updated using the text content according to the location information to obtain a target key information key-value pair.

[0020] Optionally, after inputting the second prompt word, the text content, the location information and the key information key-value pair into a preset language macro model to obtain a target key information key-value pair, the method further includes:

[0021] Performing matching verification on the target key information key-value pair based on the geometric distance key information and the position relationship key information in sequence to generate a matching verification result;

[0022] According to the matching verification result, the target key information key-value pair is corrected using a preset standardized key information name field word library to obtain a corrected key information key-value pair, and a matching verification is performed on the corrected key information key-value pair.

[0023] Optionally, performing matching verification on the target key information key-value pair based on the geometric distance key information and the position relationship key information in sequence to generate a matching verification result includes:

[0024] Determine the centroid corresponding to the key and value of the target key information key-value pair, and determine whether the matching degree between the key and the value is less than a preset threshold according to the centroid;

[0025] If the matching degree between the key and the value is less than the preset threshold, generating first prompt information so as to perform manual confirmation according to the first prompt information;

[0026] If the matching degree between the key and the value is not less than the preset threshold, the angle between the centroid line and the image wide side is obtained, and it is determined whether the angle exceeds the preset angle range; wherein the centroid line is a line between the centroid corresponding to the key and the centroid corresponding to the value; and the image wide side is a wide side of the outline rectangle where the centroid is located, determined based on optical character recognition;

[0027] If the included angle exceeds the preset angle range, a second prompt message is generated so as to perform manual confirmation according to the second prompt message;

[0028] If the angle does not exceed the preset angle range, the step of correcting the target key information key-value pair using a preset standardized key information name field vocabulary is triggered.

[0029] In a second aspect, the present application discloses a receipt recognition device, comprising:

[0030] A text recognition module, used to obtain a receipt image by a preset acquisition method, and recognize the receipt image by using a preset optical character recognition model to determine the text content in the receipt image and position information corresponding to the text content; the preset optical character recognition model is embedded with an attention mechanism module;

[0031] A spatial relationship recognition module, used to match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content and the location information into a preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross attention mechanism module;

[0032] The semantic matching correction module is used to match the information of the receipt image in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain a target key information key-value pair.

[0033] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the receipt recognition method as described above.

[0034] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the receipt recognition method as described above.

[0035] The present application provides a receipt recognition method, comprising: acquiring a receipt image through a preset acquisition method, and using a preset optical character recognition model to recognize the receipt image to determine the text content in the receipt image and the position information corresponding to the text content; the preset optical character recognition model is embedded with an attention mechanism module; matching the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and inputting the first prompt word, the text content and the position information into the preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross-attention mechanism module; matching the receipt image with information in a preset language large model prompt word library to obtain a second prompt word, and inputting the second prompt word, the text content, the position information and the key information key-value pair into the preset language large model to obtain a target key information key-value pair.

[0036] The beneficial technical effects of the present application are: by comprehensively utilizing the precise text recognition capability of OCR, the spatial understanding and relationship matching capabilities of the multimodal large model, and the universal language understanding and matching correction capabilities of the language large model, three smaller-sized models can be used to achieve the accuracy of receipt content recognition that can be completed by a single large-sized model. Greatly reduce the video memory and GPU resources occupied by recognition, and improve the interface response speed. Since the three links of text recognition, spatial relationship recognition, and semantic matching correction are respectively implemented by different models. Therefore, in the training stage, different links can be trained separately, and only the training data of the corresponding links needs to be prepared. Whether it is the required training data, the computing power consumed during training, or the video memory occupied, they are greatly reduced, which greatly improves the speed of the entire training link. In this way, the accuracy of content recognition of key fields of non-standard receipts is greatly improved, and the overall loan efficiency of used cars is improved.

[0037] In addition, the present application provides a receipt recognition device, equipment and storage medium, which correspond to the above-mentioned receipt recognition method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flow chart of a receipt recognition method disclosed in this application;

[0040] Figure 2A schematic diagram of the structure of an OCR model that introduces an attention mechanism disclosed in this application;

[0041] Figure 3 A schematic diagram of a preset multi-modal large model structure disclosed in this application;

[0042] Figure 4 A schematic diagram of a large model structure of a preset language disclosed in this application;

[0043] Figure 5 A schematic diagram of centroid identification of a target key information key-value pair disclosed in this application;

[0044] Figure 6 This is a schematic diagram of an exemplary preset angle range disclosed in this application;

[0045] Figure 7 This is a schematic diagram of the structure of a receipt recognition device disclosed in this application;

[0046] Figure 8 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Currently, there are several methods for automatic recognition of non-standard receipts for used cars, and there are corresponding problems:

[0049] The first is to use traditional OCR: pre-set as many templates as possible, and transfer to manual after recognition errors. However, there are a small number of bills with small differences, and the preset templates cannot find obvious errors, resulting in incorrectly recognized content entering the system;

[0050] The second is to promote standardized loan receipts and then perform standardized OCR recognition: the disadvantage is that the second-hand car dealers are small and change a lot, so the promotion is not timely and it is difficult to cover. In addition, the level of second-hand car practitioners varies, and there are cases where ordinary receipts are uploaded as standardized receipts, resulting in recognition errors;

[0051] The third is to use traditional OCR to recognize content and then transfer the recognition to a large-scale language model: use the understanding ability of the language model to extract and recognize key information. The disadvantage is that the large model of smaller size (such as 7B, 13B) still has insufficient understanding of spatial position, and the semantic association failure rate of information in different rows or with a little gap is high. If you expect to have strong spatial understanding ability and field relationship matching ability, the model size needs to be 60B or even 130B or more. A larger model will greatly reduce the response speed of the model during inference and increase the memory usage;

[0052] Fourth, use a large multimodal model for end-to-end image content recognition: use the multimodal large model's comprehensive capabilities such as spatial relationship understanding, information extraction, and text recognition to directly extract target data key-value pairs from the bill. The disadvantage is that since a single model covers OCR, language understanding, spatial understanding and other capabilities, in order to improve the overall effect, the model size must be larger than 72B to achieve better results. However, the model's single capability of text recognition (such as user handwritten content recognition) has limited recognition capabilities. If you want to improve the handwritten font recognition effect through training fine-tuning, or add some customized recognition strategies, fine-tuning of models larger than 72B consumes a lot of GPU (Graphics Processing Unit) resources, which is difficult for small and medium-sized enterprises to afford.

[0053] To this end, the present application provides a receipt recognition solution that can significantly improve the accuracy of identifying the content of key fields of non-standard receipts through a large model of smaller size, achieving an effect of accuracy exceeding that of a single large model of larger size.

[0054] The embodiment of the present invention discloses a receipt recognition method, see Figure 1 As shown, the method includes:

[0055] Step S11: Acquire a receipt image by a preset acquisition method, and recognize the receipt image by using a preset optical character recognition model to determine the text content in the receipt image and the position information corresponding to the text content.

[0056] In the embodiment of the present application, for the receipt material to be identified, the corresponding receipt image is obtained through a preset acquisition method. The receipt image can be photographed by any device with a shooting function such as a camera, a tablet computer or a smart phone, and the image can be rotated at various angles. After the shooting is completed, the image can be uploaded to the server through an APP, a mini program, etc.

[0057] After acquiring the receipt image, the preset optical character recognition model is used to recognize the text content and corresponding location information in the receipt image. The preset optical character recognition model is built based on traditional OCR recognition and the attention mechanism. The introduction of the attention mechanism enables the OCR model to focus on the part of the image that contains important information and reduce noise interference.

[0058] Specifically, the process of using a preset optical character recognition model to recognize the receipt image to determine the text content in the receipt image and the position information corresponding to the text content specifically includes the following steps:

[0059] Step 1: Recognize the receipt image by optical character recognition technology to obtain a single character in the receipt image and first position information corresponding to the single character;

[0060] Step 2: Performing a connected domain analysis process on the receipt image to obtain a plurality of connected domains, and determining a target connected domain from the plurality of connected domains where the interval between adjacent connected domains is less than a preset interval;

[0061] Step 3: According to the single character and the first position information corresponding to the target connected domain, the single character and the first position information are merged respectively to obtain the text content in the receipt image and the second position information corresponding to the text content.

[0062] First, use OCR to recognize individual characters in the receipt image and obtain their corresponding position information. Then, use connected domain analysis to merge adjacent characters to obtain text fragments of words and sentences, and merge their position information to obtain the final OCR result. That is, the text content and the second position information corresponding to the text content. It can be seen that the values ​​obtained in this step are various scattered text fragments, so the text recognition ability is strong.

[0063] like Figure 2 The figure shows a schematic diagram of the structure of an OCR model that introduces an attention mechanism as exemplarily provided in an embodiment of the present application. After the receipt image is input, it passes through the convolutional neural network layer, sequence encoding layer, multi-head attention mechanism layer, decoding layer, fully connected layer and SoftMax layer in sequence to obtain the final recognized text output.

[0064] Step S12: Match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content and the location information into the preset multimodal large model to obtain a key information key-value pair.

[0065] In the embodiment of the present application, in the preset multimodal large model prompt word library, the corresponding prompt word is matched according to the bill type of the receipt image - used car receipt. The prompt word contains the key information that the automated decision-making focuses on for this type of bill. Extracting information according to the bill type can improve the recognition accuracy.

[0066] Furthermore, the text content and location information obtained in step S11 are simultaneously input into the preset multimodal macro model with the matched prompt words, and the key information key-value pairs extracted are outputted by the multimodal macro model. The key information key-value pairs are key-value pairs KV consisting of key information name + field content. For example, customer name - A, product name - B, product details - C, and the relationship corresponds accurately.

[0067] It should be pointed out that the Vision Transformer (VIT) model and the Cross-Attention mechanism module are embedded in the preset multimodal large model; the Vision Transformer acts as a visual data encoder in the multimodal large model, and by applying the Transformer architecture to image processing, it realizes the effective fusion of images and other modal data such as text. The introduction of the cross-attention mechanism further improves the ability to extract key information.

[0068] like Figure 3 The figure shows a schematic diagram of the structure of a preset multimodal large model provided as an example in an embodiment of the present application. Based on the image acquisition and OCR stage, the corresponding prompt word is matched in the preset multimodal large model prompt word library according to the input bill type - used car receipt. The receipt image and the prompt word are simultaneously input into the multimodal large model, and the visual transformer model is used to introduce the cross attention mechanism to further improve the key information extraction capability. The key-value pair KV consisting of the key information name + field content is extracted through the encoding and decoding output of other internal modules.

[0069] Step S13: Match the receipt image with information in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain a target key information key-value pair.

[0070] In the embodiment of the present application, in the preset language model prompt word library, the corresponding prompt word is matched according to the bill type of the receipt image - used car receipt. The prompt word contains key information for this type of bill and the instruction requirement for merging the OCR result with the multimodal result.

[0071] Furthermore, based on the key information key-value pairs obtained by multimodal model recognition, the OCR recognition results are used as precise text, and the preset language large model is used to integrate the text position information and text semantics to accurately replace the key-value pair information, obtain the target key information key-value pairs, and achieve precise recognition of complex formats and content understanding.

[0072] Specifically, the second prompt word, the text content, the location information and the key information key-value pair are input into a preset language model, and based on the second prompt word, the key information key value is updated using the text content according to the location information to obtain a target key information key-value pair.

[0073] In the embodiment of the present application, the text content and location information obtained in step S11 + the extraction result (key information key-value pair KV) output by the preset multimodal large model in step S12 + the second prompt word matched in this step are simultaneously input into the preset language large model. Using the more accurate text recognition result of OCR, the content of the key information key-value pair is updated to obtain the target key information key-value pair K-V'.

[0074] like Figure 4 The structure diagram of a preset language model provided by an embodiment of the present application is shown. Based on the OCR results obtained in the image acquisition and OCR stages, the key information key-value pairs extracted by the multimodal model, and the prompt words matched in the preset language model prompt word library are simultaneously input into the language model, and after processing by the corresponding processing modules (embedding module, Transformer module, etc.), the target key information key-value pairs K-V' after content correction are output.

[0075] It can be seen that based on the content in the above steps, the precise text recognition ability of OCR, the spatial understanding and relationship matching ability of the multimodal large model, and the general language understanding ability and matching correction ability of the language large model are comprehensively utilized, and the accuracy of a single large model exceeding 70B is achieved with three smaller models (such as 0.1B+7B+7B), reaching 97.4%, which greatly reduces the video memory and GPU resources occupied by recognition and improves the interface response speed. At the same time, according to experiments, the present invention can greatly improve the accuracy of the content recognition of key fields of non-standard receipts, improve the overall loan efficiency of used cars, and shorten the average loan time by 25 minutes.

[0076] The present application provides a receipt recognition method, comprising: acquiring a receipt image through a preset acquisition method, and using a preset optical character recognition model to recognize the receipt image to determine the text content in the receipt image and the position information corresponding to the text content; the preset optical character recognition model is embedded with an attention mechanism module; matching the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and inputting the first prompt word, the text content and the position information into the preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross-attention mechanism module; matching the receipt image with information in a preset language large model prompt word library to obtain a second prompt word, and inputting the second prompt word, the text content, the position information and the key information key-value pair into the preset language large model to obtain a target key information key-value pair.

[0077] The beneficial technical effects of the present application are: by comprehensively utilizing the precise text recognition capability of OCR, the spatial understanding and relationship matching capabilities of the multimodal large model, and the universal language understanding and matching correction capabilities of the language large model, three smaller-sized models can be used to achieve the accuracy of receipt content recognition that can be completed by a single large-sized model. Greatly reduce the video memory and GPU resources occupied by recognition, and improve the interface response speed. Since the three links of text recognition, spatial relationship recognition, and semantic matching correction are respectively implemented by different models. Therefore, in the training stage, different links can be trained separately, and only the training data of the corresponding links needs to be prepared. Whether it is the required training data, the computing power consumed during training, or the video memory occupied, they are greatly reduced, which greatly improves the speed of the entire training link. In this way, the accuracy of content recognition of key fields of non-standard receipts is greatly improved, and the overall loan efficiency of used cars is improved.

[0078] Based on the above embodiment, this embodiment will specifically explain S11 in the above embodiment. In order to ensure the correct reading of the receipt image and have a good image quality, the process of obtaining the receipt image by the preset acquisition method may include the following steps:

[0079] Step 1: Use a terminal device with a shooting function to shoot the receipt at any angle to obtain an initial image, and recognize the initial image through a target detection model based on the YOLO algorithm to generate a corresponding recognition result;

[0080] Step 2: Based on the recognition result, the shooting angle of the initial image is automatically detected by Hough transform, and the shooting angle is rotated and corrected to an upright posture to obtain the receipt image.

[0081] Step 3: When it is detected that the image quality of the receipt image does not meet the preset shooting illumination standard, the receipt image is enhanced by using a contrast-constrained adaptive histogram equalization algorithm.

[0082] In the embodiment of the present application, a terminal device with a camera function such as a camera, a tablet computer or a smart phone is used to shoot the receipt at any angle to obtain an initial image. That is, the user is allowed to operate at any angle during shooting and accept the rotation of the image at various angles.

[0083] After obtaining the initial image, the initial image is recognized based on the YOLO (You Only Look Once) target detection model. The image clarity and integrity are ensured through corner and border detection. For incomplete images, an error message is returned and the user is required to retake the image and upload it.

[0084] For the complete initial image, the image is stored in the server, and the horizontal or vertical lines in the image are automatically detected through Hough transform to determine the shooting angle of the image. The initial image is rotated and corrected accordingly according to the shooting angle to restore the initial image to an upright posture, and the final receipt image is obtained to ensure the correct reading of the receipt image.

[0085] It is worth noting that when the image quality of the receipt image does not meet the preset shooting lighting standard, for example, the flash is turned on when shooting due to the dim indoor environment, resulting in uneven image lighting, the Contrast Limited Adaptive Histogram Equalization algorithm is used to improve the image quality.

[0086] Based on the above embodiment, in a feasible implementation, the target key information key-value pair is corrected to complete the entire bill recognition process. Specifically, after the second prompt word, the text content, the location information and the key information key-value pair are input into the preset language model to obtain the target key information key-value pair, the following steps may also be included:

[0087] Performing matching verification on the target key information key-value pair based on the geometric distance key information and the position relationship key information in sequence to generate a matching verification result;

[0088] According to the matching verification result, the target key information key-value pair is corrected using a preset standardized key information name field word library to obtain a corrected key information key-value pair, and a matching verification is performed on the corrected key information key-value pair.

[0089] In the embodiment of the present application, first, a matching check is performed based on the key information of geometric distance, and the positions are obtained based on OCR recognition, and the centroid of the positions corresponding to the key values ​​is calculated. If the matching degree is less than the threshold, a manual confirmation is prompted; if it is greater than the threshold, the subsequent automatic process continues.

[0090] Specifically, determine the centroid corresponding to the key and value of the target key information key-value pair, and judge whether the matching degree between the key and the value is less than a preset threshold based on the centroid; if the matching degree between the key and the value is less than the preset threshold, generate a first prompt information so as to perform manual confirmation based on the first prompt information.

[0091] like Figure 5 The figure shows an exemplary method of obtaining a position based on OCR recognition and calculating the centroid of the key values ​​corresponding to the position. The calculation formula of the matching degree is: ;

[0092] It is the Euclidean distance between K and V calculated by the location information of the OCR recognition result corresponding to the key information name and content; since the location information contour recognized by OCR is generally a rectangle, the distance is obtained by calculating the centroid of the two;

[0093] It is a normalization parameter for the overall image resolution. It solves the problem that the higher the image resolution, the greater the pixel distance between two fields of the same bill, which leads to a greater geometric distance. ;in, for Figure 5 The angle between the centroid line and the wide side of the image; h is the pixel value of the high side of the image; w is the pixel value of the wide side of the image.

[0094] Secondly, the matching verification is performed based on the key information of the position relationship. , set the angle range according to the regular bill type, and set the preset angle range to ,like Figure 6 If it exceeds the range, manual confirmation will be prompted.

[0095] Specifically, if the matching degree between the key and the value is not less than the preset threshold, the subsequent automatic process is continued. The angle between the centroid line and the image wide side is obtained, and it is determined whether the angle exceeds the preset angle range; wherein the centroid line is the line between the centroid corresponding to the key and the centroid corresponding to the value; the image wide side is the wide side of the outline rectangle where the centroid is located determined based on optical character recognition; if the angle exceeds the preset angle range, a second prompt information is generated so that manual confirmation is performed according to the second prompt information.

[0096] Furthermore, if the angle does not exceed the preset angle range, the step of correcting the target key information key-value pair using the preset standardized key information name field vocabulary is triggered. In an embodiment of the present application, when the angle between the centroid line and the wide side meets the set angle range of the conventional bill type, the preset standardized key information name field vocabulary is used to match and replace the key information in the target key information key-value pair K-V' to obtain the corrected K'-V' key-value pair. Finally, for the few core contents of the corrected key information key-value pair K'-V', a one-to-one precise verification is performed with the existing system through the background API interface (Application Program Interface). If the verification fails, manual processing is prompted; if successful, the entire bill recognition content is completed, and the information is output to the subsequent processing flow.

[0097] Correspondingly, the present application also discloses a receipt recognition device, see Figure 7 As shown, the device comprises:

[0098] A text recognition module 11, used to obtain a receipt image by a preset acquisition method, and recognize the receipt image by using a preset optical character recognition model to determine the text content in the receipt image and the position information corresponding to the text content; the preset optical character recognition model is embedded with an attention mechanism module;

[0099] The spatial relationship recognition module 12 is used to match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content and the location information into the preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross attention mechanism module;

[0100] The semantic matching correction module 13 is used to match the information of the receipt image in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain a target key information key-value pair.

[0101] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0102] It can be seen that, through the above scheme of this embodiment, the receipt image is acquired through the preset acquisition method, and the receipt image is recognized by using the preset optical character recognition model to determine the text content in the receipt image and the position information corresponding to the text content; the preset optical character recognition model is embedded with an attention mechanism module; the receipt image is matched with information in a preset multimodal large model prompt word library to obtain a first prompt word, and the first prompt word, the text content and the position information are input into the preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross-attention mechanism module; the receipt image is matched with information in a preset language large model prompt word library to obtain a second prompt word, and the second prompt word, the text content, the position information and the key information key-value pair are input into the preset language large model to obtain a target key information key-value pair.

[0103] The beneficial technical effects of the present application are: by comprehensively utilizing the precise text recognition capability of OCR, the spatial understanding and relationship matching capabilities of the multimodal large model, and the universal language understanding and matching correction capabilities of the language large model, three smaller-sized models can be used to achieve the accuracy of receipt content recognition that can be completed by a single large-sized model. Greatly reduce the video memory and GPU resources occupied by recognition, and improve the interface response speed. Since the three links of text recognition, spatial relationship recognition, and semantic matching correction are respectively implemented by different models. Therefore, in the training stage, different links can be trained separately, and only the training data of the corresponding links needs to be prepared. Whether it is the required training data, the computing power consumed during training, or the video memory occupied, they are greatly reduced, which greatly improves the speed of the entire training link. In this way, the accuracy of content recognition of key fields of non-standard receipts is greatly improved, and the overall loan efficiency of used cars is improved.

[0104] Furthermore, the present application also discloses an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the scope of use of the present application.

[0105] Figure 8 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the receipt recognition method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be a computer.

[0106] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0107] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The data 223 may include various data. The storage method may be temporary storage or permanent storage.

[0108] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the receipt recognition method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0109] Furthermore, the embodiment of the present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium mentioned here includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. Wherein, when the computer program is executed by the processor, the aforementioned receipt recognition method is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the aforementioned embodiment, and no further description will be given here.

[0110] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0111] The steps of the receipt recognition method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0112] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0113] The receipt recognition method, device, equipment and medium provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained in detail by using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A receipt recognition method, characterized in that: include: Acquire a receipt image by a preset acquisition method, and recognize the receipt image by using a preset optical character recognition model to determine text content in the receipt image and position information corresponding to the text content; An attention mechanism module is embedded in the preset optical character recognition model; Matching the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and inputting the first prompt word, the text content and the location information into the preset multimodal large model to obtain a key information key-value pair; The preset multimodal large model is embedded with a visual transformer and a cross-attention mechanism module; The receipt image is matched with information in a preset language model prompt word library to obtain a second prompt word, and the second prompt word, the text content, the location information and the key information key-value pair are input into the preset language model to obtain a target key information key-value pair.

2. The receipt recognition method according to claim 1, characterized in that: The method of acquiring the receipt image by a preset acquisition method includes: Using a terminal device with a photographing function to photograph the receipt at any angle to obtain an initial image, and using a target detection model based on the YOLO algorithm to recognize the initial image to generate a corresponding recognition result; Based on the recognition result, the shooting angle of the initial image is automatically detected through Hough transformation, and the shooting angle is rotated and corrected to an upright posture to obtain the receipt image.

3. The receipt recognition method according to claim 1, characterized in that: The process of acquiring the receipt image by the preset acquisition method also includes: When it is detected that the image quality of the receipt image does not meet the preset shooting illumination standard, the receipt image is enhanced by using a contrast-limited adaptive histogram equalization algorithm.

4. The receipt recognition method according to claim 1, characterized in that: The step of recognizing the receipt image by using a preset optical character recognition model to determine text content in the receipt image and position information corresponding to the text content includes: Recognize the receipt image by optical character recognition technology to obtain a single character in the receipt image and first position information corresponding to the single character; Performing a connected domain analysis process on the receipt image to obtain a plurality of connected domains, and determining a target connected domain from the plurality of connected domains, wherein the interval between adjacent connected domains is less than a preset interval; According to the single character and the first position information corresponding to the target connected domain, the single character and the first position information are merged respectively to obtain the text content in the receipt image and the second position information corresponding to the text content.

5. The receipt recognition method according to claim 1, characterized in that: The step of inputting the second prompt word, the text content, the location information, and the key information key-value pair into a preset language macro model to obtain a target key information key-value pair includes: The second prompt word, the text content, the location information and the key information key-value pair are input into a preset language model, and based on the second prompt word, the key information key value is updated using the text content according to the location information to obtain a target key information key-value pair.

6. The receipt recognition method according to any one of claims 1 to 5, characterized in that: After inputting the second prompt word, the text content, the location information and the key information key-value pair into a preset language macro model to obtain a target key information key-value pair, the method further includes: Performing matching verification on the target key information key-value pair based on the geometric distance key information and the position relationship key information in sequence to generate a matching verification result; According to the matching verification result, the target key information key-value pair is corrected using a preset standardized key information name field word library to obtain a corrected key information key-value pair, and a matching verification is performed on the corrected key information key-value pair.

7. The receipt recognition method according to claim 6, characterized in that: The matching verification of the target key information key-value pair is performed based on the geometric distance key information and the position relationship key information in sequence to generate a matching verification result, including: Determine the centroid corresponding to the key and value of the target key information key-value pair, and determine whether the matching degree between the key and the value is less than a preset threshold according to the centroid; If the matching degree between the key and the value is less than the preset threshold, generating first prompt information so as to perform manual confirmation according to the first prompt information; If the matching degree between the key and the value is not less than the preset threshold, the angle between the centroid line and the image wide side is obtained, and it is determined whether the angle exceeds the preset angle range; wherein the centroid line is a line between the centroid corresponding to the key and the centroid corresponding to the value; and the image wide side is a wide side of the outline rectangle where the centroid is located, determined based on optical character recognition; If the included angle exceeds the preset angle range, a second prompt message is generated so as to perform manual confirmation according to the second prompt message; If the angle does not exceed the preset angle range, the step of correcting the target key information key-value pair using a preset standardized key information name field vocabulary is triggered.

8. A receipt recognition device, characterized in that: include: A text recognition module, used to obtain a receipt image by a preset acquisition method, and recognize the receipt image by using a preset optical character recognition model to determine the text content in the receipt image and position information corresponding to the text content; An attention mechanism module is embedded in the preset optical character recognition model; A spatial relationship recognition module, used to match the receipt image with information in a preset multimodal large model prompt word library to obtain a first prompt word, and input the first prompt word, the text content and the location information into a preset multimodal large model to obtain a key information key-value pair; the preset multimodal large model is embedded with a visual transformer and a cross attention mechanism module; The semantic matching correction module is used to match the information of the receipt image in a preset language model prompt word library to obtain a second prompt word, and input the second prompt word, the text content, the location information and the key information key-value pair into the preset language model to obtain a target key information key-value pair.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the receipt recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein when the computer program is executed by a processor, the receipt recognition method according to any one of claims 1 to 7 is implemented.

Images