Document recognition method, device and equipment, and computer program product

Abstract

The application discloses a kind of bill identification method, device and equipment, computer program product, the method includes: obtaining handwritten bill image and using image preprocessing strategy to pre-process;The pre-processed handwritten bill image is identified using OCR model, and the identification result of OCR model is obtained;According to the identification result of OCR model, search is carried out in pre-constructed bill knowledge base, and the bill knowledge corresponding to the identification result of OCR model is obtained;According to the pre-processed handwritten bill image and bill knowledge, identification is carried out using multimodal large model, and the identification result of multimodal large model is obtained;According to the identification result of OCR and multimodal large model, using preset verification strategy determines the final bill identification result.The application improves the robustness of identification under different scenes by image preprocessing;Adopt the fusion identification architecture of large model collaborative complementation, combined with the local perception advantage of OCR and the global understanding and logical judgment ability of multimodal large model, the identification accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of invoice recognition technology, and in particular to an invoice recognition method, apparatus and equipment, and computer program product. Background Technology

[0002] In banking operations, processing various handwritten documents and vouchers is a high-frequency and crucial foundational task. Whether it's a check, deposit slip, business authorization form, or other specific voucher, staff must accurately identify the key handwritten information on the document (such as amount, date, account number, payee, etc.) and enter it into the business system to complete subsequent processing. Relying on manual visual interpretation and manual entry is not only inefficient and consumes a lot of manpower and time, but it is also prone to operational errors due to factors such as human fatigue.

[0003] To improve automation, Optical Character Recognition (OCR) technology has been introduced into the processing of handwritten banknotes. However, traditional OCR models face challenges such as low recognition accuracy and insufficient generalization ability due to factors like illegible and varied handwriting, overlapping strokes, complex backgrounds, and the presence of stamps. Furthermore, they lack information classification capabilities and are sensitive to image quality, making it difficult to meet the stringent accuracy requirements of banking operations. In recent years, multimodal large language models have demonstrated powerful contextual understanding and logical reasoning capabilities, integrating visual and textual information, providing new possibilities for verifying and correcting OCR recognition results and overcoming its technical limitations.

[0004] Based on this trend, various methods for document recognition and extraction that integrate OCR with large-scale models have been proposed. These methods recognize the limitations of a single OCR model and compensate for this deficiency by introducing external knowledge or complex models. The post-processing stage employs different technical paths to improve recognition accuracy, including large-scale model fine-tuning, rule engine verification, prompt word engineering, and text correction pipelines. However, while these solutions provide valuable insights for improving recognition accuracy, there is still room for improvement in proactive error prevention, deep integration of visual and textual information, and end-to-end process optimization when dealing with specific scenarios such as handwritten bank documents, which are characterized by high noise, high deformation, and strong domain specificity. Summary of the Invention

[0005] This application provides a method, apparatus, and computer program product for recognizing bank handwritten invoices, thereby improving the accuracy and efficiency of recognizing such invoices.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a bill recognition method, the bill recognition method comprising:

[0008] Obtain an image of the handwritten receipt;

[0009] The handwritten ticket image is preprocessed using an image preprocessing strategy to obtain a preprocessed handwritten ticket image.

[0010] The preprocessed handwritten ticket image is recognized using an optical character recognition model to obtain the recognition result of the optical character recognition model;

[0011] Based on the recognition results of the optical character recognition model, a search is performed in the pre-built invoice knowledge base to obtain the invoice knowledge corresponding to the recognition results of the optical character recognition model;

[0012] Based on the preprocessed handwritten ticket image and the corresponding ticket knowledge, a multimodal large model is used for recognition to obtain the recognition result of the multimodal large model;

[0013] Based on the recognition results of the optical character recognition model and the recognition results of the multimodal large model, the final ticket recognition result is determined using a preset verification strategy.

[0014] Optionally, the step of preprocessing the handwritten ticket image using an image preprocessing strategy to obtain a preprocessed handwritten ticket image includes:

[0015] The watermark of the seal in the handwritten ticket image is removed using a preset image restoration algorithm to obtain the restored handwritten ticket image.

[0016] The handwritten ticket image is subjected to orientation recognition using a preset image orientation recognition model to obtain the orientation category of the handwritten ticket image, and the handwritten ticket image is corrected according to the orientation category of the handwritten ticket image to obtain a corrected handwritten ticket image.

[0017] The handwritten ticket image is enhanced using a handwritten font blending enhancement algorithm to obtain an enhanced handwritten ticket image.

[0018] Optionally, the handwriting hybrid enhancement algorithm includes a horizontal projection algorithm and a time-frequency analysis algorithm. The step of using the handwriting hybrid enhancement algorithm to enhance the handwritten document image to obtain the enhanced handwritten document image includes:

[0019] The horizontal projection algorithm is used to perform horizontal projection analysis on the handwritten ticket image to obtain the tilt angle of the handwritten ticket image;

[0020] The time-frequency analysis algorithm is used to perform time-frequency analysis on local regions of the handwritten ticket image to obtain the degree of local distortion of the handwritten characters;

[0021] A nonlinear geometric transformation model is constructed based on the tilt angle of the handwritten ticket image and the degree of local distortion of the handwritten characters;

[0022] The handwritten ticket image is adaptively corrected using the nonlinear geometric transformation model to obtain the handwritten ticket image after handwriting enhancement.

[0023] Optionally, the step of retrieving the invoice knowledge corresponding to the recognition result of the optical character recognition model from a pre-built invoice knowledge base based on the recognition result of the optical character recognition model includes:

[0024] Multiple effective fields are extracted from the recognition results of the optical character recognition model as feature keywords;

[0025] The pre-built invoice knowledge base is searched using the feature keywords to obtain the invoice knowledge corresponding to the recognition result of the optical character recognition model. The pre-built invoice knowledge base has pre-set field paradigm templates, field extraction rules and prompt word configurations for various invoice types.

[0026] Optionally, determining the final ticket recognition result using a preset verification strategy based on the recognition results of the optical character recognition model and the multimodal large model includes:

[0027] The recognition results of the optical character recognition model and the recognition results of the multimodal large model are matched at the field level to obtain the field-level matching results;

[0028] Based on the field-level matching results, field-level similarity is calculated to obtain the field-level similarity calculation results;

[0029] Based on the field-level similarity calculation results, the final invoice recognition result is determined using the preset verification strategy.

[0030] Optionally, determining the final ticket recognition result based on the field-level similarity calculation result using the preset verification strategy includes:

[0031] If the field-level similarity calculation result is 1, then the recognition results of the optical character recognition model and the multimodal large model for the field corresponding to the field-level similarity will be used as the final field recognition result.

[0032] If the field-level similarity calculation result is not 1, then based on the recognition results of the field corresponding to the field-level similarity by the optical character recognition model and the multimodal big model, and the invoice knowledge, a prompt word is constructed, and based on the prompt word, a comprehensive reasoning is performed using the reasoning big model to obtain the final field recognition result.

[0033] Optionally, acquiring the handwritten ticket image includes:

[0034] Maintain handwritten ticket images through a queue of tickets to be processed;

[0035] If a handwritten ticket image to be processed is detected in the queue of tickets to be processed, the handwritten ticket image to be processed is obtained from the queue of tickets to be processed.

[0036] The invoice recognition method further includes:

[0037] Once all handwritten ticket images in the queue of pending tickets have been processed, the ticket recognition results of all processed handwritten ticket images are summarized to obtain a summary result of the ticket recognition results.

[0038] A summary analysis report on invoice recognition is generated based on the aggregated results of the invoice recognition.

[0039] Secondly, embodiments of this application also provide a bill recognition device, the bill recognition device comprising:

[0040] The acquisition unit is used to acquire images of handwritten receipts;

[0041] The preprocessing unit is used to preprocess the handwritten ticket image using an image preprocessing strategy to obtain a preprocessed handwritten ticket image.

[0042] An optical character recognition unit is used to recognize the preprocessed handwritten ticket image using an optical character recognition model, and to obtain the recognition result of the optical character recognition model.

[0043] The retrieval unit is used to retrieve the invoice knowledge base in a pre-built database based on the recognition results of the optical character recognition model, and obtain the invoice knowledge corresponding to the recognition results of the optical character recognition model.

[0044] The multimodal large model recognition unit is used to recognize the preprocessed handwritten ticket image and corresponding ticket knowledge using a multimodal large model, and obtain the recognition result of the multimodal large model.

[0045] The verification unit is used to determine the final ticket recognition result based on the recognition results of the optical character recognition model and the recognition results of the multimodal large model using a preset verification strategy.

[0046] Thirdly, embodiments of this application also provide an apparatus, comprising:

[0047] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned ticket recognition methods.

[0048] Fourthly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement any of the aforementioned ticket recognition methods.

[0049] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: The invoice recognition method of the embodiments of this application first acquires a handwritten invoice image; then, it preprocesses the handwritten invoice image using an image preprocessing strategy to obtain a preprocessed handwritten invoice image; then, it uses an optical character recognition model to recognize the preprocessed handwritten invoice image to obtain the recognition result of the optical character recognition model; then, it searches a pre-built invoice knowledge base based on the recognition result of the optical character recognition model to obtain the invoice knowledge corresponding to the recognition result of the optical character recognition model; then, based on the preprocessed handwritten invoice image and the corresponding invoice knowledge, it uses a multimodal large model for recognition to obtain the recognition result of the multimodal large model; finally, based on the recognition result of the optical character recognition model and the recognition result of the multimodal large model, it uses a preset verification strategy to determine the final invoice recognition result. The invoice recognition method of this application introduces multiple preprocessing strategies in the image preprocessing stage, improving robustness to complex backgrounds, layout variations, and differences in handwriting style, and providing high-quality standardized image input for subsequent recognition tasks. It employs a fusion recognition architecture that combines the local perception advantages of the OCR small model with the global understanding and logical judgment capabilities of the multimodal large model, thereby improving recognition accuracy. Simultaneously, it leverages AI intelligent agent technology to achieve end-to-end automated processing, reducing manual intervention. While ensuring process flexibility, it significantly improves the efficiency of the invoice recognition process, better adapting to the complex and varied situations of handwritten invoices, and providing strong support for invoice processing-related business. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a flowchart illustrating a ticket recognition method according to an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of a ticket recognition process in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of a ticket recognition device according to an embodiment of this application;

[0054] Figure 4This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0057] The technical terms used in this application mainly include:

[0058] (1) AI intelligent agent: refers to an artificial intelligence system with a certain degree of autonomy, which can perceive the environment, reason, make decisions and perform actions to complete specific goals or tasks. It goes beyond simple question answering and can achieve complex functions by calling tools and interacting with other intelligent agents or the environment.

[0059] (2) Workflow: describes a series of automated, repeatable steps and rules designed to accomplish a business objective or task. In AI applications, it specifically refers to connecting multiple AI models, tools, or manual review nodes in a specific order and logic to form an end-to-end automated processing pipeline.

[0060] (3) Knowledge base: A structured collection of information used to store, organize, and retrieve knowledge. In the field of AI, it usually refers to a proprietary data source used by large language models to enhance their retrieval capabilities, enabling them to acquire and respond based on the latest and most accurate internal knowledge, thus overcoming the lag in their internal knowledge.

[0061] (4) DotsOCR model: An optical character recognition model focused on document text recognition. It is particularly good at handling complex layouts, multilingual mixed text, and low-quality document images, and can convert text information in images into editable and searchable text data with high accuracy.

[0062] (5) Qwen2.5-VL model: A large-scale vision-language model. It can simultaneously understand and process image and text information, and complete multimodal tasks such as visual question answering, image description, and image-text reasoning. It is a powerful tool for understanding and generating text related to visual content.

[0063] (6) DeepSeek-R1 model: A large language model focused on reasoning tasks. This model adopts a special "reasoning" mode, which generates complex thought chains or performs multi-step logical deductions to deal with complex problems that require deep mathematical problem solving, code generation or logical analysis.

[0064] Existing image preprocessing solutions for bank receipts are mostly general-purpose preprocessing methods, such as image denoising and quality enhancement in "OCR Recognition and Error Correction Method Based on Large Model" (Patent No.: CN118982835A), and image binarization and geometric correction methods in "OCR Recognition and Error Correction Method Based on Large Model" (Patent No.: CN118982835A) and "Method and Device for Analyzing Traditional Chinese Medicine Prescription Data Based on OCR Recognition and Large Model" (Patent No.: CN119811698A). However, the processing effect still has bottlenecks for complex degradation cases in handwritten bank receipts, such as extreme and non-uniform lighting and shadows, deep wrinkles and distortions, and high adhesion between handwriting and seal / background.

[0065] Regarding the process of OCR recognition combined with large-scale model fusion for document recognition and extraction, the above methods recognize the limitations of a single OCR model and compensate for this deficiency by introducing external knowledge or complex models. The post-processing stage employs different technical paths to improve recognition accuracy, including large-scale model fine-tuning, rule engine verification, prompt word engineering, and text correction pipelines. However, most existing technologies are loosely combined, making it difficult to achieve the optimal balance between accuracy in text recognition and depth in semantic understanding. Furthermore, their verification mechanisms often rely on fixed thresholds or rules, and the system design is often domain-specific, exhibiting significant shortcomings in generalizability and automation. Specifically, the limitations of existing technical solutions are mainly reflected in the following key aspects:

[0066] (1) The image preprocessing process is not robust enough and is difficult to cope with complex ticket scenarios.

[0067] Existing preprocessing methods for handwritten documents are still imperfect, and their effectiveness in handling complex physical deformations and background interference present in handwritten bank documents is unsatisfactory. Specifically:

[0068] 1) Insufficient ability to handle complex backgrounds and interference factors: Existing solutions focus on basic noise removal and binarization, but lack specialized removal or dilution functions for strong interference factors commonly found on invoices, such as watermarks, stamps with overlaid text, and handwritten signatures. These interferences can seriously pollute the recognition results of OCR and even mislead the semantic understanding of subsequent large-scale models.

[0069] 2) The accuracy and efficiency of orientation correction need improvement: In image orientation correction (0° / 90° / 180° / 270°), existing solutions require inferring the image orientation based on text orientation before performing rotation correction, which is complex and inefficient. In tilt correction, relying on traditional straight line detection methods significantly reduces the correction effect when the lines on the document or table are unclear or missing. The entire geometric correction process suffers from a dual bottleneck in efficiency and accuracy.

[0070] 3) Lack of handwriting-specific optimization: Existing preprocessing workflows generally lack enhancement algorithms for handwriting features. Common problems in handwriting, such as stroke adhesion, character distortion, and overall tilt, cannot be completely solved by general preprocessing, leading to irreparable errors introduced into the OCR model during character segmentation and recognition.

[0071] (2) The similarity comparison threshold is set subjectively and rigidly, which poses a risk of misjudgment and missed judgment.

[0072] Existing solutions typically rely on a fixed similarity threshold for their core verification modules, a design with significant limitations. Specifically:

[0073] 1) Lack of adaptive threshold setting: Fixed similarity thresholds are subjective and cannot adapt to the dynamic changes in different invoice types, writing quality, and business fields (such as different accuracy requirements for account numbers and amounts). This results in a lack of structural flexibility in the system, requiring repeated manual parameter adjustments, which is costly and difficult to optimize.

[0074] 2) Threshold-based intervention mechanisms have inherent flaws: Existing solutions typically only initiate re-identification or manual review for fields with similarity below a specific threshold. This "low-sense, high-sense" strategy carries a significant risk of missed detections. Because the identification results for fields with similarity exceeding the threshold are not absolutely reliable, systematically ignoring the review of these "high-confidence" fields may lead to the incorrect final adoption of important information (such as amounts and account numbers), triggering business risks.

[0075] (3) The prompts and configurations are complex, and the system generalization and automation costs are high.

[0076] In existing solutions, the performance of multimodal large models highly depends on carefully designed prompts and configuration templates. When facing the needs of multi-scenario ticket recognition, the following problems exist:

[0077] 1) High configuration workload and lack of versatility: In the existing solution, different JSON templates, prompt word templates, and field rules need to be manually configured separately for different invoice types. This process involves a lot of repetitive and tedious work, requiring deep human involvement, resulting in high costs and time costs for extending the system to new invoice types, and making it difficult to quickly respond to changes in business needs.

[0078] 2) Insufficient process automation: Due to the strong dependencies of the above configurations, the existing system struggles to achieve true end-to-end automation. When processing various types of invoices, manual intervention is often required for process configuration or switching, limiting the system's efficiency in large-scale, multi-type invoice batch processing scenarios.

[0079] (4) The functional design is limited to single-image recognition and lacks batch processing and business insight capabilities.

[0080] Existing solutions generally focus on improving the recognition accuracy of individual tickets, failing to expand functionality from the perspective of the overall banking business process. Specifically:

[0081] 1) Inability to support efficient batch processing: Existing solutions cannot automate the batch processing of multiple types of bills. In daily banking operations, a large number of different types of bills need to be processed continuously. Existing technologies lack a unified scheduling and identification mechanism, resulting in low overall business processing efficiency.

[0082] 2) Lack of in-depth data value mining: The system design stops at information extraction and identification, lacking the summarization and statistical analysis of the identification results. It cannot automatically generate valuable business insight reports such as business volume statistics, error type analysis, and trends of specific fields, which prevents the technological achievements from being fully transformed into decision support for operation and management, thus limiting the depth of its application value.

[0083] In summary, the shortcomings of existing technologies in terms of preprocessing robustness, intelligent verification mechanisms, system versatility, and batch processing and data aggregation capabilities restrict their widespread application and effectiveness in the complex scenario of bank handwritten document recognition.

[0084] Based on this, embodiments of this application provide a method for identifying invoices, such as... Figure 1 The diagram shows a flowchart of a bill recognition method according to an embodiment of this application. The bill recognition method includes the following steps S110 to S160:

[0085] Step S110: Obtain the image of the handwritten ticket.

[0086] Combination Figure 2 This application provides a schematic diagram of a ticket recognition process according to an embodiment. Users can upload handwritten ticket files in various ways, supporting common formats such as PNG, JPG, and PDF. These uploaded files are received and used as raw data for subsequent processing, providing the basic input for the entire ticket recognition process.

[0087] Step S120: The handwritten ticket image is preprocessed using an image preprocessing strategy to obtain a preprocessed handwritten ticket image.

[0088] The preprocessing in this application embodiment may include various operations such as image denoising, binarization, incremental sharpening, watermark fading and removal, image rotation correction, and handwriting enhancement, providing a more complete image preprocessing workflow, thereby improving the adaptability to complex situations of handwritten tickets.

[0089] Step S130: The preprocessed handwritten ticket image is recognized using an optical character recognition model to obtain the recognition result of the optical character recognition model.

[0090] An optical character recognition (OCR) model, such as DotsOCR, is used as a small model to perform character-level text localization and preliminary recognition on preprocessed handwritten ticket images. This model has the advantages of accurate and fast local perception, which can quickly locate the position of characters in the image and perform preliminary character recognition, outputting preliminary recognition results.

[0091] Step S140: Based on the recognition result of the optical character recognition model, a search is performed in the pre-built invoice knowledge base to obtain the invoice knowledge corresponding to the recognition result of the optical character recognition model.

[0092] This application embodiment pre-constructs a bill knowledge base, which stores relevant information for various bill types, including field extraction specifications, prompt word configurations, and rule configurations. Based on the preliminary OCR recognition results, intelligent retrieval is performed in the bill knowledge base, automatically matching and loading the prompt word configurations, rule configurations, and other information corresponding to the recognition results into a multimodal large model, so as to perform adaptive field extraction and structured output for different bill types in the subsequent process.

[0093] Step S150: Based on the preprocessed handwritten ticket image and the corresponding ticket knowledge, a multimodal large model is used for recognition to obtain the recognition result of the multimodal large model.

[0094] A general multimodal large model, such as Qwen2.5-VL, is used to perform global contextual understanding and semantic reasoning on the preprocessed handwritten invoice image and the loaded invoice knowledge. This model can comprehensively consider image information and invoice knowledge, leveraging its powerful logical judgment ability to achieve a more comprehensive and accurate recognition of invoice content.

[0095] Step S160: Based on the recognition results of the optical character recognition model and the recognition results of the multimodal large model, the final ticket recognition result is determined using a preset verification strategy.

[0096] A collaborative and complementary large-scale model fusion recognition architecture is adopted, which integrates the recognition results of the OCR small model and the recognition results of the multimodal large model. Through a preset verification strategy, the recognition results of the two models are comprehensively analyzed, and their respective advantages are utilized to complement each other, thereby determining the final document recognition result and improving the accuracy of the recognition result.

[0097] This application embodiment can achieve the above process by introducing AI agent technology. The AI ​​agent can make independent decisions based on the input ticket image and task objectives, and can schedule and execute the entire chain of operations such as preprocessing, recognition, verification, and output to achieve end-to-end automated processing. At the same time, users can easily customize and fine-grained control the workflow, reducing human intervention while ensuring flexibility, and achieving highly automated operation without human intervention.

[0098] This application's embodiments introduce multiple preprocessing strategies in the image preprocessing stage, improving robustness to complex backgrounds, layout variations, and differences in handwriting style, providing high-quality standardized image input for subsequent recognition tasks. It employs a fusion recognition architecture that combines the local perception advantages of the OCR small model with the global understanding and logical judgment capabilities of the multimodal large model, enhancing recognition accuracy. Simultaneously, it leverages AI intelligent agent technology to achieve end-to-end automated processing, reducing manual intervention. While ensuring process flexibility, it significantly improves the efficiency of the invoice recognition process, better adapting to the complex and varied situations of handwritten invoices, and providing strong support for invoice processing-related businesses.

[0099] In some embodiments of this application, the step of preprocessing the handwritten ticket image using an image preprocessing strategy to obtain a preprocessed handwritten ticket image includes: removing the watermark from the handwritten ticket image using a preset image restoration algorithm to obtain a restored handwritten ticket image; performing direction recognition on the handwritten ticket image using a preset image direction recognition model to obtain the direction category of the handwritten ticket image, and correcting the handwritten ticket image according to the direction category of the handwritten ticket image to obtain a corrected handwritten ticket image; and enhancing the handwritten ticket image using a handwritten font blending enhancement algorithm to obtain a handwritten ticket image with enhanced handwritten font.

[0100] Continue to refer to Figure 2 The preprocessing operations performed on handwritten ticket images in this application mainly include the following:

[0101] (1) Seal removal

[0102] For example, the DocDiff-based image inpainting algorithm can be used to process seal watermarks in handwritten ticket images. The DocDiff algorithm has powerful image analysis capabilities and can accurately identify the location and features of complex background interference elements such as seals and watermarks in ticket images.

[0103] After identifying these interfering elements, the algorithm intelligently fills and repairs the interfering areas based on the image's underlying structure and texture information. During the removal of stamps and watermarks, special attention is paid to preserving the integrity of the underlying text information, ensuring that the strokes and structure of the characters are not damaged, thus obtaining a repaired handwritten ticket image and providing a clearer image foundation for subsequent recognition work.

[0104] (2) Orientation correction

[0105] For example, a lightweight model based on PP-LCNet_x1_0_doc_ori can be used for image orientation recognition. This model has been specifically trained for the task of classifying the orientation of ticket images, and can quickly and accurately determine the orientation of input ticket images. Compared with traditional line detection methods, it improves the accuracy and processing speed of orientation correction.

[0106] This network identifies the orientation of an image into four common orientation categories: 0°, 90°, 180°, and 270°. When processing an image, the model outputs a 4-dimensional probability vector, where each element corresponds to a probability value for a given orientation category. The index corresponding to the maximum value in this 4-dimensional probability vector is selected as the predicted orientation category.

[0107] Based on the predicted orientation category, a corresponding geometric rotation transformation is performed on the original image. If the recognition result is 90°, the image is rotated 90 degrees counterclockwise; if the recognition is 180°, the image is rotated 180 degrees; if the recognition is 270°, the image is rotated 90 degrees clockwise; and if the recognition is 0°, the image remains unchanged. This method effectively corrects the orientation of document images, replacing traditional correction schemes based on projection analysis or line detection, and significantly improves processing speed and anti-interference capabilities.

[0108] (3) Enhanced handwriting

[0109] For example, a hybrid algorithm combining horizontal projection and time-frequency distribution can be used to enhance handwriting. The horizontal projection algorithm can analyze the distribution characteristics of handwriting in the horizontal direction, while the time-frequency distribution can analyze the characteristics of handwriting from a time-frequency perspective, capturing details such as character tilt and distortion.

[0110] By combining these two algorithms, this method specifically addresses common issues in handwriting such as slant and distortion, character cursive writing, and inconsistent slant amplitude. Through comprehensive processing of information obtained from horizontal projection and time-frequency distribution analysis, the handwriting is intelligently adjusted to make the characters more regular and clear, resulting in an enhanced handwritten document image that provides strong support for accurate character recognition in subsequent steps.

[0111] This application embodiment comprehensively improves the quality and recognizability of handwritten ticket images through the aforementioned image preprocessing steps. The stamp removal algorithm effectively removes complex background interference on the ticket while completely preserving the text information; the orientation correction method based on a lightweight model quickly and accurately corrects the image orientation, demonstrating excellent performance in processing speed and anti-interference capability; the handwriting enhancement algorithm combining horizontal projection and time-frequency distribution specifically addresses various problems with handwritten handwriting, making the characters more regular. These preprocessing steps work together to provide a high-quality image foundation for subsequent ticket recognition, helping to improve the accuracy and stability of the entire ticket recognition process and better adapting to the complex and varied situations of handwritten tickets.

[0112] In some embodiments of this application, the handwriting hybrid enhancement algorithm includes a horizontal projection algorithm and a time-frequency analysis algorithm. The step of using the handwriting hybrid enhancement algorithm to enhance the handwritten document image to obtain the enhanced handwritten document image includes: performing horizontal projection analysis on the handwritten document image using the horizontal projection algorithm to obtain the tilt angle of the handwritten document image; performing time-frequency analysis on a local region of the handwritten document image using the time-frequency analysis algorithm to obtain the local distortion degree of the handwritten characters; constructing a nonlinear geometric transformation model based on the tilt angle of the handwritten document image and the local distortion degree of the handwritten characters; and performing adaptive correction processing on the handwritten document image using the nonlinear geometric transformation model to obtain the enhanced handwritten document image.

[0113] (1) Obtaining the tilt angle through horizontal projection analysis

[0114] For the input handwritten ticket image, the pixel distribution density is calculated row by row. Specifically, for each row of the image, the number of non-background pixels (i.e., handwritten character pixels) in that row is counted, and this is used as a measure of the pixel distribution density of that row. By analyzing the pixel distribution density of these rows, a specific algorithm (such as a method based on least squares line fitting) is used to obtain an estimate of the overall tilt angle of the document. This estimate reflects the degree of tilt of the text in the handwritten ticket image relative to the horizontal direction, providing basic parameters for subsequent correction processing.

[0115] (2) Time-frequency analysis to obtain the degree of local distortion

[0116] By selecting a local region in a handwritten document image, the discrete Wigner-Ville distribution can be applied for time-frequency analysis. The discrete Wigner-Ville distribution is a powerful time-frequency analysis tool that can simultaneously reveal the characteristics of a signal in both the time and frequency domains. Handwritten characters are treated as a special type of signal. By detecting the instantaneous frequency characteristics of the handwriting signal, the degree of distortion in a local region of the handwritten character can be accurately identified. For example, when characters are curved or deformed, their instantaneous frequency characteristics will change accordingly; the discrete Wigner-Ville distribution can capture these changes, thereby quantifying the degree of local distortion.

[0117] (3) Construct a nonlinear geometric transformation model

[0118] Based on the two key parameters of the overall tilt angle and the degree of local distortion of the handwritten characters in the previously acquired handwritten ticket image, a fusion calculation is performed. During the fusion calculation, the influence of the overall tilt on the character position and the change in character shape due to local distortion are comprehensively considered. A nonlinear geometric transformation model is constructed using specific mathematical methods, such as building a transformation model based on polynomial functions. This nonlinear geometric transformation model can adaptively generate corresponding transformation parameters according to different tilt angles and degrees of local distortion, used to correct the characters in the image and restore them to a more regular state.

[0119] (4) Adaptive correction processing and multi-level verification mechanism

[0120] An adaptive correction process for handwritten ticket images is performed using a pre-constructed nonlinear geometric transformation model. This model automatically adjusts the transformation parameters based on the tilt and distortion of characters at different positions in the image, precisely correcting each character to make the characters in the entire image more neatly arranged and more standardized in shape.

[0121] A multi-level verification mechanism was implemented during the correction process. Specifically, after performing horizontal projection analysis on the initially corrected image to obtain the horizontal projection curve, the peak-to-valley ratio of this curve was calculated. If the peak-to-valley ratio was lower than a set threshold, it indicated that the correction effect might not have reached the ideal state, and an iterative optimization process could be automatically initiated. In the iterative optimization process, the image was re-corrected by adjusting the transformation parameters of the nonlinear geometric transformation model, and the peak-to-valley ratio of the horizontal projection curve was recalculated until the peak-to-valley ratio reached or exceeded the set threshold, achieving the optimal correction effect.

[0122] This application employs a hybrid enhancement algorithm combining horizontal projection and time-frequency analysis to comprehensively and accurately process the complex characteristics of handwritten characters. Horizontal projection analysis effectively obtains the overall tilt angle of the document, providing directional guidance for overall correction; time-frequency analysis accurately identifies the degree of local distortion of handwritten characters, resolving local deformation issues. The constructed nonlinear geometric transformation model adaptively corrects the image based on overall and local information, making the characters more regular. Simultaneously, a multi-level verification mechanism ensures optimized correction results, avoiding under- or over-correction. This series of processing steps works together to significantly improve the quality of handwritten document images, providing a clear and regular image foundation for subsequent document recognition, contributing to improved accuracy and stability of the entire document processing workflow, and better adapting to the complex and varied realities of handwritten documents.

[0123] In some embodiments of this application, the step of retrieving the bill knowledge corresponding to the recognition result of the optical character recognition model from a pre-built bill knowledge base based on the recognition result of the optical character recognition model includes: extracting multiple effective fields from the recognition result of the optical character recognition model as feature keywords; and using the feature keywords to retrieve the bill knowledge corresponding to the recognition result of the optical character recognition model from the pre-built bill knowledge base. The pre-built bill knowledge base is pre-configured with field paradigm templates, field extraction rules, and prompt word configurations for various bill types.

[0124] Continue to refer to Figure 2 The system uses the DotsOCR tool to perform full-text field recognition on the pre-processed handwritten document image. DotsOCR can comprehensively recognize text information in images, not only outputting the text content but also recording the coordinate position of each text and its corresponding confidence score. The recognition results from DotsOCR, including text content, coordinate position, and confidence score, are organized in JSON format and persistently stored in the system database. This step provides a complete and structured data foundation for subsequent knowledge retrieval, facilitating the system's easy access to and analysis of the recognition results.

[0125] After entering the knowledge retrieval stage, effective fields are extracted from the OCR recognition results stored in the database as feature keywords. Specifically, the first three effective fields from the OCR recognition results can be selected. Effective fields here refer to fields with clear semantics that can represent key information about the invoice, such as invoice type, amount, and date. By extracting these key fields as feature keywords, more accurate retrieval can be performed in the invoice knowledge base, improving retrieval efficiency and accuracy.

[0126] The extracted feature keywords are used to perform searches within a pre-built invoice knowledge base. This knowledge base is meticulously designed and constructed, pre-configured with field paradigm templates for various invoice types, field extraction rules, and prompt word configurations. The field paradigm templates define the standard format and content requirements for each field in different types of invoices, such as the standardized format for fields like taxpayer identification number and product name on invoices. The field extraction rules clarify the specific methods for accurately extracting each field from complex invoice information. The prompt word configuration provides the foundation for subsequent structured prompt word assembly; appropriate prompt words can be selected and combined based on different invoice types and search criteria.

[0127] Upon successful retrieval from the invoice knowledge base, structured prompts containing invoice type features, field extraction specifications, and output format requirements are automatically assembled based on the search results. Invoice type features clearly define the specific type of invoice being processed, such as VAT invoice or general invoice; field extraction specifications detail the specific steps and requirements for extracting each field from the OCR recognition results; and output format requirements specify the format in which the final invoice knowledge should be presented, such as JSON or XML. By assembling these structured prompts, clear and accurate guidance is provided for subsequent processing and applications.

[0128] The pre-built invoice knowledge base in this application, with its rich field paradigm templates, extraction rules, and prompt word configurations, can accurately match the characteristics of different types of invoices, improving the accuracy and adaptability of retrieval. The overall solution effectively integrates the advantages of OCR recognition and knowledge retrieval, enabling the rapid and accurate conversion of textual information in handwritten invoice images into valuable invoice knowledge. This provides strong support for invoice processing, analysis, and management, enhancing the system's intelligence level and processing efficiency.

[0129] In some embodiments of this application, determining the final invoice recognition result based on the recognition results of the optical character recognition model and the recognition results of the multimodal large model using a preset verification strategy includes: performing field-level matching on the recognition results of the optical character recognition model and the recognition results of the multimodal large model to obtain a field-level matching result; performing field-level similarity calculation based on the field-level matching result to obtain a field-level similarity calculation result; and determining the final invoice recognition result based on the field-level similarity calculation result using the preset verification strategy.

[0130] Continue to refer to Figure 2Since the output structures of the optical character recognition model and the multimodal large model are different, a field mapping relationship based on the Qwen2.5-VL output structure can be established first. Qwen2.5-VL outputs data in key-value pair format. For each key-value pair, a search is performed in the full-text JSON results of DotsOCR to find the corresponding field content. This step is to clarify the correspondence between the fields in the recognition results of the two models, laying the foundation for subsequent matching and similarity calculations.

[0131] During the mapping and matching process, there may be fields that exist in Qwen2.5-VL but are not recognized by DotsOCR. In such cases, the recognition result of Qwen2.5-VL is directly adopted, and the similarity of that field is set to a baseline value of 1. This is because Qwen2.5-VL, as a multimodal large model, can integrate information from multiple aspects such as images and text for recognition, and may have better recognition capabilities for some complex or ambiguous fields. Therefore, when DotsOCR cannot recognize the field, the result of Qwen2.5-VL is trusted first.

[0132] For successfully matched fields—those recognized by both Qwen2.5-VL and DotsOCR—an improved edit distance algorithm can be used to calculate character-level similarity. The traditional Levenshtein distance algorithm simply calculates the minimum number of single-character editing operations (insertion, deletion, or replacement) required to transform one string into another. This application's embodiments introduce a character weighting mechanism, assigning higher weights to numbers and key characters. For example, in recognizing invoice amounts, the accuracy of numbers is crucial, so numeric characters have a higher weight in the similarity calculation; key characters, such as specific characters in invoice codes, are also given higher weights. In this way, the similarity between two strings at the character level can be more accurately reflected.

[0133] Semantic similarity calculation is used to further evaluate field similarity. A pre-trained semantic vector model maps text to a vector space, and then the cosine similarity between two text vectors is calculated in this space. Cosine similarity measures the similarity between two vectors in a direction; a value closer to 1 indicates greater semantic similarity. This method captures the semantic information of the text, achieving high similarity scores even when two strings are not identical at the character level but express the same or similar meaning.

[0134] The final field-level similarity score is derived by weighted fusion of the character-level similarity (e.g., weighted at 0.6) and semantic-level similarity (e.g., weighted at 0.4). This weighting method comprehensively considers both character-level accuracy and semantic-level similarity, resulting in a more comprehensive and accurate similarity calculation.

[0135] Based on the field-level similarity calculation results obtained above, the final invoice recognition result is determined using a preset verification strategy.

[0136] This application's embodiments cleverly integrate the recognition results of the optical character recognition model and the multimodal large model by establishing field mapping relationships, fully leveraging the advantages of both models. The handling of unrecognized fields improves the completeness and accuracy of information. Regarding similarity calculation, the improved edit distance algorithm, combined with a character weighting mechanism and a weighted fusion method for semantic similarity calculation, can comprehensively and accurately evaluate field similarity from both character and semantic perspectives. This avoids the limitations of single calculation methods, effectively improving the accuracy and reliability of invoice recognition, reducing the need for manual intervention, and enhancing the intelligence and efficiency of the entire invoice processing system.

[0137] In some embodiments of this application, determining the final invoice recognition result based on the field-level similarity calculation result using the preset verification strategy includes: if the field-level similarity calculation result is 1, then the recognition results of the optical character recognition model and the multimodal large model for the field corresponding to the field-level similarity are taken as the final field recognition result; if the field-level similarity calculation result is not 1, then based on the recognition results of the optical character recognition model and the multimodal large model for the field corresponding to the field-level similarity and invoice knowledge, prompt words are constructed, and based on the prompt words, a comprehensive reasoning is performed using a reasoning large model to obtain the final field recognition result.

[0138] Continue to refer to Figure 2 When the field-level similarity calculation result is 1, it indicates that the optical character recognition model and the multimodal large model have completely consistent recognition results for this field. In this case, the recognition results of these two models for this field can be directly used as the final field recognition result without additional processing or verification. This is because completely consistent recognition results can largely guarantee accuracy, and direct use can improve processing efficiency and reduce unnecessary calculation and judgment processes.

[0139] When the field-level similarity calculation result is not 1, it indicates a discrepancy in the recognition results of the two models for that field, at which point the deep inference verification process begins. First, the content of the discrepancy field is collected, namely the different recognition results of the optical character recognition model and the multimodal large model for that field. Simultaneously, the original image region corresponding to the field is obtained. The original image region contains the actual presentation information of the field in the invoice image, such as font, color, and background, which may help determine the accuracy of the recognition result. Furthermore, relevant invoice specifications are extracted from the invoice knowledge base, such as the standard format and value range of the field in specific types of invoices; and historical recognition records are also extracted. These historical recognition records may contain correct and incorrect recognition results for similar fields under different circumstances, providing a reference for inference. This multi-source information is then used to construct structured prompts. Structured prompts present various relevant information in a clear and standardized manner, facilitating understanding and processing by the inference large model.

[0140] The constructed structured prompts are input into a large-scale inference model, such as the DeepSeek-R1 inference model, for comprehensive analysis. This large-scale inference model possesses powerful language understanding and reasoning capabilities, enabling it to evaluate the reasonableness of recognition results from both optical character recognition (OCR) and multimodal models based on multi-source information. For example, it can determine whether the recognition result conforms to a standard format based on document specifications, analyze historical recognition records to determine which recognition result is more reliable in similar situations, and combine information from the original image region to determine whether the recognition result matches image features. Through comprehensive analysis, the large-scale inference model outputs the final judgment result and the corrected field content.

[0141] The results of the large-scale inference model's judgment are updated in real time to the JSON output, replacing any previously disputed fields. This creates a dynamic and intelligent error correction loop, effectively reducing the risk of misjudgments and omissions. Simultaneously, the basis for the judgment and the confidence score are recorded. The basis for the judgment can include specific information referenced by the large-scale inference model during the analysis process, such as a clause in the invoice specification or a case from historical identification records. The confidence score reflects the model's confidence in the final judgment, providing a reference for subsequent quality assessment and traceability.

[0142] This application's embodiments achieve intelligent decision-making and deep verification of invoice field recognition results by employing different processing strategies based on field-level similarity calculation results. By constructing structured prompts containing multi-source information and utilizing a large-scale reasoning model for comprehensive inference, it fully leverages invoice knowledge, historical experience, and original image information. This allows for a more comprehensive and accurate evaluation of the rationality of different recognition results, effectively resolving the problem of discrepancies in model recognition results and significantly improving the accuracy and reliability of invoice field recognition.

[0143] In some embodiments of this application, acquiring the handwritten invoice image includes: maintaining handwritten invoice images through a queue of invoices to be processed; when a handwritten invoice image to be processed is detected in the queue of invoices to be processed, acquiring the handwritten invoice image to be processed from the queue of invoices to be processed; the invoice recognition method further includes: when all handwritten invoice images to be processed in the queue of invoices to be processed have been processed, summarizing the invoice recognition results of all processed handwritten invoice images to obtain a summary result of invoice recognition results; and generating a summary analysis report of invoice recognition based on the summary result of invoice recognition results.

[0144] This application embodiment specifically maintains a queue of pending tickets, which is used to store handwritten ticket images in an orderly manner. The existence of this queue ensures that ticket processing is carried out in a certain order, avoiding chaos and disorder in the processing.

[0145] The AI ​​agent continuously monitors the status of the queue of unprocessed tickets. It uses specific detection mechanisms, such as periodically checking the queue length or using an event-driven approach, to determine if there are any handwritten ticket images in the queue awaiting processing. Once an unprocessed ticket is detected in the queue, the AI ​​agent automatically performs a retrieval operation, extracting the handwritten ticket image from the queue and sending it to the subsequent data preprocessing module to initiate a new round of ticket processing.

[0146] Continue to refer to Figure 2 Once all handwritten ticket images in the queue have been processed, the results summary and report generation are triggered. During the processing of each handwritten ticket image, the ticket recognition results are first converted to standard JSON format. JSON format is characterized by its clear structure, ease of parsing and transmission, and convenient storage and processing of various information in the recognition results, such as field names, recognition values, and similarity scores.

[0147] Next, a format converter is used to transform the JSON data into a standard Markdown table format. Markdown tables are concise, easy to read and edit, and can intuitively display the invoice recognition results. Finally, the Markdown content is appended to a specified XLSX spreadsheet file. XLSX format is a common spreadsheet format with powerful data storage and analysis capabilities, enabling convenient management and further processing of large amounts of invoice recognition results.

[0148] After summarizing all invoice recognition results, a summary analysis report is automatically generated. This report includes several aspects, among which recognition difference statistics are used to analyze the differences in recognition results between different invoices, such as the different distribution of recognition values ​​for certain fields in different invoices, to help understand the stability and consistency of recognition.

[0149] The similarity distribution displays the similarity of the recognition results for each field. By statistically analyzing the distribution range and frequency of similarity, the accuracy and reliability of the recognition results can be assessed. Anomaly markers are used to identify anomalies that occur during the recognition process, such as unrecognizable fields or fields with excessively low similarity, facilitating subsequent focused inspection and handling of these issues. The summary analysis report provides a structured data foundation for subsequent batch data analysis, helping to delve deeper into the patterns and problems in the invoice recognition process.

[0150] This application embodiment achieves automated pipeline processing of batch invoices by maintaining and monitoring a queue of invoices to be processed, ensuring the orderliness and efficiency of invoice processing. Converting the invoice recognition results into a standard format and summarizing and storing them ensures that the recognition results are saved completely and sequentially, facilitating subsequent management and retrieval. The automatically generated summary analysis report provides a rich and structured data foundation for subsequent batch data analysis, helping to gain a deeper understanding of the overall invoice recognition situation, promptly identify and address potential problems, improve the intelligence level and practicality of the entire invoice recognition system, and better meet the needs of large-scale invoice processing.

[0151] It should be noted that the above embodiments of this application employ specific models such as DotsOCR, Qwen2.5-VL, and DeepSeek-R1. Without departing from the core concept of this application, these technical components can be replaced by models with similar functionalities. Specifically, the OCR recognition model can be replaced with Tesseract, EasyOCR, or other engines with text detection and recognition capabilities; the multimodal large model can adopt alternative models with visual-language understanding capabilities such as GPT-4V, Gemini Pro Vision, and LLaVA; and the reasoning model can be replaced with GPT-4, the Claude series, or other large language models with strong logical reasoning capabilities. These alternative solutions are equivalent in implementing the overall architecture and processing flow described in this application and should all fall within the protection scope of this application.

[0152] In summary, the key points of this application are mainly as follows:

[0153] (1) Image preprocessing: Construct a more complete image preprocessing workflow, add a lightweight orientation correction method based on DocDiff stamp removal, a method based on PP-LCNet_x1_0_doc_ori, and a handwritten character enhancement algorithm that combines horizontal projection and discrete Wigner-Ville distribution.

[0154] (2) Design of collaborative recognition architecture for large and small models in general scenarios: A collaborative recognition architecture for DotsOCR small model and Qwen-2.5VL multimodal large model was constructed, and the prompt words were dynamically adjusted by establishing a knowledge base for extracting standard knowledge from the ticket fields.

[0155] (3) Intelligent agent-driven automated workflow: By introducing AI intelligent agents, end-to-end process scheduling and control are realized, a customizable loop processing mechanism is established, and users can control the process in a fine-grained manner, thus realizing a complete automation solution.

[0156] (4) Batch invoice processing and data summary analysis: A multi-format conversion pipeline of JSON→Markdown→XLSX was designed, and the functions of batch result summary and difference report generation were developed. A complete data foundation for subsequent statistical analysis was provided, forming a complete data value chain from identification to business insight, which provided strong data support for system performance optimization and business decision-making.

[0157] This application has achieved at least the following technical effects:

[0158] (1) Improved handwritten document image preprocessing technology: Added stamp removal function; optimized orientation correction function, using PPLCNet model for orientation recognition and then rotation correction; added tilt and distortion correction algorithm for handwritten characters. Effectively solved multiple problems such as stamp interference, orientation error and handwritten character deformation in document images, significantly improved the system's robustness to complex backgrounds, layout variations and differences in writing style, and provided high-quality standardized image input for subsequent recognition tasks.

[0159] (2) Applicable to a wider range of general scenarios: Different prompt words can be used according to different ticket types. A knowledge base for extracting ticket fields is created, and the knowledge base is built and called for retrieval. The corresponding prompt word configuration, rule configuration and other information are input into the Qwen2.5-VL multimodal large model, realizing the ability to intelligently recognize different ticket types in a wide range of scenarios.

[0160] (3) Using a dynamic decision-making mechanism to replace the fixed similarity threshold: By conducting in-depth reasoning analysis on disputed fields through reasoning models, the false negative rate is effectively reduced, ensuring the accuracy of the identification results.

[0161] (4) Intelligent agent automated autonomous operation: Introducing AI intelligent agents to build workflows, which can make independent decisions based on ticket images and tasks without human intervention; supporting custom loop processes, allowing users to quickly and conveniently control the process in fine granular terms.

[0162] (5) Automatic batch processing of ticket images: The system processes single images in a loop, converts the JSON results into Markdown format, then into XLSX file format, and finally converts the batch results into an Excel spreadsheet and generates a difference report, providing a reference for subsequent statistical analysis and performance optimization.

[0163] This application also provides a ticket recognition device 300, such as... Figure 3 The diagram shows a schematic representation of a document recognition device according to an embodiment of this application. The document recognition device 300 includes:

[0164] Acquisition unit 310 is used to acquire images of handwritten tickets;

[0165] Preprocessing unit 320 is used to preprocess the handwritten ticket image using an image preprocessing strategy to obtain a preprocessed handwritten ticket image.

[0166] The optical character recognition unit 330 is used to recognize the preprocessed handwritten ticket image using an optical character recognition model, and obtain the recognition result of the optical character recognition model.

[0167] The retrieval unit 340 is used to retrieve the invoice knowledge base in the pre-built invoice knowledge base according to the recognition result of the optical character recognition model, and obtain the invoice knowledge corresponding to the recognition result of the optical character recognition model;

[0168] The multimodal large model recognition unit 350 is used to perform recognition using a multimodal large model based on the preprocessed handwritten ticket image and corresponding ticket knowledge, and obtain the recognition result of the multimodal large model.

[0169] The verification unit 360 is used to determine the final ticket recognition result based on the recognition result of the optical character recognition model and the recognition result of the multimodal large model using a preset verification strategy.

[0170] In some embodiments of this application, the preprocessing unit 320 is specifically used to: remove the watermark from the handwritten ticket image using a preset image restoration algorithm to obtain a restored handwritten ticket image; perform direction recognition on the handwritten ticket image using a preset image direction recognition model to obtain the direction category of the handwritten ticket image, and correct the handwritten ticket image according to the direction category of the handwritten ticket image to obtain a corrected handwritten ticket image; and perform handwriting enhancement on the handwritten ticket image using a handwriting hybrid enhancement algorithm to obtain a handwritten ticket image with enhanced handwriting.

[0171] In some embodiments of this application, the handwritten text hybrid enhancement algorithm includes a horizontal projection algorithm and a time-frequency analysis algorithm. The preprocessing unit 320 is specifically used to: perform horizontal projection analysis on the handwritten ticket image using the horizontal projection algorithm to obtain the tilt angle of the handwritten ticket image; perform time-frequency analysis on the local region of the handwritten ticket image using the time-frequency analysis algorithm to obtain the local distortion degree of the handwritten characters; construct a nonlinear geometric transformation model based on the tilt angle of the handwritten ticket image and the local distortion degree of the handwritten characters; and perform adaptive correction processing on the handwritten ticket image using the nonlinear geometric transformation model to obtain the handwritten ticket image after handwriting enhancement.

[0172] In some embodiments of this application, the retrieval unit 340 is specifically used to: extract multiple valid fields as feature keywords from the recognition results of the optical character recognition model; and use the feature keywords to search the pre-built invoice knowledge base to obtain invoice knowledge corresponding to the recognition results of the optical character recognition model. The pre-built invoice knowledge base is pre-configured with field paradigm templates, field extraction rules, and prompt word configurations for various invoice types.

[0173] In some embodiments of this application, the verification unit 360 is specifically used to: perform field-level matching on the recognition results of the optical character recognition model and the recognition results of the multimodal large model to obtain a field-level matching result; perform field-level similarity calculation based on the field-level matching result to obtain a field-level similarity calculation result; and determine the final ticket recognition result using the preset verification strategy based on the field-level similarity calculation result.

[0174] In some embodiments of this application, the verification unit 360 is specifically used to: if the field-level similarity calculation result is 1, then use the recognition results of the optical character recognition model and the multimodal large model for the field corresponding to the field-level similarity as the final field recognition result; if the field-level similarity calculation result is not 1, then construct prompt words based on the recognition results of the optical character recognition model and the multimodal large model for the field corresponding to the field-level similarity and the invoice knowledge, and use the reasoning large model to perform comprehensive reasoning based on the prompt words to obtain the final field recognition result.

[0175] In some embodiments of this application, the acquisition unit 310 is specifically used to: maintain handwritten ticket images through a queue of tickets to be processed; and acquire the handwritten ticket image to be processed from the queue of tickets to be processed when a handwritten ticket image to be processed is detected in the queue of tickets to be processed. The ticket recognition device 300 further includes: a summarization unit, used to summarize the ticket recognition results of all processed handwritten ticket images in the queue of tickets to be processed after all processed handwritten ticket images have been processed, to obtain a summary result of ticket recognition results; and a generation unit, used to generate a summary analysis report of ticket recognition based on the summary result of ticket recognition results.

[0176] It is understood that the above-mentioned bill recognition device can implement each step of the bill recognition method provided in the foregoing embodiments. The relevant explanations of the bill recognition method are applicable to the bill recognition device and will not be repeated here.

[0177] Figure 4 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 4 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0178] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0179] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0180] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0181] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0182] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0183] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0184] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0185] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0186] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0187] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0188] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0189] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0190] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for recognizing invoices, characterized in that, The invoice recognition method includes: Obtain an image of the handwritten receipt; The handwritten ticket image is preprocessed using an image preprocessing strategy to obtain a preprocessed handwritten ticket image. The preprocessed handwritten ticket image is recognized using an optical character recognition model to obtain the recognition result of the optical character recognition model; Based on the recognition results of the optical character recognition model, a search is performed in the pre-built invoice knowledge base to obtain the invoice knowledge corresponding to the recognition results of the optical character recognition model; Based on the preprocessed handwritten ticket image and the corresponding ticket knowledge, a multimodal large model is used for recognition to obtain the recognition result of the multimodal large model; Based on the recognition results of the optical character recognition model and the recognition results of the multimodal large model, the final ticket recognition result is determined using a preset verification strategy.

2. The ticket recognition method according to claim 1, characterized in that, The step of preprocessing the handwritten ticket image using an image preprocessing strategy to obtain a preprocessed handwritten ticket image includes: The watermark of the seal in the handwritten ticket image is removed using a preset image restoration algorithm to obtain the restored handwritten ticket image. The handwritten ticket image is subjected to orientation recognition using a preset image orientation recognition model to obtain the orientation category of the handwritten ticket image, and the handwritten ticket image is corrected according to the orientation category of the handwritten ticket image to obtain a corrected handwritten ticket image. The handwritten ticket image is enhanced using a handwritten font blending enhancement algorithm to obtain an enhanced handwritten ticket image.

3. The ticket recognition method according to claim 2, characterized in that, The handwritten character hybrid enhancement algorithm includes a horizontal projection algorithm and a time-frequency analysis algorithm. The step of using the handwritten character hybrid enhancement algorithm to enhance the handwritten document image, resulting in an enhanced handwritten document image, includes: The horizontal projection algorithm is used to perform horizontal projection analysis on the handwritten ticket image to obtain the tilt angle of the handwritten ticket image; The time-frequency analysis algorithm is used to perform time-frequency analysis on local regions of the handwritten ticket image to obtain the degree of local distortion of the handwritten characters; A nonlinear geometric transformation model is constructed based on the tilt angle of the handwritten ticket image and the degree of local distortion of the handwritten characters; The handwritten ticket image is adaptively corrected using the nonlinear geometric transformation model to obtain the handwritten ticket image after handwriting enhancement.

4. The ticket recognition method according to claim 1, characterized in that, The step of retrieving the bill knowledge corresponding to the recognition result of the optical character recognition model from a pre-built bill knowledge base based on the recognition result of the optical character recognition model includes: Multiple effective fields are extracted from the recognition results of the optical character recognition model as feature keywords; The pre-built invoice knowledge base is searched using the feature keywords to obtain the invoice knowledge corresponding to the recognition result of the optical character recognition model. The pre-built invoice knowledge base has pre-set field paradigm templates, field extraction rules and prompt word configurations for various invoice types.

5. The ticket recognition method according to claim 1, characterized in that, The step of determining the final ticket recognition result based on the recognition results of the optical character recognition model and the multimodal large model, using a preset verification strategy, includes: The recognition results of the optical character recognition model and the recognition results of the multimodal large model are matched at the field level to obtain the field-level matching results; Based on the field-level matching results, field-level similarity is calculated to obtain the field-level similarity calculation results; Based on the field-level similarity calculation results, the final invoice recognition result is determined using the preset verification strategy.

6. The ticket recognition method according to claim 5, characterized in that, The step of determining the final ticket recognition result based on the field-level similarity calculation result and using the preset verification strategy includes: If the field-level similarity calculation result is 1, then the recognition results of the optical character recognition model and the multimodal large model for the field corresponding to the field-level similarity will be used as the final field recognition result. If the field-level similarity calculation result is not 1, then based on the recognition results of the field corresponding to the field-level similarity by the optical character recognition model and the multimodal big model, and the invoice knowledge, a prompt word is constructed, and based on the prompt word, a comprehensive reasoning is performed using the reasoning big model to obtain the final field recognition result.

7. The document recognition method according to any one of claims 1 to 6, characterized in that, The acquisition of the handwritten invoice image includes: Maintain handwritten ticket images through a queue of tickets to be processed; If a handwritten ticket image to be processed is detected in the queue of tickets to be processed, the handwritten ticket image to be processed is obtained from the queue of tickets to be processed. The invoice recognition method further includes: Once all handwritten ticket images in the queue of pending tickets have been processed, the ticket recognition results of all processed handwritten ticket images are summarized to obtain a summary result of the ticket recognition results. A summary analysis report on invoice recognition is generated based on the aggregated results of the invoice recognition.

8. A ticket recognition device, characterized in that, The document recognition device includes: The acquisition unit is used to acquire images of handwritten receipts; The preprocessing unit is used to preprocess the handwritten ticket image using an image preprocessing strategy to obtain a preprocessed handwritten ticket image. An optical character recognition unit is used to recognize the preprocessed handwritten ticket image using an optical character recognition model, and to obtain the recognition result of the optical character recognition model. The retrieval unit is used to retrieve the invoice knowledge base in a pre-built database based on the recognition results of the optical character recognition model, and obtain the invoice knowledge corresponding to the recognition results of the optical character recognition model. The multimodal large model recognition unit is used to recognize the preprocessed handwritten ticket image and corresponding ticket knowledge using a multimodal large model, and obtain the recognition result of the multimodal large model. The verification unit is used to determine the final ticket recognition result based on the recognition results of the optical character recognition model and the recognition results of the multimodal large model using a preset verification strategy.

9. An apparatus comprising: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the ticket recognition methods of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the ticket recognition method according to any one of claims 1 to 7.

Images