Invoice information extraction and invoice category judgment method based on large language model

Abstract

The present application relates to the field of image recognition technology, and in particular to a method for extracting invoice information and determining invoice categories based on a large language model, the method comprising obtaining an invoice file, extracting the invoice text of the invoice file; designing a first-level prompt word, inputting the first-level prompt word and the invoice text into a preset large language model, and the large language model outputs the key information of the invoice text under the guidance of the first-level prompt word; for each commodity in the key information of the invoice text, designing a second-level prompt word, the second-level prompt word includes the commodity name, the corresponding commodity category and the invoice category, inputting the second-level prompt word into the large language model, and the large language model outputs the invoice category coding result of each commodity under the guidance of the second-level prompt word; integrating and outputting the key information of the invoice text and the invoice category coding result of each commodity. The present application can achieve a significant improvement in the efficiency and accuracy of invoice processing.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for extracting invoice information and determining invoice categories based on a large language model. Background Art

[0002] With the deepening of digital transformation of enterprises, the use of electronic invoices is becoming more and more common. As an important part of modern enterprise financial management, electronic invoices can not only improve the efficiency of enterprise financial processing, but also reduce the management and storage costs brought by paper invoices. Therefore, the application scope of electronic invoices among enterprises is constantly expanding. With the popularization of electronic invoices, the number of invoices that enterprises need to process has increased dramatically, which has also put forward higher requirements for the accurate extraction and classification management of invoice information.

[0003] At present, traditional methods for extracting invoice information and determining invoice categories mainly rely on manual review and automated solutions based on template matching or rule engines. During the manual review process, financial personnel need to check invoices one by one, manually enter key information, and determine the invoice category based on the product name. In order to improve processing efficiency, some companies have adopted automated solutions based on template matching or rule engines. These solutions automatically extract key information from invoices and perform preliminary classification through predefined templates or rules. This automated processing method reduces the workload of manual review to a certain extent and improves processing speed.

[0004] Although existing technologies have improved the efficiency of invoice processing to a certain extent, there is still a major problem: insufficient adaptability and accuracy. When faced with invoices of different formats and languages, traditional methods are prone to errors and it is difficult to ensure the accuracy of information extraction and classification judgment. Summary of the invention

[0005] This application provides a method for extracting invoice information and determining invoice categories based on a large language model. This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for extracting invoice information and determining invoice categories based on a large language model, the method comprising:

[0007] Obtain an invoice file, and extract the invoice text of the invoice file;

[0008] Designing first-level prompt words, inputting the first-level prompt words and the invoice text into a preset large language model, and the large language model outputs key information of the invoice text under the guidance of the first-level prompt words;

[0009] For each commodity in the key information of the invoice text, a second-level prompt word is designed, wherein the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category, and the second-level prompt word is input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word;

[0010] The key information of the invoice text and the invoicing category coding result of each commodity are integrated and output.

[0011] In a specific implementation scheme, obtaining the invoice file and extracting the invoice text of the invoice file includes:

[0012] The obtaining of the invoice file and extracting the invoice text of the invoice file comprises:

[0013] Obtain invoice files, which include four forms: invoice images, Word files, table files, and PDF files;

[0014] For the PDF file, determine whether it is a scanned copy, detect the text length of the PDF file, and if the text length is less than a preset threshold, consider the PDF file to be a scanned copy. If the PDF file is a scanned copy, convert it into an image for subsequent processing. If the PDF file is not a scanned copy, directly use a text extraction tool to extract the invoice text from the PDF file;

[0015] For the Word file, use a text extraction tool to extract the invoice text from the Word file;

[0016] For the table file, a preset large language model is used to perform semantic understanding and information extraction to complete the extraction of the invoice text.

[0017] In a specific implementation scheme, the obtaining of the invoice document and the extraction of the invoice text of the invoice document further include:

[0018] For the invoice image and the image converted from the scanned copy in the PDF file, the text area of ​​the image is detected based on the local binary pattern and phase consistency:

[0019] The calculation formula for the LBP value of each pixel is as follows:

[0020]

[0021] Among them, I(x,y) is the gray value of the pixel (x,y), I(x i ,y i ) is the pixel point (x i ,y i), P is the number of surrounding pixels, and s(x) is the sign function, as follows:

[0022]

[0023] Phase consistency is defined as follows:

[0024]

[0025] Among them, A n (x,y) is the amplitude of the nth frequency component, is the phase, N is the total number of frequency components, and the similarity of each pixel to other pixels is calculated based on the local binary pattern and phase consistency as follows:

[0026]

[0027] After calculating the similarity of each pixel with other pixels, the text region is detected using region growing technique;

[0028] Extract the text from all text areas and combine them into the invoice text.

[0029] In a specific implementation manner, the detecting text regions using the region growing technique includes:

[0030] Select a seed pixel from the image as a starting point, and select pixels above a preset similarity threshold as seed pixels;

[0031] Determine the conditions for merging pixels into regions and define a similarity threshold. Only when the similarity of adjacent pixels is higher than this threshold, they will be merged into the same region.

[0032] Starting from the seed pixel, check its neighboring pixels. For each neighboring pixel, check its similarity with the pixels in the current region. If the similarity of the neighboring pixel is higher than the preset threshold and has not been assigned to any region, add it to the current region and mark it as visited.

[0033] This process is repeated recursively or iteratively until no more neighboring pixels that meet the criteria can be found; when all neighboring pixels have been checked, all pixels in the current region are merged into a connected region.

[0034] In a specific implementation scheme, extracting text from all text areas and integrating them into invoice text includes:

[0035] Connect pixels in the text area into character shapes and determine the boundaries of characters based on the relative positions and pixel intensities of the pixels;

[0036] Extract characters by detecting specific patterns or contours of character shapes, and perform simple pattern matching or rule matching on the extracted characters;

[0037] The recognized and extracted characters are reconstructed into complete text lines or paragraphs according to their layout order in the image, and the reconstructed text lines or paragraphs are combined into the invoice text of the entire image.

[0038] In a specific possible implementation scheme, for each commodity in the key information of the invoice text, a second-level prompt word is designed, the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category, the second-level prompt word is input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word, including:

[0039] Construct a mapping table of product names to product categories, invoicing categories to codes, and product names to codes;

[0040] Design a second-level prompt word for each commodity in the key information of the invoice text based on the mapping relationship correspondence table;

[0041] The second-level prompt words are input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt words;

[0042] Post-process the invoicing category coding results of each commodity output by the big language.

[0043] In a specific possible implementation scheme, after the second-level prompt words are input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt words, the method further includes:

[0044] For products for which the large language model cannot output the invoicing category coding results, repeatedly design broader third-level prompt words and input them into the large language model until the large language model can output the invoicing category coding results of the remaining products under the guidance of the third-level prompt words.

[0045] In the second aspect, the present application provides an invoice information extraction and invoice category judgment system based on a large language model, which adopts the following technical solutions:

[0046] An invoice information extraction and invoice category judgment system based on a large language model, comprising:

[0047] A text extraction module, used to obtain an invoice file and extract the invoice text of the invoice file;

[0048] An information extraction module, used for designing first-level prompt words, inputting the first-level prompt words and the invoice text into a preset large language model, and the large language model outputs key information of the invoice text under the guidance of the first-level prompt words;

[0049] a classification discrimination module, for designing a second-level prompt word for each commodity in the key information of the invoice text, wherein the second-level prompt word includes a commodity name, a corresponding commodity category, and an invoicing category, and inputting the second-level prompt word into the large language model, wherein the large language model outputs an invoicing category coding result for each commodity under the guidance of the second-level prompt word;

[0050] The data output module is used to integrate and output the key information of the invoice text and the invoicing category coding result of each commodity.

[0051] In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a method for extracting invoice information and determining invoice categories based on a large language model as described in the first aspect.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program is stored. When the program is executed by a processor, it is used to implement a method for extracting invoice information and determining invoice categories based on a large language model as described in the first aspect.

[0053] In summary, the beneficial effects of this application include at least:

[0054] 1) Through the carefully designed multi-level prompt words, complex tasks can be decomposed into multiple subtasks. Each subtask uses prompt words to clarify the requirements and guide the model to complete the task step by step. The first-level prompt words are used to extract basic invoice information; the second-level prompt words are used to determine the classification of goods; if necessary, the third-level prompt words can be used for further multiple rounds of inquiries. Multi-level prompt words not only improve the completion of tasks, but also enhance the controllability and accuracy of large models.

[0055] 2) By combining the mapping relationship table related to the enterprise collection field with the large language model, the accuracy of commodity classification can be significantly improved. By introducing commodity categories and standard invoicing category codes into the large model through prompt words, the large model can make full use of these domain knowledge when making commodity classification judgments, thereby greatly improving the accuracy of classification judgments.

[0056] 3) A multi-round inquiry mechanism is designed. If the first classification judgment fails, a third-level prompt word is constructed to guide the model to select relevant categories in a wider range. This multi-round inquiry mechanism can improve the success rate of classification judgment, ensure that each product can find a suitable invoicing category, and avoid omissions and misclassifications.

[0057] Acquire and process invoice files, including images and PDF documents, and obtain invoice text through preprocessing and text extraction technology. Subsequently, use the set prompt words and large language model to perform semantic understanding and information extraction on the invoice text to accurately obtain key information such as the buyer, seller and product details. In particular, for each product, determine its accurate invoicing category code through a combination of multi-level prompt words and large language models. This method improves the accuracy and adaptability of information extraction and classification through high automation and semantic understanding, and effectively addresses the limitations and error rates of traditional methods when processing diversified invoices. Finally, the structured JSON format invoice data is output, which can be directly processed and managed by subsequent systems, achieving a significant improvement in the efficiency and accuracy of invoice processing.

[0058] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flow chart of the invoice information extraction and invoice category determination method based on a large language model in an embodiment of the present application.

[0060] Figure 2 It is a flow chart of extracting invoice text from an invoice image and an image converted from a scanned copy in a PDF file in an embodiment of the present application.

[0061] Figure 3 It is a flowchart of step S103 in the embodiment of the present application.

[0062] Figure 4 It is an overall flow chart of a method for extracting invoice information and determining invoice categories based on a large language model provided in one embodiment of the present application.

[0063] Figure 5 It is a structural block diagram of the invoice information extraction and invoice category judgment system based on the large language model in the embodiment of the present application.

[0064] Figure 6 It is a block diagram of an electronic device for extracting invoice information and judging invoice categories based on a large language model in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0066] Optionally, the present application uses the large language model-based invoice information extraction and invoice category judgment method provided in each embodiment as an example of being used in an electronic device, where the electronic device is a terminal or a server. The terminal can be a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0067] Reference Figure 1 , is a flow chart of a method for extracting invoice information and determining invoice categories based on a large language model provided by an embodiment of the present application, the method comprising at least the following steps:

[0068] Step S101: Obtain an invoice file and extract the invoice text of the invoice file.

[0069] Step S102: design the first-level prompt words, input the first-level prompt words and the invoice text into a preset large language model, and the large language model outputs key information of the invoice text under the guidance of the first-level prompt words.

[0070] Step S103: For each commodity in the key information of the invoice text, a second-level prompt word is designed, and the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category. The second-level prompt word is input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word.

[0071] Step S104: Integrate and output the key information of the invoice text and the invoicing category coding result of each commodity.

[0072] In step S101, first obtain the invoice file. Invoice files usually include four forms, invoice pictures, Word files, spreadsheet files and PDF files. For PDF files, first determine whether it is a scanned copy and detect the text length of the PDF file. If the text length is less than the preset threshold, the PDF file is considered to be a scanned copy. The preset threshold is usually 50 characters. This application does not limit the specific value of the preset threshold. If the PDF file is a scanned copy, convert it into an image for subsequent processing. If the PDF file is not a scanned copy, directly use the text extraction tool to extract the invoice text from the PDF file. For Word files, also directly use the text extraction tool to extract the invoice text from the Word file.

[0073] Optionally, the present application uses existing text extraction tools, such as the PyMuPDF library, etc., or other existing technologies may be used to extract the text of a PDF file or a Word file. The present application does not limit the technology for extracting text from a PDF file.

[0074] For table files, since the table files may include a large amount of product information and the generation logic of each table is different, a preset large language model is used to perform semantic understanding and information extraction, thereby completing the extraction of invoice text for batch product information.

[0075] Optionally, the large language model of the present application is an existing large language model, such as GTP-4, etc., or other existing large language models. The present application does not limit the specific type of the large language model.

[0076] For invoice images and images converted from scanned copies in PDF files, refer to Figure 2 , is a flow chart of identifying invoice images and invoice text converted from scanned copies in PDF files in an embodiment of the present application, the method comprising at least the following steps:

[0077] S1011, pre-process the image.

[0078] Specifically, we first use non-local mean denoising to reduce the noise in the image and retain the clarity of the text, and then use the following formula to enhance the contrast and brightness of the image:

[0079]

[0080] Among them, I′(x,y) is the pixel value of the image after contrast and brightness enhancement, I(x,y) is the pixel value of the original image, and I min and I max are the minimum and maximum pixel values ​​of the image, respectively. α and β are adjustment parameters used to adjust contrast and brightness. The original pixel values ​​are normalized to the range of 0-1 so that the contrast and brightness enhancement effects can be applied to the image more accurately. The normalized pixel values ​​are exponentially transformed to enhance or weaken the contrast of the pixel values. This nonlinear transformation can make the dark and bright details in the image more prominent. Finally, Gaussian filtering and Sobel operators are used for edge detection and enhancement to assist in extracting the contour and edge features of the subsequent text.

[0081] S1012. Detect text areas in the preprocessed image based on local binary patterns and phase consistency.

[0082] Specifically, the local binary pattern is a method for describing the local texture features of an image. For each pixel, a binary number is generated by comparing its grayscale value with that of the surrounding pixels, and then the LBP value is obtained. The calculation formula for the LBP value of each pixel is as follows:

[0083]

[0084] Among them, I(x,y) is the gray value of the pixel (x,y), I(x i ,y i ) is the pixel point (x i ,y i ), P is the number of surrounding pixels, and s(x) is the sign function, as follows:

[0085]

[0086] Phase congruency is a method for detecting image structure based on phase information. It detects edge and texture features in an image by calculating the phase congruence of local frequency components. The definition formula of phase congruence is as follows:

[0087]

[0088] Among them, A n (x,y) is the amplitude of the nth frequency component, is the phase, and N is the total number of frequency components. The similarity of each pixel to other pixels is then calculated based on the local binary pattern and phase consistency as follows:

[0089]

[0090] In implementation, after calculating the similarity of each pixel with other pixels, the region growing technique is used to detect the text region. Specifically, a seed pixel is selected from the image as the starting point. Usually, pixels with high similarity or other predefined attributes are selected as seed pixels. In the present application, pixels above a preset similarity threshold are selected as seed pixels. Determine the conditions for merging pixels into regions. These conditions are usually based on the similarity or other features between pixels. In text detection, a similarity threshold can be defined, and only when the similarity of adjacent pixels is higher than this threshold, they are merged into the same region. Starting from the seed pixel, check its adjacent pixels. For each adjacent pixel, check its similarity with the pixels in the current region. If the similarity of the adjacent pixel is higher than the preset threshold and has not been assigned to any region, it is added to the current region and marked as visited. Repeat this process recursively or iteratively until no more adjacent pixels that meet the conditions can be found. When all adjacent pixels have been checked, all pixels in the current region are merged into a connected region. For the detected text regions, some post-processing steps are required, such as filling holes, removing small regions, or smoothing boundaries to improve the accuracy and continuity of the text regions.

[0091] S1013. Extract text from all text areas and integrate them into invoice text.

[0092] Specifically, for each text region, a method based on pixel intensity and shape features can be used to extract possible characters, which can be achieved through connected region analysis or pixel-based operations. The specific method includes: connecting pixels in the text region into character shapes and determining the boundaries of characters based on the relative positions of pixels and pixel intensities. Extracting characters by detecting specific patterns or contours of character shapes. Performing simple pattern matching or rule matching on the extracted characters. For example, some basic rules can be defined to identify numbers, letters, or specific symbols. These rules can determine the identity of characters based on the shape, size, and pixel distribution of characters. Reconstructing the identified and extracted characters into complete text lines or paragraphs in the order of their layout in the image. This can be done by connecting characters and sorting them according to their positions in the image. Finally, combining the reconstructed text lines or paragraphs into the invoice text of the entire image. The invoice text can be organized in the format of text lines or paragraphs and output in the form of text strings.

[0093] In addition, as a preferred method, a method combining a non-local mean denoising algorithm and a convolutional neural network can also be used to extract the invoice text of the image converted from the scanned copy in the PDF file, including: first, the image is processed using a non-local mean (NLM) denoising algorithm. The NLM algorithm reduces the noise level of the image by comparing the similarity of each pixel in the image with the surrounding pixels. This is particularly important for detecting and recognizing text in the image, because the text may be interfered by the surrounding complex background or noise. Subsequently, on the image after NLM denoising, an existing convolutional neural network, such as YOLOv3, SSD, etc., is used to detect the text area. The model can accurately locate the text area in the image and generate a bounding box to identify the position and size of the text. After the bounding box is generated using the existing convolutional neural network, the intersection and union ratio IoU between it and the real text bounding box is calculated. If the IoU value is high, it means that the model detection accuracy is high. If the IoU value is low, it is necessary to further optimize the model parameters, and the high or low IoU value is judged by comparing with the threshold. The IoU value is the ratio of the area of ​​the overlapping part of the two bounding boxes to the area combined together. Finally, the text area is detected and the existing model based on convolutional neural network and recurrent neural network structure is used to identify the text content. The CNN-RNN structure can handle various complex image scenes and different font styles. The visual features extracted by CNN can adapt to text areas under different lighting, angles and background conditions, while RNN can learn and infer the grammatical and semantic information of the text.

[0094] In implementation, the above two methods can also be used to extract the invoice text from the invoice image and the image converted from the scanned copy in the PDF file, and then the two extracted invoice texts are processed as the final invoice text. Some text areas may be missed or incompletely recognized by one method, while the other method may be able to recognize these missed text contents. Through the union processing, it can be ensured that all recognizable text information is captured and integrated as much as possible, improving the completeness and coverage of text extraction. For tasks involving large amounts of text data, such as invoice text recognition, the robustness and reliability of the system are crucial. By combining the text extraction results of multiple methods, a more robust and reliable text recognition system can be built, reducing the problem of system failure or incompleteness caused by the inadequacy of a single method.

[0095] Optionally, other existing technologies may be used to directly perform text recognition on invoice images and images converted from scanned copies in PDF files, such as OCR text recognition technology, etc. This application does not limit the technology for extracting image text.

[0096] In another feasible embodiment, after extracting the invoice text, the integrity of the invoice text is verified to determine whether the invoice text completely contains the necessary information. Specifically, the integrity score of the text is calculated using the following formula:

[0097]

[0098] Where N is the total number of required fields, f i Indicates the i-th required field, Presence(f i ) represents the field f i The number of occurrences in the text, MaxPresence(f i ) represents the field f i The maximum number of occurrences in the text, w(f i ) is the field f i Structured texts such as invoices usually have fixed fields and format requirements. Designing a formula can judge the completeness of the text based on these requirements, ensuring that all necessary fields are correctly identified and included. The above formula combines the existence of the field, the importance weight, and the proportion of the number of occurrences. This comprehensive consideration can more comprehensively evaluate the completeness of the text, rather than simply judging whether certain fields appear.

[0099] In implementation, after calculating the integrity score of the text, the integrity score is compared with the preset threshold. If it is greater than or equal to the preset threshold, the extracted invoice text is judged to be relatively complete. If it is less than the preset threshold, the invoice text is extracted again until the integrity score is greater than or equal to the preset threshold.

[0100] For example, suppose there is an invoice text that needs to be evaluated for completeness, where the necessary fields include the invoice number, date, and total amount. The invoice text contains the following information:

[0101] Invoice number: INV-2024-001;

[0102] Date: 2024-07-15;

[0103] Total amount: $500.00;

[0104] Assuming that the importance weight of each field is 1 and the preset threshold is also 1, use the improved integrity evaluation formula mentioned above:

[0105]

[0106] The final calculated completeness score is equal to the preset threshold, indicating that the invoice text is relatively complete.

[0107] In step S102, firstly, the first-level prompt words are designed, and the first-level prompt words are used to clarify the fields extracted from the invoice text by the large language model, such as the buyer information (buyer's name, address, contact information), seller information (seller's name, address, contact information), commodity name (commodity name, quantity, unit price, total price), etc. Then, the first-level prompt words in JSON format and the invoice text are input into the preset large language model. Under the guidance of the first-level prompt words, the large language model performs semantic understanding and information extraction, and outputs the key information of the invoice text in JSON format.

[0108] Optionally, the large language model of the present application is an existing large language model, such as GTP-4, etc., or other existing large language models. The present application does not limit the specific type of the large language model.

[0109] In the implementation, after the large language model outputs the key information of the invoice text in JSON format, the JSON data output by the large language model is post-processed, including cleaning up useless information, handling abnormal situations, etc., to ensure that the data type and format of each field meet the expected requirements. By parsing the JSON string, it is converted into a structured data object that the program can understand and process. For example, a JSON parser or library in a programming language is used to convert the JSON formatted text into a corresponding data structure, such as an object, array, or hash table.

[0110] In step S103, refer to Figure 3 , is a flow chart of step S103 in an embodiment of the present application, and the method includes at least the following steps:

[0111] S1031. Construct a mapping relationship table between product name and product category, invoicing category and code, and product name and code.

[0112] For example, the product category corresponding to the product name citrus is fruit, and its corresponding code is 001. The invoicing category corresponding to code 001 is orange.

[0113] S1032. Design a second-level prompt word for each commodity in the key information of the invoice text based on the mapping relationship correspondence table.

[0114] Specifically, the second-level prompt words include the product name, the corresponding product category, and the invoicing category. For example, if the product name is citrus, the corresponding second-level prompt words should be the product name citrus, the product category fruit, and the invoicing category orange.

[0115] S1033. Input the second-level prompt words into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt words.

[0116] Specifically, the second-level prompt words in JSON format are input into the large language model, and the large language model outputs the invoicing category coding result of each commodity in JSON format under the guidance of the second-level prompt words. For example, if the input commodity is citrus, the corresponding model output result is orange and 001.

[0117] S1034. For commodities for which the large language model cannot output invoicing category coding results, repeatedly design broader third-level prompt words and input them into the large language model until the large language model can output invoicing category coding results for the remaining commodities under the guidance of the third-level prompt words.

[0118] Specifically, if the invoicing category of some goods cannot find the corresponding exact category or code in the standard code, it is necessary to enter the stage of multiple rounds of inquiry. The purpose of this stage is to guide the model to find relevant or broader categories from the standard code by designing the third-level prompt words, so as to determine the invoicing category and code for the goods as accurately as possible. First, for goods whose invoicing categories or codes cannot be determined, new third-level prompt words are designed, which may guide the model to find relevant categories or codes in a wider range. These prompt words may need to describe the characteristics or classification requirements of the goods more broadly. Then the designed third-level prompt words are input into the large language model, and the model is called again for classification judgment. This step is similar to the previous call to the large language model, but the goal is to find a more suitable invoicing category or code. If there are still undetermined situations, it may be necessary to call the large model iteratively multiple times until the invoicing categories and codes of all goods can be determined.

[0119] For example, under the premise that the second-level prompt words are the product name citrus, the product category fruit, and the invoicing category orange, if the large model cannot output the invoicing category coding result of the product, then a new third-level prompt word is redesigned as the product name citrus, the product category agricultural product, and the invoicing category fruit.

[0120] S1035. Post-process the invoicing category coding result of each commodity output by the big language.

[0121] Specifically, the validity of the JSON string output by the model is verified. This includes checking whether the syntactic structure of the JSON is correct and ensuring that all keys and values ​​meet the format requirements of the JSON. Data cleaning may be performed, which may remove extra spaces, special characters, or other unnecessary content. This ensures that the output JSON string is clean and contains only necessary data. In addition, any possible exceptions are handled, such as handling missing data fields, undefined categories or encodings. These situations may require appropriate processing based on business logic, such as filling in default values, providing warnings or error messages, etc. Finally, by parsing the JSON string, convert it into a structured data object that the program can understand and process. For example, use a JSON parser or library in a programming language to convert JSON-formatted text into a corresponding data structure, such as an object, array, or hash table.

[0122] In practice, we sometimes encounter some special products whose names cannot find accurate matches in the standard invoicing categories. In order to solve this problem, this application designs a multi-round inquiry mechanism. If the first classification judgment fails, a third-level prompt word is constructed to guide the model to select relevant categories in a wider range. This multi-round inquiry mechanism can effectively improve the success rate of classification judgment, and to a certain extent ensure that each product can find a suitable invoicing category to avoid omissions and misclassifications.

[0123] In step S104, the key information of the invoice text and the invoicing category coding result of each commodity are integrated into structured invoice data, and the structured invoice data is output in JSON format for use by the subsequent invoice processing system. JSON is a lightweight data exchange format suitable for cross-platform and cross-language data exchange. The output in JSON format enables the entire invoice processing system to provide the identified, classified and output information to other business systems or users in a format that is easy to manage and use.

[0124] In summary, refer to Figure 4, is an overall flow chart of an invoice information extraction and invoicing category judgment method based on a large language model provided by an embodiment of the present application. First, the invoice file, including pictures and PDF documents, is obtained and processed, and the invoice text is obtained through preprocessing and text extraction technology. Subsequently, the invoice text is semantically understood and information extracted using the set prompt words and large language model to accurately obtain key information such as the buyer, seller and product details. In particular, for each product, its accurate invoicing category code is determined by combining multi-level prompt words and a large language model. This method improves the accuracy and adaptability of information extraction and classification through high automation and semantic understanding, and effectively addresses the limitations and error rate problems of traditional methods in processing diversified invoices. Finally, the structured JSON format invoice data is output, which can be directly processed and managed by subsequent systems, achieving a significant improvement in the efficiency and accuracy of invoice processing.

[0125] Figure 5 This is a structural block diagram of an invoice information extraction and invoice category judgment system based on a large language model provided by an embodiment of the present application. The device includes at least the following modules:

[0126] The text extraction module is used to obtain the invoice file and extract the invoice text of the invoice file.

[0127] The information extraction module is used to design the first-level prompt words, input the first-level prompt words and the invoice text into a preset large language model, and the large language model outputs the key information of the invoice text under the guidance of the first-level prompt words.

[0128] The classification and discrimination module is used to design a second-level prompt word for each commodity in the key information of the invoice text. The second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category. The second-level prompt word is input into the large language model. The large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word.

[0129] The data output module is used to integrate and output the key information of the invoice text and the invoicing category coding results of each commodity.

[0130] For relevant details, refer to the above method embodiment.

[0131] Figure 6 4 is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0132] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0133] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the invoice information extraction and invoicing category judgment method based on a large language model provided in the method embodiment of the present application.

[0134] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0135] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0136] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the invoice information extraction and invoicing category judgment method based on a large language model of the above method embodiment.

[0137] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the invoice information extraction and invoicing category judgment method based on a large language model of the above-mentioned method embodiment.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for extracting invoice information and determining invoice categories based on a large language model, characterized in that: The method comprises: Obtain an invoice file, and extract the invoice text of the invoice file; Designing first-level prompt words, inputting the first-level prompt words and the invoice text into a preset large language model, and the large language model outputs key information of the invoice text under the guidance of the first-level prompt words; For each commodity in the key information of the invoice text, a second-level prompt word is designed, the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category, the second-level prompt word is input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word; for each commodity in the key information of the invoice text, a second-level prompt word is designed, the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category, the second-level prompt word is input into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word, including: Construct a mapping relationship table from product name to product category, from invoicing category to code, and from product name to code; design a second-level prompt word for each product in the key information of the invoice text based on the mapping relationship table; input the second-level prompt word into the large language model, and the large language model outputs the invoicing category coding result of each product under the guidance of the second-level prompt word; for products for which the large language model cannot output the invoicing category coding result, repeatedly design a broader third-level prompt word and input it into the large language model until the large language model can output the invoicing category coding result of the remaining products under the guidance of the third-level prompt word; post-process the invoicing category coding result of each product output by the large language; The key information of the invoice text and the invoicing category coding result of each commodity are integrated and output.

2. The invoice information extraction and invoice category determination method based on a large language model according to claim 1 is characterized in that: The obtaining of the invoice file and extracting the invoice text of the invoice file comprises: Obtain invoice files, which include four forms: invoice images, Word files, table files, and PDF files; For the PDF file, determine whether it is a scanned copy, detect the text length of the PDF file, and if the text length is less than a preset threshold, consider the PDF file to be a scanned copy. If the PDF file is a scanned copy, convert it into an image for subsequent processing. If the PDF file is not a scanned copy, directly use a text extraction tool to extract the invoice text from the PDF file; For the Word file, use a text extraction tool to extract the invoice text from the Word file; For the table file, a preset large language model is used to perform semantic understanding and information extraction to complete the extraction of the invoice text.

3. The invoice information extraction and invoice category determination method based on a large language model according to claim 2 is characterized in that: The obtaining of the invoice file and extracting the invoice text of the invoice file further comprises: For the invoice image and the image converted from the scanned copy in the PDF file, the text area of ​​the image is detected based on the local binary pattern and phase consistency: The calculation formula for the LBP value of each pixel is as follows: Among them, I(x,y) is the gray value of the pixel (x,y), I(x i ,y i ) is the pixel point (x i ,y i ), P is the number of surrounding pixels, and s(x) is the sign function, as follows: Phase consistency is defined as follows: Among them, A n (x,y) is the amplitude of the nth frequency component, is the phase, N is the total number of frequency components, and the similarity of each pixel to other pixels is calculated based on the local binary pattern and phase consistency as follows: After calculating the similarity of each pixel with other pixels, the text region is detected using region growing technique; Extract the text from all text areas and combine them into the invoice text.

4. The invoice information extraction and invoice category determination method based on a large language model according to claim 3 is characterized in that: The method of detecting the text region using the region growing technique includes: Select a seed pixel from the image as a starting point, and select pixels above a preset similarity threshold as seed pixels; Determine the conditions for merging pixels into regions and define a similarity threshold. Only when the similarity of adjacent pixels is higher than this threshold, they will be merged into the same region. Starting from the seed pixel, check its neighboring pixels. For each neighboring pixel, check its similarity with the pixels in the current region. If the similarity of the neighboring pixel is higher than the preset threshold and has not been assigned to any region, add it to the current region and mark it as visited. This process is repeated recursively or iteratively until no more neighboring pixels that meet the criteria can be found; when all neighboring pixels have been checked, all pixels in the current region are merged into a connected region.

5. The invoice information extraction and invoice category determination method based on a large language model according to claim 3 is characterized in that: The method of extracting the text from all text areas and integrating them into invoice text includes: Connect pixels in the text area into character shapes and determine the boundaries of characters based on the relative positions and pixel intensities of the pixels; Extract characters by detecting specific patterns or contours of character shapes, and perform simple pattern matching or rule matching on the extracted characters; The recognized and extracted characters are reconstructed into complete text lines or paragraphs according to their layout order in the image, and the reconstructed text lines or paragraphs are combined into the invoice text of the entire image.

6. A system for extracting invoice information and determining invoice categories based on a large language model, characterized in that: include: A text extraction module, used to obtain an invoice file and extract the invoice text of the invoice file; An information extraction module, used for designing first-level prompt words, inputting the first-level prompt words and the invoice text into a preset large language model, and the large language model outputs key information of the invoice text under the guidance of the first-level prompt words; The classification discrimination module is used to design a second-level prompt word for each commodity in the key information of the invoice text, wherein the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category, and input the second-level prompt word into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word; the second-level prompt word is designed for each commodity in the key information of the invoice text, wherein the second-level prompt word includes the commodity name, the corresponding commodity category and the invoicing category, and input the second-level prompt word into the large language model, and the large language model outputs the invoicing category coding result of each commodity under the guidance of the second-level prompt word. Construct a mapping relationship table from product name to product category, from invoicing category to code, and from product name to code; design a second-level prompt word for each product in the key information of the invoice text based on the mapping relationship table; input the second-level prompt word into the large language model, and the large language model outputs the invoicing category coding result of each product under the guidance of the second-level prompt word; for products for which the large language model cannot output the invoicing category coding result, repeatedly design a broader third-level prompt word and input it into the large language model until the large language model can output the invoicing category coding result of the remaining products under the guidance of the third-level prompt word; post-process the invoicing category coding result of each product output by the large language; The data output module is used to integrate and output the key information of the invoice text and the invoicing category coding result of each commodity.

7. An electronic device, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a method for extracting invoice information and determining invoicing categories based on a large language model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a program, which, when executed by a processor, is used to implement a method for extracting invoice information and determining invoicing categories based on a large language model as described in any one of claims 1 to 5.

Images