A Method and System for Recognizing Expense Details Based on OCR and LLM

Abstract

This invention relates to a method and system for recognizing expense details based on OCR and LLM, comprising: acquiring an image to be processed; detecting the expense detail region of the image and cropping a first sub-image; using OCR to recognize the first sub-image to obtain a first set; constructing an affine transformation matrix based on the first set to perform horizontal image correction and obtaining a horizontally corrected second sub-image; using OCR to recognize the second sub-image to obtain a second set; performing row and column correction on the text lines in the second sub-image based on the second set; and constructing a prompt based on the third set, and combining it with a large language model to obtain expense detail recognition information. This invention significantly improves text recognition accuracy under complex conditions through multi-stage image processing and OCR text recognition optimization; it introduces a large language model to empower spatial relationship understanding, achieving an intelligent leap from text data to specific information extraction, reducing manual intervention and improving efficiency.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and medical information technology, and in particular relates to a method and system for recognizing expense details based on OCR and LLM. Background Technology

[0002] Accurately and quickly parsing medical expense details is a major challenge in the medical expense reimbursement process. The inconsistencies in format and field names among different medical institutions further complicate the automatic identification and parsing of these details.

[0003] Traditional manual data entry is not only time-consuming and laborious, but also prone to errors. While current OCR technology can initially recognize text information, its accuracy is limited when processing non-standard format, tilted, or distorted documents. Furthermore, it lacks contextual understanding of the recognized data, making it difficult to directly apply it to the extraction of specific information.

[0004] To address this issue, large-scale LLM language models demonstrate powerful capabilities in text processing and contextual understanding, and can be used to interpret and extract information from OCR results. However, expense detail lists often contain a large number of data items, presented in rows and columns, with each column typically having a specific meaning, such as item name, unit price, quantity, and amount. Therefore, while LLM models have strong contextual understanding capabilities, they often fail to perform well for tasks requiring a certain understanding of spatial relationships to better parse textual information. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for identifying expense details based on OCR and LLM, so as to solve the problem that image distortion under various shooting angles makes it impossible to accurately extract specific information using the LLM model.

[0006] To achieve one of the above-mentioned objectives, an embodiment of the present invention provides a method for identifying expense detail lists based on OCR and LLM, the method comprising:

[0007] Acquire the image to be processed, detect the cost details area of ​​the image to be processed, and crop the first sub-image;

[0008] The first sub-image is identified using OCR to obtain a first set. An affine transformation matrix is ​​constructed based on the first set to perform horizontal image correction, and a horizontally corrected second sub-image is obtained.

[0009] The second sub-image is identified using OCR to obtain a second set. Based on the second set, the text lines in the second sub-image are corrected in rows and columns. The corrected text line information constitutes a third set.

[0010] Based on the third set, a prompt is constructed, and a large language model is used to obtain detailed cost identification information.

[0011] As a further improvement of one embodiment of the present invention, the method further includes using a target detection model to detect the cost detail region in the image to be processed, and cropping the image to be processed based on the bounding boxes in the detection results to generate a first sub-image containing only the cost detail region.

[0012] As a further improvement to one embodiment of the present invention, the method further includes, in which the step of using OCR to identify the first sub-graph / second sub-graph to obtain the first set / second set includes,

[0013] OCR recognition is used to perform text recognition on the first sub-image / second sub-image to obtain the position information of the minimum bounding rectangle of the text line and the text content recognition result. The position information of the minimum bounding rectangle of the text line and the text content are stored in the first set / second set.

[0014] As a further improvement to one embodiment of the present invention, the method further includes, in that the construction of the affine transformation matrix based on the first set includes,

[0015] Select the longest text line in the first set. Calculate the tilt angle θ of the longest text line based on the coordinates of the four vertices of its minimum bounding rectangle. Use this tilt angle θ as the tilt angle of the first sub-graph. Then, use the center point (x...) of the first sub-graph as the tilt angle. c ,y c Using θ as the center of rotation and the tilt angle θ as the rotation angle, construct the affine transformation matrix M;

[0016] The formula for calculating the tilt angle θ is expressed as follows:

[0017] The affine transformation matrix M is represented as:

[0018]

[0019] Where θ represents the tilt angle, (x1, y1) and (x2, y2) represent the coordinates of the top-left and top-right vertices of the minimum bounding rectangle of the longest text line; M represents the affine transformation matrix, (x c ,y c ) represents the coordinates of the center point of the first subgraph.

[0020] As a further improvement to one embodiment of the present invention, the method further includes, in which the step of performing image horizontal correction and obtaining the horizontally corrected second sub-image includes,

[0021] The first sub-image is subjected to an affine transformation using an affine transformation matrix M to obtain a candidate sub-image after horizontal correction. It is then determined whether the candidate sub-image is upright. If the candidate sub-image is upright, it is used as the second sub-image. If the candidate sub-image is inverted, it is rotated 180° around the center point of the image to obtain the second sub-image.

[0022] The affine transformation is expressed as:

[0023]

[0024] Where M is the affine transformation matrix, (X i ,Y i () represents the coordinates of the i-th pixel in the first sub-image. Let be the coordinates of the i-th pixel after affine transformation.

[0025] As a further improvement to one embodiment of the present invention, the method further includes: the line correction of the text lines in the second sub-image includes,

[0026] S31: Iterate through the text lines in the second set, performing the iteration on each text line encountered. j According to text line l j The coordinate information of the four vertices of the minimum bounding rectangle is used to calculate the text line l. j Center point coordinates (x j ,y j ), text line l j height h j Text line l j tilt angle α j and construct text lines l j The row domain Q j ;

[0027] S32: Calculate the center point coordinates of all text lines in the second set, and filter out the text lines whose center point coordinates are located in line region Q. j All text lines within the range; and will be located in line field Q j The ordinate of the center point of all text lines within the range is changed to the ordinate of the text line l. j The ordinate y j Then, the text lines with the same center point ordinate are sorted in ascending order according to the x-coordinate of the center point of the text line. The modified center point coordinates of the text lines, the coordinates of the four vertices of the smallest bounding rectangle of the text lines, and the character information recognized by OCR are recorded in the third set.

[0028] S33: Remove the text lines from the third set from the second set;

[0029] S34: Repeat steps S31 to S33 until all text lines in the second set have been moved to the third set;

[0030] The construction text line l j The row domain Q j Includes: connecting text lines j Find the top-left and top-right vertices of the smallest bounding rectangle, and place the text line l j Let s1 be the line containing the top left and top right vertices of the smallest bounding rectangle, and connect the text lines l. j The bottom right and bottom left vertices of the smallest bounding rectangle will be used to define the text line l. j The line containing the lower left and lower right vertices of the smallest bounding rectangle is denoted as s2. The region enclosed by lines s1, s2, and the left and right boundaries of the second subgraph is taken as the text line l. j The row domain Q j .

[0031] As a further improvement to one embodiment of the present invention, the method further includes the text line l j Center point coordinates (x j ,y j The calculation formula is expressed as follows:

[0032]

[0033] The text line l j height h j The calculation formula is expressed as follows:

[0034] The text line l j Inclination angle α j The calculation formula is expressed as:

[0035] Where j∈[1,N], N is the number of text lines in the second set, (x j1 ,y j1 ),(x j2 ,y j2 ),(x j3 ,y j3 (x) represents the coordinates of the top-left, top-right, and bottom-right vertices of the minimum bounding rectangle of the j-th text line in the second set. j ,y j ) represents the coordinates of the center point of the j-th text line.

[0036] As a further improvement to one embodiment of the present invention, the method further includes, in that, column correction of the text rows in the second sub-image includes,

[0037] S35: Iterate through the text lines in the third set, performing the iteration on each text line encountered. k Where k∈[1,N], and N is the number of text lines in the third set, based on text line l k Obtain the coordinate information of the four vertices of the minimum bounding rectangle to retrieve the text line l k center point coordinates ( x k,y k And construct column field W k ;

[0038] S36: Filter out text lines whose center point is located in column field W from the third set. k All text lines within the range will be located in column field W. k The x-coordinate of the center point of all text lines within the range is changed to the x-coordinate of the text line l. k The x-coordinate of the center point k And mark the modified text lines;

[0039] S37: Repeat the process of S35 to S36, skipping the marked text lines in S36, until the traversal of the third set is completed;

[0040] The constructed column field W k Includes: connecting text lines k Find the top-left and bottom-left vertices of the smallest bounding rectangle, and place the text line l k Let s3 be the line containing the top left and top right vertices of the smallest bounding rectangle, and connect the text lines l. k The top-right and bottom-right vertices of the smallest bounding rectangle will be used to define the text line l. k The line containing the lower left and lower right vertices of the smallest bounding rectangle is denoted as s4. The region enclosed by lines s3, s4, and the upper and lower boundaries of the second subgraph is taken as the text line l. k column field W K .

[0041] As a further improvement to one embodiment of the present invention, the method further includes, in the step of constructing a prompt based on the third set and obtaining cost detail identification information using a large language model, the following steps are included:

[0042] Based on the center point coordinates of the text lines in the third set and the OCR content recognition results, a prompt is constructed, including setting a task description, providing task-related contextual information or display, setting the input format, setting placeholders, setting output format requirements, inputting the constructed prompt into a large language model for contextual understanding and positional relationship analysis, extracting cost details recognition information, and outputting the results.

[0043] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a cost detail list recognition system based on OCR and LLM, the system comprising:

[0044] Image acquisition module: used to acquire the image to be processed;

[0045] Target detection module: used to detect the cost detail region of the image to be processed and extract the first sub-image;

[0046] OCR recognition module: used to perform OCR recognition on the first sub-image to obtain the first set; and to perform OCR recognition on the second sub-image to obtain the second set;

[0047] Correction processing module: used to perform horizontal correction on the first sub-image and row and column correction on the text lines in the second sub-image;

[0048] Large language model processing module: used to build a prompt and use a large language model to obtain detailed expense identification information;

[0049] The image acquisition module is connected to the target detection module and transmits the acquired image data to the target detection module; the target detection module is connected to the OCR recognition module and transmits the target region screenshot data to the OCR recognition module; the output of the OCR recognition module is connected to the correction processing module and transmits the recognition result to the correction processing module for correction; the output of the correction processing module is connected to the large language model processing module and sends the corrected text location information and text content into the large language model for analysis, outputting target information.

[0050] Compared with existing technologies, the OCR and LLM-based expense detail list recognition method provided by this invention significantly improves the accuracy of text recognition under complex conditions while also enhancing text recognition precision through multi-stage image processing and OCR text recognition optimization. It introduces a large language model to empower spatial relationship understanding, achieving an intelligent leap from text data to specific information extraction, reducing manual intervention and improving efficiency. This invention can effectively process medical expense detail images from different sources and of different qualities, enhancing the system's flexibility and versatility in practical applications. Through algorithm optimization, processing time is shortened while ensuring the reliability of data extraction, providing an efficient tool for medical expense management. Attached Figure Description

[0051] Figure 1 This is an overall flowchart of the cost detail list recognition method based on OCR and LLM described in this invention.

[0052] Figure 2This is a cost detail area sub-map provided in an embodiment of the cost detail list recognition method based on OCR and LLM described in this invention.

[0053] Figure 3 This is a flowchart of the image horizontal correction process in the OCR and LLM-based expense detail list recognition method described in this invention.

[0054] Figure 4 This is a flowchart of row and column correction of text rows in the second sub-graph of the cost detail list recognition method based on OCR and LLM described in this invention.

[0055] Figure 5 This is a schematic diagram of the text line field Q in the cost detail list recognition method based on OCR and LLM described in this invention.

[0056] Figure 6 This is a schematic diagram of the system structure of the expense detail list recognition system based on OCR and LLM described in this invention. Detailed Implementation

[0057] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0058] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0059] In Embodiment 1 of the present invention, the present invention provides a method for identifying expense detail lists based on OCR and LLM, such as... Figures 1-5 As shown, the method includes,

[0060] S1: Acquire the image to be processed, detect the cost details area of ​​the image to be processed, and crop the first sub-image;

[0061] In one specific embodiment of the present invention, such as Figure 2 As shown, "detecting the cost detail region of the image to be processed and cropping the first sub-image" includes: using an object detection model to perform object detection on the image to be processed, determining whether there is a cost detail region in the image, and if there is a cost detail region, cropping the image according to the bounding box in the detection result to generate a first sub-image A containing only the cost detail region, and deleting non-cost regions to eliminate interference.

[0062] In one specific embodiment of the present invention, the "object detection model" includes training an object detection model for the cost details region using a deep learning neural network.

[0063] In one specific embodiment of the present invention, "training the target detection model" includes collecting expense detail images uploaded by customers, manually annotating them using LabelImg, dividing the dataset into training, validation, and test sets in an 8:1:1 ratio, training the dataset using the PP-YOLOE neural network model, and outputting a target detection model for the expense detail area.

[0064] In one specific embodiment of the present invention, the structure of the target detection model includes: selecting the PP-YOLOE model as the target detection model; feeding the image to be detected as input into the target detection model, performing feature extraction through a convolutional neural network (CNN) to extract meaningful feature information from the image, including edges, textures, shapes, etc.; generating a series of candidate regions based on feature extraction, which may contain the target to be detected; performing classification and regression for each candidate region, where classification is used to determine whether the region contains the target, and regression is used to determine the pixel position of the target in the image; after classification and regression, the model may output multiple overlapping bounding boxes, and to eliminate these redundant bounding boxes, a non-maximum suppression algorithm is used to eliminate the redundant bounding boxes, finally outputting the detected target and its corresponding bounding box information.

[0065] It should be noted that choosing a lightweight backbone network can effectively reduce the complexity of the model while ensuring feature extraction capabilities, thereby improving detection speed; using feature pyramid fusion technology to integrate feature information from different levels can enhance the model's ability to detect targets at different scales; by optimizing the loss function, the target detection model can better balance positive and negative samples during training, improving the model's convergence speed and performance; the comprehensive application of these structures makes PP-YOLOE perform excellently in target detection tasks, ensuring both detection accuracy and speed.

[0066] It should also be noted that PP-YOLOE employs a multi-scale prediction head, enabling the model to predict targets of different sizes simultaneously, further improving the accuracy and robustness of detection.

[0067] S2: Use OCR to recognize the first sub-image to obtain the first set, construct an affine transformation matrix based on the first set to perform horizontal image correction, and obtain the horizontally corrected second sub-image;

[0068] In one specific embodiment of the present invention, such as Figure 3 As shown, "perform OCR recognition on the first sub-image to obtain the first set" includes,

[0069] Using OCR, text recognition is performed on the first sub-image A of the cost details area to obtain the position information of the minimum bounding rectangle of the text line and the text content recognition result, which constitutes the text line information list set L1 in the second sub-image, i.e., the first set is represented as: {"points":[(x1,y1),(x2,y2),(x3,y3),(x4,y4)],"text":"text line content"}, where points represent the coordinates of the four vertices of the minimum bounding rectangle of the text line, corresponding to the top left, top right, bottom right, and bottom left vertices, and text represents the text information of the text line.

[0070] In one specific embodiment of the present invention, "constructing an affine transformation matrix based on a first set" includes,

[0071] Select the longest text line l1 in the first set L1, and calculate the tilt angle θ of text line l1 based on the coordinates of the four vertices of the minimum bounding rectangle of the longest text line. The calculation formula is as follows: Using this tilt angle θ as the tilt angle of the entire text region, and the center point (x) of the first sub-image A c ,y c Using θ as the center of rotation and the tilt angle θ as the rotation angle, the affine transformation matrix M is constructed as follows:

[0072]

[0073] Where (x1,y1), (x2,y2), (x3,y3), (x4,y4) represent the coordinates of the top-left, top-right, bottom-right, and bottom-left vertices of the smallest bounding rectangle of the longest text line; (x c ,y c ) represents the coordinates of the center point of the first subgraph.

[0074] In one specific embodiment of the present invention, "obtaining the second sub-graph B for horizontal correction" includes: performing an affine transformation on the first sub-graph A using an affine transformation matrix M to obtain a candidate sub-graph for horizontal correction; and determining whether the candidate sub-graph is positive.

[0075] Affine transformation is represented as:

[0076]

[0077] Where M is the affine transformation matrix, (X i ,Y i () represents the coordinates of the i-th pixel in the first sub-image. Let be the coordinates of the i-th pixel after affine transformation.

[0078] The specific steps for determining whether a candidate subgraph is positive are as follows:

[0079] The longest text line l1 in the first set L1 is subjected to an affine transformation using the affine transformation matrix M to obtain the horizontally corrected text line l2.

[0080] Then, a general text orientation classification model is used to perform binary classification of text line l2 to obtain the orientation of text line l2. If text line l2 is positive, it indicates that the candidate sub-image is positive and is used as the second sub-image B; if text line l2 is negative, it indicates that the candidate sub-image is negative and is rotated 180° around the center point of the image to obtain the second sub-image B.

[0081] In one specific embodiment of the present invention, a text direction classification model is used to classify the direction of text lines. This includes cropping a sub-image of the text line based on the coordinate information of the minimum bounding rectangle of the text line, using the cropped sub-image as input, and feeding it into the ch_ppocr_mobile_v2.0_cls network model for algorithm inference. The inference result indicates whether the text line is forward or backward.

[0082] S3: Use OCR to recognize the second sub-image to obtain the second set. Based on the second set, perform text line correction and text row and column correction on the text in the second sub-image. The corrected text line information constitutes the third set.

[0083] In a specific embodiment of the present invention, the second set is represented as:

[0084] {"points":[(x1,y1),(x2,y2),(x3,y3),(x4,y4)],"text":"Text line content"}

[0085] Here, "points" represents the coordinates of the four vertices of the smallest bounding rectangle of the text line, corresponding to the top left, top right, bottom right, and bottom left vertices, and "text" represents the text information of the text line.

[0086] In one specific embodiment of the present invention, such as Figure 4 As shown, text line correction of the text in the second subgraph based on the second set includes,

[0087] S301: Iterate through the text lines in the second set, performing the iteration on each text line encountered. j Where j∈[1,N], and N is the number of text lines in the second set, based on text line l j The location information is the coordinates of the four vertices of the minimum bounding rectangle [(x j1 ,y j1 ),(x j2 ,y j2 ),(x j3 ,yj3 ),(x j4 ,y j4 )] Calculate text line l j Center point coordinates (x j ,y j ), text line l j height h j Text line l j Horizontal tilt angle α j and construct text lines l j The row domain Q j ;

[0088] text line l j Center point coordinates (x j ,y j The calculation formula is expressed as follows:

[0089]

[0090] text line l j height h j The calculation formula is expressed as follows:

[0091] text line l j Horizontal tilt angle α j The calculation formula is expressed as:

[0092] Among them, (x j1 ,y j1 ),(x j2 ,y j2 ),(x j3 ,y j3 ),(x j4 ,y j4 (x) represents the coordinates of the top-left, top-right, bottom-right, and bottom-left vertices of the minimum bounding rectangle of the j-th text line in the second set. j ,y j ) represents the coordinates of the center point of the j-th text line.

[0093] S302: Calculate the center coordinates of all text lines in the second set L2, and filter out the text lines whose center coordinates are located in the line region Q. j The text lines within the range; and the text lines located in line field Q j The y-coordinate of the center point of the text line within the range is changed to the text line l. j The ordinate y j The text lines are then sorted according to the x-coordinate of their center point, and the coordinates of the center point, the coordinates of the four vertices of the smallest bounding rectangle, and the character information are recorded in a third set L3.

[0094] S303: Remove the text lines in the third set L3 from the second set L2;

[0095] S304: Repeat steps S301 to S303 until all text lines in the second set L2 are moved to the third set L3.

[0096] In a specific real-time scenario of this invention, the third set L3 constructed after text line correction is represented as:

[0097] {"center":(x0,y0),"points":[(x1,y1),(x2,y2),(x3,y3),(x4,y4)],"text":"text line content"}, where center represents the coordinates of the center point of the text line, points represent the coordinates of the four vertices of the smallest bounding rectangle of the text line, corresponding to the top left, top right, bottom right, and bottom left vertices, and text represents the text information of the text line;

[0098] In one specific embodiment of the present invention, such as Figure 5 As shown, “Constructing text line l” j The row domain Q j "Including connecting text lines l" j Find the top-left and top-right vertices of the smallest bounding rectangle, and place the text line l j Let s1 be the line containing the top left and top right vertices of the smallest bounding rectangle, and connect the text lines l. j The bottom right and bottom left vertices of the smallest bounding rectangle will be used to define the text line l. j The line containing the lower left and lower right vertices of the smallest bounding rectangle is denoted as s2. The region enclosed by lines s1, s2, and the left and right boundaries of the second subgraph is taken as the text line l. j The row domain Q j .

[0099] In one specific embodiment of the present invention, text row and column correction based on a third set includes,

[0100] S305: Iterate through the text lines in the third set, performing the iteration on each text line encountered. k Where k∈[1,N], and N is the number of text lines in the third set, based on text line l k Obtain the coordinate information of the four vertices of the minimum bounding rectangle to retrieve the text line l k Center point coordinates (x k ,y k And construct column field W k ;

[0101] S306: Filter out text lines whose center point is located in column field W from the third set. k All text lines within the range will be located in column field W. k The x-coordinate of the center point of all text lines within the range is changed to the x-coordinate of the text line l. k The x-coordinate of the center point k And record which lines of text were modified;

[0102] S307: Repeat the process of S305 to S306, skipping the modified text lines recorded in S306, until the traversal of the third set is completed.

[0103] In one specific embodiment of the present invention, "constructing column domain W" K "Including connecting text lines l" k Find the top-left and bottom-left vertices of the smallest bounding rectangle, and place the text line l k Let s3 be the line containing the top left and top right vertices of the smallest bounding rectangle, and connect the text lines l. k The top-right and bottom-right vertices of the smallest bounding rectangle will be used to define the text line l. k The line containing the lower left and lower right vertices of the smallest bounding rectangle is denoted as s4. The region enclosed by lines s3, s4, and the upper and lower boundaries of the second subgraph is taken as the text line l. k column field W K .

[0104] In a specific implementation scenario of the present invention, the third set L3 after text line correction and text column correction is represented as: [{"p":[1462,830],"t":"Medical Insurance Coverage"},{"p":[1658,833],"t":"Outside Medical Insurance"},{"p":[146,857],"t":"Level II Nursing Care"}];

[0105] Where p represents the coordinates of the center point of the text line, and t represents the text character information recognized by OCR of the text line.

[0106] It should be noted that in the third set L3, the y-coordinates of the center points of the same line of text are consistent, and the x-coordinates of the center points of the same column of text are consistent. Text lines with consistent y-coordinates are sorted in order of their x-coordinate values, and text lines with consistent x-coordinates are sorted in order of their y-coordinate values.

[0107] S4: Construct a prompt based on the third set and use a large language model to obtain detailed cost identification information.

[0108] In one specific embodiment of the present invention, "building a prompt" includes: setting a task description: briefly describing the task that the model is expected to perform; setting a context or example: providing contextual information related to the task or showing one or more examples to help the model understand how to parse the cost details; setting an input format description: detailing the format and structure of the input data so that the model can parse it correctly; setting placeholders: reserving spaces in the prompt template for inserting actual cost detail data; and setting output format requirements: explicitly specifying the output format and structure expected to be returned by the model.

[0109] In a specific implementation scenario of this invention, the prompt data is constructed as follows: "Your current task is to extract the key information I specify from the OCR text recognition results of the expense details image. The OCR text recognition results are enclosed in ``` symbols and contain the recognized text and its coordinate information in the image, in order from left to right and from top to bottom in the original image; an example of the OCR text recognition results: [{"p":[1462,830],"t":"Medical Insurance Within"},{"p":[1658,833],"t":"Medical Insurance Outside"}], where p is the horizontal and vertical pixel coordinates of the text, and t is the recognized text content. The key information I specify is enclosed in [] symbols; please note the OCR text recognition..." The results may contain issues such as long sentences being truncated, unreasonable word segmentation, misaligned correspondences, and multiple results being paginated. You need to make a comprehensive judgment based on the contextual semantics and text coordinate information to extract accurate key information. When returning results, use JSON format, containing a key-value pair. The key is the key information I specify, and the value is the extracted result. The value is a string array. If it is believed that the key information is missing from the OCR recognition result, then assign "No relevant information found" to the value. Please only output the results in JSON format and do not include any other extra text! Let's begin: OCR text: ``````Key information to extract: [Details of expenses where the data in the medical insurance column is 0.00]"

[0110] In one specific embodiment of the present invention, "obtaining expense detail identification information using a large language model" includes: placing a third set L3 into a constructed prompt template, enclosing it with ``` symbols; sending the generated prompt data into a large language model LLM for model inference; the LLM analyzing the input prompt, understanding its context, and identifying the positional relationships between various fields; based on contextual understanding and positional relationship analysis, the LLM attempting to extract the values ​​of the target fields; and outputting the extracted field values ​​in a specified JSON format, with the output result being expense detail identification information.

[0111] It should be noted that the LLM large model adopts the Transformer architecture, especially the architecture based on the self-attention mechanism, which enables LLM to capture complex dependencies in language and generate high-quality natural language text. The advantages of this architecture are its parallel computing power, long-distance dependency capture ability and interpretability, which has enabled the LLM large model to achieve remarkable results in the field of NLP.

[0112] In Embodiment 2 of the present invention, the present invention provides a method and system for identifying expense detail lists based on OCR and LLM, such as... Figure 6 As shown, the system includes,

[0113] Image acquisition module: used to acquire the image to be processed;

[0114] Target detection module: used to detect the cost detail region of the image to be processed and extract the first sub-image;

[0115] OCR recognition module: used to perform OCR recognition on the first sub-image to obtain the first set; and to perform OCR recognition on the second sub-image to obtain the second set;

[0116] Correction processing module: used to perform horizontal correction on the first sub-image and row and column correction on the text lines in the second sub-image;

[0117] Large language model processing module: used to construct prompt data and use a large language model to obtain detailed expense identification information;

[0118] Output module: Used to output the information extracted by the large language model processing module in a visual or editable format for users to view or use.

[0119] It should be noted that the image acquisition module is connected to the target detection module, transmitting the acquired image data to the target detection module; the target detection module is connected to the OCR recognition module, transmitting the target region screenshot data to the OCR recognition module; the output of the OCR recognition module is connected to the correction processing module, transmitting the recognition result to the correction processing module for correction; the output of the correction processing module is connected to the large language model processing module, sending the corrected text location information and text content into the LLM large model for analysis; the output of the large language model processing module is connected to the output module, outputting the extracted information to the user interface.

[0120] In a third embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the cost detail list identification method based on OCR and LLM as described above.

[0121] In a fourth embodiment of the present invention, the present invention provides a storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps in the cost detail list identification method based on OCR and LLM as described above.

[0122] In summary, the expense detail list recognition method provided by this invention significantly improves the accuracy of text recognition under complex conditions while also enhancing text recognition precision through multi-stage image processing and OCR text recognition optimization. The introduction of a large language model empowers spatial relationship understanding, achieving an intelligent leap from text data to specific information extraction, reducing manual intervention and improving efficiency. This invention can effectively process medical expense detail images from different sources and of different qualities, enhancing the system's flexibility and versatility in practical applications. Algorithm optimization shortens processing time while ensuring the reliability of data extraction, providing an efficient tool for medical expense management.

[0123] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0124] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0125] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0126] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0127] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An OCR and LLM based expense itemization identification method, characterized in that: The method comprises the steps of: Specifically, a target detection model is used to detect the expense item area in the image to be processed, and the image to be processed is intercepted according to the boundary box in the detection result to generate a first sub-image containing only the expense item area. An OCR is used to recognize the first sub-image to obtain a first set, and an affine transformation matrix is constructed based on the first set to correct the image horizontally and obtain a second sub-image after horizontal correction. An OCR is used to recognize the second sub-image to obtain a second set, and the text lines in the second sub-image are corrected in rows and columns based on the second set, and the corrected text line information constitutes a third set. A prompt is constructed based on the third set, and a large language model is used to obtain expense item recognition information. The method comprises the steps of: An OCR is used to recognize the first sub-image to obtain a first set, and an affine transformation matrix is constructed based on the first set to correct the image horizontally and obtain a second sub-image after horizontal correction. The affine transformation is represented as: The method comprises the steps of: S33: Remove the text lines in the third set from the second set. S34: Repeat steps S31-S33 until all text lines in the second set are moved to the third set. Select the longest text line in the first set, and calculate the tilt angle of the longest text line according to the coordinate information of the four vertexes of the minimum circumscribed rectangle of the longest text line , the tilt angle , the tilt angle of the first subgraph as the center point of the first subgraph , the tilt angle as the rotation center, the rotation angle , the rotation angle, and construct an affine transformation matrix ; Tilt angle The calculation formula is expressed as: ; Affine transformation matrix is expressed as: ； wherein, denotes the tilt angle, denotes the coordinates of the top-left vertex and the coordinates of the top-right vertex of the minimum bounding rectangle of the longest text line; denotes the affine transformation matrix, is the center point coordinate of the first subgraph; The method comprises the steps of: Using an affine transformation matrix affine transformation is performed on the first subgraph to obtain a horizontal correction subgraph; it is judged whether the subgraph is forward or not, if the subgraph is forward, the subgraph is taken as the second subgraph; if the subgraph is reverse, the second subgraph is obtained by rotating the subgraph around the center point of the image by 180°. S37: Repeat S35-S36, and skip the text lines marked in S36 until the traversal processing of the third set is completed. ； wherein, is an affine transformation matrix, is the coordinate of the i-th pixel point in the first subgraph, is the coordinate of the i-th pixel point after the affine transformation. The method comprises the steps of: S31: Iterate through the text lines in the second set, performing the iteration on each text line encountered. According to the text line Calculate the text line using the coordinates of the four vertices of the minimum bounding rectangle. center point coordinates Text lines height Text lines tilt angle and construct text lines domain ; S32: Calculate the center point coordinates of all text lines in the second set, and select all text lines whose center point coordinates in the second set are located in the line domain within the range; and change the vertical coordinates of the center point coordinates of all text lines located in the line domain within the range to the vertical coordinates of the text lines , and sort the text lines with the same center point vertical coordinates in ascending order of the horizontal coordinates of the text line center points. Record the changed text line center point coordinates, the four vertex coordinates of the minimum bounding rectangle of the text line, and the character information recognized by OCR in the third set.​ The prompt is constructed based on the text line center point coordinates and the character information recognized by the OCR, including setting a task description, providing context information or demonstrations related to the task, setting an input format, setting a placeholder, setting an output format requirement, inputting the constructed prompt into a large language model for context understanding and position relationship analysis, extracting expense item recognition information, and outputting the result. The method comprises the steps of: The construction text line domain Includes: connecting text lines Find the top-left and top-right vertices of the smallest bounding rectangle, and place the text line... The line containing the top-left and top-right vertices of the smallest bounding rectangle is denoted as . Connect text lines The bottom right and bottom left vertices of the smallest bounding rectangle will be used to define the text lines. The line containing the lower left and lower right vertices of the smallest bounding rectangle is denoted as . ,straight line ,straight line The area enclosed by the left and right boundaries of the second subgraph is used as the text line. domain ; The text line The center point coordinates of the text line The calculation formula is represented as: ； the height of the text line The calculation formula is expressed as: ;​ The text line The inclination angle The calculation formula is expressed as: ; wherein, , is the number of text lines of the second set, , , denotes the top-left, top-right, bottom-right vertex coordinates of the minimum bounding rectangle of the th text line of the second set, denotes the center point coordinates of the th text line. An image acquisition module is configured to acquire an image to be processed. S35: Iterate through the text lines in the third set, performing the iteration on each text line encountered. ,in , The number of text lines in the third set, based on the number of text lines. Obtain the coordinates of the four vertices of the minimum bounding rectangle to retrieve the text line. center point coordinates And construct column fields ; S36: Select all text lines whose center points are located in the column domain of the third set, and change the horizontal coordinates of the center points of all text lines located in the column domain of the third set to the horizontal coordinates of the text lines whose center points are located in the column domain of the third set, and mark the modified text lines; A target detection module is configured to detect an expense item area in the image to be processed and intercept a first sub-image. The constructed column field Comprise: connecting the top left vertex and the bottom left vertex of the minimum bounding rectangle of the text line The straight line where the top left vertex and the bottom left vertex of the minimum bounding rectangle of the text line Are located is recorded as Connecting the top right vertex and the bottom right vertex of the minimum bounding rectangle of the text line The straight line where the top right vertex and the bottom right vertex of the minimum bounding rectangle of the text line Are located is recorded as The straight line The straight line The area surrounded by the straight line The straight line And the upper and lower boundaries of the second subgraph is taken as the column field of the text line An OCR recognition module is configured to perform OCR recognition on the first sub-image to obtain a first set, and perform OCR recognition on the second sub-image to obtain a second set. A correction processing module is configured to correct the first sub-image horizontally, and correct the text lines in the second sub-image in rows and columns.

2. An OCR and LLM based expense itemization identification system based on the method of claim 1, characterized by: ​ ​ ​ ​ ​ The large language model processing module is used for constructing a prompt and obtaining expense item identification information by using a large language model; The image acquisition module is connected with the target detection module, and image data acquired is transmitted to the target detection module; the target detection module is connected with the OCR recognition module, and target region screenshot data is transmitted to the OCR recognition module; the output of the OCR recognition module is connected with the correction processing module, and the recognition result is transmitted to the correction processing module for correction; the output of the correction processing module is connected with the large language model processing module, and the corrected text position information and text content are transmitted to the large language model for analysis, and target information is output.

Images