Electronic invoice information extraction method, device, electronic device and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and provides an electronic invoice information extraction method, device, electronic device and storage medium, the method comprising: recognizing an electronic invoice image, and semantically segmenting the recognition result to obtain cell information of multiple cells; classifying labels based on the cell information of multiple cells to obtain a label of each cell; recognizing multiple labels of multiple cells to determine the mapping relationship between each cell; normalizing the first target text information of each cell in the electronic invoice image to obtain second target text information; and extracting electronic invoice information from the second target text information according to the mapping relationship between each cell. The present invention semantically segments the recognition result, segments two overlapping boundary boxes, and then extracts electronic invoice information respectively, thereby improving the accuracy of the extracted electronic invoice information.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an electronic invoice information extraction method, device, electronic device and storage medium. Background Art

[0002] The charging items in medical electronic invoices are generally presented in a table format, and the table format is relatively fixed. In order to extract information from the electronic invoice, the existing technology usually performs structured processing on medical fee receipts in the form of pictures, and then collects and manages the receipt information.

[0003] However, there may be overlapping problems in the tables in medical electronic invoices. If each medical billing receipt is directly extracted after structured processing, it cannot be ensured that the extracted charging items match the charging amounts, resulting in confusing extracted electronic invoice information and low accuracy.

[0004] Therefore, it is necessary to propose a method that can accurately extract electronic invoice information. Summary of the invention

[0005] In view of the above, it is necessary to propose an electronic invoice information extraction method, device, electronic device and storage medium. By semantically segmenting the recognition results, the two overlapping boundary boxes are segmented and the electronic invoice information is extracted separately, thereby improving the accuracy of the extracted electronic invoice information.

[0006] A first aspect of the present invention provides a method for extracting electronic invoice information, the method comprising:

[0007] Receiving an electronic invoice image of the text to be extracted, and identifying the electronic invoice image to obtain a recognition result;

[0008] Performing semantic segmentation on the recognition result to obtain cell information of multiple cells of the electronic invoice image;

[0009] Performing label classification based on the cell information of the multiple cells to obtain a label for each cell;

[0010] Identify multiple labels of the multiple cells and determine a mapping relationship between the cells;

[0011] Normalizing the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image;

[0012] According to the mapping relationship between each cell, the electronic invoice information is extracted from the multiple second target text information of the multiple cells of the electronic invoice image.

[0013] Optionally, the semantic segmentation of the recognition result to obtain cell information of multiple cells of the electronic invoice image includes:

[0014] Performing sequence labeling on the first text information of each bounding box in the recognition result;

[0015] Inputting the first text information with sequence annotation into a pre-trained sequence annotation model for recognition to obtain the second text information of each bounding box;

[0016] The label in the second text information of each bounding box is identified, and semantic segmentation is performed on each corresponding bounding box to obtain cell information of multiple cells of the electronic invoice image.

[0017] Optionally, the identifying the label in the second text information of each bounding box, performing semantic segmentation on each corresponding bounding box, and obtaining the cell information of multiple cells of the electronic invoice image includes:

[0018] When it is recognized that the second text information of any one of the plurality of bounding boxes contains a plurality of labels, segmenting the any one of the bounding boxes according to the labels to obtain a plurality of cells;

[0019] Performing coordinate transformation on the first coordinate information of any one of the bounding boxes to obtain second coordinate information of each of the plurality of cells;

[0020] Determine first target text information of each cell according to second coordinate information of each cell in the plurality of cells;

[0021] The recognition result is updated according to the second coordinate information and the first target text information of each cell in the multiple cells to obtain the cell information of the multiple cells of the electronic invoice image.

[0022] Optionally, performing coordinate transformation on the first coordinate information of any one of the bounding boxes to obtain second coordinate information of each of the multiple cells includes:

[0023] Identify the character types to which all characters in the second text information in any one of the bounding boxes belong;

[0024] Determine the standard character of each character according to the character type to which each character belongs;

[0025] According to the character types to which all the characters belong and the corresponding standard characters of each of the characters, all the characters in the second text information of the any one boundary box are converted into standard characters, and the sum of the number of characters of the standard characters is calculated to determine the sum of the number of characters of the second text information of the any one boundary box;

[0026] According to the first coordinate information of any one of the boundary boxes and the sum of the number of standard characters of the second text information of any one of the boundary boxes, a preset formula is used to calculate the coordinate information of each character in any one of the boundary boxes, and the second coordinate information of each cell in the multiple cells is calculated according to the coordinate information of each character.

[0027] Optionally, performing label classification based on the cell information of the multiple cells to obtain a label for each cell includes:

[0028] The cell information is input into a pre-trained label classification model to obtain a label for each cell, wherein the training process of the label classification model includes:

[0029] Get historical cell information;

[0030] Extracting historical text information and coordinate information of each cell from the historical cell information;

[0031] Determine the basic features of each cell according to the historical text information of each cell, and determine the column alignment area features, relative position features and row adjacent cell features of each cell according to the coordinate information of each cell;

[0032] The basic features, column alignment region features, relative position features and row adjacent cell features of each cell are associated to obtain the target features of each cell;

[0033] Determine a training set and a test set from target features of a plurality of cells;

[0034] Training a preset fine-tuning model based on the training set to obtain a label classification model;

[0035] Input the test set into the label classification model for testing, and calculate the test pass rate;

[0036] If the test pass rate is greater than or equal to the preset pass rate threshold, it is determined that the label classification model training is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the label classification model training is re-performed.

[0037] Optionally, determining the column alignment region feature, relative position feature and row adjacent cell feature of each cell according to the coordinate information of each cell includes:

[0038] Randomly select any cell in any column as a target cell, and calculate the column alignment feature of the next row of cells of the target cell, wherein the calculation of the column alignment feature of the next row of cells of the target cell includes: recursively starting from the target cell, traversing the next row of cells of the target cell in turn, and calculating the column height of the next row of cells of the target cell to obtain the column height; calculating the column distance difference between the target cell and the next row of cells to obtain the column distance difference; calculating the overlap rate between the target cell and the next row of cells to obtain the row overlap rate; when the quotient of the column distance difference and the column height is less than or equal to a preset first threshold, and the row overlap rate is greater than or equal to a preset second threshold, it is determined that the target cell and the next row of cells of the target cell constitute an alignment area; when the target cell and the next row of cells of the target cell constitute an alignment area, calculate the average value of the basic features of the next row of cells of the target cell, and determine the average value as the column alignment feature of the next row of cells of the target cell; repeatedly perform the calculation of the column alignment feature of the next row of cells of the target cell until the column alignment features of all cells are extracted;

[0039] Identify the label of each cell to determine the title cell and the information cell, calculate the row relative distance between each information cell and each title cell, and extract the row relative distance to determine it as the relative position feature of the corresponding information cell;

[0040] The basic features, basic features of the left adjacent cell and basic features of the right adjacent cell of each cell are merged to obtain a merged basic feature, and the merged basic feature is determined as the row adjacent cell feature of each cell.

[0041] Optionally, the normalizing the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image includes:

[0042] Extracting a plurality of first keywords that hit a preset dictionary from the first target text information of each cell;

[0043] Extracting a preset number of second keywords from the preset dictionary according to the plurality of first keywords;

[0044] Calculate the similarity between any first keyword and any second keyword;

[0045] A second keyword with the greatest similarity is selected from the calculated similarities to be determined as the second target text information of the corresponding cell.

[0046] A second aspect of the present invention provides an electronic invoice information extraction device, the device comprising:

[0047] A receiving and identifying module, used for receiving an electronic invoice image of the text to be extracted, and identifying the electronic invoice image to obtain an identification result;

[0048] A segmentation module, used for performing semantic segmentation on the recognition result to obtain cell information of multiple cells of the electronic invoice image;

[0049] A classification module, used for performing label classification based on the cell information of the plurality of cells to obtain a label of each cell;

[0050] A determination module, used to identify multiple labels of the multiple cells and determine the mapping relationship between the cells;

[0051] A normalization processing module, used for performing normalization processing on the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image;

[0052] The extraction module is used to extract the electronic invoice information from the multiple second target text information of the multiple cells of the electronic invoice image according to the mapping relationship between the various cells.

[0053] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the electronic invoice information extraction method when executing a computer program stored in the memory.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the electronic invoice information extraction method when executed by a processor.

[0055] In summary, the electronic invoice information extraction method, device, electronic device and storage medium of the present invention obtain cell information of multiple cells of the electronic invoice image by identifying the electronic invoice image and semantically segmenting the identification result, thereby solving the problem that a bounding box contains multiple cells and improving the accuracy of the subsequently extracted electronic invoice information. Label classification is performed based on the cell information of the multiple cells to obtain the label of each cell, thereby avoiding the classification of labels of different categories into the same group, improving the accuracy of label classification, and thus ensuring the accuracy of each label. The first target text information of each cell in the electronic invoice image is normalized, and the normalization refers to updating the first target text information in each cell to ensure that the extracted electronic invoice information is a standard value, thereby improving the readability of the electronic invoice information. According to the mapping relationship between each cell, the electronic invoice information is extracted from the second target text information of multiple cells of the electronic invoice image, thereby avoiding the phenomenon of confusion of electronic invoice information caused by separate electronic invoice information, thereby improving the accuracy and efficiency of electronic invoice information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the electronic invoice information extraction method provided in the first embodiment of the present invention.

[0057] Figure 2 It is a structural diagram of the electronic invoice information extraction device provided in the second embodiment of the present invention.

[0058] Figure 3 It is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0059] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0061] Embodiment 1

[0062] Figure 1 It is a flow chart of the electronic invoice information extraction method provided in the first embodiment of the present invention.

[0063] In this embodiment, the electronic invoice information extraction method can be applied to electronic devices. For electronic devices that need to extract electronic invoice information, the electronic invoice information extraction function provided by the method of the present invention can be directly integrated into the electronic device, or run in the electronic device in the form of a software development kit (SDK).

[0064] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, as well as machine learning and deep learning.

[0066] like Figure 1 As shown, the electronic invoice information extraction method specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0067] S11, receiving an electronic invoice image of the text to be extracted, and identifying the electronic invoice image to obtain a recognition result.

[0068] In this embodiment, when the user extracts electronic invoice information, the user sends the electronic invoice image of the text to be extracted to the client through the client. Specifically, the client can be a smart phone, IPAD or other existing smart devices, and the server can be an electronic invoice information extraction subsystem. During the electronic invoice information extraction process, the client can send the electronic invoice image of the text to be extracted to the electronic invoice information extraction subsystem. When the electronic invoice information extraction subsystem receives the electronic invoice image of the text to be extracted, it identifies the electronic invoice image.

[0069] In this embodiment, in the field of digital medical technology, the electronic invoice image can be an invoice for payment items, an electronic invoice for medical fees, an outpatient fee receipt, or a physical examination report, for example, a routine blood test report, a routine urine test report, or other physical examination reports.

[0070] In an optional embodiment, the step of identifying the electronic invoice image to obtain an identification result includes:

[0071] Recognize the electronic invoice image using OCR to obtain information of multiple bounding boxes, wherein each bounding box information includes first coordinate information, confidence level, and first text information of each bounding box;

[0072] The information of the plurality of bounding boxes is determined as a recognition result.

[0073] In this embodiment, the first coordinate information of each bounding box includes coordinate information of the upper left corner, coordinate information of the lower left corner, coordinate information of the upper right corner, and coordinate information of the lower right corner of each bounding box.

[0074] In this embodiment, for outpatient fee receipts, the first text information may be other information such as item name, amount, remarks, etc.

[0075] S12, performing semantic segmentation on the recognition result to obtain cell information of multiple cells of the electronic invoice image.

[0076] In this embodiment, the cell information includes the second coordinate information and the first target text information of each cell. Cells are generally divided by blank areas. A bounding box represents a table unit, that is, a cell. There are also special cases. When a cell text is long, it may be spatially adhered to adjacent cells. There is a phenomenon that a bounding box contains multiple cells. It is necessary to perform semantic segmentation on the cells, which solves the problem of a bounding box containing multiple cells and improves the accuracy of the subsequently extracted electronic invoice information.

[0077] In an optional embodiment, the semantic segmentation of the recognition result to obtain the cell information of multiple cells of the electronic invoice image includes:

[0078] Performing sequence labeling on the first text information of each bounding box in the recognition result;

[0079] Inputting the first text information with sequence annotation into a pre-trained sequence annotation model for recognition to obtain the second text information of each bounding box;

[0080] The label in the second text information of each bounding box is identified, and semantic segmentation is performed on each corresponding bounding box to obtain cell information of multiple cells of the electronic invoice image.

[0081] In this embodiment, BIO sequence annotation may be used to perform sequence annotation on the first text information of each bounding box. The BIO sequence annotation is a prior art and will not be described in detail in this case.

[0082] Furthermore, the identifying of the label in the second text information of each bounding box, performing semantic segmentation on each corresponding bounding box, and obtaining the cell information of multiple cells of the electronic invoice image includes:

[0083] When it is recognized that the second text information of any one of the plurality of bounding boxes contains a plurality of labels, segmenting the any one of the bounding boxes according to the labels to obtain a plurality of cells;

[0084] Performing coordinate transformation on the first coordinate information of any one of the bounding boxes to obtain second coordinate information of each of the plurality of cells;

[0085] Determine first target text information of each cell according to second coordinate information of each cell in the plurality of cells;

[0086] The recognition result is updated according to the second coordinate information and the first target text information of each cell in the multiple cells to obtain the cell information of the multiple cells of the electronic invoice image.

[0087] In this embodiment, by semantically segmenting each bounding box, two overlapping bounding boxes can be segmented and electronic invoice information can be extracted separately, thereby solving the problem of confusing extracted electronic invoice information caused by directly extracting electronic invoice information from the bounding box and improving the accuracy of the extracted electronic invoice information.

[0088] Furthermore, the performing coordinate transformation on the first coordinate information of any one of the bounding boxes to obtain the second coordinate information of each of the plurality of cells includes:

[0089] Identify the character types to which all characters in the second text information in any one of the bounding boxes belong;

[0090] Determine the standard character of each character according to the character type to which each character belongs;

[0091] According to the character types to which all the characters belong and the corresponding standard characters of each of the characters, all the characters in the second text information of the any one boundary box are converted into standard characters, and the sum of the number of characters of the standard characters is calculated to determine the sum of the number of characters of the second text information of the any one boundary box;

[0092] According to the first coordinate information of any one of the boundary boxes and the sum of the number of standard characters of the second text information of any one of the boundary boxes, a preset formula is used to calculate the coordinate information of each character in any one of the boundary boxes, and the second coordinate information of each cell in the multiple cells is calculated according to the coordinate information of each character.

[0093] Specifically, using the following preset formula to calculate the coordinate information of the upper left corner and the coordinate information of the upper right corner of the nth character in any one of the boundary boxes includes:

[0094]

[0095]

[0096] x n_0 =x n-1_3 ,

[0097] y n_0 =y n-1_3 ,

[0098]

[0099]

[0100] x 0_0 =x zri_0 ,

[0101] y 0_0 =y ori_0 ,

[0102] in, Represents the coordinate information of the upper left corner of any bounding box. represents the coordinate information of the upper right corner of any bounding box, all_normal_char represents the sum of the standard characters of the second text information of any bounding box; normal_char n Represents the standard number of characters for the nth character, (x n_0 ,y n_0 ) represents the coordinate information of the upper left corner of the nth character, (x n_3 ,y n_3 ) represents the coordinate information of the upper right corner of the nth character.

[0103] For example, the standard character of a Chinese character is 1, the standard character of an uppercase English character is 0.75, the standard character of a lowercase English character is 0.5, the standard character of a punctuation mark is 0.5, and the normal_character is 1. nThe standard character representing the nth character. For example, if the second text information is "Hailun Eye Drops Levofloxacin Hydrochloride" and the nth character is set to "salt", then normal_char n The labeled character 1 representing the 6th character.

[0104] Specifically, the principle of calculating the coordinate information of the lower left corner and the lower right corner of the nth character in any one of the bounding boxes is the same as that of calculating the coordinate information of the upper left corner and the upper right corner of the nth character in any one of the bounding boxes.

[0105] In this embodiment, in order to ensure the accuracy of semantic segmentation, the first text information in each bounding box is subjected to sequence labeling and then semantic segmentation. At the same time, the first text information in each bounding box is converted into standard characters and then coordinate conversion is performed, so as to obtain the cell information of multiple cells of the electronic invoice image. On the one hand, through sequence labeling, the accuracy of the cells after semantic segmentation can be ensured. On the other hand, by converting the first text information into standard characters and then performing coordinate conversion, the coordinate information of each cell is obtained, ensuring the integrity of the cell information of each cell after semantic segmentation.

[0106] S13, perform label classification based on the cell information of the multiple cells to obtain the label of each cell.

[0107] In this embodiment, after obtaining the cell information, since the cell information includes the second coordinate information and the first target text information of each cell, the label of each cell can be obtained according to the first target text information of each cell.

[0108] In an optional embodiment, the performing label classification based on the cell information of the multiple cells to obtain the label of each cell includes:

[0109] Input the cell information of the multiple cells into a pre-trained label classification model to obtain the label of each cell.

[0110] Specifically, the training process of the label classification model includes:

[0111] Obtain historical cell information;

[0112] Extract the historical text information and coordinate information of each cell from the historical cell information;

[0113] Determine the basic features of each cell according to the historical text information of each cell, and determine the column alignment region feature, relative position feature and row adjacent cell feature of each cell according to the coordinate information of each cell;

[0114] The basic features, column alignment region features, relative position features and row adjacent cell features of each cell are associated to obtain the target features of each cell;

[0115] Determine a training set and a test set from target features of a plurality of cells;

[0116] Training a preset fine-tuning model based on the training set to obtain a label classification model;

[0117] Input the test set into the label classification model for testing, and calculate the test pass rate;

[0118] If the test pass rate is greater than or equal to the preset pass rate threshold, it is determined that the label classification model training is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the label classification model training is re-performed.

[0119] In this embodiment, a fine-tuning model may be pre-set, and the fine-tuning model is pre-set based on basic features of historical cells, column alignment region features, relative position features, and row adjacent cell features.

[0120] Furthermore, determining the basic features of each cell according to the historical text information of each cell includes:

[0121] When the historical text information of each cell hits a keyword in a preset dictionary, the first basic feature of each cell is determined to be 1; or, when the historical text information of each cell does not hit a keyword in the preset dictionary, the first basic feature of each cell is determined to be 0;

[0122] Counting the first number of times that any two characters in the historical text information of each cell are sorted and hit the keywords in the preset dictionary, and determining the first number as the second basic feature of each cell;

[0123] Counting the second number of times each word in the historical text information of each cell hits a keyword in a preset dictionary, and determining the second number as the third basic feature of each cell;

[0124] When the historical text information of each cell matches the preset amount regular expression, the fourth basic feature of each cell is determined to be 1; or, when the historical text information of each cell does not match the preset amount regular expression, the fourth basic feature of each cell is determined to be 0;

[0125] When the value in the historical text information of each cell is 1, the fifth basic feature of each cell is determined to be 1; or, when the value in the historical text information of each cell is not 1, the fifth basic feature of each cell is determined to be 0.

[0126] Further, determining the column alignment region feature, relative position feature and row adjacent cell feature of each cell according to the coordinate information of each cell includes:

[0127] Randomly select any cell in any column as a target cell, and calculate the column alignment feature of the next row of cells of the target cell, wherein the calculation of the column alignment feature of the next row of cells of the target cell includes: recursively starting from the target cell, traversing the next row of cells of the target cell in turn, and calculating the column height of the next row of cells of the target cell to obtain the column height; calculating the column distance difference between the target cell and the next row of cells to obtain the column distance difference; calculating the overlap rate between the target cell and the next row of cells to obtain the row overlap rate; when the quotient of the column distance difference and the column height is less than or equal to a preset first threshold, and the row overlap rate is greater than or equal to a preset second threshold, it is determined that the target cell and the next row of cells of the target cell constitute an alignment area; when the target cell and the next row of cells of the target cell constitute an alignment area, calculate the average value of the basic features of the next row of cells of the target cell, and determine the average value as the column alignment feature of the next row of cells of the target cell; repeatedly perform the calculation of the column alignment feature of the next row of cells of the target cell until the column alignment features of all cells are extracted;

[0128] Identify the label of each cell to determine the title cell and the information cell, calculate the row relative distance between each information cell and each title cell, and extract the row relative distance to determine it as the relative position feature of the corresponding information cell;

[0129] The basic features, basic features of the left adjacent cell and basic features of the right adjacent cell of each cell are merged to obtain a merged basic feature, and the merged basic feature is determined as the row adjacent cell feature of each cell.

[0130] In this embodiment, the preset first threshold may be 2, and the preset second threshold may be 0.8, which is not limited in this embodiment.

[0131] In this embodiment, the column height refers to the distance between the upper boundary line and the lower boundary line of each cell; the column distance difference refers to the difference between the minimum y-axis coordinate of the next cell of the target cell and the maximum y-axis coordinate of the target cell; the row overlap rate refers to the quotient of the difference between the upper boundary line of the target cell and the lower boundary line of the next cell of the target cell divided by the sum of the column height of the target cell and the column height of the next cell of the target cell.

[0132] Further, after determining that the target cell and the cells in the next row of the target cell form an alignment area, the method further includes:

[0133] If the quotient of the difference between the minimum value of the column of the cells in the next row of the target cell and the minimum value of the column of the target cell and the column height is less than a preset third threshold, it is determined that the target cell and the cells in the next row of the target cell are left-aligned; or

[0134] If the quotient of the difference between the maximum value of the column of the cells in the next row of the target cell and the maximum value of the column of the target cell and the column height is less than the preset third threshold, it is determined that the target cell and the cells in the next row of the target cell are right-aligned; or

[0135] If the quotient of the difference between the column middle value of the cells in the next row of the target cell and the column middle value of the target cell and the column height is less than the preset third threshold, it is determined that the target cell and the cells in the next row of the target cell are middle-aligned.

[0136] In this embodiment, the preset third threshold may be 1, which is not limited in this embodiment.

[0137] Furthermore, the method further comprises:

[0138] The quotient of the difference between the column minimum value of the cells in the next row of calculated target cells and the column minimum value of the target cell and the column height is greater than or equal to a preset third threshold, or the quotient of the difference between the column maximum value of the cells in the next row of calculated target cells and the column maximum value of the target cell and the column height is greater than or equal to the preset third threshold, or the quotient of the difference between the column middle value of the cells in the next row of calculated target cells and the column middle value of the target cell and the column height is greater than or equal to the preset third threshold, and it is determined that the target cell and the cells in the next row of calculated target cells do not constitute column alignment.

[0139] In this embodiment, the position coordinates of the corresponding cells can be adjusted by determining left alignment, right alignment and center alignment. When the electronic invoice information is extracted later, the position coordinates of the adjusted cells are taken into account, thereby improving the accuracy of extracting the electronic invoice information.

[0140] In this embodiment, when labeling and classifying the cell information of multiple cells, four aspects are considered from the basic features, column alignment area features, relative position features and row adjacent cell features in the cell information of each cell. Specifically, the basic features are determined from five dimensions: whether the historical text information of each cell hits the key in the preset dictionary, the number of times any two words are sorted and hit the key words in the preset dictionary, the number of times each word hits the key words in the preset dictionary, whether it matches the preset amount regular expression, and whether the value is 1, thereby ensuring the integrity of the basic features; the column alignment area features are considered from whether each cell is a column, thereby ensuring that the acquired cells will not be misplaced; the relative position features are considered from the relative position of each cell to the header cell while ensuring column alignment, thereby further ensuring the accuracy of the position coordinates of each cell; the row adjacent cell features expand the basic features of each cell and increase the features of each cell.

[0141] In this embodiment, the label of each cell is determined by considering multiple dimensions, thereby avoiding the classification of labels of different categories into the same group, improving the accuracy of label classification, and further ensuring the accuracy of each label.

[0142] S14, identifying multiple labels of the multiple cells, and determining a mapping relationship between the cells.

[0143] In this embodiment, a row relationship recognition algorithm can be used to determine the mapping relationship between each cell. Specifically, the row relationship recognition algorithm divides the multiple cells into multiple cells corresponding to each row, and identifies the positional relationship between the multiple cells corresponding to each row, thereby determining the mapping relationship between each cell.

[0144] S15, normalizing the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image.

[0145] In this embodiment, the normalization processing refers to updating the first target text information in each cell to ensure that the extracted electronic invoice information is a standard value, thereby improving the readability of the electronic invoice information.

[0146] In an optional embodiment, the normalizing the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image includes:

[0147] Extracting a plurality of first keywords that hit a preset dictionary from the first target text information of each cell;

[0148] Extracting a preset number of second keywords from the preset dictionary according to the plurality of first keywords;

[0149] Calculate the similarity between any first keyword and any second keyword;

[0150] A second keyword with the greatest similarity is selected from the calculated similarities to be determined as the second target text information of the corresponding cell.

[0151] In this embodiment, since the preset dictionary contains standard values, for example, if the first target text information is Chinese herbal medicine pieces, the Chinese herbal medicine pieces are normalized to Chinese medicine fees.

[0152] S16, extracting electronic invoice information from multiple second target text information of multiple cells of the electronic invoice image according to the mapping relationship between the cells.

[0153] In this embodiment, for the electronic invoice for charges, the mapping relationship refers to the association between the charge item and the amount or quantity. For example, the extracted charge item is the bed fee, and the mapping relationship of the corresponding amount is: XXXX yuan.

[0154] In this embodiment, by determining the mapping relationship between each cell and extracting the electronic invoice information according to the mapping relationship, the phenomenon of confusion of electronic invoice information caused by extracting the electronic invoice information separately is avoided, thereby improving the accuracy and efficiency of extracting the electronic invoice information.

[0155] In summary, the electronic invoice information extraction method described in this embodiment, by identifying the electronic invoice image, semantically segmenting the identification result, and obtaining the cell information of multiple cells of the electronic invoice image, solves the problem that a bounding box contains multiple cells, and improves the accuracy of the subsequently extracted electronic invoice information. Label classification is performed based on the cell information of the multiple cells to obtain the label of each cell, avoiding the classification of labels of different categories into the same group, improving the accuracy of label classification, and thus ensuring the accuracy of each label. The first target text information of each cell in the electronic invoice image is normalized, and the normalization refers to updating the first target text information in each cell to ensure that the extracted electronic invoice information is a standard value, thereby improving the readability of the electronic invoice information. According to the mapping relationship between each cell, the electronic invoice information is extracted from the multiple second target text information of the multiple cells of the electronic invoice image, avoiding the phenomenon of confusion of electronic invoice information caused by the separate electronic invoice information, thereby improving the accuracy and efficiency of the electronic invoice information extraction.

[0156] Embodiment 2

[0157] Figure 2 It is a structural diagram of the electronic invoice information extraction device provided in the second embodiment of the present invention.

[0158] In some embodiments, the electronic invoice information extraction device 20 may include multiple functional modules composed of program code segments. The program code of each program segment in the electronic invoice information extraction device 20 may be stored in the memory of the electronic device and executed by the at least one processor to perform (see Figure 1 Description) Function of extracting electronic invoice information.

[0159] In this embodiment, the electronic invoice information extraction device 20 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: a receiving and identifying module 201, a segmentation module 202, a classification module 203, a determination module 204, a normalization processing module 205 and an extraction module 206. The module referred to in the present invention refers to a series of computer-readable instruction segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0160] The receiving and identifying module 201 is used to receive an electronic invoice image of the text to be extracted, and identify the electronic invoice image to obtain a recognition result.

[0161] In this embodiment, when the user extracts electronic invoice information, the user sends the electronic invoice image of the text to be extracted to the client through the client. Specifically, the client can be a smart phone, IPAD or other existing smart devices, and the server can be an electronic invoice information extraction subsystem. During the electronic invoice information extraction process, the client can send the electronic invoice image of the text to be extracted to the electronic invoice information extraction subsystem. When the electronic invoice information extraction subsystem receives the electronic invoice image of the text to be extracted, it identifies the electronic invoice image.

[0162] In this embodiment, in the field of digital medical technology, the electronic invoice image can be an invoice for payment items, an electronic invoice for medical fees, an outpatient fee receipt, or a physical examination report, for example, a routine blood test report, a routine urine test report, or other physical examination reports.

[0163] In an optional embodiment, the receiving and identifying module 201 identifies the electronic invoice image, and the identification result obtained includes:

[0164] Recognize the electronic invoice image using OCR to obtain information of multiple bounding boxes, wherein each bounding box information includes first coordinate information, confidence level, and first text information of each bounding box;

[0165] The information of the plurality of bounding boxes is determined as a recognition result.

[0166] In this embodiment, the first coordinate information of each bounding box includes coordinate information of the upper left corner, coordinate information of the lower left corner, coordinate information of the upper right corner, and coordinate information of the lower right corner of each bounding box.

[0167] In this embodiment, for outpatient fee receipts, the first text information may be other information such as item name, amount, remarks, etc.

[0168] The segmentation module 202 is used to perform semantic segmentation on the recognition result to obtain cell information of multiple cells of the electronic invoice image.

[0169] In this embodiment, the cell information includes the second coordinate information and the first target text information of each cell. Cells are generally divided by blank areas. A bounding box represents a table unit, that is, a cell. There are also special cases. When a cell text is long, it may be spatially adhered to adjacent cells. There is a phenomenon that a bounding box contains multiple cells. It is necessary to perform semantic segmentation on the cells, which solves the problem of a bounding box containing multiple cells and improves the accuracy of the subsequently extracted electronic invoice information.

[0170] In an optional embodiment, the segmentation module 202 performs semantic segmentation on the recognition result, and obtains cell information of multiple cells of the electronic invoice image including:

[0171] Performing sequence labeling on the first text information of each bounding box in the recognition result;

[0172] Inputting the first text information with sequence annotation into a pre-trained sequence annotation model for recognition to obtain the second text information of each bounding box;

[0173] The label in the second text information of each bounding box is identified, and semantic segmentation is performed on each corresponding bounding box to obtain cell information of multiple cells of the electronic invoice image.

[0174] In this embodiment, BIO sequence annotation may be used to perform sequence annotation on the first text information of each bounding box. The BIO sequence annotation is a prior art and will not be described in detail in this case.

[0175] Furthermore, the identifying of the label in the second text information of each bounding box, performing semantic segmentation on each corresponding bounding box, and obtaining the cell information of multiple cells of the electronic invoice image includes:

[0176] When it is recognized that the second text information of any one of the plurality of bounding boxes contains a plurality of labels, segmenting the any one of the bounding boxes according to the labels to obtain a plurality of cells;

[0177] Performing coordinate transformation on the first coordinate information of any one of the bounding boxes to obtain second coordinate information of each of the plurality of cells;

[0178] Determine first target text information of each cell according to second coordinate information of each cell in the plurality of cells;

[0179] The recognition result is updated according to the second coordinate information and the first target text information of each cell in the multiple cells to obtain the cell information of the multiple cells of the electronic invoice image.

[0180] In this embodiment, by semantically segmenting each bounding box, two overlapping bounding boxes can be segmented and electronic invoice information can be extracted separately, thereby solving the problem of confusing extracted electronic invoice information caused by directly extracting electronic invoice information from the bounding box and improving the accuracy of the extracted electronic invoice information.

[0181] Furthermore, the performing coordinate transformation on the first coordinate information of any one of the bounding boxes to obtain the second coordinate information of each of the plurality of cells includes:

[0182] Identify the character types to which all characters in the second text information in any one of the bounding boxes belong;

[0183] Determine the standard character of each character according to the character type to which each character belongs;

[0184] According to the character types to which all the characters belong and the corresponding standard characters of each of the characters, all the characters in the second text information of the any one boundary box are converted into standard characters, and the sum of the number of characters of the standard characters is calculated to determine the sum of the number of characters of the second text information of the any one boundary box;

[0185] According to the first coordinate information of any one of the boundary boxes and the sum of the number of standard characters of the second text information of any one of the boundary boxes, a preset formula is used to calculate the coordinate information of each character in any one of the boundary boxes, and the second coordinate information of each cell in the multiple cells is calculated according to the coordinate information of each character.

[0186] Specifically, using the following preset formula to calculate the coordinate information of the upper left corner and the coordinate information of the upper right corner of the nth character in any one of the boundary boxes includes:

[0187]

[0188]

[0189] x n_0 = x n-1_3 ,

[0190] y n_0 = y n-1_3 ,

[0191]

[0192]

[0193] x 0_0 = x ori_0 ,

[0194] y 0_0 = y ori_0 ,

[0195] wherein, represents the coordinate information of the upper left corner of any one bounding box, represents the coordinate information of the upper right corner of any one bounding box, and all_normal_char represents the sum of the standard characters of the second text information of the any one bounding box; normal_char n represents the number of standard characters of the nth character, (x n_0 , y n_0 ) represents the coordinate information of the upper left corner of the nth character, (x n_3 , y n_3 ) represents the coordinate information of the upper right corner of the nth character.

[0196] Exemplarily, the standard character of a Chinese character is 1, the standard character of an uppercase English character is 0.75, the standard character of a lowercase English character is 0.5, the standard character of a punctuation mark is 0.5, and normal_char n represents the standard character of the nth character. For example, if the second text information is "Hailun Eye Drops Levofloxacin Hydrochloride", and the nth character is set as "salt", then normal_char n represents the marked character 1 of the 6th character.

[0197] Specifically, the principle of calculating the coordinate information of the lower left corner and the lower right corner of the nth character in the any one bounding box is the same as the principle of calculating the coordinate information of the upper left corner and the upper right corner of the nth character in the any one bounding box.

[0198] In this embodiment, in order to ensure the accuracy of semantic segmentation, the first text information in each bounding box is sequence-labeled and then semantically segmented. At the same time, the first text information in each bounding box is converted into standard characters and then coordinates are transformed to obtain cell information of multiple cells of the electronic invoice image. On the one hand, the accuracy of the cells after semantic segmentation can be ensured through sequence labeling. On the other hand, the coordinate information of each cell is obtained by converting the first text information into standard characters and then performing coordinate transformation, thereby ensuring the integrity of the cell information of each cell after semantic segmentation.

[0199] The classification module 203 is used to perform label classification based on the cell information of the multiple cells to obtain a label of each cell.

[0200] In this embodiment, after the cell information is obtained, since the cell information includes the second coordinate information and the first target text information of each cell, the label of each cell can be obtained according to the first target text information of each cell.

[0201] In an optional embodiment, the classification module 203 performs label classification based on the cell information of the multiple cells, and obtains the label of each cell including:

[0202] The cell information of the multiple cells is input into a pre-trained label classification model to obtain a label for each cell.

[0203] Specifically, the training process of the label classification model includes:

[0204] Get historical cell information;

[0205] Extracting historical text information and coordinate information of each cell from the historical cell information;

[0206] Determine the basic features of each cell according to the historical text information of each cell, and determine the column alignment area features, relative position features and row adjacent cell features of each cell according to the coordinate information of each cell;

[0207] The basic features, column alignment region features, relative position features and row adjacent cell features of each cell are associated to obtain the target features of each cell;

[0208] Determine a training set and a test set from target features of a plurality of cells;

[0209] Training a preset fine-tuning model based on the training set to obtain a label classification model;

[0210] Input the test set into the label classification model for testing, and calculate the test pass rate;

[0211] If the test pass rate is greater than or equal to the preset pass rate threshold, it is determined that the label classification model training is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the label classification model training is re-performed.

[0212] In this embodiment, a fine-tuning model may be pre-set, and the fine-tuning model is pre-set based on basic features of historical cells, column alignment region features, relative position features, and row adjacent cell features.

[0213] Furthermore, determining the basic features of each cell according to the historical text information of each cell includes:

[0214] When the historical text information of each cell hits a keyword in a preset dictionary, the first basic feature of each cell is determined to be 1; or, when the historical text information of each cell does not hit a keyword in the preset dictionary, the first basic feature of each cell is determined to be 0;

[0215] Counting the first number of times that any two characters in the historical text information of each cell are sorted and hit the keywords in the preset dictionary, and determining the first number as the second basic feature of each cell;

[0216] Counting the second number of times each word in the historical text information of each cell hits a keyword in a preset dictionary, and determining the second number as the third basic feature of each cell;

[0217] When the historical text information of each cell matches the preset amount regular expression, the fourth basic feature of each cell is determined to be 1; or, when the historical text information of each cell does not match the preset amount regular expression, the fourth basic feature of each cell is determined to be 0;

[0218] When the value in the historical text information of each cell is 1, the fifth basic feature of each cell is determined to be 1; or, when the value in the historical text information of each cell is not 1, the fifth basic feature of each cell is determined to be 0.

[0219] Further, the determining of the column alignment region feature, the relative position feature and the row adjacent cell feature of each cell according to the coordinate information of each cell includes:

[0220] Randomly select any cell in any column as a target cell, and calculate the column alignment feature of the next row of cells of the target cell, wherein the calculation of the column alignment feature of the next row of cells of the target cell includes: recursively starting from the target cell, traversing the next row of cells of the target cell in turn, and calculating the column height of the next row of cells of the target cell to obtain the column height; calculating the column distance difference between the target cell and the next row of cells to obtain the column distance difference; calculating the overlap rate between the target cell and the next row of cells to obtain the row overlap rate; when the quotient of the column distance difference and the column height is less than or equal to a preset first threshold, and the row overlap rate is greater than or equal to a preset second threshold, it is determined that the target cell and the next row of cells of the target cell constitute an alignment area; when the target cell and the next row of cells of the target cell constitute an alignment area, calculate the average value of the basic features of the next row of cells of the target cell, and determine the average value as the column alignment feature of the next row of cells of the target cell; repeatedly perform the calculation of the column alignment feature of the next row of cells of the target cell until the column alignment features of all cells are extracted;

[0221] Identify the label of each cell to determine the title cell and the information cell, calculate the row relative distance between each information cell and each title cell, and extract the row relative distance to determine it as the relative position feature of the corresponding information cell;

[0222] The basic features, basic features of the left adjacent cell and basic features of the right adjacent cell of each cell are merged to obtain a merged basic feature, and the merged basic feature is determined as the row adjacent cell feature of each cell.

[0223] In this embodiment, the preset first threshold may be 2, and the preset second threshold may be 0.8, which is not limited in this embodiment.

[0224] In this embodiment, the column height refers to the distance between the upper boundary line and the lower boundary line of each cell; the column distance difference refers to the difference between the minimum y-axis coordinate of the next cell of the target cell and the maximum y-axis coordinate of the target cell; the row overlap rate refers to the quotient of the difference between the upper boundary line of the target cell and the lower boundary line of the next cell of the target cell divided by the sum of the column height of the target cell and the column height of the next cell of the target cell.

[0225] Further, after determining that the target cell and the cells in the next row of the target cell form an alignment area, the method further includes:

[0226] If the quotient of the difference between the minimum value of the column of the cells in the next row of the target cell and the minimum value of the column of the target cell and the column height is less than a preset third threshold, it is determined that the target cell and the cells in the next row of the target cell are left-aligned; or

[0227] If the quotient of the difference between the maximum value of the column of the cells in the next row of the target cell and the maximum value of the column of the target cell and the column height is less than the preset third threshold, it is determined that the target cell and the cells in the next row of the target cell are right-aligned; or

[0228] If the quotient of the difference between the column middle value of the cells in the next row of the target cell and the column middle value of the target cell and the column height is less than the preset third threshold, it is determined that the target cell and the cells in the next row of the target cell are middle-aligned.

[0229] In this embodiment, the preset third threshold may be 1, which is not limited in this embodiment.

[0230] Further, the quotient of the difference between the column minimum value of the cells in the next row of the target cell and the column minimum value of the target cell and the column height is greater than or equal to a preset third threshold, or the quotient of the difference between the column maximum value of the cells in the next row of the target cell and the column maximum value of the target cell and the column height is greater than or equal to the preset third threshold, or the quotient of the difference between the column middle value of the cells in the next row of the target cell and the column middle value of the target cell and the column height is greater than or equal to the preset third threshold, and it is determined that the target cell and the cells in the next row of the target cell do not constitute column alignment.

[0231] In this embodiment, the position coordinates of the corresponding cells can be adjusted by determining left alignment, right alignment and center alignment. When the electronic invoice information is extracted later, the position coordinates of the adjusted cells are taken into account, thereby improving the accuracy of extracting the electronic invoice information.

[0232] In this embodiment, when labeling and classifying the cell information of multiple cells, four aspects are considered from the basic features, column alignment area features, relative position features and row adjacent cell features in the cell information of each cell. Specifically, the basic features are determined from five dimensions: whether the historical text information of each cell hits the key in the preset dictionary, the number of times any two words are sorted and hit the key words in the preset dictionary, the number of times each word hits the key words in the preset dictionary, whether it matches the preset amount regular expression, and whether the value is 1, thereby ensuring the integrity of the basic features; the column alignment area features are considered from whether each cell is a column, thereby ensuring that the acquired cells will not be misplaced; the relative position features are considered from the relative position of each cell to the header cell while ensuring column alignment, thereby further ensuring the accuracy of the position coordinates of each cell; the row adjacent cell features expand the basic features of each cell and increase the features of each cell.

[0233] In this embodiment, the label of each cell is determined by considering multiple dimensions, thereby avoiding the classification of labels of different categories into the same group, improving the accuracy of label classification, and further ensuring the accuracy of each label.

[0234] The determination module 204 is used to identify the multiple labels of the multiple cells and determine the mapping relationship between the cells.

[0235] In this embodiment, a row relationship recognition algorithm can be used to determine the mapping relationship between each cell. Specifically, the row relationship recognition algorithm divides the multiple cells into multiple cells corresponding to each row, and identifies the positional relationship between the multiple cells corresponding to each row, thereby determining the mapping relationship between each cell.

[0236] The normalization processing module 205 is used to perform normalization processing on the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image.

[0237] In this embodiment, the normalization processing refers to updating the first target text information in each cell to ensure that the extracted electronic invoice information is a standard value, thereby improving the readability of the electronic invoice information.

[0238] In an optional embodiment, the normalization processing module 205 performs normalization processing on the first target text information of each cell in the electronic invoice image, and obtains the second target text information of each cell in the electronic invoice image, including:

[0239] Extracting a plurality of first keywords that hit a preset dictionary from the first target text information of each cell;

[0240] Extracting a preset number of second keywords from the preset dictionary according to the plurality of first keywords;

[0241] Calculate the similarity between any first keyword and any second keyword;

[0242] A second keyword with the greatest similarity is selected from the calculated similarities to be determined as the second target text information of the corresponding cell.

[0243] In this embodiment, since the preset dictionary contains standard values, for example, if the first target text information is Chinese herbal medicine pieces, the Chinese herbal medicine pieces are normalized to Chinese medicine fees.

[0244] The extraction module 206 is used to extract the electronic invoice information from the multiple second target text information of the multiple cells of the electronic invoice image according to the mapping relationship between the various cells.

[0245] In this embodiment, for the electronic invoice for charges, the mapping relationship refers to the association between the charge item and the amount or quantity. For example, the extracted charge item is the bed fee, and the mapping relationship of the corresponding amount is: XXXX yuan.

[0246] In this embodiment, by determining the mapping relationship between each cell and extracting the electronic invoice information according to the mapping relationship, the phenomenon of confusion of electronic invoice information caused by extracting the electronic invoice information separately is avoided, thereby improving the accuracy and efficiency of extracting the electronic invoice information.

[0247] In summary, the electronic invoice information extraction device described in this embodiment, by identifying the electronic invoice image, semantically segmenting the identification result, and obtaining the cell information of multiple cells of the electronic invoice image, solves the problem that a boundary box contains multiple cells, and improves the accuracy of the subsequently extracted electronic invoice information. Label classification is performed based on the cell information of the multiple cells to obtain the label of each cell, avoiding the classification of labels of different categories into the same group, improving the accuracy of label classification, and thus ensuring the accuracy of each label. The first target text information of each cell in the electronic invoice image is normalized, and the normalization refers to updating the first target text information in each cell to ensure that the extracted electronic invoice information is a standard value, thereby improving the readability of the electronic invoice information. According to the mapping relationship between each cell, the electronic invoice information is extracted from the multiple second target text information of the multiple cells of the electronic invoice image, avoiding the phenomenon of confusion of electronic invoice information caused by the separate electronic invoice information, thereby improving the accuracy and efficiency of the electronic invoice information extraction.

[0248] Embodiment 3

[0249] See also Figure 3 FIG. 2 is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31 , at least one processor 32 , at least one communication bus 33 and a transceiver 34 .

[0250] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention, and may be a bus structure or a star structure. The electronic device 3 may also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0251] In some embodiments, the electronic device 3 is an electronic device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 may also include client devices, which include but are not limited to any electronic product that can interact with a client through a keyboard, mouse, remote control, touchpad, or voice-controlled device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0252] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0253] In some embodiments, the memory 31 is used to store program codes and various data, such as the electronic invoice information extraction device 20 installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0254] In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The at least one processor 32 is the control core (Control Unit) of the electronic device 3, and uses various interfaces and lines to connect various components of the entire electronic device 3, and executes various functions and processes data of the electronic device 3 by running or executing programs or modules stored in the memory 31, and calling data stored in the memory 31.

[0255] In some embodiments, the at least one communication bus 33 is configured to implement connection and communication between the memory 31 and the at least one processor 32, etc.

[0256] Although not shown, the electronic device 3 may also include a power source (such as a battery) for supplying power to each component. Optionally, the power source may be logically connected to the at least one processor 32 through a power management device, so that the power management device can manage charging, discharging, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0257] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0258] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) or a processor to execute a part of the method described in each embodiment of the present invention.

[0259] In a further embodiment, in combination Figure 2 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed application programs (such as the electronic invoice information extraction device 20), program codes, etc., for example, the above-mentioned modules.

[0260] The memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to execute related functions. For example, Figure 2 The modules described are program codes stored in the memory 31 and are executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of extracting electronic invoice information.

[0261] Exemplarily, the program code may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the program code in the electronic device 3. For example, the program code may be divided into a receiving and identifying module 201, a segmenting module 202, a classification module 203, a determining module 204, a normalization processing module 205, and an extraction module 206.

[0262] In one embodiment of the present invention, the memory 31 stores a plurality of computer-readable instructions, and the plurality of computer-readable instructions are executed by the at least one processor 32 to implement the function of extracting electronic invoice information.

[0263] Specifically, the specific implementation method of the at least one processor 32 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0264] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0265] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0266] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0267] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or, and the singular does not exclude the plural. The multiple units or devices stated in the present invention can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0268] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for extracting electronic invoice information, characterized in that: The method comprises: Receiving an electronic invoice image of the text to be extracted, and identifying the electronic invoice image to obtain a recognition result; The recognition result is semantically segmented to obtain cell information of multiple cells of the electronic invoice image, including: performing sequence labeling on the first text information of each bounding box in the recognition result; inputting the sequence-labeled first text information into a pre-trained sequence labeling model for recognition to obtain second text information of each bounding box; when it is recognized that the second text information of any one of the multiple bounding boxes contains multiple labels, semantically segmenting the any one bounding box according to the multiple labels to obtain multiple cells; performing coordinate conversion on the first coordinate information of the any one bounding box to obtain second coordinate information of each of the multiple cells, including: identifying all the characters in the second text information of the any one bounding box; The method comprises the steps of: determining a character type to which a character belongs; determining a standard character for each character according to the character type to which each character belongs; converting all characters in the second text information of any one of the boundary boxes into standard characters according to the character type to which all the characters belong and the corresponding standard characters of each character, and calculating the sum of the number of characters of the standard characters to determine the sum of the number of characters of the second text information of any one of the boundary boxes; calculating the coordinate information of each character in any one of the boundary boxes according to the first coordinate information of the any one of the boundary boxes and the sum of the number of characters of the standard characters of the second text information of the any one of the boundary boxes by using a preset formula, and obtaining the second coordinate information of each cell in the multiple cells according to the coordinate information of each character; Determine first target text information of each cell according to second coordinate information of each cell in the plurality of cells; Update the recognition result according to the second coordinate information of each cell in the plurality of cells and the first target text information to obtain cell information of the plurality of cells of the electronic invoice image; Performing label classification based on the cell information of the multiple cells to obtain a label for each cell; Identify multiple labels of the multiple cells and determine a mapping relationship between the cells; Normalizing the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image; According to the mapping relationship between each cell, the electronic invoice information is extracted from the multiple second target text information of the multiple cells of the electronic invoice image.

2. The method for extracting electronic invoice information according to claim 1, characterized in that: The performing label classification based on the cell information of the plurality of cells to obtain a label for each cell includes: The cell information is input into a pre-trained label classification model to obtain a label for each cell, wherein the training process of the label classification model includes: Get historical cell information; Extracting historical text information and coordinate information of each cell from the historical cell information; Determine the basic features of each cell according to the historical text information of each cell, and determine the column alignment area features, relative position features and row adjacent cell features of each cell according to the coordinate information of each cell; The basic features, column alignment region features, relative position features and row adjacent cell features of each cell are associated to obtain the target features of each cell; Determine a training set and a test set from target features of a plurality of cells; Training a preset fine-tuning model based on the training set to obtain a label classification model; Input the test set into the label classification model for testing, and calculate the test pass rate; If the test pass rate is greater than or equal to the preset pass rate threshold, it is determined that the label classification model training is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the label classification model training is re-performed.

3. The method for extracting electronic invoice information as claimed in claim 2, characterized in that: Determining the column alignment region feature, relative position feature, and row adjacent cell feature of each cell according to the coordinate information of each cell includes: Randomly select any cell in any column as a target cell, and calculate the column alignment feature of the next row of cells of the target cell, wherein the calculation of the column alignment feature of the next row of cells of the target cell includes: recursively starting from the target cell, traversing the next row of cells of the target cell in turn, and calculating the column height of the next row of cells of the target cell to obtain the column height; calculating the column distance difference between the target cell and the next row of cells to obtain the column distance difference; calculating the overlap rate between the target cell and the next row of cells to obtain the row overlap rate; when the quotient of the column distance difference and the column height is less than or equal to a preset first threshold, and the row overlap rate is greater than or equal to a preset second threshold, it is determined that the target cell and the next row of cells of the target cell constitute an alignment area; when the target cell and the next row of cells of the target cell constitute an alignment area, calculate the average value of the basic features of the next row of cells of the target cell, and determine the average value as the column alignment feature of the next row of cells of the target cell; repeatedly perform the calculation of the column alignment feature of the next row of cells of the target cell until the column alignment features of all cells are extracted; Identify the label of each cell to determine the title cell and the information cell, calculate the row relative distance between each information cell and each title cell, and extract the row relative distance to determine it as the relative position feature of the corresponding information cell; The basic features, basic features of the left adjacent cell and basic features of the right adjacent cell of each cell are merged to obtain a merged basic feature, and the merged basic feature is determined as the row adjacent cell feature of each cell.

4. The method for extracting electronic invoice information according to claim 1, characterized in that: The normalizing process of the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image includes: Extracting a plurality of first keywords that hit a preset dictionary from the first target text information of each cell; Extracting a preset number of second keywords from the preset dictionary according to the plurality of first keywords; Calculate the similarity between any first keyword and any second keyword; A second keyword with the greatest similarity is selected from the calculated similarities to be determined as the second target text information of the corresponding cell.

5. An electronic invoice information extraction device, characterized in that: The device is used to implement the electronic invoice information extraction method according to any one of claims 1 to 4, and the device includes: A receiving and identifying module, used for receiving an electronic invoice image of the text to be extracted, and identifying the electronic invoice image to obtain an identification result; A segmentation module, used for performing semantic segmentation on the recognition result to obtain cell information of multiple cells of the electronic invoice image; A classification module, used for performing label classification based on the cell information of the plurality of cells to obtain a label of each cell; A determination module, used to identify multiple labels of the multiple cells and determine the mapping relationship between the cells; A normalization processing module, used for performing normalization processing on the first target text information of each cell in the electronic invoice image to obtain the second target text information of each cell in the electronic invoice image; The extraction module is used to extract the electronic invoice information from the multiple second target text information of the multiple cells of the electronic invoice image according to the mapping relationship between the various cells.

6. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor is used to implement the electronic invoice information extraction method as described in any one of claims 1 to 4 when executing the computer program stored in the memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the electronic invoice information extraction method as described in any one of claims 1 to 4 is implemented.

Images