Method for extracting bill information and its device, equipment, medium and product

Abstract

The present application discloses a method for extracting bill information and its device, equipment, medium, and product, the method comprising: performing text recognition on bill images, obtaining the coordinate information of the text boxes of each text area therein and the text information in the text boxes; constructing graph structure data, wherein each node represents the coordinate information of a single text box, the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes; using a preset graph convolutional neural network to iteratively update the feature information of the nodes and edges in the graph structure data for multiple times, obtaining the updated graph structure data; inputting the feature information of the nodes in the updated graph structure data into a classification network for classification, and determining the classification labels corresponding to the text information of each text box. The present application can accurately extract the text information required for order data from bill images.

Description

Technical Field

[0001] The present application relates to the field of e-commerce information technology, and in particular to a bill information extraction method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] For some offline e-commerce scenarios, consumer users of the e-commerce platform may provide the ticket information corresponding to their e-commerce orders through third-party application screenshots or paper electronic pictures, such as the product information required for placing an order or the remittance payment information corresponding to a certain order. Based on this, it is necessary to obtain order-related data based on the pictures.

[0003] Obtaining the corresponding data of the order from the image involves the rapid extraction and structuring of key information in the transaction order, which helps merchants to quickly collect all transaction orders and quickly assist merchants to complete the statistics of order information, so as to help merchants grasp the overall transaction status and formulate corresponding promotion, delivery, production and other measures in a targeted manner. At the same time, it can also quickly grasp commercial confidential information to avoid omissions and leaks.

[0004] The industry generally processes image-based bill information by acquiring text information based on image text recognition, and then structuring the data in combination with conventional rule matching. This method is relatively primitive, and the processing process is complex and not intelligent enough. Especially when it is necessary to provide bill information extraction services in multiple languages ​​to multiple regions, it is not universal, so improvements are needed. Summary of the invention

[0005] The primary purpose of the present application is to solve the above-mentioned problems and to provide a bill information extraction method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A method for extracting bill information provided to meet one of the purposes of this application includes the following steps:

[0008] Performing text recognition on the bill image to obtain coordinate information of the text box of each text area therein and the text information in the text box;

[0009] Constructing graph structure data, wherein each node represents the coordinate information of a single text box and the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes;

[0010] Using a preset graph convolutional neural network to iteratively update the feature information of nodes and edges in the graph structure data multiple times to obtain updated graph structure data;

[0011] The feature information of the nodes in the updated graph structure data is input into the classification network for classification, and the classification label corresponding to the text information of each text box is determined.

[0012] A bill information extraction device provided to meet one of the purposes of the present application includes a text recognition module, a graph structuring module, an information updating module, and an information classification module. The text recognition module is used to perform text recognition on the bill image to obtain the coordinate information of the text box of each text area and the text information in the text box; the graph structuring module is used to construct graph structure data, in which each node represents the coordinate information of a single text box and the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes; the information updating module is used to use a preset graph convolutional neural network to perform multiple iterations on the feature information of the nodes and edges in the graph structure data to obtain the updated graph structure data; the information classification module is used to input the feature information of the nodes in the updated graph structure data into the classification network for classification, and determine the classification labels corresponding to the text information of each text box.

[0013] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the bill information extraction method described in the present application.

[0014] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the ticket information extraction method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0015] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the bill information extraction method described in any embodiment of the present application.

[0016] Compared with the prior art, the technical solution of this application has at least the following technical advantages:

[0017] First, the present application performs text recognition on the bill image, obtains the coordinate information of the text box corresponding to each text area and the text information in the text box, constructs the corresponding graph structure data, uses the nodes therein to store the coordinate information of the text box and the feature information of the text information, and defines the edges therein as the feature information of the relative distance information between any two nodes, and then uses a graph convolutional neural network to iteratively update the graph structure data multiple times, continuously extracts the high-level semantic information of the entire graph structure data, and classifies the feature information of the updated nodes in the graph structure data on the basis of realizing the update of the entire graph structure data, and correspondingly obtains the classification labels of the text information of each text box, thereby realizing the classification of the text information contained in the bill image. Since the high-level semantic information extracted by multiple iterations of the graph structure data is classified based on the graph convolutional neural network, it is not limited by the language of the text information in the bill image, and can realize cross-language recognition of the structured data in the bill image, obtain accurate classification results, and facilitate computer equipment to efficiently process these structured data.

[0018] Secondly, when constructing graph structure data, the present application realizes the association between the text boxes in the bill image and the text information therein through nodes, and establishes the association between each text box through the relative distance information of the text boxes between each two nodes. Therefore, it can guide the graph convolutional neural network to extract the corresponding deep semantics and realize the precise semantic representation of the relationship between the text and its position in the bill image, thereby ensuring accurate classification.

[0019] Furthermore, the present application integrates the text information in the text box in the bill image and the relative distance information of the text box through graph structure data, thereby realizing the effective fusion of information from multiple modalities. On this basis, reasoning about order data can be performed, which can provide richer reference information and guide the reasoning process to obtain more accurate data recognition results, thereby improving the accuracy of order data recognition.

[0020] In addition, the bill image of the present application can usually be obtained by users offline and provided online. Therefore, the e-commerce business model that combines online and offline is realized, which can assist merchants in quickly processing transaction order information, facilitate merchants to collect confidential information obtained by consumer users offline through online, and enhance merchants' information processing capabilities for global transactions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a typical embodiment of the method for extracting bill information of the present application;

[0023] Figure 2A schematic diagram of the process of detecting and identifying travel text from a bill image in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the process of constructing graph structure data of a bill image in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a process for constructing feature information of a node in the graph structure data in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the working process of the graph convolutional neural network in the embodiment of the present application;

[0027] Figure 6 An exemplary principle block diagram of a graph convolutional neural network in an embodiment of the present application;

[0028] Figure 7 A flowchart of a process for obtaining update information of a node in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a process of updating feature information of an edge in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the process of obtaining a bill image from an original image in an embodiment of the present application;

[0031] Fig.10 This is a screenshot example of an exemplary electronic payment voucher for this application;

[0032] Fig.11 A flow chart of the process of starting an e-commerce business process according to a structure obtained by graph convolutional neural network classification in an embodiment of the present application;

[0033] Fig.12 This is a principle block diagram of the bill information extraction device of this application;

[0034] Fig.13 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0036] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0037] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0038] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0039] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0040] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0041] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0042] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0043] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0044] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0045] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0046] A bill information extraction method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the application scenarios of the e-commerce platform including live e-commerce in the present application, it is generally deployed in a server for implementation. The method can be executed by accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface.

[0047] See also Figure 1 The bill information extraction method of the present application, in its typical embodiment, comprises the following steps:

[0048] Step S1100: Perform text recognition on the bill image to obtain coordinate information of the text box of each text area and text information in the text box;

[0049] Determine a bill image. In the exemplary e-commerce business scenario of the present application, the bill image is usually an image containing text content corresponding to the order data. The order data is various types of data related to one or more business links of the order business process of the e-commerce platform. For example, for the offline remittance payment link of the e-commerce order, the order data may include payment date, payment amount, payment amount name, handling fee amount, handling fee amount name, total amount, total amount name, remarks information and other different categories of data; for the logistics link of the e-commerce order, the order data may include express number, logistics carrier and other different categories of data. All of the above can be regarded as order data.

[0050] It should be noted that offline as referred to in this application refers to offline operations relative to e-commerce transactions, that is, operations that are not performed on the e-commerce platform referred to in this application, and does not mean that the user's operations must not rely on the Internet. For example, in the above example, if the user takes a screenshot to obtain the bill image after performing the corresponding payment operation through a third-party application, it is regarded as the offline remittance operation referred to in this application.

[0051] The receipt image may be obtained by intercepting the area where the image containing the order data is located from the original image submitted by the user of the e-commerce platform. The image determined as the receipt image is generally an image that can be manually identified as containing the order data, but for the computer equipment of the e-commerce platform, it can be determined in advance whether it contains the order data through technical identification. For images that do not contain the order data, no subsequent processing is required. For images that contain the order data, the receipt image can be obtained and further subjected to text recognition processing.

[0052] In one embodiment, to determine whether the original image contains the order data, a neural network model that has been pre-trained to a convergence state can be used. The neural network model has the ability to perform representation learning on the original image to obtain its corresponding image feature information. The image feature information is then input into a binary classifier for classification and judgment, thereby determining whether the original image contains the order data, and obtaining a bill image from the original image containing the order data.

[0053] It is not difficult to understand that in the bill image containing order data, there are various text areas containing text information, and the image corresponding to each text area is a line text image. When performing text recognition on the bill image containing order data, the text boxes corresponding to each text area in the bill image can be detected first, so as to obtain the corresponding line text image. The text box can be represented by the coordinate information of the corresponding text area. For example, the coordinates corresponding to the four corner points of the text area can be used to express the text box corresponding to the text area. Subsequently, the corresponding line text image can be cut out from the bill image according to the coordinate information, and then text recognition is performed on each line text image respectively, so as to obtain the text information corresponding to each line text image.

[0054] Determining the line text image from the bill image and recognizing the corresponding text information based on the line text image can be achieved by using the optical image recognition technology (OCR) in traditional technology, or by extracting and implementing it using a neural network model based on deep learning.

[0055] It can be known that the coordinate information of the text box of each line of text image indicates the image information corresponding to the order data, constituting one modality; the text information corresponding to each line of text image provides the text content corresponding to the order data, constituting another modality. Therefore, through this step, information of two modalities is obtained.

[0056] Step S1200, constructing graph structure data, wherein each node represents the coordinate information of a single text box and the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes;

[0057] In order to achieve a comprehensive representation of the information of the two modes obtained in the previous step, the coordinate information and text information corresponding to the text box in the bill image obtained in the previous step can be structured using a preset graph structure to obtain corresponding graph structure data.

[0058] In the graph structure data, corresponding to each node of the graph structure data constructed by each text box, each node can store and represent the feature information of the coordinate information of the corresponding text box, and store the feature information of the text information in the text box. The feature information of the coordinate information of the text box can be obtained by vectorizing the coordinate information. The feature information of the text information can be obtained by embedding the text information into a word to obtain its embedding vector.

[0059] In one embodiment, when embedding the text information, a multilingual dictionary is provided in advance, the text information is segmented, and then the feature values ​​corresponding to each segmented word are obtained by searching the multilingual dictionary, and the embedding vector is formed by these feature values. The multilingual dictionary can be a plurality of basic dictionaries independently provided for different languages, or a collection of dictionaries corresponding to all languages ​​into the same comprehensive dictionary.

[0060] The connection information between nodes in the graph structure data, that is, the edge between any two nodes, can be represented by calculating the relative distance between the two text boxes based on the coordinate information of the text boxes of each two nodes. For example, the relative distance between the two text boxes on the horizontal and vertical axes can be calculated by using the coordinates of the center points of each text box to jointly constitute the relative distance information.

[0061] It can be seen that in the graph structure data, the comprehensive representation of the feature information of the two modes in the bill information can be achieved, so that the order data contained in the bill image can be initially formed into semantically related structured representation information.

[0062] Step S1300: using a preset graph convolutional neural network to iteratively update the feature information of nodes and edges in the graph structure data for multiple times to obtain updated graph structure data;

[0063] For the graph structure data, a graph convolutional neural network (GCN) can be used to extract features to obtain deep semantic information corresponding to each node. After accessing the classification network, the graph convolutional neural network can be trained in advance using a sufficient number of training samples, trained to a convergent state, and learned to extract deep semantic information of the feature information of each node from the given graph structure data, and then put into use in this application.

[0064] When training the graph convolutional neural network, an appropriate amount of bill images can be prepared to obtain the coordinate information and text information of the text boxes of each text area therein, and then input them into the graph convolutional neural network for multiple iterations of representation learning and updating of the graph structure data. The feature information in the nodes in the graph structure data finally obtained is classified and mapped by the classification network to obtain its corresponding classification label. Then, the preset supervision label is used, and the cross entropy loss function is applied to calculate the model loss value corresponding to the classification label. Then, the graph convolutional neural network is back-propagated according to the model loss value to implement gradient update. In this way, the graph convolutional neural network is trained by iterative cycles of multiple training samples until the graph convolutional neural network model is judged to have reached a convergence state according to the model loss value obtained from the training, and the training can be terminated.

[0065] The graph convolutional neural network takes the feature information of the nodes and the feature information of the edges in the graph structure data as input, and extracts the deep semantic information of the feature information of each node and the feature information of the edge through a convolution operation, and then updates the feature information of the corresponding nodes and the feature information of the edges with the corresponding deep semantic information, thereby updating the graph structure data.

[0066] In one embodiment, in the process of applying the graph convolutional neural network to update the graph structure data, the graph structure data can be updated multiple times to improve the semantic depth of the updated feature information. The specific number of iterations can be flexibly set by those skilled in the art according to the principles disclosed above in this application.

[0067] Step S1400: input the feature information of the nodes in the updated graph structure data into the classification network for classification, and determine the classification label corresponding to the text information of each text box.

[0068] After completing the update of the graph structure data, the feature information of the nodes in the graph structure data can be input into a preset classification network for classification mapping. The classification network has been pre-configured to the number of classifications during the stage of participating in the graph convolutional neural network training. Therefore, the feature information of each node can be mapped to its corresponding classification, and the classification label of the text box corresponding to each node can be obtained. The classification of the text information in the text area in the bill image is completed, and the corresponding structured data is obtained, which can be used as order data.

[0069] According to the above embodiments, it can be seen that the present application has many advantages, including but not limited to:

[0070] First, the present application performs text recognition on the bill image, obtains the coordinate information of the text box corresponding to each text area and the text information in the text box, constructs the corresponding graph structure data, uses the nodes therein to store the coordinate information of the text box and the feature information of the text information, and defines the edges therein as the feature information of the relative distance information between any two nodes, and then uses a graph convolutional neural network to iteratively update the graph structure data multiple times, continuously extracts the high-level semantic information of the entire graph structure data, and classifies the feature information of the updated nodes in the graph structure data on the basis of realizing the update of the entire graph structure data, and correspondingly obtains the classification labels of the text information of each text box, thereby realizing the classification of the text information contained in the bill image. Since the high-level semantic information extracted by multiple iterations of the graph structure data is classified based on the graph convolutional neural network, it is not limited by the language of the text information in the bill image, and can realize cross-language recognition of the structured data in the bill image, obtain accurate classification results, and facilitate computer equipment to efficiently process these structured data.

[0071] Secondly, when constructing graph structure data, the present application realizes the association between the text boxes in the bill image and the text information therein through nodes, and establishes the association between each text box through the relative distance information of the text boxes between each two nodes. Therefore, it can guide the graph convolutional neural network to extract the corresponding deep semantics and realize the precise semantic representation of the relationship between the text and its position in the bill image, thereby ensuring accurate classification.

[0072] Furthermore, the present application integrates the text information in the text box in the bill image and the relative distance information of the text box through graph structure data, thereby realizing the effective fusion of information from multiple modalities. On this basis, reasoning about order data can be performed, which can provide richer reference information and guide the reasoning process to obtain more accurate data recognition results, thereby improving the accuracy of order data recognition.

[0073] In addition, the bill image of the present application can usually be obtained by users offline and provided online. Therefore, the e-commerce business model that combines online and offline is realized, which can assist merchants in quickly processing transaction order information, facilitate merchants to collect confidential information obtained by consumer users offline through online, and enhance merchants' information processing capabilities for global transactions.

[0074] See also Figure 2 In an embodiment further developed on the basis of any of the above embodiments, the step S1100 performs text recognition on the bill image, including the following steps:

[0075] Step S1110: calling a preset text detection model to detect the bill image, and obtaining coordinate information of text boxes corresponding to each text area therein;

[0076] In order to perform text recognition on the bill image, it is first necessary to obtain the line text images corresponding to each text area therein, and based on this, a preset text detection model can be used to detect the line text images of the bill image. Similarly, the text detection model constrains the size of the bill image input therein through input parameters, so the bill image containing the order data can be preprocessed according to its input parameter requirements, and adjusted to a specific size, such as 1024*1024 pixels, through operations such as cropping and scaling, to obtain the preprocessed bill image to meet the input requirements of the text detection model.

[0077] The text detection model can be implemented using any basic neural network model such as CNN, Resnet, etc. that can perform representation learning on images and combine with a classifier to achieve text detection. Similarly, the text detection model is pre-trained to a convergence state and can be trained accordingly by technical personnel in this field using a sufficient amount of corresponding training samples to enable it to learn the ability to identify text boxes corresponding to multiple lines of text images in a given pre-processed bill image.

[0078] After calling the text detection model to perform text recognition on the bill image, the text boxes corresponding to the various text areas therein can be obtained. The text boxes are described by the coordinates of the four corner points corresponding to the text areas, which constitute the coordinate information of the text boxes.

[0079] Step S1120, extracting line text images corresponding to each text area from the bill image according to the coordinate information of the text box;

[0080] The position of the corresponding line text image in the preprocessed bill image can be determined by the coordinates of the four corner points of each text box. Similarly, the specific position of the text box in the bill image before preprocessing can be further determined according to the scaling ratio relationship before and after the bill image preprocessing. Therefore, the line text images corresponding to each text box can be cut out from the bill image before or after preprocessing according to the coordinates of the corresponding four corner points of each text box, and these line text images are regarded as containing text information.

[0081] Step S1130 inputs the line text image into a text recognition model that has been pre-trained to a convergent state to perform text recognition to obtain corresponding text information.

[0082] A text recognition model is prepared for performing text recognition on each line of text images. The text recognition model can be implemented using any basic neural network model that can realize text detection, such as CRNN or AttentionOCR, wherein the RNN in the CRNN uses a bidirectional LSTM basic model as the backbone to enhance feature extraction, while the CNN part uses a general ordinary convolutional neural network model.

[0083] Similarly, the text recognition model is pre-trained to a convergence state, and can be trained accordingly by technicians in this field using a sufficient amount of corresponding training samples so that it can learn the ability to recognize text information in a given line text image.

[0084] At this point, by performing text recognition on the line text images corresponding to each text box using the text recognition model, the text information in each text area existing in the bill image can be obtained.

[0085] In this embodiment, a text detection model is first used to detect the bill image to determine the presence of multiple text boxes with text information, thereby obtaining the coordinate information of the line text image in the bill image, and then the corresponding line text image in the bill image is cropped according to the coordinate information of the text box, and a text recognition model is used to perform text recognition on each line text image to obtain the corresponding text content, that is, the corresponding text information, thereby completing the extraction of two modal information corresponding to the order data in the bill image, providing an effective data source for the present application to achieve fast and accurate identification of order data.

[0086] See also Figure 3 In an embodiment further developed on the basis of any of the above embodiments, the step S1200, constructing graph structure data, comprises the following steps:

[0087] Step S1210: Create a graph structure of graph structure data, wherein the graph structure includes nodes and edges for describing connection information between any two nodes;

[0088] The graph structure required for constructing graph structure data can be pre-defined, mainly defining the nodes and edges therein. The nodes are mainly used to represent the comprehensive features corresponding to the position of each text box and its text information, and the edges are mainly used to represent the relative distance features between any two text boxes.

[0089] In one embodiment, the node may be initialized to store the coordinate information of its corresponding text box, such as the coordinates of the four contact points of the text box; and the edge may be initialized to store the relative distance information between any two text boxes, such as the longitudinal relative distance and the lateral relative distance between the two text boxes.

[0090] Step S1220, obtaining a feature vector of coordinate information corresponding to each text box and a feature vector of text information corresponding to the feature vector stored as feature information in a node of the graph structure, wherein the coordinate information includes coordinates of four corner points of the text box;

[0091] Corresponding to each node in the graph structure data, the coordinate information of the text box stored in each node and the text information in the text box are vectorized, and then the nodes in the graph structure data are updated accordingly.

[0092] For example, for the coordinate information of the corresponding text box in each node, it can be directly converted into a feature vector to obtain the feature information of the coordinate information; for the text information in the corresponding text box in each node, it can be converted into a corresponding embedding vector as feature information using a preset multilingual dictionary, or further extracted from the embedding vector with the help of a text feature extraction model to obtain the corresponding deep semantic features as the feature information of the text information. Then, the feature information of the coordinate information of the text box corresponding to each node and the feature information of the text information in the text box are combined as the feature information corresponding to the node and stored in the node to implement the update of the node.

[0093] Step S1230, calculate the feature vector of the relative distance information of any two text boxes based on the coordinate information between the two text boxes, and store it as the feature information of the edge between the nodes corresponding to the two text boxes, wherein the relative distance information includes the longitudinal relative distance and the lateral relative distance between the two text boxes.

[0094] Corresponding to each edge in the graph structure data, for the two nodes corresponding to each edge, according to the coordinate information between the two text boxes corresponding to the two nodes, first determine the two numerical values ​​of the longitudinal relative distance and the lateral relative distance between the two coordinate information, and then vectorize them into a feature vector and store them in the feature information of the edge of the two nodes.

[0095] When determining the relative distance information between the two nodes corresponding to each edge, the coordinates of the center points of the two nodes can be determined based on the coordinate information of the text boxes of the two nodes. The principle is as follows:

[0096] Assume that the coordinates of the four corner points of each text box are: (x0, y0), (x1, y1), (x2, y2), (x3, y3), then the coordinates of the center point of each text box can be calculated according to the following formula:

[0097]

[0098]

[0099] Among them, x center is the horizontal coordinate of the center point of the text box, y center is the vertical coordinate of the center point of the text box, (x center ,y center ) are the center coordinates of the text box.

[0100] After determining the coordinates of the center points of the two text boxes, it is not difficult to understand that we only need to calculate the difference between their respective vertical and horizontal coordinates to obtain the corresponding values ​​of the vertical relative distance and the horizontal relative distance, and on this basis, vectorize to obtain the corresponding edge feature information.

[0101] Through the above process, the construction of the graph structure data can be realized, and the text information and position information corresponding to each text area in the bill image can be converted into feature information, and stored in the graph structure data according to the predefined graph structure, thereby realizing the structured and effective representation of multiple modal information of the text content contained in the bill image.

[0102] See also Figure 4 In an embodiment specific to any of the above embodiments, the step S1220, obtaining the feature vector of the coordinate information corresponding to each text box and the feature vector of the text information corresponding to the feature vector stored as feature information in a node in the graph structure, includes the following steps:

[0103] Step S1221, converting the coordinate information corresponding to each text box into a feature vector;

[0104] For each text box, its coordinate information includes the coordinates of its four corner points, and each coordinate includes two numerical values, the horizontal coordinate and the vertical coordinate. Based on this, the coordinates of the four corner points can be arranged in a certain preset order to obtain the feature vector of each text box. In one embodiment, the text box feature vector can also be normalized and encoded to obtain a new feature vector. Regardless of the method used, as long as the coordinate information of each text box is converted into a numerical feature to obtain a feature vector, the feature vector can be used as the feature information of the coordinate information of the text box.

[0105] Step S1222: Encode the embedding vector of the text information corresponding to each text box according to a preset multilingual dictionary, and use a preset text feature extraction model to extract each embedding vector to obtain a feature vector of the text information of each text box;

[0106] As mentioned above, in order to realize the vectorization of the text information in each text box, a preset multilingual dictionary can be used to encode the text information in each text box, obtain the feature values ​​of the word segments in the text information in the multilingual dictionary, and construct the feature values ​​of the text information into a feature vector according to the order of appearance of each word segment in the text information. The feature vector can theoretically be used as the feature information of the text information. In another embodiment, a text feature extraction model based on LSTM, Bi LSTM, etc. can be used to further extract the deep semantic information of the embedded vector of the text information of each text box, and obtain a new feature vector as the feature information of the text information, so that the feature information further represents the deep semantic information of the text information. Of course, the text feature extraction model is pre-trained in a convergence state.

[0107] Step S1223: construct the feature vector of the coordinate information of each text box and the feature vector of the text information of the text box into feature information, and store the feature information in a single node of the graph structure.

[0108] For each text box, after obtaining the feature vector of its coordinate information and the feature vector of the text information in the text box, the two can be combined into the feature information corresponding to the text box, which is stored in the node corresponding to the text box, replacing the original feature information of the node, and completing the construction of the corresponding node.

[0109] This embodiment constructs the feature information of each node step by step. During the construction process, the coordinate information and text information of the text box corresponding to the node are vectorized to obtain the feature information. While storing the feature information of the text information in the node, the coordinate information of the text box is also stored in an associated manner, so that the feature information of each node can more comprehensively represent the multimodal information corresponding to each text area in the bill image, which can more effectively guide the semantic extraction process of the graph convolutional neural network and help to achieve accurate classification of each text information later.

[0110] See also Figure 5 In an embodiment further developed on the basis of any of the above embodiments, the step S1300, using a preset graph convolutional neural network to iteratively update the feature information of the nodes and edges in the graph structure data for multiple times, comprises the following steps:

[0111] Step S1310: using the information extraction subnet in the graph convolutional neural network to obtain update information of nodes in the graph structure data, the update information is obtained by superimposing the first high-level semantic information of the feature information of the node on the significant information of its second high-level semantic information, and the significant information is obtained by weighted extraction of the second high-level semantic information matching the feature information of the edge in the graph structure data;

[0112] See also Figure 6 The schematic diagram of the exemplary graph convolutional neural network of the present application is shown, and it can be seen that the graph convolutional neural network includes an information extraction subnet, an edge update subnet, and a node update subnet. The graph convolutional neural network can perform feature extraction on the graph structure data in a cyclic iteration within a preset number of times. Each iteration uses the information extraction subnet to extract update information from the feature information of nodes and edges in the graph structure data, and then updates the graph structure data through the edge update subnet and the node update subnet.

[0113] When the information extraction subnet obtains the update information, in one embodiment, Figure 7 As shown, the following process can be used, including the following steps:

[0114] Step S1311, encoding the graph structure data to obtain a node feature information matrix and an edge feature information matrix;

[0115] The feature information of each node in the graph structure data is constructed as a node feature information matrix, wherein the feature information of each node can be used as a row vector in the matrix. Similarly, the feature information of the edges in the graph structure data is also constructed as an edge feature information matrix. Since the edge connection has symmetry, the edge feature information matrix constructed according to the edge connection information is a symmetric matrix. The node feature information matrix and the edge feature information matrix can be used as inputs of the information extraction subnet.

[0116] Step S1312: extracting first high-level semantic information of the node feature information matrix using the first fully connected layer in the information extraction subnet;

[0117] In the information extraction subnet, a first fully connected layer is provided, and the first fully connected layer is used to perform linear mapping on the node feature information matrix of the graph structure data, so as to extract its corresponding first high-level semantic information.

[0118] Step S1313: extracting the second high-level semantic information of the node feature information matrix using the second fully connected layer in the information extraction subnet;

[0119] In the information extraction subnet, a second fully connected layer is provided, and the second fully connected layer is used to perform linear mapping on the node feature information matrix of the graph structure data, so as to extract its corresponding second high-level semantic information.

[0120] Here, the first fully connected layer and the second fully connected layer are divided into two paths to linearly map the node feature information matrix respectively, and respectively match the weights learned during the training process of the graph convolutional neural network model. Therefore, the two high-level semantic information obtained can be used for different purposes.

[0121] Step S1314: normalize the edge feature information matrix using a normalization layer in the information extraction subnet to obtain edge feature semantic information;

[0122] In the information extraction subnet, a normalization layer constructed based on the Sigmoid function is also provided to normalize the edge feature information matrix, standardizing each feature information therein to a numerical space of [0,1], thereby obtaining corresponding edge feature semantic information, which can be used as a weight for extracting significant information from the second high-level semantic information.

[0123] Step S1315: In the information extraction subnet, obtain significant information by multiplying the second high-level semantic information by the product matrix of the edge feature semantic information, sum it with the first high-level semantic information and activate the output to obtain update information of the nodes in the graph structure data.

[0124] The edge feature semantic information can be used as a weight for extracting significant information from the second high-level semantic information. Therefore, the second high-level semantic information can be multiplied by the edge feature semantic information to obtain a corresponding matrix as the significant information, thereby extracting important features of nodes in the graph structure data from the second high-level semantic information.

[0125] Furthermore, the first high-level semantic information and the salient information are summed by matrix addition, and then activated and outputted by means of an activation layer. The information obtained is the update information corresponding to the nodes in the graph structure data.

[0126] Step S1320: using the edge update subnet in the graph convolutional neural network to update the edge feature information in the graph structure data, wherein the updated edge feature information is obtained by superimposing high-level semantic information of the edge feature information before the update and multi-channel high-level semantic information of the node;

[0127] When the edge update subnet is used to update the feature information of the edge in the graph structure data, in one embodiment, Figure 8 As shown, the following process can be used, including the following steps:

[0128] Step S1321, using the first fully connected layer of the edge update subnet to extract the first high-level semantic information of the node feature information matrix;

[0129] In the edge update subnet, a first fully connected layer is provided, and the first fully connected layer is used to perform linear mapping on the node feature information matrix of the graph structure data, so as to extract its corresponding first high-level semantic information.

[0130] Step S1322: extracting the second high-level semantic information of the node feature information matrix using the second fully connected layer of the edge update subnet;

[0131] In the edge update subnet, a second fully connected layer is provided, and the second fully connected layer is used to perform linear mapping on the node feature information matrix of the graph structure data, so as to extract its corresponding second high-level semantic information.

[0132] Step S1323: extract high-level semantic information of the edge feature information matrix using the third fully connected layer of the edge update subnet;

[0133] In the edge update subnet, a third fully connected layer is provided, and the third fully connected layer is used to linearly map the edge feature information matrix of the graph structure data, so as to extract its corresponding high-level semantic information, which can be called the third high-level semantic information.

[0134] Step S1324: sum the high-level semantic information of the edge feature information matrix and each high-level semantic information of the node feature information matrix, and then activate and output the summed ...

[0135] In order to obtain richer reference information for the feature information of the edges in the graph structure data, the first high-level semantic information, the second high-level semantic information and the third high-level semantic information can be summed by matrix addition, and then the activation output is applied to the sum result. The output result can be used as the feature information of the new edge, thereby realizing the update of the feature information of the edges of the graph structure data.

[0136] According to the above process, further flexible processing can be performed. For example, the high-level semantic information of the node feature information matrix is ​​not limited to being obtained by using two fully connected layers, but can also be obtained by using a single fully connected layer or more than three fully connected layers.

[0137] In addition, when all high-level semantic information is added to a matrix, it can also be replaced by implementing mean pooling. Technical personnel in this field can handle this flexibly. In short, it can be implemented as the reference information provided by the feature information of the edges in the graph structure data combined with the feature information of the nodes in the graph structure data.

[0138] Step S1330: Use the node update subnet in the graph convolutional neural network to splice the feature information of the node with its update information to update the feature information of the node of the graph structure data, and continue to iterate the above process within a preset number of times until the preset number of times is reached.

[0139] After the feature information of the edge in the graph structure data is updated, the feature information of the node in the graph structure data can be further updated. Specifically, the feature information of the node and its update information are spliced ​​using a splicing layer provided in the node update subnet, and the new feature information after the approximation is the feature information of the node in the graph structure data.

[0140] After the feature information of the edges and nodes in the graph structure data is updated through a single iteration, it is determined whether multiple iterations within a preset number of times have been completed. If the preset number of times has not been reached, the loop can continue to start from step S1310, and the graph structure data can be iteratively updated until the preset number of times is reached. The number of times for controlling iterations can be flexibly set by those skilled in the art, for example, any value between 3 and 10 times.

[0141] According to the disclosure of the above embodiments, the present application updates the feature information of the nodes and edges in the graph structure data through multiple iterations of a loop, and each iteration deepens the semantics of the feature information of the nodes and edges by one layer, thereby continuously deepening the semantics of the feature information of the nodes and edges of the graph structure data, thereby achieving accurate representation of the information in the graph structure data, helping to make the updated graph structure data of the graph convolutional neural network easier to be accurately classified, and making the trained graph convolutional neural network have better generalization ability and robustness, thereby realizing a single network suitable for text recognition services in multiple languages.

[0142] See also Fig. 9 In an embodiment expanded on the basis of any of the above embodiments, before step S1100, the step of performing text recognition on the bill image, the following steps are included:

[0143] Step S1010: Obtain the original image submitted by the user;

[0144] In the e-commerce platform application scenario of this application, taking a more specific e-commerce live broadcast scenario as an example, after the audience user of the live broadcast room, that is, the consumer user, completes the payment service for the order corresponding to a certain product in the live broadcast room from a third-party application, obtains the electronic payment voucher and stores it in a screenshot. In the screenshot of the electronic payment voucher, Fig.10 The example shown includes three basic information items, namely payment date, payment amount and remarks, which constitute the order data of this application, as well as other information that may not be paid attention to in this embodiment.

[0145] The audience user submits the screenshot of the electronic payment voucher as the original image to the backend of the e-commerce platform, and then the server can obtain the original image.

[0146] Step S1020, performing image preprocessing on the original image according to the input parameters of the order classification model pre-trained to a convergence state to obtain a to-be-detected image;

[0147] The electronic payment voucher represents the order data in the form of a bitmap. Therefore, although it can be recognized manually, for computer equipment, it is necessary to use technology to determine whether it contains the order data. In this embodiment, an order classification model is prepared to determine whether the original image contains order data. Accordingly, it is necessary to adapt to the input requirements of the order classification model and perform image preprocessing on the original image, including cropping, scaling, etc., to adjust it to a specific size, such as 224*224 pixels, to become the image to be detected.

[0148] Step S1030: Use the order classification model to classify and judge the image to be detected, determine whether the image to be detected and the original image contain order description information, and determine the original image containing the order description information as a bill image.

[0149] The order classification model is implemented based on a neural network basic model. It obtains image feature information by performing representation learning on the image to be detected, and then uses a binary classifier to discriminate the image feature information to determine whether the image to be detected, that is, the original image, contains the expected order data. For the original image containing order data, it is determined to be a bill image.

[0150] The order classification model can be implemented based on any basic neural network model such as traditional CNN, Resnet, MaskCNN, etc., and it can be trained to a convergence state in advance. When training the order classification model, pictures containing order data are used as positive samples, and pictures not containing order data are used as negative samples, and sufficient samples are used to perform iterative training on it until it converges.

[0151] In this embodiment, the original image submitted by the user is automatically and intelligently identified with the help of the order classification model, and whether the original image submitted by the user contains order data is quickly determined, and the original image containing the order data is determined as the bill image of this application. This realizes the automatic identification function of the original image submitted by the user. In some complex scenarios, for example, the audience user of the live broadcast room directly sends his original image through the chat interface of his live broadcast room, which is easily regarded as a general expression image. However, due to the support of the order classification model, the original image can be directly called from the user chat record for intelligent automatic identification. Once the original image is found to be the image to be detected, the subsequent steps can be completed accordingly, eliminating the manual processing process.

[0152] See also Fig.11In an embodiment extended on the basis of any of the above embodiments, after the step of determining the classification label corresponding to the text information of each text box, the following steps are included:

[0153] Step S1500: converting each text information into order data according to the classification label;

[0154] After the text information corresponding to each text box in the bill image is labeled with corresponding classification labels by the classification network, each text information becomes attribute data corresponding to the corresponding field in the order data. Therefore, the correspondence between the field and the attribute data can be used to construct the text information in the bill image into structured order data.

[0155] Step S1600: Generate an e-commerce transaction order according to the order data, and push the e-commerce transaction order to the terminal device that submits the bill image to start a preset e-commerce business process.

[0156] For the backend of the e-commerce platform, after obtaining the structured order data, it can use this order data to continue its e-commerce transaction orders.

[0157] Still Fig.10 Corresponding scenario example, when the user completes the offline payment and uploads his electronic payment voucher to the e-commerce platform to obtain the corresponding structured order data, the payment date, payment amount and remarks information are determined. The remarks information is generally used to mark the order number of the corresponding e-commerce transaction order. Accordingly, the order data will contain three data entries, which are three key-value pairs with payment date, payment amount and remarks information as key data (which can be regarded as fields). From the value data of these three key-value pairs, the corresponding specific payment date, specific payment amount and specific order number can be obtained. Based on this, the e-commerce order record created by the user in advance can be marked with a payment mark, so as to generate a valid e-commerce transaction order based on the e-commerce order record, and then push it to the terminal device that submitted the original image for the corresponding consumer user to review. After the user reviews and confirms, the background can start the preset e-commerce business process accordingly, such as pushing the e-commerce transaction order to the logistics delivery interface for billing.

[0158] This embodiment further uses the structured order data obtained from the bill image in this application to improve the e-commerce business process, so that the e-commerce business process can achieve a closed loop with the assistance of the order data, thereby realizing the combination of offline payment and online transactions.

[0159] See also Fig.12, a bill information extraction device provided to meet one of the purposes of the present application is a functional embodiment of the bill information extraction method of the present application, and the device includes a text recognition module 1100, a graph structuring module 1200, an information updating module 1300, and an information classification module 1400. The text recognition module 1100 is used to perform text recognition on the bill image, and obtain the coordinate information of the text box of each text area and the text information in the text box; the graph structuring module 1200 is used to construct graph structure data, wherein each node represents the coordinate information of a single text box and the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes; the information updating module 1300 is used to use a preset graph convolutional neural network to iteratively update the feature information of the nodes and edges in the graph structure data for multiple times to obtain the updated graph structure data; the information classification module 1400 is used to input the feature information of the nodes in the updated graph structure data into the classification network for classification, and determine the classification label corresponding to the text information of each text box.

[0160] In an embodiment further developed on the basis of any of the above embodiments, the text recognition module 1100 comprises: a text detection unit, used to call a preset text detection model to detect the bill image, and obtain coordinate information of text boxes corresponding to each text area therein; an image cropping unit, used to extract line text images corresponding to each text area from the bill image according to the coordinate information of the text boxes; a text recognition unit, used to input the line text image into a text recognition model pre-trained to a convergent state for text recognition, and obtain corresponding text information.

[0161] In an embodiment further developed on the basis of any of the above embodiments, the graph structuring module 1200 comprises: a structure creation unit, for creating a graph structure of graph structure data, wherein the graph structure comprises nodes and edges for describing connection information between any two nodes; a node construction unit, for obtaining a feature vector of coordinate information corresponding to each text box and a feature vector of text information corresponding to the feature information stored as feature information in a node in the graph structure, wherein the coordinate information comprises the coordinates of the four corner points of the text box; an edge construction unit, for calculating a feature vector of relative distance information of the two text boxes according to the coordinate information between the two text boxes, and storing the feature information as feature information of the edges between the nodes corresponding to the two text boxes, wherein the relative distance information comprises a longitudinal relative distance and a lateral relative distance between the two text boxes.

[0162] In an embodiment concretized on the basis of any of the above embodiments, the node construction unit includes: a coordinate conversion subunit, used to convert the coordinate information corresponding to each text box into a feature vector; a text conversion subunit, used to encode the embedding vector of the text information corresponding to each text box according to a preset multilingual dictionary, and use a preset text feature extraction model to extract each embedding vector to obtain the feature vector of the text information of each text box; a feature construction subunit, used to construct the feature vector of the coordinate information of each text box and the feature vector of the text information of the text box into feature information, and store the feature information in a single node of the graph structure.

[0163] In an embodiment further developed on the basis of any of the above embodiments, the information update module 1300 comprises: an update preparation unit, for obtaining update information of nodes in the graph structure data by using the information extraction subnet in the graph convolutional neural network, wherein the update information is obtained by superimposing the first high-level semantic information of the feature information of the node on the significant information of its second high-level semantic information, wherein the significant information is obtained by weight extraction by matching the second high-level semantic information with the feature information of the edge in the graph structure data; an edge update unit, for updating the feature information of the edge in the graph structure data by using the edge update subnet in the graph convolutional neural network, wherein the updated feature information of the edge is obtained by superimposing the high-level semantic information of the feature information of the edge before the update on the multi-channel high-level semantic information of the node; a node update unit, for updating the feature information of the nodes of the graph structure data by using the node update subnet in the graph convolutional neural network to splice the feature information of the node with its update information, and continuing to iterate the above process within a preset number of times until the preset number of times is reached.

[0164] In an embodiment concretized on the basis of any of the above embodiments, the update preparation unit includes: a matrix encoding subunit, used to encode the graph structure data to obtain a node feature information matrix and an edge feature information matrix; a first extraction subunit, used to extract the first high-level semantic information of the node feature information matrix using the first fully connected layer in the information extraction subnet; a second extraction subunit, used to extract the second high-level semantic information of the node feature information matrix using the second fully connected layer in the information extraction subnet; a normalization subunit, used to normalize the edge feature information matrix using the normalization layer in the information extraction subnet to obtain edge feature semantic information; an information processing subunit, used to obtain significant information by multiplying the second high-level semantic information and the edge feature semantic information in the information extraction subnet, and activate the output after summing it with the first high-level semantic information to obtain update information of the nodes in the graph structure data.

[0165] In an embodiment concretized based on any of the above embodiments, the edge update unit includes: a first extraction subunit, used to extract the first high-level semantic information of the node feature information matrix using the first fully connected layer of the edge update subnet; a second extraction subunit, used to extract the second high-level semantic information of the node feature information matrix using the second fully connected layer of the edge update subnet; a third extraction subunit, used to extract the high-level semantic information of the edge feature information matrix using the third fully connected layer of the edge update subnet; and an update execution subunit, used to sum the high-level semantic information of the edge feature information matrix with the respective high-level semantic information of the node feature information matrix and then activate and output it as the updated feature information of the edge in the graph structure data.

[0166] In an embodiment expanded on the basis of any of the above embodiments, prior to the text recognition module 1100, it includes: an original image acquisition module, used to acquire the original image submitted by the user; a classification detection module, used to perform image preprocessing on the original image according to the input parameters of an order classification model pre-trained to a convergence state, to obtain a picture to be detected; a bill determination module, used to classify and distinguish the picture to be detected using the order classification model, to determine whether the picture to be detected and the original picture contain order description information, and to determine the original picture containing the order description information as a bill image.

[0167] In an embodiment expanded on the basis of any of the above embodiments, the information classification module 1400 includes: a data conversion module for converting each text information into order data according to the classification label; an order push module for generating an e-commerce transaction order according to the order data, and pushing the e-commerce transaction order to the terminal device that submits the bill image to start a preset e-commerce business process.

[0168] In order to solve the above technical problems, the present application also provides a computer device. Fig.13 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a ticket information extraction method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the ticket information extraction method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Fig.13The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] In this embodiment, the processor is used to execute Fig.12 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the bill information extraction device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0170] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the ticket information extraction method of any embodiment of the present application.

[0171] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0172] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0173] To sum up, the present application can identify order data expressed in multiple languages ​​from bill images, which can be directly called by e-commerce order business processes, thereby improving the processing efficiency of identifying order data from images and executing e-commerce order business processes, and facilitating the implementation of e-commerce order business processes that combine online and offline.

[0174] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0175] The above is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for extracting bill information, characterized in that: The steps include: Performing text recognition on the bill image to obtain coordinate information of the text box of each text area therein and the text information in the text box; Constructing graph structure data, wherein each node represents the coordinate information of a single text box and the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes; Using a preset graph convolutional neural network to iteratively update the feature information of nodes and edges in the graph structure data multiple times to obtain updated graph structure data; Input the feature information of the nodes in the updated graph structure data into the classification network for classification, and determine the classification label corresponding to the text information of each text box; The preset graph convolutional neural network is used to iteratively update the feature information of the nodes and edges in the graph structure data for multiple times, including the following steps: The information extraction subnet in the graph convolutional neural network is used to obtain update information of nodes in the graph structure data, wherein the update information is obtained by superimposing the first high-level semantic information of the feature information of the node on the significant information of the second high-level semantic information, and the significant information is obtained by weighted extraction of the second high-level semantic information matching the feature information of the edge in the graph structure data; The edge update subnet in the graph convolutional neural network is used to update the feature information of the edge in the graph structure data, wherein the updated feature information of the edge is obtained by superimposing high-level semantic information of the feature information of the edge before the update and multi-channel high-level semantic information of the node; The node update subnet in the graph convolutional neural network is used to splice the feature information of the node with its update information to update the feature information of the node of the graph structure data, and the above process is iterated within a preset number of times until the preset number is reached.

2. The method for extracting bill information according to claim 1, characterized in that: Text recognition for bill images includes the following steps: Calling a preset text detection model to detect the bill image, and obtaining coordinate information of text boxes corresponding to each text area therein; Cut out the line text images corresponding to each text area from the bill image according to the coordinate information of the text box; The line text image is input into a text recognition model that has been pre-trained to a convergent state to perform text recognition and obtain corresponding text information.

3. The method for extracting bill information according to claim 1, characterized in that: Constructing graph structure data includes the following steps: Creating a graph structure of graph structure data, wherein the graph structure includes nodes and edges for describing connection information between any two nodes; Obtaining a feature vector of coordinate information corresponding to each text box and a feature vector of text information correspondingly stored as feature information in a node in the graph structure, wherein the coordinate information includes coordinates of four corner points of the text box; The feature vector of the relative distance information of any two text boxes is calculated based on the coordinate information between the two text boxes, and stored as the feature information of the edge between the nodes corresponding to the two text boxes. The relative distance information includes the longitudinal relative distance and the lateral relative distance between the two text boxes.

4. The method for extracting bill information according to claim 3, characterized in that: Obtaining the feature vector of the coordinate information corresponding to each text box and the feature vector of the text information correspondingly stored as feature information in a node in the graph structure includes the following steps: Convert the coordinate information corresponding to each text box into a feature vector; Encode the embedding vector of the text information corresponding to each text box according to a preset multilingual dictionary, and extract each embedding vector using a preset text feature extraction model to obtain a feature vector of the text information of each text box; The feature vector of the coordinate information of each text box and the feature vector of the text information of the text box are constructed as feature information, and the feature information is stored in a single node of the graph structure.

5. The method for extracting bill information according to claim 4, characterized in that: Using the information extraction subnet in the graph convolutional neural network to obtain update information of nodes in the graph structure data includes the following steps: Encoding the graph structure data to obtain a node feature information matrix and an edge feature information matrix; Using the first fully connected layer in the information extraction subnet to extract the first high-level semantic information of the node feature information matrix; Using a second fully connected layer in the information extraction subnet to extract second high-level semantic information of the node feature information matrix; Normalizing the edge feature information matrix using a normalization layer in the information extraction subnet to obtain edge feature semantic information; In the information extraction subnet, the product matrix of the second high-level semantic information and the edge feature semantic information is used to obtain significant information, which is summed with the first high-level semantic information and then activated and output to obtain update information of the nodes in the graph structure data.

6. The method for extracting bill information according to claim 5, characterized in that: Using the edge update subnet in the graph convolutional neural network to update the feature information of the edges in the graph structure data includes the following steps: Using a first fully connected layer of an edge update subnet to extract first high-level semantic information of the node feature information matrix; The second fully connected layer of the edge update subnet is used to extract the second high-level semantic information of the node feature information matrix; Using the third fully connected layer of the edge update subnet to extract high-level semantic information of the edge feature information matrix; The high-level semantic information of the edge feature information matrix and each high-level semantic information of the node feature information matrix are summed and activated and output as updated feature information of the edge in the graph structure data.

7. A bill information extraction device, characterized in that: include: A text recognition module is used to perform text recognition on the bill image and obtain the coordinate information of the text box of each text area and the text information in the text box; A graph structuring module, used to construct graph structure data, wherein each node represents the coordinate information of a single text box and the feature information of the text information, and the edge between any two nodes represents the feature information of the relative distance information between the text boxes of the two nodes; An information updating module, used to iteratively update the feature information of nodes and edges in the graph structure data multiple times using a preset graph convolutional neural network to obtain updated graph structure data; An information classification module is used to input the feature information of the nodes in the updated graph structure data into a classification network for classification, and determine the classification label corresponding to the text information of each text box; The information updating module comprises: An update preparation unit, used for obtaining update information of nodes in the graph structure data by using the information extraction subnet in the graph convolutional neural network, wherein the update information is obtained by superimposing the first high-level semantic information of the feature information of the node on the significant information of the second high-level semantic information, and the significant information is obtained by weight extraction by matching the feature information of the edge in the graph structure data with the second high-level semantic information; An edge updating unit, configured to update feature information of edges in the graph structure data using an edge updating subnet in the graph convolutional neural network, wherein the updated feature information of edges is obtained by superimposing high-level semantic information of the feature information of edges before updating and multi-channel high-level semantic information of nodes; The node updating unit is used to update the feature information of the node of the graph structure data by splicing the feature information of the node with its update information using the node updating subnet in the graph convolutional neural network, and continue to iterate the above process within a preset number of times until the preset number of times is reached.

8. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images