Bill processing method and device, model training method and equipment, medium and product

Abstract

The invention provides a bill processing method and device, a model training method and device, a medium and a product, which can be applied to the technical field of artificial intelligence. The bill processing method comprises the following steps: extracting text information and image information from a bill image, and fusing the text information and the image information to obtain fused information; a target cluster to which the bill image belongs is determined based on the fusion information, and the target cluster comprises at least one reference image; and determining the reference image of which the similarity with the bill image is greater than a preset threshold in the at least one reference image as a target reference image, the target reference image being used for guiding the input of the structured data of the bill image. The invention further provides a bill processing device, a model training method, equipment, a medium and a product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically to a bill processing method, apparatus, model training method, equipment, medium and product. Background Art

[0002] In banking business, when business personnel receive bills from customers, they often need to enter them into the system to form structured data. However, due to the different clarity of images and the large variety of them, manual classification and recognition require reliance on previous experience and knowledge and consume manpower. Although deep learning fully automatic methods have a high accuracy rate in the classification and recognition of bills in the current automated optical character recognition, for banking business, in order to avoid risks, manual operation or confirmation is still required from image data to structured data. Moreover, for a large variety of bill images, different styles of layout and content information may interfere with recognition. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a bill processing method, apparatus, model training method, equipment, medium and product.

[0004] According to the first aspect of the present disclosure, a bill processing method is provided, the method comprising: extracting text information and image information from a bill image, fusing the text information and the image information to obtain fused information; determining a target cluster to which the bill image belongs based on the fused information, wherein the target cluster includes at least one reference image; determining a reference image in at least one reference image whose similarity with the bill image is greater than a preset threshold as a target reference image, wherein the target reference image is used to guide the input of structured data of the bill image.

[0005] According to an embodiment of the present disclosure, text information and image information are extracted from a bill image, and the text information and image information are fused to obtain fused information, including: in response to an input bill image, extracting a first feature vector and a second feature vector of the bill image, wherein the first feature vector is used to represent the text information of the bill; the second feature vector is used to represent the image information of the bill; and determining a first fused feature vector based on the first feature vector and the second feature vector, wherein the first fused feature vector is used to represent fused information including text information and image information.

[0006] According to an embodiment of the present disclosure, a first fused feature vector is determined based on a first feature vector and a second feature vector, including: flattening the first feature vector and the second feature vector, respectively; fusing the flattened first feature vector and the flattened second feature vector to obtain a third feature vector, wherein the third feature vector is a feature vector obtained by first fusing the first feature vector and the second feature vector; and determining the first fused feature vector based on the third feature vector.

[0007] According to an embodiment of the present disclosure, determining the first fused feature vector according to the third feature vector includes: performing a second fusion process on the third feature vector to obtain the first fused feature vector.

[0008] According to an embodiment of the present disclosure, determining a target cluster of a bill image based on fusion information includes: respectively determining distances between a first fusion feature vector and multiple cluster centers; and determining a target cluster of the bill image from multiple clusters based on the distances.

[0009] According to an embodiment of the present disclosure, a reference image in at least one reference image whose similarity with a bill image is greater than a preset threshold is determined as a target reference image, and the target reference image is used to guide the input of structured data of the bill image, including: obtaining a second fused feature vector of at least one reference image, the second fused feature vector being a fused feature vector of at least one reference image; respectively determining the similarity between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image; and determining the reference image whose similarity is greater than a preset threshold as the target reference image.

[0010] According to an embodiment of the present disclosure, respectively determining the similarity between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image includes: respectively calculating the Euclidean distance between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image.

[0011] A second aspect of the present disclosure provides a training method for a deep clustering model, which is used to implement the above method. The training method includes: obtaining training samples from a training sample set, where the training samples are bill images; determining a first feature vector and a second feature vector from the input training samples; determining a first fused feature vector based on the first feature vector and the second feature vector; determining a cluster to which the training sample belongs based on the first fused feature vector; calculating a cross entropy loss function based on the probability of the cluster and the cluster predicted by the classification model; and performing back propagation to update the weights based on the cross entropy loss function until the model training is completed.

[0012] The third aspect of the present disclosure provides a bill processing device, which includes: an extraction module for extracting text information and image information from a bill image, and fusing the text information and the image information to obtain fused information; a processing module for determining a target cluster to which the bill image belongs based on the fused information, wherein the target cluster includes at least one reference image; and a clustering module for determining a reference image in at least one reference image whose similarity with the bill image is greater than a preset threshold as a target reference image, wherein the target reference image is used to guide the input of structured data of the bill image.

[0013] A fourth aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fifth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0015] The sixth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 A diagram schematically illustrates an application scenario of a method, apparatus, device, medium, and program product for bill processing according to an embodiment of the present disclosure;

[0018] Figure 2 Schematically shows a flowchart of a bill processing method according to Embodiment 1 of the present disclosure

[0019] Figure 3 The flowchart of the bill processing method according to the third embodiment of the present disclosure is schematically shown;

[0020] Figure 4 The flowchart of the bill processing method according to the fourth embodiment of the present disclosure is schematically shown;

[0021] Figure 5 The flowchart of the bill processing method according to the fifth embodiment of the present disclosure is schematically shown;

[0022] Figure 6 The flowchart of the bill processing method according to the sixth embodiment of the present disclosure is schematically shown;

[0023] Figure 7 A flowchart of a method for training a deep clustering model according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 8 A schematic diagram of a structure of a bill processing device according to an embodiment of the present disclosure is shown; and

[0025] Fig. 9 A block diagram of an electronic device suitable for implementing bill processing according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0032] Figure 1 The application scenario diagram of bill processing according to an embodiment of the present disclosure is schematically shown.

[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a collection device 101, a storage device 102, a network 103, and a server 104. The network 103 is used to provide a medium for a communication link between the collection device 101, the storage device 102, and the server 104. The network 103 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0034] The acquisition device 101 can be used, for example, to acquire image information including bills.

[0035] The storage device 102 may be used to store acquired image information.

[0036] The server 104 may be a server that provides various services, such as a background management server that processes based on bill image information (for example only). The background management server may analyze and process the received image data, and feed back the processing result (for example, the processing result based on the bill image, etc.) to the terminal device.

[0037] It should be noted that the bill processing method provided in the embodiment of the present disclosure can generally be executed by the server 104. Accordingly, the bill processing device provided in the embodiment of the present disclosure can generally be set in the server 104. The bill processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 104 and can communicate with the collection device 101, the storage device 102 and / or the server 104. Correspondingly, the bill processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 104 and can communicate with the collection device 101, the storage device 102 and / or the server 105.

[0038] It should be understood that Figure 1The number of acquisition devices, storage devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0039] In the banking business process, when business personnel receive bill images from customers, they need to enter them into the system to form structured data. However, due to the different clarity of bill images and the wide variety of bills, classification and identification requires rich experience and knowledge, which consumes manpower.

[0040] Traditional methods for processing bill images mostly rely on automatic character recognition to directly obtain the category to which the bill image belongs. Although the deep learning fully automatic method has achieved very good accuracy, for banking business, it is necessary to avoid risks. Any wrong data entry may lead to serious financial consequences, such as mis-transfer of funds and confusion of accounts. Therefore, manual operation or confirmation is still required from image data to structured data. Moreover, for a large number of bill images, different styles of layout and content information may interfere with recognition.

[0041] The method for processing bill images provided by the embodiment of the present disclosure, on the one hand, combines various comprehensive features to achieve comprehensive extraction of image information; on the other hand, by using the extracted features for similarity matching, similar images and corresponding structured data can be quickly acquired, which can assist in the entry of structured data.

[0042] The following will be based on Figure 1 The scene described by Figure 2~Figure 6 The method for processing bills of the disclosed embodiment is described in detail.

[0043] Figure 2 The flowchart of the bill processing method according to the first embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the bill processing of this embodiment includes operations S210 to S230, and the transaction processing method can be executed by the server 104.

[0045] In operation S210, text information and image information are extracted from the bill image, and the text information and the image information are fused to obtain fused information;

[0046] In operation S220, a target cluster to which the bill image belongs is determined based on the fusion information, wherein the target cluster includes at least one reference image;

[0047] In operation S230, a reference image whose similarity with the bill image is greater than a preset threshold among the at least one reference image is determined as a target reference image, and the target reference image is used to guide the input of structured data of the bill image.

[0048] In an embodiment of the present disclosure, a bill image may include different types of bills, which may be invoices, receipts, bank statements, checks, etc. For example, when the bill is an invoice, the structured data may include the invoice number, the date of issue, the seller information, the buyer information, the details of the goods or services, the tax rate or the tax amount, etc. For example, when the bill is a receipt, the structured data may include the receipt number, the payee information, the payee information, the item, the amount or the signature, etc. For example, when the bill is a bank statement, the structured data may include the account information, the transaction date, the transaction description, the transaction amount, the balance, the branch information, etc. For example, when the bill is a check, the structured data may include the check number, the issue date, the payee information, the payee information, the amount or the signature, etc.

[0049] Text information is the text content extracted from the bill image, which can be grammar and sentence information, etc., and can be extracted through a convolutional recurrent neural network. For example, when the bill is an invoice, the text information may include the invoice number, such as "F2023123456"; the invoice date, such as "January 15, 2024"; the seller's information such as: company name, "ABC Co., Ltd.", address, "No. 123, XX Road, XX District, XX City", and tax identification number, "91310115MA1H7XXXXX"; the buyer's information such as: company name: "XYZ Technology Company", address: "No. 456, XX Street, XX District, XX City", and tax identification number: "91110105MA000XXXXX"; details of goods or services such as product description: "Laptop XX", quantity: "1 unit", unit price: "¥8,999.00"; total amount: "¥8,999.00"; tax rate and tax amount such as "VAT rate 13%, tax amount ¥1,169.87", total amount: "¥10,168.87"; remarks: such as "freight included", etc.

[0050] Image information is the visual features extracted from the bill image, which can be color distribution and texture pattern, etc., extracted through convolutional neural network. For example, when the bill is an invoice, the image information can include colors such as red seal, blue font, etc.; the texture of the invoice paper, watermarks and anti-counterfeiting marks, etc.; the position and style of visual elements such as titles, table lines, borders, etc.

[0051] Fusion information combines text information and image information to form a comprehensive information, which is used to fully describe the content of the bill image. For example, the text information and image information of the above invoice can be fused to generate fusion information. By combining text information and image information, more subtle but key feature differences can be captured, reducing the probability of misclassification caused by misleading single features.

[0052] Through a clustering algorithm, such as a k-means clustering algorithm, with k clusters, cluster analysis is performed on the fused information of the above invoices to determine the target cluster a to which the invoice belongs, and there are b image information in the target cluster, where 1≤a≤k, 1≤b, and the similarity between it and all historical invoices in the same cluster is calculated, and the invoice with a similarity greater than a preset threshold is selected as the target reference image. For example, the preset threshold is t, and the similarity of c image information among b image information is greater than the preset threshold t, 1≤c≤b, so the above c images can be determined as the target reference images of the input invoice. Based on this, the structured data input of the new invoice can be completed quickly and accurately based on the found target reference invoice and its corresponding structured data. For example, the relevant information of the new invoice can be confirmed based on the product name, quantity, unit price, tax rate, etc.

[0053] According to the bill image processing method of the embodiment of the present disclosure, by fusing text information and image information, combined with cluster analysis and similarity matching, it effectively solves the problems of time-consuming and laborious manual classification and recognition, image quality problems, a wide variety of bills, and the risks of automated methods. This method not only improves the accuracy and robustness of classification, can better handle low-quality or blurred images, and optimizes the input process of structured data, but also can use the information of these target reference images to guide the structured data input process of new bill images, thereby improving image processing efficiency.

[0054] Figure 3 The flowchart of the bill processing method according to the third embodiment of the present disclosure is schematically shown.

[0055] like Figure 3 As shown, the bill processing of this embodiment includes operations S310 to S320, and the transaction processing method can be executed by the server 104.

[0056] According to an embodiment of the present disclosure, text information and image information are extracted from a bill image, and the text information and image information are fused to obtain fused information including:

[0057] In operation S310, in response to the input bill image, extract a first feature vector and a second feature vector of the bill image, wherein the first feature vector is used to represent text information of the bill; and the second feature vector is used to represent image information of the bill;

[0058] In operation S320, a first fused feature vector is determined according to the first feature vector and the second feature vector, where the first fused feature vector is used to represent fused information including text information and image information.

[0059] In an embodiment of the present disclosure, the first feature vector of the bill image can be extracted by a trained convolutional recurrent neural network (CRNN):

[0060]

[0061] Among them, σ is the parameter of CRNN, is the CRNN network mapping, and X is the input bill image.

[0062] The second eigenvector of the bill image extracted by the trained convolutional neural network (CNN) is:

[0063]

[0064] Among them, θ is the parameter of CNN, is the network mapping of CNN, is the input bill image.

[0065] In the CNN model, the input image is preprocessed, such as resizing and normalization, to ensure the consistency of the input.

[0066] When extracting the second feature vector, for example, a larger convolution kernel, such as 7x7, can be used in the first layer or the first few layers. These larger receptive fields can help the network quickly obtain large-scale structural information of the image, such as the overall layout of the bill. Gradually adopting smaller convolution kernels, such as 3x3, in subsequent layers to gradually refine the features can capture finer local features, such as text or table lines in a specific area. In this process, each convolution operation is usually followed by a pooling layer to reduce the spatial dimension of the feature map and enhance the robustness to spatial changes. Pooling not only reduces the computational complexity, but also helps retain the most important visual information while reducing the risk of overfitting. Through multiple combinations of convolution and pooling, feature vectors rich in important visual information of the image are gradually generated.

[0067] According to an embodiment of the present disclosure, the first feature vector and the second feature vector are fused to generate a first fused feature vector, so that a more comprehensive and rich bill representation can be obtained, so that the model can simultaneously utilize both text and image types of information, reducing the risk of misclassification that may be caused by a single information source. At the same time, the fused feature vector can more accurately measure the similarity between two bills.

[0068] Figure 4 The flowchart of the bill processing method according to the fourth embodiment of the present disclosure is schematically shown.

[0069] like Figure 4 As shown, the bill processing of this embodiment includes operations S410 to S420. The transaction processing method can be executed by the server 104.

[0070] According to an embodiment of the present disclosure, determining a first fused feature vector according to the first feature vector and the second feature vector includes:

[0071] In operation S410, the first eigenvector and the second eigenvector are flattened respectively; the flattened first eigenvector and the flattened second eigenvector are fused to obtain a third eigenvector, wherein the third eigenvector is a eigenvector obtained by first fusion of the first eigenvector and the second eigenvector;

[0072] In operation S420, a first fused feature vector is determined according to the third feature vector.

[0073] In the embodiment of the present disclosure, for example, the first feature vector The dimension is m×n, and the second eigenvector The dimension is d×k. The third eigenvector is determined to be (mn+dk)×1 by flattening m×n to mn×1, flattening d×k to dk×1, and then concatenating the mn×1 and dk×1 dimensions. The first fused eigenvector is determined based on (mn+dk)×1.

[0074] According to the embodiments of the present disclosure, since the first eigenvector is a sequence variable and the second eigenvector is an image feature variable, the dimensions of the two vectors are inconsistent and it is difficult to directly fuse or add them. Therefore, in this embodiment, by flattening the first eigenvector and the second eigenvector, different types of feature vectors can be effectively combined to achieve the fusion of text information and image information.

[0075] According to an embodiment of the present disclosure, determining the first fused feature vector according to the third feature vector includes: performing a second fusion process on the third feature vector to obtain the first fused feature vector.

[0076] In the embodiment of the present disclosure, after the third feature vector is input into the fully connected layer network for the second fusion process, the first fused feature vector is output as:

[0077]

[0078] in, is the first fusion feature vector.

[0079] By further fusing the third eigenvectors, the dimension of the variables can be reduced again, which can significantly reduce the number of parameters required for the subsequent step, improve computational efficiency, and prevent overfitting caused by too many parameters.

[0080] Figure 5 The flowchart of the bill processing method according to the fifth embodiment of the present disclosure is schematically shown.

[0081] like Figure 5 As shown, the bill processing of this embodiment includes operations S510 to S520. The transaction processing method can be executed by the server 104.

[0082] According to an embodiment of the present disclosure, a target cluster of a bill image is determined based on fusion information, wherein the target cluster includes at least one reference image, including:

[0083] In operation S510, distances between the first fused feature vector and a plurality of cluster centers are respectively determined;

[0084] In operation S520, a target cluster of the bill image is determined from among a plurality of clusters according to the distance.

[0085] In the embodiment of the present disclosure, the cluster center is the average value of the feature vectors of all data points in the cluster. For example, the distance between the first fused feature vector of the input bill image and multiple cluster centers can be calculated by Euclidean distance or cosine similarity, and based on the calculated multiple distances, the cluster with the closest distance is selected as the target cluster of the input bill image and the bill image is assigned to this cluster.

[0086] According to the embodiments of the present disclosure, the above method can quickly and effectively assign a new bill image to the most suitable cluster. At the same time, the cluster center can be continuously updated and adapted to new bill types based on the input bill image, thereby improving the flexibility and stability of the system.

[0087] Figure 6 The flowchart of the bill processing method according to the sixth embodiment of the present disclosure is schematically shown.

[0088] like Figure 6 As shown, the bill processing of this embodiment includes operations S610 to S630, and the transaction processing method can be executed by the server 104.

[0089] According to an embodiment of the present disclosure, a reference image whose similarity with a bill image is greater than a preset threshold in at least one reference image is determined as a target reference image, and the target reference image is used to guide the input of structured data of the bill image, including:

[0090] In operation S610, a second fused feature vector of at least one reference image is obtained, where the second fused feature vector is a fused feature vector of at least one reference image;

[0091] In operation S620, similarities between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image are respectively determined;

[0092] In operation S630, a reference image having a similarity greater than a preset threshold is determined as a target reference image.

[0093] In the embodiment of the present disclosure, for example, the target cluster obtained is the ath cluster, and there are b image information in the ath cluster, where 1≤a≤k, 1≤b, and the similarity between the first fused feature vector of the input bill image and the b images in the ath cluster is calculated respectively, and the images are sorted according to at least one calculated similarity. Based on a preset similarity threshold t, a reference image with a similarity higher than the threshold is selected from the sorted list as the target reference image.

[0094] According to the embodiments of the present disclosure, by obtaining the similarity between images in the same cluster, historical bill images with high similarity can be found quickly and effectively, providing reliable structured data input guidance for new bill images, and ultimately achieving efficient and accurate bill processing.

[0095] According to an embodiment of the present disclosure, respectively determining the similarity between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image includes:

[0096] The Euclidean distance between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image is calculated respectively.

[0097] In the embodiments of the present disclosure, the Euclidean distance is a way to measure the straight-line distance between two points. For example, the first fused feature vector C1=(x1, x2, …, xn), the second fused feature vector C2=(y1, y2, …, yn), then the Euclidean distance between C1 and C2 is:

[0098]

[0099] By calculating the distance between C1 and multiple C2s, the smaller the distance, the higher the similarity, the similarity is determined. In this embodiment, by calculating the Euclidean distance, the similarity between the reference image and the bill image can be quantified, thereby helping the system identify the target reference image that is most likely to serve as a guide for the input of bill image structured data.

[0100] Figure 7 The flowchart of the training method of the deep clustering model according to the embodiment of the present disclosure is schematically shown.

[0101] like Figure 7As shown, the bill processing of this embodiment includes operations S710 to S760, and the transaction processing method can be executed by the server 104.

[0102] A second aspect of the present disclosure provides a training method for a deep clustering model, the deep clustering model is used to implement the above method, and the training method includes:

[0103] In operation S710, a training sample in a training sample set is obtained, where the training sample is a bill image;

[0104] In operation S720, a first feature vector and a second feature vector are determined from the input training sample;

[0105] In operation S730, a first fused feature vector is determined based on the first feature vector and the second feature vector;

[0106] In operation S740, determining the cluster to which the training sample belongs according to the first fused feature vector;

[0107] In operation S750, a cross entropy loss function is calculated based on the cluster and the probability of the cluster predicted by the classification model;

[0108] In operation S760, back propagation is performed to update the weights according to the cross entropy loss function until the training of the model is completed.

[0109] In the embodiments of the present disclosure, for example, a training sample set containing a large number of bill images can be prepared, where each bill image is used as a training sample, and necessary preprocessing operations such as resizing and normalization are performed on all bill images to ensure that they can be used as inputs of the model. Based on the input bill image, the first feature vector can be extracted using the trained CRNN. , where the parameters of CRNN remain unchanged during the training process; CNN is used to extract the second feature vector from the same input bill image , and determine the first fused feature vector Xc based on the first eigenvector and the second eigenvector, wherein the principle of the method of flattening the first eigenvector and the second eigenvector and then fusing them has been explained in the above embodiment and will not be repeated here.

[0110] Set the predefined category parameter k, that is, the number of clusters, and randomly initialize k cluster centers to form a cluster center matrix , where d is the feature dimension. Apply the K-means algorithm to the first fused feature vectors of all training samples, calculate the Euclidean distance between each sample and each cluster center, assign each sample to the nearest cluster, and update the cluster center position until convergence or the maximum number of iterations is reached.

[0111] According to the final clustering results, a pseudo label is assigned to each training sample. , indicating which cluster this training sample belongs to. For example, if a sample is assigned to the i-th cluster, only the i-th bit of its pseudo-label yn is 1, and the rest are 0.

[0112] Build a classifier, such as a softmax layer, to predict the probability distribution of each training sample belonging to each cluster.

[0113] The cross entropy loss function is used to evaluate the difference between the predicted probability distribution and the true pseudo label yn, and the clustering loss function is as follows:

[0114]

[0115] in, represents the first fused feature vector, which is the feature extracted after the fusion layer thickness, where X is the feature extracted in the above embodiment. , represents the pseudo label of sample Xn, N is the total number of samples, is the cluster center matrix.

[0116] After each clustering, the generated pseudo-label is used as the real label, and the probability distribution of each training sample belonging to each cluster is predicted by the classifier to calculate the clustering loss function. Back propagation is performed based on the calculated clustering loss function to update the weights of the CNN and fusion layers.

[0117] In some possible embodiments, it is assumed that there is a training sample set containing 10,000 bill images, and the number of clusters k is preset to be 10. The training process can be performed according to the following steps:

[0118] Set the total number of clustering times Q=20, the number of network training times after each clustering P=5, and the initial learning rate L=0.001.

[0119] K-means clustering is applied to the first fused feature vectors of all bill images to generate initial pseudo labels.

[0120] The network is trained for 5 rounds using these pseudo labels to update the weights of the CNN and fusion layers, where the fusion layer is used to fuse the first feature vector and the second feature vector to generate a first fused feature vector. Each time the weight is updated, the extracted second feature vector and the first fused feature vector need to be updated based on the same training sample.

[0121] The above clustering and training process was repeated 20 times, during which the learning rate was adjusted as needed to ensure that the model gradually converged to the optimal state.

[0122] In the embodiments of the present disclosure, the above-mentioned training method can effectively combine the advantages of clustering and supervised learning, continuously optimize the model, enable it to better understand and parse the content of the bill image, and at the same time improve the classification accuracy.

[0123] Based on the above-mentioned bill image processing method, the present disclosure also provides a bill processing device. Figure 8 The device is described in detail.

[0124] Figure 8 The structural framework diagram of the bill processing device according to the embodiment of the present disclosure is schematically shown.

[0125] like Figure 8 As shown, the bill image processing device of this embodiment includes an extraction module 810 , a processing module 820 and a clustering module 830 .

[0126] The extraction module 810 is used to extract text information and image information from the bill image, and fuse the text information and image information to obtain fused information. In one embodiment, the extraction module 810 can be used to perform the operation S210 described above, which will not be described in detail here.

[0127] The processing module 820 is used to determine the target cluster to which the bill image belongs based on the fusion information, wherein the target cluster includes at least one reference image. In one embodiment, the processing module 820 can be used to perform the operation S220 described above, which will not be described in detail here.

[0128] The clustering module 830 is used to determine the reference image whose similarity with the bill image is greater than a preset threshold in at least one reference image as a target reference image, and the target reference image is used to guide the input of structured data of the bill image. In one embodiment, the clustering module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0129] According to an embodiment of the present disclosure, the extraction module 810 includes:

[0130] The first extraction unit 811 is used to extract the first feature vector and the second feature vector of the bill image in response to the input bill image, wherein the first feature vector is used to represent the text information of the bill; and the second feature vector is used to represent the image information of the bill. In one embodiment, the first extraction unit 811 can be used to perform the operation S310 described above, which will not be repeated here.

[0131] The second extraction unit 812 is used to determine a first fused feature vector based on the first feature vector and the second feature vector, wherein the first fused feature vector is used to represent fused information including text information and image information; including flattening the first feature vector and the second feature vector respectively; fusing the flattened first feature vector and the flattened second feature vector to obtain a third feature vector, wherein the third feature vector is a feature vector after the first fusion of the first feature vector and the second feature vector; and determining the first fused feature vector based on the third feature vector. In one embodiment, the second extraction unit 812 can be used to perform the operations S320, S410, and S420 described above, which will not be repeated here.

[0132] According to an embodiment of the present disclosure, the processing module 820 includes:

[0133] The first processing unit 821 is used to respectively determine the distances between the first fused feature vector and the multiple cluster centers. In one embodiment, the first processing unit 821 can be used to perform the operation S510 described above, which will not be described in detail here.

[0134] The second processing unit 822 is used to determine the target cluster of the bill image from the multiple clusters according to the distance. In one embodiment, the first processing unit 822 can be used to perform the operation S520 described above, which will not be described in detail here.

[0135] According to an embodiment of the present disclosure, the clustering module 830 includes:

[0136] The reference unit 831 is used to obtain a second fused feature vector of at least one reference image, where the second fused feature vector is a fused feature vector of at least one reference image. In one embodiment, the reference unit 831 can be used to perform the operation S610 described above, which will not be described in detail herein.

[0137] The calculation unit 832 is used to respectively determine the similarity between the second fused feature vector of at least one reference image and the first fused feature vector of the bill image; and determine the reference image whose similarity is greater than a preset threshold as the target reference image. In one embodiment, the calculation unit 832 can be used to perform the operation S620 described above, which will not be repeated here.

[0138] According to the embodiments of the present disclosure, the specific implementation details and technical effects of the bill image device are the same as the specific implementation details and technical effects of the bill image processing method.

[0139] According to the embodiment of the present disclosure, any multiple modules of the extraction module 810, the processing module 820 and the clustering module 830 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to the embodiment of the present disclosure, at least one of the extraction module 810, the processing module 820 and the clustering module 830 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the extraction module 810, the processing module 820 and the clustering module 830 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0140] Fig. 9 A block diagram of an electronic device suitable for implementing a bill image processing method according to an embodiment of the present disclosure is schematically shown.

[0141] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0142] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0143] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0144] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0145] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0146] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method for bill image processing provided by the embodiment of the present disclosure.

[0147] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0148] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0149] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0150] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0151] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0153] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A bill processing method, characterized in that: The method comprises: Extracting text information and image information from the bill image, and fusing the text information and the image information to obtain fused information; Determine the target cluster to which the bill image belongs based on the fusion information, wherein the target cluster includes at least one reference image; A reference image among the at least one reference image whose similarity with the bill image is greater than a preset threshold is determined as a target reference image, and the target reference image is used to guide the input of structured data of the bill image.

2. The method according to claim 1, characterized in that The extracting of text information and image information from the bill image and fusing the text information and image information to obtain fused information includes: In response to an input bill image, extract a first feature vector and a second feature vector of the bill image, wherein the first feature vector is used to represent text information of the bill; and the second feature vector is used to represent image information of the bill; A first fused feature vector is determined according to the first feature vector and the second feature vector, where the first fused feature vector is used to represent fused information including the text information and the image information.

3. The method according to claim 2, characterized in that The determining a first fused feature vector according to the first feature vector and the second feature vector includes: Flattening the first eigenvector and the second eigenvector respectively; The first flattened feature vector and the second flattened feature vector are fused to obtain a third feature vector, wherein the third feature vector is a feature vector obtained by first fusion of the first feature vector and the second feature vector; The first fused feature vector is determined according to the third feature vector.

4. The method according to claim 3, characterized in that Determining the first fused feature vector according to the third feature vector includes: The third feature vector is subjected to a second fusion process to obtain the first fused feature vector.

5. The method according to claim 2, characterized in that: Determining the target cluster of the bill image based on the fusion information includes: respectively determining the distances between the first fused feature vector and a plurality of cluster centers; The target cluster of the receipt image is determined from the plurality of clusters according to the distance.

6. The method according to claim 5, characterized in that The step of determining, among the at least one reference image, a reference image whose similarity with the bill image is greater than a preset threshold as a target reference image, wherein the target reference image is used to guide the input of structured data of the bill image, comprises: Acquire a second fused feature vector of at least one reference image, where the second fused feature vector is a fused feature vector of the at least one reference image; respectively determining the similarity between the second fused feature vector of the at least one reference image and the first fused feature vector of the bill image; The reference image whose similarity is greater than a preset threshold is determined as the target reference image.

7. The method according to claim 6, characterized in that The respectively determining the similarity between the second fused feature vector of the at least one reference image and the first fused feature vector of the bill image comprises: The Euclidean distance between the second fused feature vector of the at least one reference image and the first fused feature vector of the bill image is calculated respectively.

8. A method for training a deep clustering model, characterized in that: The deep clustering model is used to implement the method according to any one of claims 1 to 7, and the training method comprises: Obtaining a training sample from a training sample set, wherein the training sample is a bill image; Determine a first eigenvector and a second eigenvector from the input training sample; Determine the first fused feature vector based on the first feature vector and the second feature vector; Determine the cluster to which the training sample belongs according to the first fused feature vector; Calculating a cross entropy loss function based on the clusters and the probabilities of the clusters predicted by the classification model; Back propagation is performed to update the weights according to the cross entropy loss function until the model training is completed.

9. A bill processing device, characterized in that: The device comprises: An extraction module, used for extracting text information and image information from the bill image, and fusing the text information and image information to obtain fused information; a processing module, configured to determine a target cluster to which the bill image belongs based on the fusion information, wherein the target cluster includes at least one reference image; and The clustering module is used to determine a reference image whose similarity with the bill image is greater than a preset threshold among the at least one reference image as a target reference image, wherein the target reference image is used to guide the input of structured data of the bill image.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images