Bill information extraction method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a bill information extraction method and device, equipment, a medium and a program product, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a bill image of a target bill; identifying a text in the bill image to obtain bill text information; inputting the bill image and the bill text information into a pre-trained multi-modal information extraction model to enable the multi-modal information extraction model to execute the following steps: respectively extracting features of the bill image and the bill text information to obtain image features and text features; quantifying an image uncertainty index and a text uncertainty index; determining weights of the image features and the text features based on the image uncertainty index and the text uncertainty index; and based on the weights of the image features and the text features, fusing the features to generate a fusion representation of the target bill. The information of the two dimensions of the image and the text is dynamically fused, so that the information of the two dimensions complements each other, and the accuracy of an extraction result is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, medium, and program product for extracting invoice information. Background Technology

[0002] Negotiable instruments (such as bank drafts, commercial drafts, checks, and insurance policies) are key payment and settlement tools for financial transactions and credit circulation, widely used in scenarios such as inter-enterprise trade settlement, personal fund transfers, and bank custody. In the processing of physical negotiable instruments, banks and other financial institutions often need to extract key information (such as amount, drawer's account number, validity period, and the bank to which the seal belongs) from the physical instrument before processing transactions to ensure compliance and fund security.

[0003] Currently, when extracting information from invoices, especially physical invoices, image recognition is the primary method used to extract visual features such as seals and serial numbers. However, due to physical wear and tear during the circulation of physical invoices and limitations in the accuracy of scanning equipment, the extracted information often suffers from missing, damaged, or misidentified features, affecting the accuracy of the extracted characteristics. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and program product for extracting invoice information. By extracting information from both the image and text dimensions of the invoice separately, and then dynamically fusing the features of the two dimensions based on the uncertainty of each dimension, the two dimensions can complement each other. This avoids the problem of inaccurate final extraction results caused by the lack, damage, or misidentification of features in a single dimension, thereby improving the accuracy of invoice information extraction.

[0005] Firstly, this application provides a method for extracting invoice information, including:

[0006] Obtain the image of the target invoice;

[0007] Identify the text in the ticket image to obtain the ticket text information;

[0008] The invoice image and invoice text information are input into a pre-trained multimodal information extraction model, which then performs the following steps to obtain a fused representation of the target invoice:

[0009] Features of the invoice image and invoice text information are extracted separately to obtain image features and text features;

[0010] Image uncertainty indices for quantifying image features, and text uncertainty indices for text features;

[0011] The weights of image features and text features are determined based on image uncertainty index and text uncertainty index.

[0012] Based on the weights of image features and text features, the image features and text features are fused to generate a fused representation of the target document.

[0013] Secondly, this application provides a method for verifying the authenticity of a negotiable instrument, including:

[0014] Obtain the bill fusion representation of the target bill; the bill fusion representation is obtained through the method provided in the first aspect above.

[0015] Based on the fusion representation of the bills, extract the key features of the bills;

[0016] Calculate the similarity between the key features of each invoice and the typical features of the corresponding class in the preset feature library;

[0017] If the similarity meets the preset conditions, the target ticket is determined to be a genuine ticket.

[0018] Thirdly, this application provides a device for extracting invoice information, comprising:

[0019] The acquisition module is used to acquire the image of the target ticket;

[0020] The text recognition module is used to recognize the text in the ticket image and obtain the ticket text information;

[0021] The fusion representation generation module is used to input the invoice image and invoice text information into a pre-trained multimodal information extraction model, so that the multimodal information extraction model performs the following steps to obtain the fusion representation of the target invoice: extracting features from the invoice image and invoice text information respectively to obtain image features and text features; quantifying the image uncertainty index of the image features and the text uncertainty index of the text features; determining the weights of the image features and the text features based on the image uncertainty index and the text uncertainty index; and fusing the image features and text features based on the weights of the image features and the text features to generate the fusion representation of the target invoice.

[0022] Fourthly, this application provides a device for verifying the authenticity of invoices, comprising:

[0023] The fusion representation acquisition module is used to acquire the fusion representation of the target bill; the fusion representation is obtained through the method provided in the first aspect above.

[0024] The key feature extraction module is used to extract key features of invoices based on the invoice fusion representation;

[0025] The similarity calculation module is used to calculate the similarity between the key features of each invoice and the typical features of the corresponding class in the preset feature library;

[0026] The authenticity verification module is used to determine that the target ticket is genuine if the similarity meets the preset conditions.

[0027] Fifthly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0028] The memory stores instructions that the computer executes;

[0029] The processor executes computer-executable instructions stored in memory to implement the methods provided in the first and / or second aspects above.

[0030] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the first and / or second aspects above.

[0031] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the first and / or second aspects above.

[0032] The method, apparatus, equipment, medium, and program products for extracting invoice information provided in this application obtain two dimensions of information—invoice image and invoice text—by acquiring an image of the target invoice and recognizing the text within the image. This leverages the complementary advantages of these two dimensions, overcoming the limitations of single-dimensional information. The invoice image and text information are then input into a pre-trained multimodal information extraction model, which extracts features from the input information. Uncertainty quantification is then applied to these features. Finally, based on this uncertainty, the model determines the weights of the image and text features and fuses them to obtain a fused representation of the target invoice. This application achieves adaptive fusion of image and text information, allowing them to complement and synergize. When information in one dimension is missing, damaged, or misidentified, information from the other dimension can supplement and correct it, improving the accuracy of the information extraction results. Furthermore, by dynamically adjusting the weights during fusion based on the uncertainty of the information, information with lower uncertainty contributes more to the fused representation. This effectively solves the problem of poor fusion results due to uneven data quality, thus improving the robustness of the fusion and further enhancing the reliability of the information extraction results. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0034] Figure 1 A flowchart illustrating a method for extracting invoice information provided in an embodiment of this application;

[0035] Figure 2 A schematic diagram of a ticket image provided for an embodiment of this application;

[0036] Figure 3 A flowchart illustrating another method for extracting invoice information provided in this application embodiment;

[0037] Figure 4 A flowchart illustrating a method for verifying the authenticity of a document provided in this application embodiment;

[0038] Figure 5 A flowchart illustrating another method for verifying the authenticity of invoices provided in this application embodiment;

[0039] Figure 6 This is a schematic diagram of the structure of a bill information extraction device provided in an embodiment of this application;

[0040] Figure 7 A schematic diagram of a document authenticity verification device provided in this application embodiment;

[0041] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0045] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0046] It should be noted that the bill information extraction method, device, equipment, medium and program products provided in this application can be used in the field of artificial intelligence, or in any field other than artificial intelligence. The application field of the bill information extraction method, device, equipment, medium and program products in this application is not limited.

[0047] Negotiable instruments (such as bank drafts, commercial drafts, checks, and insurance policies) are legally issued securities and payment vouchers. They are key payment and settlement tools for financial transactions and credit circulation, widely used in inter-enterprise trade settlements, personal fund transfers, and bank custody. This application applies to scenarios involving the processing of negotiable instruments, specifically in situations where information needs to be extracted from the instruments for subsequent operations, such as extracting information to verify the authenticity of the instruments or extracting information from the instruments to conduct transaction operations, such as remittances.

[0048] Currently, the extraction of key information from invoices mainly relies on image recognition, which involves identifying information in the invoice image, such as visual features like seals, serial numbers, and amounts, and then determining the identified information as the final key information extracted.

[0049] However, since some of these documents are physical, they are highly susceptible to physical wear and tear during circulation, leading to missing or damaged information. For example, the document number may become illegible due to creases or folds, or the document itself may be damaged, resulting in information loss. Furthermore, when converting physical documents to electronic information, such as by photographing or scanning them, factors like the precision of the photographing / scanning equipment, human error, or environmental conditions can also cause missing or incorrect information. For instance, a poor shooting angle may result in a blurry image, or uneven lighting may weaken information in certain areas. Consequently, key information obtained solely through image recognition may be missing, damaged, or misidentified, affecting the accuracy of information extraction and potentially impacting subsequent operations, such as remittances, thus posing a risk to financial security.

[0050] The invoice information extraction method provided in this application aims to solve the above-mentioned technical problems. Specifically, it acquires an image of the invoice and identifies the text within the image, obtaining information in both image and text dimensions. Then, it inputs the invoice image and the identified text into a multimodal information extraction model. Leveraging the model's powerful capabilities, it extracts features from both the image and text, quantifies the uncertainty of each dimension's features, determines the weights of each dimension based on this uncertainty, and finally dynamically fuses the image and text features. This method uses image and text features to complement and correct each other, avoiding errors in the final result due to incorrect information extracted from a single dimension, thus improving the accuracy of information extraction. Simultaneously, by dynamically determining weights based on uncertainty, it achieves dynamic adjustment according to feature quality, improving the robustness of the fusion and consequently enhancing the reliability of information extraction.

[0051] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0052] Figure 1 This is a flowchart illustrating a method for extracting invoice information according to an embodiment of this application. The invoice information extraction method provided in this application can be executed by an electronic device with corresponding processing capabilities. Figure 1 As shown, the method for extracting invoice information provided in this application includes the following steps:

[0053] Step S101: Obtain the image of the target ticket.

[0054] Negotiable instruments refer to securities and payment vouchers issued in accordance with the law, including but not limited to bank drafts, commercial drafts, checks, invoices, and insurance policies. Negotiable instruments can be classified into physical negotiable instruments and virtual negotiable instruments (also known as electronic negotiable instruments) according to their form. The negotiable instruments in this application can be either physical or virtual (also known as electronic negotiable instruments), but specifically physical negotiable instruments. The target negotiable instrument is a negotiable instrument pending processing, i.e., a negotiable instrument from which information needs to be extracted.

[0055] A bill image refers to a visualized digital image corresponding to the information on a bill. A bill image includes images of the main body area, seal area, serial number area, and watermark area of ​​the bill. For example, Figure 2 This is a schematic diagram of a ticket image provided in an embodiment of this application. Figure 2 As shown, the image of the ticket includes, but is not limited to, the main body of the ticket, the seal, the serial number, and the watermark.

[0056] Specifically, for physical tickets, technologies such as photography and scanning can be used to convert the ticket information into a digital image, i.e., a ticket image. For virtual tickets, the ticket image can be obtained from the corresponding electronic document.

[0057] In some embodiments, after acquiring the image of the target ticket, the ticket image can be preprocessed to optimize the quality of the ticket image, reduce noise interference to subsequent processing, and ensure that the information of the ticket can be effectively extracted.

[0058] Specifically, Gaussian filtering and adaptive histogram equalization can be used to denoise and enhance the contrast of the ticket image, thereby improving its quality.

[0059] In some embodiments, damaged areas in the ticket image can also be identified, and edge repair can be performed on the damaged areas.

[0060] Damaged areas in a document image are regions where visual features are incompletely displayed due to physical wear. Damaged areas can be identified using edge detection algorithms or by analyzing abrupt changes in grayscale or brightness values ​​within the document image.

[0061] Step S102: Identify the text in the ticket image to obtain the ticket text information.

[0062] The text information on a negotiable instrument refers to the text present on the instrument, including but not limited to information about the drawer, amount, purpose, type, and validity period. Based on the text type in the instrument image, the text information includes both printed text and handwritten text on the instrument.

[0063] Continue to refer to Figure 2 The images of receipts often also include textual information, such as... Figure 2 The text on the document contains information such as the type of bill, payee, issuer, amount, date of issue, validity period, and handwritten purpose information. This textual information also carries crucial details about the bill. By obtaining this textual information, the image of the bill can be supplemented and verified.

[0064] In this step, OCR (Optical Character Recognition) technology is used to recognize the text in the ticket image and obtain the ticket text information.

[0065] It is also possible to use intelligent enhanced recognition technology to identify the text in the ticket image and obtain the ticket text information.

[0066] Intelligent augmented recognition technology is based on traditional OCR technology and integrates deep learning, computer vision and semantic understanding technologies to achieve text extraction and recognition.

[0067] Specifically, an intelligent augmented recognition model is constructed based on intelligent augmented recognition technology. This model is trained using a training dataset, enabling it to learn the features of each character. Then, based on these features, the model extracts the text information from the document image.

[0068] To improve the accuracy and generalization of text recognition, the training dataset can include images of different fonts (including various types of printed text and various styles of handwritten text), special font images, mixed language text images, and text images in various scenarios (such as text images under different lighting conditions, different degrees of wear, and different levels of clarity).

[0069] In some embodiments, the bill text information may also include other text associated with the target bill, including but not limited to the text of other documents associated with the bill, such as the bill issuance application form, verification record, and accompanying business instructions.

[0070] In some embodiments, after obtaining the document text information, the document text information can be stored in a structured format, such as JSON, to facilitate computer understanding.

[0071] Optionally, after recognizing the text in the invoice image and obtaining the invoice text information, the method further includes: performing structured field validation on the invoice text information to identify abnormal fields in the invoice text information; correcting and / or semantically completing the abnormal fields; and then performing the step of inputting the invoice image and invoice text information into a multimodal information extraction model.

[0072] Structured field validation verifies the consistency of field format, values, and logic through preset rules.

[0073] Abnormal fields are those in the document text that do not conform to the preset format, rules, and logic. In real-world scenarios, abnormal fields include fields containing typos and fields with missing content.

[0074] In this embodiment, structured field validation is performed on the document text information to identify fields that do not conform to preset rules and determine them as abnormal fields. If the abnormal field contains typos, it is corrected, and the corrected field replaces the abnormal field in the document text information. If the abnormal field contains missing content, semantic completion is performed, and the completed field replaces the abnormal field in the document text information.

[0075] For example, if there is an abnormal field that is “amount of ten thousand”, then the abnormal field is semantically completed, such as being completed as “amount of ten thousand yuan”.

[0076] In some embodiments, this step can be performed by a large language model. A large language model (LLM) is a deep learning model trained on massive amounts of text data, whose core function is to achieve understanding and generation by learning language rules.

[0077] Specifically, the invoice text information is input into the large model, which performs structured field validation on the invoice text information, identifies abnormal fields in the invoice text information, and corrects and / or completes the semantics of the abnormal fields after identification. Finally, the large model outputs the invoice text information after correction and / or semantic completion.

[0078] In some embodiments, after obtaining the document text information, the document text information can be standardized, that is, converted into a unified format. For example, the unified validity period format is "YYYY-MM-DD".

[0079] By correcting and / or semantically completing abnormal fields in the invoice text information, the data quality of the invoice text information can be improved, thereby enhancing its usability and providing a high-quality data foundation for subsequent steps.

[0080] Step S103: Input the ticket image and ticket text information into the pre-trained multimodal information extraction model so that the multimodal information extraction model can perform the following steps S1031 to S1034 to obtain the fused representation of the target ticket.

[0081] Multimodal information extraction models are artificial intelligence models capable of understanding, processing, and generating information in multiple different modalities (such as images and text). By training the model, semantic associations between different modalities can be established, such as associating the stamp corresponding to the ticket issuer (e.g., when the issuer is xx).

[0082] The multimodal information extraction model is used to extract features from the input ticket image and ticket text information, quantify uncertainty indicators, dynamically assign weights, and fuse features to finally obtain a fused representation of the target ticket.

[0083] Specifically, the multimodal information extraction model performs the following steps to ultimately obtain the fused features of the target document:

[0084] Step S1031: Extract features from the ticket image and ticket text information respectively to obtain image features and text features.

[0085] In this step, features of the invoice image are extracted to obtain image features; features of the invoice text information are extracted to obtain text features.

[0086] The features of the bill image are extracted, and the resulting image features can be obtained by performing layered feature extraction on the bill image, prioritizing the extraction of key visual features such as seal texture features, serial number outline features, and watermark features.

[0087] Specifically, the ticket image is segmented to obtain at least one sub-region image from which features need to be extracted; features are extracted from the at least one sub-region image to obtain image features.

[0088] One method for segmenting a document image is to pre-set the coordinate positions of each sub-region in the document image, such as the coordinate positions of the document type or the seal; then, the document image is segmented according to the preset coordinate positions.

[0089] Alternatively, the edge contours of each sub-region in the ticket image can be identified by the differences in attribute parameters (such as grayscale values) or edge detection algorithms (such as the Canny operator), and the sub-region images can be obtained by locating and segmenting based on the edge contours.

[0090] Features are extracted from at least one sub-region image to obtain image features. Each sub-region image is then input into a deep learning-based feature extraction model, which can be a Convolutional Neural Network (CNN), ResNet (Residual Network), or VGG (Visual Geometry Group). The feature extraction model extracts features from each sub-region image based on key features learned in advance from a large number of image samples, thus obtaining image features.

[0091] For example, the images of each sub-region are input into a convolutional neural network. The convolutional layers of the convolutional neural network capture local detail features in the sub-region images by sliding the convolutional kernels. The pooling layers downsample the local detail features, that is, retain key information and reduce dimensionality. Then, the deep network fuses the shallow features to finally generate high-level features that can accurately describe the sub-region images.

[0092] The image features can be obtained by extracting features from at least one sub-region image. Alternatively, based on the attribute parameters (such as grayscale values) of the sub-region image, the spatial distribution of pixels corresponding to the attribute parameters can be statistically analyzed to identify the texture features in the sub-region image, and finally, the image features can be obtained.

[0093] For example, the sub-region image is binarized, that is, the grayscale / color image is converted into a binary image containing only black and white pixels. Based on the spatial distribution, quantity ratio and morphological structure of black and white pixels, the features of the sub-region are extracted to obtain the image features.

[0094] Extracting features from the textual information of a negotiable instrument yields textual features. These features can be derived using natural language processing techniques to extract the core semantic information, which constitutes the textual features. The core semantic information includes at least the issuer's name, monetary keywords, and validity period.

[0095] For example, the fields corresponding to the ticket issuer's name, amount keywords, and validity period are matched according to preset rules, and then the corresponding fields are converted into vectors, which are the text features.

[0096] For example, the text features are matched to corresponding fields according to preset rules, such as locating the corresponding fields by keywords (amount, name, etc.) or fixed identifiers. Then, BERT (Bidirectional Encoder Representations from Transformers) is used to convert the corresponding fields into vectors, forming text features.

[0097] Step S1032: Quantify the image uncertainty index of image features and the text uncertainty index of text features.

[0098] The uncertainty index is used to characterize the uncertainty of image and text features. The higher the uncertainty index, the higher the probability that the corresponding image or text features are incorrect, i.e., the lower the accuracy.

[0099] In this step, the multimodal information extraction model quantifies the degree of deviation between the obtained image features and text features and the normal pattern based on the pre-learned normal pattern. Finally, the degree of deviation is converted into a numerical value to obtain the image uncertainty index of image features and the text uncertainty index of text features.

[0100] The uncertainty of image features can be quantified by analyzing the distribution of their attribute parameters. The text uncertainty index can be determined based on text confidence and the information entropy of text features. Confidence is an indicator reflecting the accuracy of text recognition when identifying text in a document image and obtaining document text information.

[0101] Specifically, the discreteness and anomalies of the attribute parameters of statistical image features are quantified to obtain the uncertainty of image features. Discreteness can be obtained by calculating the standard deviation of the gray-level histogram of the image features. Anomalies can be determined by calculating the degree of deviation of the distribution of attribute parameters from a preset distribution (such as the difference in mean or variance). The preset distribution is a pre-defined standard distribution of the attribute parameters of the image features under clear and complete conditions.

[0102] If the image features are more blurred, such as the higher the blur of a seal or watermark, the more it will disrupt the regular distribution of gray levels, resulting in more dispersed attribute parameters and higher discreteness. If the image features are missing or damaged, it will naturally increase the degree of deviation from the standard distribution. Therefore, discreteness and anomalies can effectively reflect the image uncertainty index of image features.

[0103] In some embodiments, the image features include multiple features, such as seal texture features, serial number contour features, and watermark features. For each feature, an uncertainty index can be determined. The image uncertainty index is obtained by weighting the uncertainty indices of each feature.

[0104] For example, the uncertainty index is set to 0.2 for clear seal texture features and 0.1 for unclear number outline features and 0.1 for watermark features. Finally, the uncertainty indices of each item are weighted to obtain the image uncertainty index. The clarity of each item can be determined based on whether the discreteness and anomaly of its attribute parameters meet preset parameter conditions. These preset parameter conditions can be that the discreteness is less than a discreteness threshold and the anomaly is less than an anomaly threshold.

[0105] Step S1033: Determine the weights of image features and text features based on image uncertainty index and text uncertainty index.

[0106] The correspondence between image uncertainty index and image feature weights, as well as the correspondence between text uncertainty index and text feature weights, can be preset, such as linear or non-linear relationships, so that features with lower uncertainty indices have higher corresponding weights.

[0107] In this step, the image uncertainty index is substituted into the correspondence to obtain the weights of the image features. Similarly, the text uncertainty index is substituted into the correspondence to obtain the weights of the text features.

[0108] In some embodiments, a correspondence between image uncertainty indicators, text uncertainty indicators, and the weights of image features can be pre-defined. The image uncertainty indicators and text uncertainty indicators are then substituted into this correspondence to obtain the weights of the image features. Similarly, a correspondence between image uncertainty indicators, text uncertainty indicators, and the weights of text features can be pre-defined. The image uncertainty indicators and text uncertainty indicators are then substituted into this correspondence to obtain the weights of the text features.

[0109] By dynamically adjusting the fusion weights, the contribution of each feature to the fusion representation is adjusted according to the uncertainty of image and text features. This avoids low-quality features interfering with the fusion results and increases the value of high-quality features, thereby improving the accuracy and robustness of the fusion representation.

[0110] Step S1034: Based on the weights of image features and text features, the image features and text features are fused to generate a fused representation of the target document.

[0111] In this step, the multimodal information extraction model uses the weights of image features and the weights of text features to fuse image features and text features, and finally outputs a fused representation of the target document.

[0112] One possible fusion method is to convert image features and text features into vectors of the same dimension in the same feature space, and then perform a weighted combination of the elements in the image features of the same dimension with the corresponding elements in the text features to finally obtain the fused representation of the target document.

[0113] Another possible fusion method is to convert image features and text features into vectors of the same dimension in the same feature space, identify the elements with higher weights in the image features and corresponding elements in the text features as target elements, and concatenate the target elements to obtain the fused representation of the target ticket.

[0114] For example, a Transformer layer can be used to implement weights based on image features and text features, and to fuse image features and text features to generate a fused representation of the target document.

[0115] By fusing image and text features, the resulting fused representation includes both visual features such as seals and serial numbers from the image and semantic features such as issuer information and amount from the text. If the seal is blurry, the issuer's name in the text can help identify the seal information. This effectively avoids the risk of errors in the final extraction result due to incorrect information in a single dimension, improving the reliability and accuracy of the fused representation.

[0116] In some embodiments, after generating a fusion representation of the target document, information weaknesses in the fusion representation can be identified, such as ambiguous monetary features or unclear seal ownership information. Then, image features and text features are used to enhance the information weaknesses to strengthen their features.

[0117] The invoice information extraction method provided in this application obtains two dimensions of information—invoice image and invoice text—by acquiring the invoice image of the target invoice and recognizing the text in the invoice image. This allows the complementary advantages of the two dimensions to overcome the limitations of single-dimensional information. Then, the invoice image and invoice text information are input into a pre-trained multimodal information extraction model, which extracts features from the input information. The uncertainty of these features is then quantified, and finally, based on this uncertainty, the model determines the weights of the image and text features and fuses them to obtain a fused representation of the target invoice. This application achieves adaptive fusion of image and text information, enabling these two dimensions to complement and synergize. When information in one dimension is missing, damaged, or misidentified, information in the other dimension can supplement and correct it, improving the accuracy of the information extraction results. Simultaneously, by dynamically adjusting the weights during fusion based on the uncertainty of the information, information with lower uncertainty contributes more to the fused representation. This approach effectively solves the problem of poor fusion results due to uneven data quality, thus improving the robustness of the fusion and further enhancing the reliability of the information extraction results.

[0118] Figure 3 This is a flowchart illustrating another method for extracting invoice information provided in an embodiment of this application. The method provided in this embodiment is... Figure 1 Based on the illustrated embodiment, the steps performed by the multimodal information extraction model have been refined. For example... Figure 3 As shown, after inputting the ticket image and ticket text information into the pre-trained multimodal information extraction model, the multimodal information extraction model specifically performs the following steps:

[0119] Step S301: Extract features from the ticket image and ticket text information respectively to obtain image features and text features.

[0120] Step S302: Calculate the sharpness of the image features based on the attribute parameters of the image features.

[0121] In this step, the gradient value of the image feature can be calculated based on the attribute parameters of the image feature (such as grayscale value); the sharpness of the image feature can be determined based on the statistical values ​​of the gradient value of the image feature, such as the mean, maximum or variance.

[0122] For example, the Sobel operator is used to calculate the rate of change of gray values ​​in the horizontal or vertical direction in the image features to obtain multiple gradient values; the variance of multiple gradient values ​​is calculated, and the sharpness is determined according to the correspondence between variance and sharpness, such as a linear relationship, so that the larger the variance, the higher the sharpness.

[0123] Sharpness can be expressed as: Sharpness = k Variance + b. Where k and b are pre-set configurable parameters.

[0124] Step S303: Calculate the similarity between the image features and the preset standard image features.

[0125] The preset standard image features are the image features of a clear and complete ticket image that have been set in advance.

[0126] In this step, the average of the squared differences between the attribute parameters corresponding to the image features and the preset standard image features is calculated. The smaller the value, the more similar the images are, i.e., the higher the similarity.

[0127] Specifically, the average of the squared differences between the attribute parameters corresponding to the image features and the preset standard image features is calculated. This average is then substituted into a linear relationship to obtain the similarity. The parameters in the linear relationship are pre-set configurable parameters.

[0128] Step S304: Determine the image uncertainty index of image features based on sharpness and similarity.

[0129] In this step, sharpness and similarity are weighted to determine the image uncertainty index of image features.

[0130] For example, the weight of sharpness is set to 0.6, the weight of similarity is set to 0.4, and the image uncertainty index is set to 0.6. Clarity +0.4 Similarity.

[0131] Step S305: Quantify the first uncertainty index of text features through text field integrity analysis.

[0132] Text field integrity analysis is a technique used to identify whether an input text field contains missing parts. It can count the proportion of blank values ​​and placeholders to the total number of characters, and check whether the text field conforms to preset rules through regular expressions or rule matching.

[0133] In this step, text field integrity analysis can be performed by the text field integrity analysis layer. The text field integrity analysis layer determines the preset rules that the text features do not conform to by calculating the proportion of whitespace values ​​and placeholders in the total number of characters, and by matching regular expressions or rules. Based on this proportion, the number of preset rules that do not conform, and the type of preset rules that do not conform, a first uncertainty index for the text features is determined.

[0134] For example, a weight is determined based on the type of preset rule that is not conformed to. This weight is then weighted by the number of preset rules that are not conformed to, resulting in a first uncertainty index.

[0135] Step S306: Quantify the second uncertainty index of text features through semantic consistency verification.

[0136] Semantic consistency verification is a technique used to determine whether one or more segments of text are consistent in their core meaning, logical relationship, or information orientation, rather than simply comparing whether the texts are the same.

[0137] Specifically, semantic consistency verification can be achieved by converting text features and standard text features into fixed-dimensional semantic vectors using a large model, and then measuring the differences using cosine similarity or Euclidean distance.

[0138] Standard text features are the text features of pre-set standard invoice text information.

[0139] In this step, semantic consistency verification can be implemented by a semantic consistency verification layer. The semantic consistency verification layer obtains the standard text features stored internally, converts the text features and standard text features into fixed-dimensional semantic vectors, and calculates the difference using cosine similarity or Euclidean distance. Based on this difference, a second uncertainty index is determined.

[0140] Step S307: Weight the first uncertainty index and the second uncertainty index to obtain the text uncertainty index of the text features.

[0141] In this step, the first uncertainty index and the second uncertainty index are weighted using pre-set weight values ​​to obtain the text uncertainty index of the text features.

[0142] Step S308: Obtain the initial weights of image features and text features.

[0143] The initial weights of image features and text features are pre-set configurable parameters. For example, the initial weight of image features is set to 0.6, and the initial weight of text features is set to 0.4.

[0144] The initial weights of image features and text features can be stored in the system. The initial weights of image features and text features are obtained by reading the initial weights stored in the system.

[0145] Step S309: Adjust the initial weights of image features and text features based on the comparison results of image uncertainty index and text uncertainty index.

[0146] Step S310: The initial weights of the adjusted image features and the initial weights of the adjusted text features are determined as the weights of the image features and the weights of the text features.

[0147] In this step, the initial weights of the image features and the initial weights of the text features are adjusted based on the difference between the image uncertainty index and the text uncertainty index, and / or the feature corresponding to the maximum value.

[0148] In some embodiments, if the feature corresponding to the maximum value is an image feature, the initial weight of the image feature is added to the first adjustment value to obtain the weight of the image feature, for example, the weight of the image feature is adjusted to 0.7; the initial weight of the text feature is subtracted from the second adjustment value to obtain the weight of the text feature.

[0149] In some embodiments, if the feature corresponding to the maximum value is a text feature, the initial weight of the text feature is added to the third adjustment value to obtain the weight of the text feature, such as adjusting the weight of the text feature to 0.65; the initial weight of the image feature is subtracted from the fourth adjustment value to obtain the weight of the image feature.

[0150] The first, second, third, and fourth adjustment values ​​can be fixed or dynamic. For example, the first to fourth adjustment values ​​can be set to 0.1, or they can be determined based on the difference between the image uncertainty index and the text uncertainty index.

[0151] For example, the higher the difference, the higher the first to fourth adjustment values.

[0152] Step S311: Among the image features and text features that have a correlation, the items with higher weights are determined as main feature items, and the items with lower weights among the image features and text features that have a correlation are determined as auxiliary feature items.

[0153] Image features and text features include multiple components, and there is a correlation between image features and text features.

[0154] For example, image features include, but are not limited to, seal texture features, serial number outline features, and watermark features, while text features include, but are not limited to, issuer name features, amount features, bill type features, and validity period features.

[0155] The seal on a negotiable instrument is the exclusive tool of the issuer, and there is a one-to-one correspondence between the issuer and the seal. Therefore, the characteristics of the issuer's name are related to the texture characteristics of the seal. The watermark on a negotiable instrument is a security feature embedded in the paper or printing layer. Different types of negotiable instruments correspond to different watermarks; therefore, the watermark characteristics are related to the instrument type. The instrument's serial number is named according to certain rules. Generally, relevant information about the instrument, such as its type, can be obtained through the serial number. Therefore, the outline characteristics of the serial number are related to the instrument type.

[0156] In this step, if the weight of image features is greater than the weight of text features, then the image feature items with a correlation are determined as primary feature items, and the text feature items with a correlation are determined as secondary feature items. If the weight of image features is less than the weight of text features, then the text feature items with a correlation are determined as primary feature items, and the image feature items with a correlation are determined as secondary feature items.

[0157] Step S312: If the weight of the main feature item is less than the preset weight threshold, and the difference between the weight of the main feature item and the weight of the auxiliary feature item is less than the preset difference threshold, then the main feature item is supplemented according to the auxiliary feature item to generate the fusion representation of the target document.

[0158] The preset weight threshold is a pre-set configurable parameter. When the weight is less than the preset weight threshold, it indicates that the information of the main feature item is relatively weak, and there is a risk of extraction errors.

[0159] In this step, if the weight of the main feature term is less than the preset weight threshold, and the difference between the weight of the main feature term and the weight of the auxiliary feature term is less than the preset difference threshold, it indicates that the main feature term is relatively vague. However, although the auxiliary feature term has a low weight, it contains effective information. Therefore, the auxiliary feature term can be used to supplement the main feature term.

[0160] Specifically, if the weight of the main feature item is less than the preset weight threshold, and the difference between the weight of the main feature item and the weight of the auxiliary feature item is less than the preset difference threshold, then based on the auxiliary feature item, the missing details in the main feature item with the relationship are supplemented, the key information is strengthened, and the deviation is corrected. The supplemented main feature item is then determined as the item for the fusion representation of the target document.

[0161] By supplementing missing details, enhancing key information, and correcting biases in main features that are related, auxiliary features can be fused with main features to achieve the same purpose.

[0162] For example, if the main feature item includes a seal feature and the seal ownership information is unclear, the seal ownership information in the main feature item can be supplemented by the issuer's name feature.

[0163] Optionally, the method further includes: if the weight of the main feature term is greater than or equal to a preset threshold, or if the difference between the weight of the main feature term and the weight of the auxiliary feature term is greater than or equal to a preset difference threshold, then a fusion representation of the target document is generated based on the main feature term.

[0164] In this embodiment, if the weight of the main feature term is greater than or equal to a preset threshold, it indicates that the information of the main feature term is relatively clear and explicit, and the fusion representation of the target document can be generated based solely on the main feature term. If the difference between the weight of the main feature term and the weight of the auxiliary feature term is greater than or equal to a preset difference threshold, it indicates that the information of the auxiliary feature term is relatively weak, with a high risk of extraction errors and limited effective information. Continuing to fuse the auxiliary feature term would reduce the effective information in the fusion representation, and the fusion representation of the target document can also be generated based on the main feature term.

[0165] By generating a fusion representation of the target document based solely on the main feature terms when the main feature terms are relatively clear and accurate, or when the effective information of the auxiliary feature terms is limited, invalid fusion is avoided, that is, invalid information is avoided from interfering with the fusion representation, thus improving the reliability and robustness of the fusion representation.

[0166] By defining primary and secondary feature terms, the core position of features with low uncertainty in the fusion representation is ensured. This prioritization makes the fusion more aligned with real-world scenarios, rather than simply a superposition of data dimensions, resulting in stronger interpretability. Furthermore, supplementing the primary feature terms with secondary feature terms enhances their information completeness and accuracy. Additionally, by evaluating the clarity of image features and their similarity to preset standard image features, the image uncertainty index is quantified from both visual and image completeness perspectives, making the evaluation of uncertainty indicators more comprehensive and objective. Simultaneously, the text uncertainty index is quantified from the aspects of text completeness and semantic consistency, making the text uncertainty index more comprehensive and objective. Moreover, the initial weights are dynamically adjusted based on the comparison results of uncertainty indicators, ensuring that the resulting weights better reflect the differences between image and text features, avoiding the problem of deviating from actual differences due to separately determining weights, and improving the adaptability of the weights.

[0167] In one possible implementation, the multimodal information extraction model includes an image uncertainty index generation layer and a text uncertainty index generation layer; quantifying the image uncertainty index of image features and the text uncertainty index of text features includes: inputting image features into the image uncertainty index generation layer to obtain the image uncertainty index of image features; inputting text features and the image uncertainty index into the text uncertainty index generation layer to obtain the text uncertainty index of text features.

[0168] The multimodal information extraction model comprises multiple layers, each responsible for implementing a corresponding step. In this embodiment, the multimodal information extraction model includes an image uncertainty index generation layer and a text uncertainty index generation layer.

[0169] The image uncertainty index generation layer is used to quantify the image uncertainty index of image features; the text uncertainty index generation layer is used to quantify the text uncertainty index of text features.

[0170] In this embodiment, after the multimodal information extraction model extracts features from the invoice image and invoice text information respectively, obtaining image features and text features, it inputs the image features into the image uncertainty index generation layer. The image uncertainty index generation layer quantizes the image features to obtain the image uncertainty index of the image features. The quantization method can be the quantization method provided in the above embodiment, which will not be elaborated here. After obtaining the image uncertainty index, the multimodal information extraction model inputs the image uncertainty index and text features into the text uncertainty index generation layer. The text uncertainty index generation layer quantizes the text features based on the image uncertainty index to obtain the text uncertainty index of the text features.

[0171] Based on the image uncertainty index, text features are quantized to obtain the text uncertainty index of the text features. The text features are quantized using the method described above to obtain an initial text uncertainty index. The initial text uncertainty index is then adjusted using the image uncertainty index to obtain the final text uncertainty index of the text features.

[0172] The text uncertainty index can be obtained by quantifying text features based on image uncertainty index. Alternatively, it can be obtained by adjusting the parameters of the text uncertainty index generation layer, such as weighting coefficients, based on the image uncertainty index; and then quantifying the text features based on the adjusted parameters.

[0173] In some scenarios, since the text information of a document is identified from the document image, the image uncertainty index can affect the text uncertainty index. This embodiment achieves coupling of uncertainty indices by using an image uncertainty index generation layer and a text uncertainty index generation layer to work together, quantifying the uncertainty indices of image features and text features, thus making the determined uncertainty indices more accurate.

[0174] Figure 4 This is a flowchart illustrating a method for verifying the authenticity of a document according to an embodiment of this application. The method provided in this embodiment can be executed by an electronic device with corresponding processing capabilities. Figure 4 As shown, the method provided in this embodiment includes the following steps:

[0175] Step S401: Obtain the bill fusion representation of the target bill.

[0176] The invoice fusion representation is obtained through the invoice information extraction method provided in the above embodiments.

[0177] Step S402: Extract key features of the invoice based on the invoice fusion representation.

[0178] Key features of a negotiable instrument are the information required in the instrument, which may include the drawer's name, drawer's account number, amount, validity period, and instrument number.

[0179] In some embodiments, key features may also include the payee's name, payee's account number, invoice date, and purpose of payment.

[0180] In this step, key features of the invoice can be extracted from the invoice fusion representation based on keyword matching.

[0181] Step S403: Calculate the similarity based on the key features of each invoice and the typical features of the corresponding class in the preset feature library.

[0182] The preset feature library is a pre-set knowledge base that stores key features of a large number of real invoices and their association rules. Association rules are the rules that link various elements in the integrated representation, such as the correspondence between the issuing party (e.g., xx) and the issuing party's account (e.g., xx).

[0183] In this step, cosine similarity or Euclidean distance is used to calculate the similarity between the key features of each document and the typical features of the corresponding class in the preset feature library.

[0184] In some embodiments, it can also be determined whether there are items among the key features of the bill that do not satisfy the association rule. For example, assuming the issuer in the key features of the bill is xx and the issuer's account number is yy, it can be determined whether there are items among the key features of the bill that do not satisfy the association rule.

[0185] Step S404: If the similarity meets the preset conditions, the target ticket is determined to be a genuine ticket.

[0186] In this embodiment, if the similarity meets the preset conditions, such as the similarity being greater than or equal to the similarity threshold, then the target ticket is determined to be a genuine ticket.

[0187] In some embodiments, a target bill may be determined to be a genuine bill when the similarity meets a preset condition and there are no items among the key features of the bill that do not meet the association rules.

[0188] In some embodiments, if the similarity does not meet the preset conditions, such as the similarity being less than the similarity threshold, or if there are items among the key features of the bill that do not meet the association rules, then the target bill is determined to be an abnormal bill.

[0189] After determining whether the target invoice is genuine or abnormal, the verification result can be displayed on the operation interface to indicate subsequent operations, and a verification report can be generated at the same time.

[0190] In some embodiments, after determining that the target bill is a genuine bill, the key features of the target bill may also be stored in a feature library for archiving.

[0191] In some embodiments, after determining that the target bill is genuine, the system of the bank and other financial institutions can be connected to initiate fund settlement and complete the fund operation corresponding to the target bill.

[0192] The document authenticity verification method provided in this embodiment extracts the key features of the fusion representation of the target document, and then calculates the similarity with the features in the preset feature library. When the similarity meets the preset conditions, the target document is determined to be a genuine document, thus realizing automatic authenticity verification of the target document with high verification efficiency.

[0193] Figure 5 This is a flowchart illustrating another method for verifying the authenticity of invoices provided in an embodiment of this application. Figure 5 As shown, this embodiment is... Figure 4 Based on the illustrated embodiment, the method for extracting invoice information provided in the above embodiment is added. For example... Figure 5 As shown, the document authenticity verification method provided in this embodiment includes the following steps:

[0194] Step S501: Obtain the image of the target ticket.

[0195] Step S502: Recognize the text in the ticket image to obtain the ticket text information.

[0196] Step S503: Preprocess the ticket image and process the ticket text information.

[0197] In this step, Gaussian filtering and adaptive histogram equalization are used to denoise and enhance the contrast of the ticket image, thereby improving its quality. Additionally, structured field validation is performed on the ticket text information to identify abnormal fields; these abnormal fields are then corrected and / or semantically completed.

[0198] Step S504: Extract features from the ticket image and ticket text information respectively to obtain image features and text features.

[0199] Steps S504 to S507 are executed by the multimodal information extraction model.

[0200] Step S505: Quantify the image uncertainty index of image features and the text uncertainty index of text features.

[0201] Step S506: Determine the weights of image features and text features based on image uncertainty index and text uncertainty index.

[0202] Step S507: Based on the weights of image features and text features, the image features and text features are fused to generate a fused representation of the target document.

[0203] Step S508: Extract key features of the invoice based on the invoice fusion representation.

[0204] Step S509: Calculate the similarity based on the key features of each invoice and the typical features of the corresponding class in the preset feature library.

[0205] Step S510: Determine whether the similarity meets the preset conditions. If it does, proceed to step S511; otherwise, proceed to step S512.

[0206] Step S511: Determine that the target bill is a genuine bill.

[0207] Step S512: Determine that the target ticket is an abnormal ticket.

[0208] The document authenticity verification method provided in this embodiment can refer to the document authenticity verification method and document information extraction method provided in the above embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0209] Figure 6 This is a schematic diagram of a bill information extraction device provided in an embodiment of this application. Figure 6 As shown, the invoice information extraction device provided in this embodiment includes an acquisition module 601, a text recognition module 602, and a fusion representation generation module 603.

[0210] The acquisition module 601 is used to acquire the image of the target ticket; the text recognition module 602 is used to recognize the text in the ticket image to obtain the ticket text information; the fusion representation generation module 603 is used to input the ticket image and ticket text information into a pre-trained multimodal information extraction model so that the multimodal information extraction model can perform subsequent steps to obtain the fusion representation of the target ticket.

[0211] The fusion representation generation module 603 specifically includes a feature extraction unit, an uncertainty index determination unit, a weight determination unit, and a fusion unit, wherein:

[0212] The feature extraction unit is used to extract features from the invoice image and invoice text information respectively, to obtain image features and text features; the uncertainty index determination unit is used to quantify the image uncertainty index of the image features and the text uncertainty index of the text features; the weight determination unit is used to determine the weight of the image features and the weight of the text features based on the image uncertainty index and the text uncertainty index; the fusion unit is used to fuse the image features and the text features based on the weight of the image features and the weight of the text features to generate a fused representation of the target invoice.

[0213] Optionally, the image features and text features include multiple components, and there is a correlation between the image features and text features; this fusion unit is specifically used for:

[0214] Among the image and text features that are related, the items with higher weights are identified as main features, and the items with lower weights among the image and text features that are related are identified as auxiliary features. If the weight of the main feature is less than a preset weight threshold, and the difference between the weight of the main feature and the weight of the auxiliary feature is less than a preset difference threshold, then the main feature is supplemented according to the auxiliary features to generate a fusion representation of the target document.

[0215] Optionally, the fusion unit is also used for:

[0216] If the weight of the main feature term is greater than or equal to a preset threshold, or if the difference between the weight of the main feature term and the weight of the auxiliary feature term is greater than or equal to a preset difference threshold, then a fusion representation of the target document is generated based on the main feature term.

[0217] Optionally, the uncertainty index determination unit further includes an image uncertainty index determination subunit, which is used for:

[0218] Based on the attribute parameters of the image features, calculate the sharpness of the image features; calculate the similarity between the image features and the preset standard image features; and determine the image uncertainty index of the image features based on the sharpness and similarity.

[0219] Optionally, the uncertainty index determination unit further includes a text uncertainty index determination subunit, which is used for:

[0220] The first uncertainty index of text features is quantified through text field integrity analysis; the second uncertainty index of text features is quantified through semantic consistency verification; and the first and second uncertainty indices are weighted to obtain the text uncertainty index of the text features.

[0221] Optionally, the multimodal information extraction model includes an image uncertainty index generation layer and a text uncertainty index generation layer; the uncertainty index determination unit is specifically used for:

[0222] Image features are input into the image uncertainty index generation layer to obtain the image uncertainty index of the image features; text features and the image uncertainty index are input into the text uncertainty index generation layer to obtain the text uncertainty index of the text features.

[0223] Optional, weight determination unit, specifically used for:

[0224] Obtain the initial weights of image features and text features; adjust the initial weights of image features and text features based on the comparison results of image uncertainty index and text uncertainty index; determine the adjusted initial weights of image features and text features as the weights of image features and text features, respectively.

[0225] Optionally, the ticket information extraction device further includes a text processing module, which is used for:

[0226] After recognizing the text in the invoice image and obtaining the invoice text information, the invoice text information is subjected to structured field validation to identify abnormal fields in the invoice text information; abnormal fields include fields with typos and fields with missing content; error correction and / or semantic completion are performed on abnormal fields, and after processing, the step of inputting the invoice image and invoice text information into the multimodal information extraction model is executed.

[0227] The bill information extraction device provided in this application can be used to execute the bill information extraction method provided in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0228] Figure 7 This is a schematic diagram of a document authenticity verification device provided in an embodiment of this application. Figure 7 As shown, the document authenticity verification device provided in this embodiment includes: a fusion characterization acquisition module 701, a key feature extraction module 702, a similarity calculation module 703, and an authenticity verification module 704.

[0229] The fusion representation acquisition module 701 is used to acquire the fusion representation of the target bill; the fusion representation is obtained by the bill information extraction method provided in the above embodiment; the key feature extraction module 702 is used to extract the key features of the bill based on the fusion representation; the similarity calculation module 703 is used to calculate the similarity between the key features of each bill and the typical features of the corresponding class in the preset feature library; the authenticity verification module 704 is used to determine that the target bill is a genuine bill if the similarity meets the preset conditions.

[0230] The document authenticity verification device provided in this application embodiment can be used to execute the document authenticity verification method provided in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0231] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device of this embodiment may include: at least one processor 801; and a memory 802 communicatively connected to the at least one processor; wherein the memory 802 stores instructions that can be executed by the at least one processor 801, and the instructions are executed by the at least one processor 801 to cause the electronic device to perform the method as described in any of the above embodiments.

[0232] Optionally, the memory 802 can be either standalone or integrated with the processor 801. When the memory 802 is configured independently, the device also includes a bus for connecting the memory 802 and the processor 801.

[0233] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0234] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the methods provided in any of the foregoing embodiments can be implemented.

[0235] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0236] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0237] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0238] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0239] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0240] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0241] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0242] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0243] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0244] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for extracting invoice information, characterized in that, include: Obtain the image of the target invoice; The text in the ticket image is identified to obtain the ticket text information; The invoice image and the invoice text information are input into a pre-trained multimodal information extraction model, which then performs the following steps to obtain a fused representation of the target invoice: Features are extracted from the ticket image and the ticket text information respectively to obtain image features and text features; The image uncertainty index quantifies the image features, and the text uncertainty index quantifies the text features; Based on the image uncertainty index and the text uncertainty index, the weights of the image features and the text features are determined. Based on the weights of the image features and the weights of the text features, the image features and the text features are fused to generate a fused representation of the target ticket.

2. The method according to claim 1, characterized in that, The image features and text features include multiple components, and the image features and text features are correlated; the process of fusing the image features and text features based on the weights of the image features and the weights of the text features to generate a fused representation of the target document includes: Among the image and text features that are related, the items with higher weights are identified as primary features, and the items with lower weights among the image and text features that are related are identified as secondary features. If the weight of the main feature item is less than a preset weight threshold, and the difference between the weight of the main feature item and the weight of the auxiliary feature item is less than a preset difference threshold, then the main feature item is supplemented according to the auxiliary feature item to generate the fusion representation of the target document.

3. The method according to claim 2, characterized in that, The method further includes: If the weight of the main feature item is greater than or equal to a preset threshold, or if the difference between the weight of the main feature item and the weight of the auxiliary feature item is greater than or equal to a preset difference threshold, then a fusion representation of the target document is generated based on the main feature item.

4. The method according to claim 1, characterized in that, The image uncertainty index for quantifying the image features includes: The sharpness of the image features is calculated based on their attribute parameters. Calculate the similarity between the image features and preset standard image features; Based on the sharpness and the similarity, an image uncertainty index is determined for the image features.

5. The method according to claim 1, characterized in that, The text uncertainty index for quantifying the text features includes: The first uncertainty index of the text features is quantified through text field integrity analysis. A second uncertainty index for the text features is quantified through semantic consistency verification; The first uncertainty index and the second uncertainty index are weighted to obtain the text uncertainty index of the text feature.

6. The method according to claim 1, characterized in that, The multimodal information extraction model includes an image uncertainty index generation layer and a text uncertainty index generation layer; the image uncertainty index that quantifies the image features and the text uncertainty index that quantifies the text features include: The image features are input into the image uncertainty index generation layer to obtain the image uncertainty index of the image features; The text features and the image uncertainty index are input into the text uncertainty index generation layer to obtain the text uncertainty index of the text features.

7. The method according to any one of claims 1-6, characterized in that, The step of determining the weights of the image features and the text features based on the image uncertainty index and the text uncertainty index includes: Obtain the initial weights of the image features and the initial weights of the text features; Based on the comparison results of the image uncertainty index and the text uncertainty index, adjust the initial weights of the image features and the initial weights of the text features; The adjusted initial weights of the image features and the adjusted initial weights of the text features are determined as the weights of the image features and the weights of the text features, respectively.

8. The method according to any one of claims 1-6, characterized in that, After identifying the text in the ticket image and obtaining the ticket text information, the method further includes: The document text information is subjected to structured field validation to identify abnormal fields in the document text information; the abnormal fields include fields containing typos and fields with missing content; The abnormal fields are corrected and / or semantically completed. After processing, the steps of inputting the ticket image and the ticket text information into the multimodal information extraction model are performed.

9. A method for verifying the authenticity of a document, characterized in that, include: Obtain the fusion representation of the target document; The fused representation of the bill is obtained by the method provided in any one of claims 1-8; Based on the aforementioned invoice fusion representation, extract key features of the invoice; Calculate the similarity between the key features of each invoice and the typical features of the corresponding class in the preset feature library; If the similarity meets the preset conditions, the target ticket is determined to be a genuine ticket.

10. A device for extracting ticket information, characterized in that, include: The acquisition module is used to acquire the image of the target ticket; The text recognition module is used to recognize the text in the ticket image and obtain the ticket text information; The fusion representation generation module is used to input the bill image and the bill text information into a pre-trained multimodal information extraction model, so that the multimodal information extraction model performs the following steps to obtain a fusion representation of the target bill: extracting features from the bill image and the bill text information respectively to obtain image features and text features; quantifying the image uncertainty index of the image features and the text uncertainty index of the text features; and determining the weights of the image features and the text features based on the image uncertainty index and the text uncertainty index. Based on the weights of the image features and the weights of the text features, the image features and the text features are fused to generate a fused representation of the target ticket.

11. A device for verifying the authenticity of a receipt, characterized in that, include: The fusion representation acquisition module is used to acquire the fusion representation of the target document. The fused representation of the bill is obtained by the method provided in any one of claims 1-8; The key feature extraction module is used to extract key features of the invoice based on the invoice fusion representation. The similarity calculation module is used to calculate the similarity between the key features of each document and the typical features of the corresponding class in the preset feature library; The authenticity verification module is used to determine that the target ticket is a genuine ticket if the similarity meets the preset conditions.

12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

Images