Cross-border single image auditing method and device, electronic equipment and storage medium

Abstract

The application relates to the field of artificial intelligence and financial technology, and provides a cross-border document image auditing method, device and equipment and a storage medium. The method comprises the following steps: acquiring a plurality of cross-border document images of a target cross-border business; classifying the cross-border document images through a preset integrated model to obtain a document image category of each cross-border document image; performing character recognition on the cross-border document images through a preset character recognition model to obtain document character content contained in each cross-border document image; performing information extraction on the document character content contained in each cross-border document image based on information extraction rules of each document image category to obtain document key information of each cross-border document image; and performing auditing on each document key information based on a preset general large language model to obtain an auditing result of the target cross-border business, wherein the auditing result comprises passing auditing and failing auditing. The disclosed embodiment can improve the auditing efficiency and accuracy of cross-border document images.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and fintech, and in particular to a method, apparatus, electronic device, and storage medium for cross-border document image verification. Background Technology

[0002] With the rise of artificial intelligence technology, image recognition technology based on AI models has been widely used in the review of various financial transactions. Currently, cross-border transactions involve many cross-border documents, such as import letters of credit and export letters of credit. However, due to the complexity of cross-border transaction rules and the wide variety of cross-border documents, existing image recognition models cannot complete the review tasks, forcing cross-border transaction systems to rely on manual review.

[0003] Cross-border documents involve complex laws and regulations governing cross-border trade, with numerous rules and intricate terms. Furthermore, there are many types of cross-border documents, including more than 40 major categories such as bills of lading, insurance policies, and packing lists, and there is no uniform format for documents within each category.

[0004] The manual review process requires comparing and verifying documents at each stage to ensure the consistency and rationality of all elements, resulting in a long manual review cycle and low accuracy. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, and storage medium for cross-border document image verification, which can improve the efficiency and accuracy of cross-border document image verification.

[0006] The first aspect of this disclosure provides a method for cross-border document image verification, including:

[0007] Acquire multiple cross-border document images for the target cross-border business;

[0008] The cross-border document images are classified using a preset integrated model to obtain the document image category for each cross-border document image.

[0009] The text content of each cross-border document image is obtained by performing text recognition on the cross-border document image using a preset text recognition model.

[0010] Based on the information extraction rules for each document image category, information is extracted from the document text content contained in each cross-border document image to obtain the key document information for each cross-border document image.

[0011] The key information of each document is reviewed based on a preset general language model to obtain the review result of the target cross-border business. The review result includes approval and failure.

[0012] A second aspect of this disclosure provides a cross-border document image verification device, the cross-border document image verification device comprising:

[0013] The first acquisition module is used to acquire multiple cross-border document images of the target cross-border business;

[0014] The image classification module is used to classify the cross-border document images using a preset integrated model to obtain the document image category of each cross-border document image.

[0015] The text recognition module is used to perform text recognition on the cross-border document images using a preset text recognition model to obtain the document text content contained in each cross-border document image.

[0016] The information extraction module is used to extract information from the document text content contained in each cross-border document image based on the information extraction rules of each document image category, so as to obtain the key document information of each cross-border document image.

[0017] The review module is used to review the key information of each document based on a preset general language model to obtain the review result of the target cross-border business. The review result includes approval and failure.

[0018] A third aspect of this disclosure provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the cross-border document image verification method of the first aspect described above.

[0019] A fourth aspect of this disclosure provides a computer-readable storage medium storing instructions and a computer program, which, when executed by a processor, implements the cross-border document image verification method of the first aspect described above.

[0020] The technical solution provided in this disclosure involves acquiring multiple cross-border document images of a target cross-border business; classifying the cross-border document images using a preset integrated model to obtain the document image category for each cross-border document image; performing text recognition on the cross-border document images using a preset text recognition model to obtain the document text content contained in each cross-border document image; extracting information from the document text content contained in each cross-border document image based on information extraction rules for each document image category to obtain the key document information for each cross-border document image; and reviewing each key document information based on a preset general language model to obtain the review result of the target cross-border business, including whether the review passed or failed. This disclosure, through a preset integrated model, text recognition model, and general language model, achieves automatic review of multiple cross-border document images of a target cross-border business, improving the review efficiency and accuracy of cross-border document images. Attached Figure Description

[0021] Figure 1 This is a framework diagram of an application system according to an embodiment of the present disclosure;

[0022] Figure 2 This is a flowchart illustrating a cross-border document image verification method in an embodiment of this disclosure;

[0023] Figure 3 As described in this embodiment of the disclosure Figure 2 A detailed flowchart of step 220;

[0024] Figure 4 As described in this embodiment of the disclosure Figure 3 A detailed flowchart of step 310;

[0025] Figure 5 As described in this embodiment of the disclosure Figure 3 A detailed flowchart of step 330;

[0026] Figure 6 As described in this embodiment of the disclosure Figure 3 A detailed flowchart of step 340;

[0027] Figure 7 As described in this embodiment of the disclosure Figure 2 A detailed flowchart preceding step 230;

[0028] Figure 8 As described in this embodiment of the disclosure Figure 2 A detailed flowchart of step 250;

[0029] Figure 9 This is a schematic diagram of a cross-border document image verification device in an embodiment of this disclosure;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device in an embodiment of this disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0032] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0033] Cross-border documents: Cross-border documents refer to a series of supporting documents used in cross-border trade to prove information such as the origin, quality, price, quantity, transportation details, and compliance of goods. These documents play a crucial role in cross-border transactions. They are not only essential for customs clearance, tax processing, and payment settlement, but also important tools for protecting the rights and interests of both buyers and sellers and ensuring the legality and smooth progress of transactions.

[0034] Optical Character Recognition (OCR) is a technology that converts printed or handwritten characters into editable text. OCR works by scanning or capturing images and using image processing and pattern recognition algorithms to convert text in the image into computer-readable text information.

[0035] Currently, for international cross-border transactions, financial institutions are required to verify the authenticity of the transactions before payment is made. The primary basis for this verification is the cross-border documents provided by both parties. In practice, each party provides the financial institution with multiple images of the target cross-border documents, which are then reviewed by the institution's staff. This process is inefficient and prone to errors. To address these issues, this disclosure provides a method for verifying cross-border document images. This method automatically verifies cross-border document images using a pre-set model, improving both the efficiency and accuracy of the verification process.

[0036] Figure 1 This is a system architecture diagram of the cross-border document image verification method according to embodiments of this disclosure. It includes a terminal 110, an internet connection 120, a gateway 130, and a server 140, etc.

[0037] Terminal 110 is used to capture images of cross-border documents and send these images to server 140, where server 140 automatically reviews the images. For example, a user can use terminal 110 to capture and upload multiple images of cross-border documents for a target cross-border business. Terminal 110 can be a single device or a combination of multiple devices. Terminal 110 can communicate with the Internet 120 via wired or wireless means to exchange data.

[0038] Server 140 refers to a computer system that can provide certain services to terminal 110. Compared to ordinary terminal 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0039] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages (cross-border document images) sent from terminal 110 to server 140 must be forwarded to the corresponding server 140 via gateway 130. Messages (approved or unapproved) sent from server 140 to terminal 110 must also be forwarded to the corresponding terminal 110 via gateway 130.

[0040] General Description of Embodiments in this Disclosure

[0041] According to one embodiment of this disclosure, a method for cross-border document image verification is provided.

[0042] The cross-border document image verification method of this disclosure can be executed on terminal 110, on server 140, or partly on terminal 110 and partly on server 140.

[0043] Please see Figure 2 A flowchart of a cross-border document image verification method provided in this disclosure embodiment specifically includes:

[0044] 210. Obtain multiple cross-border document images for the target cross-border business;

[0045] 220. Classify cross-border document images using a pre-defined ensemble model to obtain the document image category for each cross-border document image;

[0046] 230. Perform text recognition on cross-border document images using a preset text recognition model to obtain the document text content contained in each cross-border document image;

[0047] 240. Based on the information extraction rules for each document image category, extract information from the document text content contained in each cross-border document image to obtain the key document information for each cross-border document image.

[0048] 250. Based on a pre-defined general language model, review the key information of each document to obtain the review results of the target cross-border business. The review results include those that pass the review and those that fail the review.

[0049] In step 210, multiple cross-border document images may exist for the same target cross-border transaction. For example, if the cross-border document images are uploaded by the seller, they may include contract images, invoice images, packing list images, letter of credit images, etc. Among them, the contract image may contain multiple consecutive pages of images, with each page being a cross-border document image.

[0050] In step 220, since the multiple cross-border document images obtained in step 210 were uploaded by the user all at once without being categorized or sorted, it is necessary to classify these uploaded cross-border document images according to preset categories to facilitate subsequent review. Because there are related images across multiple pages in the cross-border document images, some images have high similarity and are difficult to distinguish. This embodiment integrates the various sub-models in the integrated model to classify all cross-border document images separately, and then weights the results of each classification to obtain the document image category for each cross-border document image. Because the classification results of multiple sub-models are combined, the accuracy of the obtained document image categories is higher.

[0051] In step 230, the document text content contained in each cross-border document image is extracted. The preset text recognition model can adopt a fine-tuned convolutional recurrent neural network (CRNN) model to make the text recognition model more adaptable to large text content in cross-border document images and improve the recognition accuracy of document text content.

[0052] It is understandable that other models with text recognition capabilities can also be used for recognition; no limitation is made here.

[0053] In step 240, each document image category corresponds to an information extraction rule. For example, for a letter of credit image, key information such as the document name, issuing bank name, letter of credit number, expiry date, consignee's name and address, goods name, quantity, unit price, total amount, mode of transport, and payment terms need to be extracted. For an export invoice image, key information such as the product name, quantity, unit price, total amount, mode of transport, shipping address, and order number need to be extracted.

[0054] In step 250, document review rules need to be developed based on the Uniform Customs and Practice for Documentary Credits (UCP) 600, the International Standard Banking Practice for the Examination of Documents under Documentary Credits (ISBP) 745, and established document review experience. These rules are then pre-set in the general language model, enabling it to review documents and complete automated review tasks. UCP 600, as an international practice in the field of letters of credit, provides comprehensive, specific, and operational rules and guidance for letter of credit operations, playing a crucial role in facilitating the smooth operation of cross-border trade. ISBP 745 represents the latest international practice for letter of credit operations in cross-border trade. It comprehensively, detailedly, and specifically stipulates the requirements and interpretations of letter of credit documents, providing strong guarantees for the smooth operation of cross-border trade. As an important supplement to UCP 600, ISBP 745 plays a vital role in cross-border trade. It not only provides a unified international standard for the preparation and review of letter of credit documents, but also reduces the risk of non-payment due to discrepancies in documents, thus facilitating the smooth operation of cross-border trade.

[0055] In the above embodiments, cross-border document images are classified using an integrated model to obtain the document image category for each image, thus improving the accuracy of document image category classification. Then, text recognition is performed on the cross-border document images to obtain the document text content contained in each image. Information is extracted from the document text content according to the information extraction rules for document image categories to obtain the key document information for each image, thus improving the accuracy of the key document information. Finally, each key document information is reviewed based on a preset general language model to obtain the review result for the target cross-border business. The review result includes pass and fail, improving the efficiency and accuracy of cross-border document image review.

[0056] The steps in steps 210-250 are described in detail below.

[0057] In one embodiment, such as Figure 3 As shown, step 220 may include:

[0058] 310. The business features of cross-border document images are identified by the rule recognition sub-model in the preset integrated model, and the cross-border document images are classified based on the identified target business features to obtain the first image category of each cross-border document image.

[0059] 320. The image recognition sub-model in the preset integrated model is used to identify the representational features of cross-border document images. Based on the identified target representational features, the cross-border document images are classified to obtain the second image category of each cross-border document image.

[0060] 330. Perform semantic recognition on cross-border document images through the text recognition sub-model in the preset integrated model, and classify the cross-border document images based on the recognized target text semantics to obtain the third image category of each cross-border document image.

[0061] 340. Simultaneously perform text recognition, visual recognition, and spatial feature recognition on cross-border document images using the multimodal recognition sub-model in the preset integrated model, and classify the cross-border document images based on the recognized text features, visual features, and spatial features to obtain the fourth image category for each cross-border document image.

[0062] 350. Based on the first image category, the second image category, the third image category, and the fourth image category, a weighted calculation is performed to determine the document image category of each cross-border document image.

[0063] In step 310, the rule-based recognition sub-model classifies each cross-border document image to obtain a first image category for each image. The rule-based recognition sub-model pre-sets business expert experience and feature statistics for each document image category. Business expert experience refers to the main business characteristics summarized by document review experts for each document image category. For example, the main characteristic of an export invoice image is the invoice number, and the main characteristic of a packing list image is the packing method. These main business characteristics are the review experience summarized by business experts and can be adjusted according to actual circumstances.

[0064] In step 320, each cross-border document image is classified using an image recognition sub-model to obtain a second image category for each cross-border document image. The image recognition sub-model is trained using the ConvNeXt model, which can identify the image's representational features for classification. These representational features can be image pixel features, and the format or table features appearing in each cross-border document image are determined based on these pixel features.

[0065] In step 330, each cross-border document image is classified using a text recognition sub-model to obtain a third image category for each cross-border document image. The text recognition sub-model is trained using the BERT text classification model. For each cross-border document image, it is necessary to determine the semantic meaning of each text segment within the image. For example, for a cross-border document image, if it is determined that the semantic meaning of the text in each text box indicates content related to invoices, then the cross-border document image can be classified as an export invoice image; if it is determined that the semantic meaning of the text in each text box indicates content related to packing, then the cross-border document image can be classified as a packing list image.

[0066] In step 340, each cross-border document image is classified using a multimodal recognition sub-model to obtain a fourth image category for each cross-border document image. Multimodal recognition includes text recognition, visual recognition, and spatial feature recognition, which can be performed simultaneously. Simultaneous text recognition, visual recognition, and spatial feature recognition avoids missing semantic parts of the text. In this embodiment, the multimodal recognition sub-model includes a first input layer, a second input layer, a third input layer, and multiple transformation layers. The first input layer is connected to each transformation layer, the second input layer is connected to each transformation layer, and the third input layer is connected to the first transformation layer. The first input layer is used to input two-dimensional spatial coordinates, absolute spatial coordinates, and one-dimensional relationships; the second input layer is used to input visual features; and the third input layer is used to input text features. Because the dependencies between spatial and visual features may vary from layer to layer, in one embodiment, the step of classifying the cross-border document image based on the identified text features, visual features, and spatial features includes:

[0067] The residual connections of spatial and visual features are passed to each transformation layer; then the outputs of each transformation layer are fused to obtain the fourth image category for each cross-border document image. Within each transformation layer, visual and textual features undergo self-attention processing with shared spatial features.

[0068] In step 350, different weights are assigned to the first image category, the second image category, the third image category, and the fourth image category. The weighted image category is then used as the document image category for each cross-border document image. For example, because the multimodal recognition sub-model has higher accuracy, the weight of the fourth image category can be set higher.

[0069] The advantage of steps 310-350 is that by having each sub-model in the integrated model sequentially identify all cross-border document images, and then weighting the identification results of each sub-model to obtain the document image category for each cross-border document image, the accuracy of the document image category is improved, while the error tolerance rate in the process of classifying cross-border document images is increased.

[0070] The above embodiments describe the classification process of the ensemble model in detail. The classification process of each sub-model will be described below.

[0071] In one embodiment, such as Figure 4 As shown, step 310 may include:

[0072] 410. Perform OCR recognition on cross-border document images to obtain multiple strings contained in each cross-border document image;

[0073] 420. Based on the rule recognition sub-model in the preset integrated model and the preset regular expression, the strings are matched to obtain the target business features of each cross-border document image. The regular expression is used to recognize the preset business features.

[0074] 430. Based on the target business characteristics, determine the document category to which each cross-border document image belongs, and obtain the first image category.

[0075] In step 410, OCR recognition is first performed on the cross-border document images to obtain the text information in each cross-border document image, which is then extracted in string form.

[0076] In step 420, multiple regular expressions are preset, each of which can identify a specific business feature, i.e., a specific string. The extracted strings are matched against the multiple regular expressions in sequence to determine the business feature corresponding to each string. Based on the multiple business features, one business feature is selected as the target business feature for the cross-border document image.

[0077] In step 430, a document category can be pre-defined for each target business characteristic. For example, cross-border document images with the target business characteristic of invoice number can be classified as export invoices, and cross-border document images with the target business characteristic of packing method can be classified as packing lists.

[0078] The advantage of steps 410-430 is that cross-border document images are automatically classified according to the experience of business experts and the statistical results of business characteristics. They can be classified according to pre-set document categories, avoiding unnecessary document categories and improving the accuracy of the first image category.

[0079] In one embodiment, such as Figure 5 As shown, step 330 may include:

[0080] 510. Perform OCR recognition on cross-border document images to obtain the target text content contained in each cross-border document image;

[0081] 520. Semantic recognition of the target text content is performed using the text recognition sub-model in the preset integrated model to obtain the target text semantics of each cross-border document image;

[0082] 530. Determine the document category to which each cross-border document image belongs based on the semantics of the target text to obtain the third image category.

[0083] In step 510, OCR recognition is first performed on the cross-border document images to obtain the target text content contained in each cross-border document image.

[0084] In step 520, semantic recognition of the target text content is performed by the text recognition sub-model. Unlike the rule recognition sub-model, the text recognition sub-model is trained using machine learning methods and does not consider manually set rules. Therefore, the output of the text recognition sub-model is not limited to the manually set document image categories, but forms different document image categories based on different semantics.

[0085] In step 530, a document category can be pre-defined for each type of target text semantic. For example, cross-border document images describing invoice information can be classified as export invoices, and cross-border document images describing packing methods can be classified as packing lists.

[0086] The advantage of steps 510-530 is that by classifying cross-border document images through different target text semantics, the number of document categories is increased, and the diversity of the third image category is improved.

[0087] In one embodiment, such as Figure 6 As shown, step 340 may include:

[0088] 610. Perform OCR recognition on cross-border document images to obtain the target text content contained in each cross-border document image;

[0089] 620. The target text content is identified by the multimodal sub-model in the preset integrated model to obtain the text features, visual features and spatial features of each cross-border document image;

[0090] 630. Based on textual features, visual features, and spatial features, determine the document category to which each cross-border document image belongs, thus obtaining the fourth image category.

[0091] In step 610, OCR recognition is first performed on the cross-border document images to obtain the target text content contained in each cross-border document image.

[0092] In step 620, the Docformer multimodal model can be used to extract text features, recognition features, and spatial features simultaneously.

[0093] In step 630, the Docformer multimodal model can directly fuse text features, recognition features, and spatial features through a fusion layer to output a fourth image category for each cross-border document image.

[0094] The advantage of steps 610-630 is that by using a multimodal sub-model for recognition, the correlation between text features, recognition features, and spatial features is improved, while the number of training parameters is reduced, thus improving the accuracy of the fourth image category.

[0095] In one embodiment, such as Figure 7 As shown, prior to step 230, the cross-border document image verification method may also include:

[0096] 710. Obtain the training dataset, which includes multiple cross-border document images;

[0097] 720. Perform format preprocessing on the training dataset to obtain the preprocessed training dataset;

[0098] 730. Freeze all convolutional layer parameters in the preset convolutional recurrent neural network model, and adjust the long short-term memory network and fully connected layer to obtain the adjusted convolutional recurrent neural network model.

[0099] 740. The adjusted convolutional recurrent neural network model is trained based on the preprocessed training dataset to obtain a trained text recognition model.

[0100] In step 710, the training dataset is obtained, and the cross-border document images in the training dataset are the original images.

[0101] In step 720, the cross-border document images in the training dataset are preprocessed: each cross-border document image is scaled to obtain a fixed-size cross-border document image; Gaussian noise is added to each fixed-size cross-border document image, and then a radiometric transformation is performed to obtain the processed cross-border document image. For example, the height of the cross-border document image is set to 32 and the width is set to 100, resulting in an image with a size of 32*100; then Gaussian noise is added to the image with the uniform size, where the mean of the Gaussian noise is 0 and the variance is 1; finally, the cross-border document image with added Gaussian noise is subjected to a radiometric transformation with random rotations between (-10, 10).

[0102] In step 730, an initial convolutional recurrent neural network model is obtained and fine-tuned: first, all convolutional layer parameters are frozen, and then the Long Short-Term Memory (LSTM) network and fully connected layers are fine-tuned; finally, the initial learning rate is set to 1e-4, and the parameters of the Adam optimizer are set to beta1 = 0.9, beta2 = 0.999, and epsilon = 1e-8; the loss function is set to the CTC loss function.

[0103] In step 740, before training, training parameters need to be set. For example, the batch size is set to 64, the maximum number of training characters per epoch for all samples is set to 100, and the learning rate strategy is set to stepLR. Then, the adjusted convolutional recurrent neural network model is trained based on the preprocessed training dataset and training parameters to obtain a trained text recognition model.

[0104] The advantage of steps 710-740 is that the training dataset is preprocessed first, and then the initial convolutional recurrent neural network model is fine-tuned and the corresponding training parameters are set, which improves the recognition accuracy of the trained text recognition model.

[0105] In one embodiment, such as Figure 8 As shown, step 250 may further include:

[0106] 810. Obtain key information for each document in the target cross-border business;

[0107] 820. Store key information of documents in the short-term memory of the memory units in the preset general language model;

[0108] 830. Based on key information of the documents, the intelligent agent is determined to be a document review expert. The document review expert is used to call the key information of the documents and the preset document review database. The document review database is stored in the long-term memory of the memory unit.

[0109] 840. By using the review expert role to call the planning unit to perform task planning, an review plan is obtained, which includes multiple review tasks to be executed sequentially.

[0110] 850. By using the review expert role to invoke the execution unit to execute the review plan, the review result of the target cross-border business is determined as either "passed" or "failed".

[0111] It should be noted that the memory unit includes long-term memory and short-term memory. Short-term memory is used to store the context of recently acquired information, while long-term memory stores external knowledge. External knowledge can be a knowledge database related to various roles; for example, a knowledge database could be a review database, which includes review policies, case studies, and common pitfalls. Review expert roles possess expertise in review data, processes, methods, and policies, and can access the review database corresponding to their role. The general large language model pre-defines agents, memory units, planning units, and execution units.

[0112] It should be noted that if the review is successful, more detailed review results can be output, such as whether the documents are complete, whether the documents are consistent with each other, and whether the documents are consistent within each other.

[0113] The advantages of steps 810-850 are that, based on an agent using a large language model, document review rules are formulated according to UCP600, ISBP745 and document review experience, the review plan is executed, and the review results (pass or fail) are output. The use of an agent to replace senior domain experts in document review improves the efficiency of cross-border document image review and standardizes the cross-border document image review process, thereby improving the accuracy of cross-border document image review.

[0114] In one embodiment, prior to step 240, an extraction model can be trained to extract information from the text content of each cross-border document image according to the information extraction rules for each document image category. The training process of the extraction model is as follows:

[0115] Prepare the data: Construct a list of JSON objects representing instructions, queries, and answers;

[0116] The initial model was adjusted as follows: A large Chat Generative Language Mode (ChatGLM) model was selected, and a bypass was added next to the original large model to create a link between the dimensionality-reduced matrix A and the dimensionality-incremented matrix B; matrix A was initialized with a random Gaussian distribution, and matrix B was initialized with a zero matrix. This ensures that during training, the newly added bypass BA = 0, thus having no impact on the model results.

[0117] Train the adjusted model: change the forward process h=Wx of the base large model to h=Wx+BAx, the output layer w out =W+BA. During inference, the results of the left and right parts are added together, that is, h=Wx+BAx=(W+BA)x. Therefore, it is only necessary to add the trained matrix product BA to the original weight matrix W as new weight parameters, and replace the weight parameters of the ChatGLM base large model, without adding additional computing resources. Wherein, x is the input, h is the output; A is a dimension reduction matrix with a dimension of d*r, B is a dimension lifting matrix with a dimension of r*k, r<<min(d,k), and W is the weight parameter of the ChatGLM base large model.

[0118] Based on the trained large model, write a Prompt for each type of cross-border document image to extract the key document information of that type of cross-border document image.

[0119] The above embodiments provide a training method for an extraction model, which uses the extraction model for information extraction, and improves the efficiency of information extraction.

[0120] Description of apparatus and devices according to embodiments of this disclosure

[0121] It can be understood that although the steps in the above respective flowcharts are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the sequence indicated by the arrows. Unless there is a clear description in the present embodiment, there is no strict sequence restriction on the execution of these steps, and these steps can be executed in other sequences. Moreover, at least part of the steps in the above flowcharts may include multiple sub-steps or multiple stages, these steps or stages are not necessarily executed and completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but may be executed in turn or alternately with at least a part of other steps or sub-steps or stages in other steps.

[0122] It should be noted that in each specific embodiment of the present application, when processing related matters based on data related to user identity or characteristics such as device information, device behavior data, device historical data, and device location information, the user's permission or consent will be obtained in advance, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or redirecting to a confirmation page, and after the user's separate permission or separate consent is clearly obtained, the necessary user-related data for enabling the embodiments of the present disclosure to operate normally will be acquired.

[0123] Reference to Figure 9 , Figure 9 This is a schematic diagram of the structure of the cross-border document image verification device 900 provided in this embodiment of the disclosure. The cross-border document image verification device 900 includes:

[0124] The first acquisition module 910 is used to acquire multiple cross-border document images of the target cross-border business;

[0125] The image classification module 920 is used to classify cross-border document images using a preset integrated model to obtain the document image category of each cross-border document image.

[0126] The text recognition module 930 is used to perform text recognition on cross-border document images through a preset text recognition model to obtain the document text content contained in each cross-border document image.

[0127] The information extraction module 940 is used to extract information from the text content of each cross-border document image based on the information extraction rules of each document image category, so as to obtain the key document information of each cross-border document image.

[0128] The review module 950 is used to review the key information of each document based on a preset general language model, and obtain the review results of the target cross-border business. The review results include approval and failure.

[0129] In one embodiment, the image classification module 920 is specifically used for:

[0130] The business features of cross-border document images are identified by the rule recognition sub-model in the preset integrated model, and the cross-border document images are classified based on the identified target business features to obtain the first image category of each cross-border document image.

[0131] The image recognition sub-model in the preset integrated model is used to identify the representational features of cross-border document images. Based on the identified target representational features, the cross-border document images are classified to obtain the second image category of each cross-border document image.

[0132] The text recognition sub-model in the preset integrated model is used to perform semantic recognition on cross-border document images, and the cross-border document images are classified based on the recognized target text semantics to obtain the third image category of each cross-border document image.

[0133] The multimodal recognition sub-model in the preset integrated model is used to simultaneously perform text recognition, visual recognition and spatial feature recognition on cross-border document images. Based on the recognized text features, visual features and spatial features, the cross-border document images are classified to obtain a fourth image category for each cross-border document image.

[0134] The document image category for each cross-border document image is determined by weighted calculation based on the first image category, the second image category, the third image category, and the fourth image category.

[0135] In one embodiment, the image classification module 920 is specifically used for:

[0136] OCR recognition is performed on cross-border document images to obtain multiple strings contained in each cross-border document image;

[0137] The strings are matched based on the rule recognition sub-model in the preset integrated model and the preset regular expression to obtain the target business features of each cross-border document image. The regular expression is used to recognize the preset business features.

[0138] Based on the characteristics of the target business, the document category to which each cross-border document image belongs is determined, thus obtaining the first image category.

[0139] In one embodiment, the image classification module 920 is specifically used for:

[0140] OCR recognition is performed on cross-border document images to obtain the target text content contained in each cross-border document image;

[0141] The target text content is semantically recognized by the text recognition sub-model in the preset integrated model, and the target text semantics of each cross-border document image are obtained.

[0142] Based on the semantics of the target text, the document category to which each cross-border document image belongs is determined, resulting in a third image category.

[0143] In one embodiment, the image classification module 920 is specifically used for:

[0144] OCR recognition is performed on cross-border document images to obtain the target text content contained in each cross-border document image;

[0145] The target text content is identified by using the multimodal sub-model in the preset integrated model to obtain the text features, visual features and spatial features of each cross-border document image;

[0146] Based on textual features, visual features, and spatial features, the document category to which each cross-border document image belongs is determined, resulting in a fourth image category.

[0147] In one embodiment, the cross-border document image verification device further includes:

[0148] The second acquisition module is used to acquire the training dataset, which includes multiple cross-border document images.

[0149] The preprocessing module is used to preprocess the training dataset to obtain a preprocessed training dataset.

[0150] The adjustment module is used to freeze all convolutional layer parameters in the preset convolutional recurrent neural network model and adjust the long short-term memory network and fully connected layers to obtain the adjusted convolutional recurrent neural network model.

[0151] The training module is used to train the adjusted convolutional recurrent neural network model based on the preprocessed training dataset to obtain a trained text recognition model.

[0152] In one embodiment, the audit module 950 is specifically used for:

[0153] Obtain key information for each document in the target cross-border business;

[0154] Key information of the document is stored in the short-term memory of the memory unit in the pre-set general language model;

[0155] Based on the key information of the documents, the intelligent agent is identified as the document review expert role. The document review expert role is used to call the key information of the documents and the preset document review database. The document review database is stored in the long-term memory of the memory unit.

[0156] By using the review expert role to call the planning unit to plan tasks, a review plan is obtained, which includes multiple review tasks to be executed sequentially.

[0157] By using the role of an audit expert, the execution unit is invoked to execute the audit plan and determine whether the audit result for the target cross-border business is passed or failed.

[0158] Please see Figure 10 This disclosure also provides a hardware structure for an electronic device, the electronic device comprising:

[0159] The processor 1001 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0160] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the cross-border document image verification method of the embodiments of this disclosure.

[0161] Input / output interface 1003 is used to implement information input and output;

[0162] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0163] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0164] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0165] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described cross-border document image verification method.

[0166] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned cross-border document image verification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0168] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0169] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0170] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0171] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0173] Finally, it should be noted that the above embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for cross-border document image verification, characterized in that, The method includes: Acquire multiple cross-border document images for the target cross-border business; The cross-border document images are classified using a preset integrated model to obtain the document image category for each cross-border document image. The text content of each cross-border document image is obtained by performing text recognition on the cross-border document image using a preset text recognition model. Based on the information extraction rules for each document image category, information is extracted from the document text content contained in each cross-border document image to obtain the key document information for each cross-border document image. The key information of each document is reviewed based on a preset general language model to obtain the review result of the target cross-border business. The review result includes approval and failure. The method of classifying the cross-border document images using a preset ensemble model to obtain the document image category for each cross-border document image includes: The cross-border document images are used to identify business features through the rule recognition sub-model in the preset integrated model, and the cross-border document images are classified based on the identified target business features to obtain the first image category of each cross-border document image. The image recognition sub-model in the preset integrated model is used to identify the representational features of the cross-border document images. Based on the identified target representational features, the cross-border document images are classified to obtain a second image category for each cross-border document image. The text recognition sub-model in the preset integrated model is used to perform semantic recognition on the cross-border document images, and the cross-border document images are classified based on the recognized target text semantics to obtain a third image category for each cross-border document image. The cross-border document images are simultaneously subjected to text recognition, visual recognition, and spatial feature recognition using a multimodal recognition sub-model in a pre-defined integrated model. Based on the recognized text, visual, and spatial features, the cross-border document images are classified to obtain a fourth image category for each cross-border document image. The multimodal recognition sub-model includes a first input layer, a second input layer, a third input layer, and multiple transformation layers. The first input layer is connected to each of the transformation layers, the second input layer is connected to each of the transformation layers, and the third input layer is connected to the first transformation layer. The first input layer is used to input two-dimensional spatial coordinates, absolute spatial coordinates, and two-dimensional relationships; the second input layer is used to input visual features; and the third input layer is used to input text features. The residual connection between the spatial features and visual features is passed to each of the transformation layers, and the outputs of each transformation layer are fused to obtain the fourth image category. In each transformation layer, the visual features and the text features undergo self-attention processing with shared spatial features. The document image category of each cross-border document image is determined by weighted calculation based on the first image category, the second image category, the third image category, and the fourth image category. The method uses a preset general language model to review the key information of each document, thereby obtaining the review result of the target cross-border business. The review result includes approved and unapproved results, including: Obtain key information for each document in the target cross-border business; The key information of the document is stored in the short-term memory of the memory unit in the preset general large language model; Based on the key information of the document, the intelligent agent is determined to be a document review expert role. The document review expert role is used to call the key information of the document and the preset document review database. The document review database is stored in the long-term memory of the memory unit. The review expert role calls the planning unit to perform task planning and obtain a review plan, which includes multiple review tasks to be executed sequentially. The review expert role invokes the execution unit to execute the review plan and determines whether the review result of the target cross-border business is approved or not.

2. The cross-border document image verification method according to claim 1, characterized in that, The process involves identifying business features of the cross-border document images using a rule-based recognition sub-model within a pre-defined integrated model, and classifying the cross-border document images based on the identified target business features to obtain a first image category for each cross-border document image, including: The cross-border document images are subjected to OCR recognition to obtain multiple strings contained in each cross-border document image; The string is matched based on the rule recognition sub-model in the preset integrated model and the preset regular expression to obtain the target business features of each cross-border document image. The regular expression is used to identify the preset business features. Based on the target business characteristics, the document category to which each cross-border document image belongs is determined, resulting in the first image category.

3. The cross-border document image verification method according to claim 1, characterized in that, The text recognition sub-model in the preset integrated model performs semantic recognition on the cross-border document images, and classifies the cross-border document images based on the recognized target text semantics to obtain a third image category for each cross-border document image, including: The cross-border document images are subjected to OCR recognition to obtain the target text content contained in each cross-border document image; The target text content is semantically recognized by the text recognition sub-model in the preset integrated model to obtain the target text semantics of each cross-border document image; Based on the semantics of the target text, the document category to which each cross-border document image belongs is determined, resulting in a third image category.

4. The cross-border document image verification method according to claim 1, characterized in that, The method involves simultaneously performing text recognition, visual recognition, and spatial feature recognition on the cross-border document images using a multimodal recognition sub-model within a pre-defined integrated model. Based on the recognized text, visual, and spatial features, the cross-border document images are classified to obtain a fourth image category for each cross-border document image, including: The cross-border document images are subjected to OCR recognition to obtain the target text content contained in each cross-border document image; The target text content is identified by the multimodal sub-model in the preset integrated model, and the text features, visual features and spatial features of each cross-border document image are obtained. Based on the text features, visual features, and spatial features, the document category to which each cross-border document image belongs is determined, resulting in a fourth image category.

5. The cross-border document image verification method according to claim 1, characterized in that, Before performing text recognition on the cross-border document image using a preset text recognition model, the text recognition model is trained. Obtain a training dataset, which includes multiple cross-border document images; The training dataset is preprocessed to obtain a preprocessed training dataset. The parameters of all convolutional layers in the preset convolutional recurrent neural network model are frozen, and the long short-term memory network and fully connected layers are adjusted to obtain the adjusted convolutional recurrent neural network model. The adjusted convolutional recurrent neural network model is trained based on the preprocessed training dataset to obtain a trained text recognition model.

6. A cross-border document image verification device, which can be used in the cross-border document image verification method as described in any one of claims 1 to 5, characterized in that, The cross-border document image verification device includes: The first acquisition module is used to acquire multiple cross-border document images of the target cross-border business; The image classification module is used to classify the cross-border document images using a preset integrated model to obtain the document image category of each cross-border document image. The text recognition module is used to perform text recognition on the cross-border document images using a preset text recognition model to obtain the document text content contained in each cross-border document image. The information extraction module is used to extract information from the document text content contained in each cross-border document image based on the information extraction rules of each document image category, so as to obtain the key document information of each cross-border document image. The review module is used to review the key information of each document based on a preset general language model to obtain the review result of the target cross-border business. The review result includes approval and failure.

7. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the cross-border document image verification method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the cross-border document image verification method according to any one of claims 1 to 5.

Images