Cross-border document image auditing method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence and financial science and technology, and provides a cross-border document image auditing method, device and equipment and a storage medium, and the method comprises the steps: obtaining a plurality of cross-border document images of a target cross-border business; classifying the cross-border document images through a preset integration model to obtain a document image category of each cross-border document image; performing character recognition on the cross-border document images through a preset character recognition model to obtain document character content contained in each cross-border document image; performing information extraction on document text content contained in each cross-border document image based on an information extraction rule of each document image category to obtain document key information of each cross-border document image; and each piece of document key information is audited based on a preset general large language model to obtain an audit result of the target cross-border business, and the audit result comprises passing of audit and non-passing of audit. According to the embodiment of the invention, the checking efficiency and checking accuracy of the cross-border document image can be improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and financial technology, and in particular to a cross-border document image review method, device, electronic device and storage medium. Background Art

[0002] With the rise of artificial intelligence technology, image recognition technology based on artificial intelligence models has been widely used in various financial business audits. At present, cross-border business involves many cross-border documents, such as import letters of credit, export letters of credit, etc. However, due to the complexity of cross-border business rules, there are many types of cross-border documents. The existing image recognition model cannot complete the audit task, resulting in the cross-border transaction system relying on manual audit.

[0003] Since cross-border documents involve complex laws and regulations on cross-border trade business, there are numerous rules and complicated clauses. In addition, there are many types of cross-border documents, including bills of lading, insurance policies, packing lists and more than 40 major categories of documents, and there is no unified format for documents in each category.

[0004] During the manual review process, it is necessary to compare and review the documents between each link to ensure the consistency and rationality of all elements, which leads to a long manual review cycle and low accuracy. Summary of the invention

[0005] The present invention provides a cross-border document image review method, device, electronic device and storage medium for improving the review efficiency and review accuracy of cross-border document images.

[0006] A first aspect of an embodiment of the present disclosure provides a cross-border document image review method, including:

[0007] Obtain multiple cross-border document images of the target cross-border business;

[0008] Classifying the cross-border document images through a preset integrated model to obtain a document image category of each cross-border document image;

[0009] Performing text recognition on the cross-border document images through a preset text recognition model to obtain the document text content contained in each cross-border document image;

[0010] Extracting information from the document text content contained in each cross-border document image based on the information extraction rule of each document image category to obtain key document information of each cross-border document image;

[0011] Based on a preset general large language model, the key information of each document is reviewed to obtain the review result of the target cross-border business, and the review result includes passed and failed.

[0012] A second aspect of the disclosed embodiment provides a cross-border document image review device, the cross-border document image review device comprising:

[0013] A first acquisition module is used to acquire multiple cross-border document images of a target cross-border business;

[0014] An image classification module, used to classify the cross-border document images through a preset integrated model to obtain a document image category of each cross-border document image;

[0015] A text recognition module, used to perform text recognition on the cross-border document image through a preset text recognition model to obtain the document text content contained in each cross-border document image;

[0016] An information extraction module, configured to extract information from the document text content contained in each cross-border document image based on the information extraction rules of each document image category, and obtain key document information of each cross-border document image;

[0017] The audit module is used to audit the key information of each document based on a preset general large language model to obtain the audit results of the target cross-border business, and the audit results include passed and failed.

[0018] A third aspect of an embodiment of the present disclosure provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the cross-border document image review method of the first aspect when executing the computer program.

[0019] The fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium, in which instructions are stored. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-border document image review method of the first aspect mentioned above is implemented.

[0020] In the technical solution provided by the embodiment of the present disclosure, multiple cross-border document images of the target cross-border business are obtained; the cross-border document images are classified by a preset integrated model to obtain the document image category of each cross-border document image; the cross-border document images are recognized by a preset text recognition model to obtain the document text content contained in each cross-border document image; the document text content contained in each cross-border document image is extracted based on the information extraction rules of each document image category to obtain the document key information of each cross-border document image; each document key information is reviewed based on a preset general large language model to obtain the review result of the target cross-border business, and the review result includes passed review and failed review. The embodiment of the present disclosure realizes automatic review of multiple cross-border document images of the target cross-border business through a preset integrated model, text recognition model and large language model, thereby improving the review efficiency and accuracy of cross-border document images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is an application system framework diagram of an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a process of a cross-border document image review method in an embodiment of the present disclosure;

[0023] Figure 3 In the embodiments of the present disclosure Figure 2 A specific flow chart of step 220;

[0024] Figure 4 In the embodiments of the present disclosure Figure 3 A specific flow chart of step 310;

[0025] Figure 5 In the embodiments of the present disclosure Figure 3 A specific flow chart of step 330;

[0026] Figure 6 In the embodiments of the present disclosure Figure 3 A specific flow chart of step 340;

[0027] Figure 7 In the embodiments of the present disclosure Figure 2 A specific flow chart before step 230;

[0028] Figure 8 In the embodiments of the present disclosure Figure 2 A specific flow chart of step 250;

[0029] Fig. 9 A schematic diagram of a cross-border document image review device in an embodiment of the present disclosure;

[0030] Fig.10 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0032] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0033] Cross-border documents: Cross-border documents refer to a series of certification documents used to prove the source, quality, price, quantity, transportation conditions, compliance and other information of goods in the process of cross-border trade. These documents play a vital role in cross-border transactions. They are not only necessary materials for customs clearance, tax processing, payment settlement and other links, but also important tools to protect the rights and interests of buyers and sellers and ensure the legality and smooth progress of transactions.

[0034] OCR recognition: Optical Character Recognition (OCR) is a technology that converts printed or handwritten characters into editable text. OCR recognition converts the text in the image into computer-recognizable text information by scanning or photographing the image and using image processing and pattern recognition algorithms.

[0035] At present, for international cross-border business, financial institutions need to review and confirm the authenticity of cross-border business, and then make payments after passing the review. The main basis for the review is the cross-border documents provided by both parties to the transaction. In the actual review process, any transaction party provides multiple cross-border document images of the target cross-border business to the financial institution, and then the staff of the financial institution reviews the authenticity of the target cross-border business, which is inefficient and prone to errors. In order to solve the above problems, the present disclosure provides a cross-border document image review method, which can automatically review cross-border document images through a preset model, thereby improving the review efficiency and accuracy of cross-border document images.

[0036] Figure 1 This is a system architecture diagram of the cross-border document image review method according to the embodiment of the present disclosure, including a terminal 110, the Internet 120, a gateway 130, and a server 140.

[0037] The terminal 110 is used to take images of cross-border documents and send the obtained cross-border document images to the server 140, and the cross-border document images are automatically reviewed by the server 140. For example, a user can take and upload multiple cross-border document images of a target cross-border business through the terminal 110. In addition, the terminal 110 can be a single device or a combination of multiple devices. The terminal 110 can communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0038] The server 140 refers to a computer system that can provide certain services to the terminal 110. Compared with the ordinary terminal 110, the server 140 has higher requirements in terms of stability, security, performance, etc. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0039] The gateway 130 is also called an internetwork connector or a protocol converter. The gateway 130 realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway 130 can also provide filtering and security functions. The message (cross-border document image) sent by the terminal 110 to the server 140 must be sent to the corresponding server 140 through the gateway 130. The message (passed or failed to pass the review) sent by the server 140 to the terminal 110 must also be sent to the corresponding terminal 110 through the gateway 130.

[0040] General description of the disclosed embodiments

[0041] According to one embodiment of the present disclosure, a cross-border document image review method is provided.

[0042] The cross-border document image review method of the disclosed embodiment may be executed on the terminal 110 or on the server 140 , or may be executed partially on the terminal 110 and the other partially on the server 140 .

[0043] See also Figure 2 , a flow chart of a cross-border document image review method provided by an embodiment of the present disclosure, specifically including:

[0044] 210. Obtain multiple cross-border document images of target cross-border business;

[0045] 220. Classify the cross-border document images through a preset integrated model to obtain the document image category of each cross-border document image;

[0046] 230. Perform text recognition on the cross-border document image through a preset text recognition model to obtain the document text content contained in each cross-border document image;

[0047] 240. Extract information from the document text content contained in each cross-border document image based on the information extraction rule of each document image category to obtain the key document information of each cross-border document image;

[0048] 250. Based on the preset general large language model, the key information of each document is reviewed to obtain the review results of the target cross-border business, which include passed and failed.

[0049] In step 210, there may be multiple cross-border document images for the same target cross-border business. For example, if the cross-border document image is uploaded by the seller, the cross-border document image may include a contract image, an invoice image, a packing list image, a letter of credit image, etc. Among them, the contract image may contain multiple pages of continuous images, and each page of the image is a cross-border document image.

[0050] In step 220, since the multiple cross-border document images obtained in step 210 are uploaded by the user at one time, they are not classified and sorted. In order to facilitate subsequent review, these uploaded cross-border document images need to be classified according to preset categories. Since there are multiple pages of related images in the cross-border document images, some cross-border document images are highly similar and difficult to distinguish. This embodiment classifies all cross-border document images separately through each sub-model in the integrated model, and then weights the results of each classification to obtain the document image category of each cross-border document image. Because the classification results of multiple sub-models are combined, the accuracy of the obtained document image category is higher.

[0051] In step 230, the document text content contained in each cross-border document image is extracted, wherein the preset text recognition model can adopt a fine-tuned Convolutional Recurrent Neural Network (CRNN) model to make the text recognition model more suitable for large sections of text content in cross-border document images, thereby improving the recognition accuracy of the document text content.

[0052] It is understandable that other models with text recognition functions can also be used for recognition, which is not limited here.

[0053] In step 240, each document image category corresponds to an information extraction rule. For example, for a letter of credit image, it is necessary to extract key document information such as the name of the document, the name of the issuing bank, the letter of credit number, the expiration date, the name and address of the consignee, the name of the goods, the quantity of the goods, the unit price, the total amount, the mode of transportation, and the payment terms from the letter of credit image. For an export invoice image, it is necessary to extract key document information such as the name of the product, the quantity of the goods, the unit price, the total amount, the mode of transportation, the shipping address, and the order number from the export invoice image.

[0054] In step 250, it is necessary to formulate document review rules based on Uniform Customs and Practice for Documentary Credits (UCP) 600, International Standard Banking Practice for the Examination of Documents under Documentary Credits (ISBP) 745, and document review experience, and preset them in the general large language model, so that the general large language model has the ability to review documents and complete the automatic review task. Among them, UCP600, as an international practice in the field of letters of credit, provides comprehensive, specific, and operational rules and guidance for letter of credit operations, which is of great significance for promoting the smooth progress of cross-border trade business. ISBP745 is the latest international practice for letter of credit operations in current cross-border trade business. ISBP745 comprehensively, detailed and specifically stipulates various requirements and interpretations of letter of credit documents, providing a strong guarantee for the smooth progress of cross-border trade business. As an important supplement to UCP600, ISBP745 plays a vital role in cross-border trade business. It not only provides a unified international standard for the preparation and review of letter of credit documents, but also reduces the risk of non-payment due to discrepancies between documents, and promotes the smooth progress of cross-border trade business.

[0055] In the above embodiment, the cross-border document images are classified by the integrated model to obtain the document image category of each cross-border document image, thereby improving the classification accuracy of the document image category. Then, the cross-border document images are subjected to text recognition to obtain the document text content contained in each cross-border document image; information extraction is performed on the document text content according to the information extraction rules of the document image category to obtain the document key information of each cross-border document image, thereby improving the accuracy of the document key information. Finally, the key information of each document is reviewed based on the preset general large language model to obtain the review results of the target cross-border business. The review results include passed and failed reviews, thereby improving the review efficiency and review accuracy of cross-border document images.

[0056] The steps in step 210 to step 250 are described in detail below.

[0057] In one embodiment, if Figure 3 As shown, step 220 may include:

[0058] 310. Perform business feature recognition on the cross-border document image by using the rule recognition sub-model in the preset integrated model, and classify the cross-border document image based on the recognized target business features to obtain a first image category of each cross-border document image;

[0059] 320. Recognize the representation features of the cross-border document image by using the image recognition sub-model in the preset integrated model, classify the cross-border document image based on the recognized target representation features, and obtain a second image category for each cross-border document image;

[0060] 330. Perform semantic recognition on the cross-border document image by using the text recognition sub-model in the preset integrated model, and classify the cross-border document image based on the recognized target text semantics to obtain a third image category of each cross-border document image;

[0061] 340. Perform text recognition, visual recognition and spatial feature recognition on the cross-border document image simultaneously through the multimodal recognition sub-model in the preset integrated model, and classify the cross-border document image based on the recognized text features, visual features and spatial features to obtain a fourth image category of each cross-border document image;

[0062] 350. Perform a weighted calculation based on the first image category, the second image category, the third image category, and the fourth image category to determine the document image category of each cross-border document image.

[0063] In step 310, each cross-border document image is classified by the rule recognition sub-model to obtain the first image category of each cross-border document image. Among them, the business expert experience and the feature statistics corresponding to each document image category are pre-set in the rule recognition sub-model. The business expert experience is the main business feature corresponding to each document image category summarized by the document review experts. For example, the main feature of the export invoice image is the invoice number, and the main feature of the packing list image is the packing method. The main business features are the review experience summarized by the business experts and can be adjusted according to the actual situation.

[0064] In step 320, each cross-border document image is classified by the image recognition sub-model to obtain a second image category for each cross-border document image. The image recognition sub-model is trained using a ConvNeXt model, and can recognize the representation features of the image, thereby performing classification. The representation features here can be image pixel features, and the format or table features appearing in each cross-border document image are determined based on the image pixel features.

[0065] In step 330, each cross-border document image is classified by a text recognition sub-model to obtain a third image category of each cross-border document image. The text recognition sub-model is trained using the Bert text classification model. For each cross-border document image, it is necessary to determine the text semantics of each text segment in the cross-border document image. For example, for a cross-border document image, if it is determined that the text semantics in each text box in the cross-border document image indicates content related to the invoice, then the cross-border document image can be classified as an export invoice image; if it is determined that the text semantics in each text box in the cross-border document image indicates content related to packing, then the cross-border document image can be classified as a packing list image.

[0066] In step 340, each cross-border document image is classified by a multimodal recognition sub-model to obtain a fourth image category of each cross-border document image. Among them, multimodal recognition includes text recognition, visual recognition and spatial feature recognition, and recognition can be performed simultaneously. Performing text recognition, visual recognition and spatial feature recognition at the same time can avoid missing the semantics of some texts. In this embodiment, the multimodal recognition sub-model includes a first input layer, a second input layer, a third input layer, and multiple conversion layers, wherein the first input layer is connected to each conversion layer respectively, the second input layer is connected to each conversion layer respectively, and the third input layer is connected to the first conversion layer. The first input layer is used to input two-dimensional spatial coordinates, absolute spatial coordinates and one-dimensional relationships, the second input layer is used to input visual features, and the third input layer is used to input text features. Because the dependency relationship between spatial features and visual features may vary from layer to layer, in one embodiment, the step of classifying cross-border document images based on the recognized text features, visual features and spatial features includes:

[0067] The residual connection of spatial features and visual features is passed to each conversion layer; then the output of each conversion layer is fused to obtain the fourth image category of each cross-border document image. In each conversion layer, the visual features and text features are processed by self-attention with shared spatial features.

[0068] In step 350, different weights are set for the first image category, the second image category, the third image category, and the fourth image category, and the image category after weighted calculation is used as the document image category of each cross-border document image. For example, because the accuracy of the multimodal recognition sub-model is higher, the weight of the fourth image category can be set higher.

[0069] The advantage of step 310 to step 350 is that all cross-border document images are identified in turn by each sub-model in the integrated model, and then the identification results of each sub-model are weighted to obtain the document image category of each cross-border document image, thereby improving the accuracy of the document image category and increasing the fault tolerance rate in the classification process of cross-border document images.

[0070] The above embodiment describes the classification process of the integrated model in detail. The classification process of each sub-model is described below.

[0071] In one embodiment, if Figure 4 As shown, step 310 may include:

[0072] 410. Perform OCR recognition on the cross-border document image to obtain multiple character strings contained in each cross-border document image;

[0073] 420. Matching the character string based on the rule recognition sub-model in the preset integrated model and the preset regular expression to obtain the target business features of each cross-border document image. The regular expression is used to identify the preset business features;

[0074] 430. Determine the document category to which each cross-border document image belongs based on the target business feature, and obtain a first image category.

[0075] In step 410, the cross-border document images are first subjected to OCR recognition to obtain the text information in each cross-border document image and extract it in the form of a character string.

[0076] In step 420, multiple regular expressions are preset, each of which can identify a specific business feature, that is, can identify a specific string. The extracted multiple strings are matched with multiple regular expressions in sequence to determine the business feature corresponding to each string, and a business feature is selected based on the multiple business features as the target business feature of the cross-border document image.

[0077] In step 430, a document category may be set in advance for each target business feature. For example, a cross-border document image with a target business feature such as an invoice number may be classified into an export invoice category, and a cross-border document image with a target business feature such as a packing method may be classified into a packing list category.

[0078] The advantage of step 410 to step 430 is that the cross-border document images are automatically classified according to the experience of business experts and the statistical results of business characteristics. The images can be classified according to pre-set document categories, thus avoiding unnecessary document categories and improving the accuracy of the first image category.

[0079] In one embodiment, if Figure 5 As shown, step 330 may include:

[0080] 510. Perform OCR recognition on the cross-border document image to obtain the target text content contained in each cross-border document image;

[0081] 520. Perform semantic recognition on the target text content through the text recognition sub-model in the preset integrated model to obtain the target text semantics of each cross-border document image;

[0082] 530. Determine the document category to which each cross-border document image belongs based on the target text semantics to obtain a third image category.

[0083] In step 510, the cross-border document images are first subjected to OCR recognition to obtain the target text content contained in each cross-border document image.

[0084] In step 520, semantic recognition is performed on the target text content through the text recognition sub-model. Different from the rule recognition sub-model, the text recognition sub-model is trained using machine learning methods and does not consider manually set rules. Therefore, the output results of the text recognition sub-model are not limited to manually set document image categories, but are based on different semantics to form different document image categories.

[0085] In step 530, a document category may be set in advance for each target text semantics. For example, a cross-border document image with a target text semantics such as describing invoice information may be classified into an export invoice category, and a cross-border document image with a target text semantics such as describing packing method may be classified into a packing list category.

[0086] The advantage of step 510 to step 530 is that the cross-border document images are classified by different target text semantics, which increases the number of document categories and improves the diversity of the third image category.

[0087] In one embodiment, if Figure 6 As shown, step 340 may include:

[0088] 610. Perform OCR recognition on the cross-border document image to obtain the target text content contained in each cross-border document image;

[0089] 620. Recognize the target text content through the multimodal sub-model in the preset integrated model to obtain the text features, visual features and spatial features of each cross-border document image;

[0090] 630. Determine the document category to which each cross-border document image belongs based on text features, visual features, and spatial features to obtain a fourth image category.

[0091] In step 610, the cross-border document images are first subjected to OCR recognition to obtain the target text content contained in each cross-border document image.

[0092] In step 620, the Docformer multimodal model may be used to simultaneously extract text features, recognition features, and spatial features.

[0093] In step 630, the Docformer multimodal model may directly fuse the text features, the recognition features, and the spatial features through a fusion layer, thereby outputting a fourth image category of each cross-border document image.

[0094] The advantage of steps 610-630 is that the recognition is performed through a multimodal sub-model, which improves the correlation between text features, recognition features and spatial features, while reducing the number of training parameters and improving the accuracy of the fourth image category.

[0095] In one embodiment, if Figure 7 As shown, before step 230, the cross-border document image review method may also include:

[0096] 710. Obtain a training data set, where the training data set includes multiple cross-border document images;

[0097] 720. Preprocess the format of the training data set to obtain a preprocessed training data set;

[0098] 730. Freeze all convolution layer parameters in the preset convolution recursive neural network model, and adjust the long short-term memory network and the fully connected layer to obtain an adjusted convolution recursive neural network model;

[0099] 740. The adjusted convolutional recurrent neural network model is trained based on the preprocessed training data set to obtain a trained text recognition model.

[0100] In step 710, a training data set is obtained, where the cross-border document images in the training data set are original images.

[0101] In step 720, the cross-border document images in the training data set are preprocessed: each cross-border document image is scaled to obtain a cross-border document image of a fixed size; Gaussian noise is added to each cross-border document image of a fixed size, and then a radial transformation is performed to obtain a processed cross-border document image. For example, the height of the cross-border document image is set to 32, and the width is set to 100, that is, an image with a size of 32*100 is obtained; then Gaussian noise is added to the image with unified size, where the mean of the Gaussian noise is 0 and the variance is 1; finally, the cross-border document image with Gaussian noise added is subjected to a radial transformation with random rotation between (-10,10).

[0102] In step 730, the initial convolutional recurrent neural network model is obtained and fine-tuned: first, all convolutional layer parameters are frozen, and then the long short-term memory network (LSTM) and the fully connected layer are fine-tuned; finally, the initial learning rate is set to 1e-4, and the parameters of the Adam optimizer are set to beta1=0.9, beta2=0.999, epsilon=1e-8; and the loss function is set to the CTC loss function.

[0103] In step 740, before training, it is necessary to set the training parameters, for example, set the batch size to 64, set the maximum number of epochs for all samples to 100, and set the learning rate strategy to stepLR. Then, the adjusted convolutional recurrent neural network model is trained based on the preprocessed training data set and training parameters to obtain a trained text recognition model.

[0104] The advantage of step 710 to step 740 is that the training data set is first preprocessed, and then the initial convolutional recurrent neural network model is fine-tuned and the corresponding training parameters are set, thereby improving the recognition accuracy of the trained text recognition model.

[0105] In one embodiment, if Figure 8 As shown, step 250 may also include:

[0106] 810. Obtain key information of each document in the target cross-border business;

[0107] 820. Storing key information of documents through short-term memory of memory units in a preset universal large language model;

[0108] 830. Based on the key information of the document, the intelligent agent is determined to be a document review expert role. The document review expert role is used to call the key information of the document and the preset document review database. The document review database is stored in the long-term memory of the memory unit;

[0109] 840. The audit expert role calls the planning unit to perform task planning and obtain an audit plan, which includes multiple audit tasks to be executed in sequence.

[0110] 850. Call the execution unit through the audit expert role to execute the audit plan and determine whether the audit result of the target cross-border business is passed or failed.

[0111] It should be noted that the memory unit includes long-term memory and short-term memory, where the short-term memory is used to save the recently acquired information context, and the long-term memory is used to save external knowledge. The external knowledge can be a knowledge database related to various roles. For example, the knowledge database can be a review database, which includes the review system, review cases, and error points. The review expert role has the characteristics of being proficient in review data, processes, methods, and systems, and can call the review database corresponding to the review expert role. The general large language model is preset with intelligent agents, memory units, planning units, and execution units.

[0112] It should be noted that if the audit is passed, more detailed audit results can also be output, such as whether the documents are complete, whether the documents are consistent, and whether the documents are consistent.

[0113] The advantage of step 810-step 850 is that, based on the intelligent agent (Agent) through the large language model, document review rules are formulated according to UCP600, ISBP745 and document review experience, the review plan is executed, and the review results of passing or failing the review are output. The intelligent agent is used to replace senior domain experts to review documents, which improves the review efficiency of cross-border document images, and standardizes the cross-border document image review process, thereby improving the review accuracy of cross-border document images.

[0114] In one embodiment, before step 240, an extraction model may be trained to extract information from the document text content contained in each cross-border document image according to the information extraction rule of each document image category through the extraction model. The training process of the extraction model is as follows:

[0115] Prepare data: construct a JSON object array list of instructions, questions, and answers;

[0116] Adjust the initial model: select the Chat Generative Language Mode (ChatGLM) large model, add a bypass next to the original large model to link the dimension reduction matrix A and the dimension increase matrix B; initialize matrix A with random Gaussian distribution and initialize B with 0 matrix. This ensures that the newly added bypass BA=0 during training, which has no effect on the model results.

[0117] Train the adjusted model: Change the forward process of the basic large model h = Wx to h = Wx + BAx, and the output layer w out = W + BA. During inference, add the results of the left and right parts together, that is, h = Wx + BAx = (W + BA)x. Therefore, as long as the matrix product BA after training is added to the original weight matrix W as the new weight parameter and replaces the weight parameter of the ChatGLM basic large model, no additional computing resources will be added. Among them, x is the input, h is the output; A is the dimensionality reduction matrix with dimensions d*r, B is the dimensionality increase matrix with dimensions r*k, r << min(d,k), and W is the weight parameter of the ChatGLM basic large model.

[0118] Based on the trained large model, for each type of cross-border document image, write a prompt to extract the key information of the cross-border document image.

[0119] The above embodiments provide a training method for an extraction model, which uses the extraction model for information extraction and improves the efficiency of information extraction.

[0120] Description of the apparatus and device of the present disclosure

[0121] It can be understood that although the steps in the above flowcharts are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this embodiment, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0122] It should be noted that in each specific implementation manner of this application, when it comes to relevant processing based on data related to the user's identity or characteristics such as device information, device behavior data, device historical data, and device location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through pop-up windows or jumping to the confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of this disclosure embodiment will be obtained.

[0123] Refer to Fig. 9 , Fig. 9 The cross-border document image review device 900 provided in the embodiment of the present disclosure is a schematic diagram of the structure of the cross-border document image review device 900, which includes:

[0124] A first acquisition module 910 is used to acquire multiple cross-border document images of a target cross-border business;

[0125] An image classification module 920 is used to classify the cross-border document images through a preset integrated model to obtain a document image category of each cross-border document image;

[0126] The text recognition module 930 is used to perform text recognition on the cross-border document image by using a preset text recognition model to obtain the document text content contained in each cross-border document image;

[0127] The information extraction module 940 is used to extract the document text content contained in each cross-border document image based on the information extraction rules of each document image category to obtain the key document information of each cross-border document image;

[0128] The audit module 950 is used to audit the key information of each document based on a preset general large language model to obtain the audit results of the target cross-border business, and the audit results include passed and failed.

[0129] In one embodiment, the image classification module 920 is specifically used to:

[0130] The cross-border document image is identified by the rule recognition sub-model in the preset integrated model, and the cross-border document image is classified based on the identified target business features to obtain a first image category of each cross-border document image;

[0131] Recognize the representation features of the cross-border document image through the image recognition sub-model in the preset integrated model, classify the cross-border document image based on the recognized target representation features, and obtain the second image category of each cross-border document image;

[0132] The cross-border document image is semantically recognized by the text recognition sub-model in the preset integrated model, and the cross-border document image is classified based on the recognized target text semantics to obtain the third image category of each cross-border document image;

[0133] The cross-border document image is simultaneously subjected to text recognition, visual recognition and spatial feature recognition by a multimodal recognition sub-model in a preset integrated model, and the cross-border document image is classified based on the recognized text features, visual features and spatial features to obtain a fourth image category of each cross-border document image;

[0134] A weighted calculation is performed based on the first image category, the second image category, the third image category and the fourth image category to determine the document image category of each cross-border document image.

[0135] In one embodiment, the image classification module 920 is specifically used to:

[0136] Perform OCR recognition on cross-border document images to obtain multiple character strings contained in each cross-border document image;

[0137] Match the character string based on the rule recognition sub-model in the preset integrated model and the preset regular expression to obtain the target business features of each cross-border document image. The regular expression is used to identify the preset business features.

[0138] The document category to which each cross-border document image belongs is determined based on the target business feature to obtain a first image category.

[0139] In one embodiment, the image classification module 920 is specifically used to:

[0140] Perform OCR recognition on cross-border document images to obtain the target text content contained in each cross-border document image;

[0141] The target text content is semantically recognized through the text recognition sub-model in the preset integrated model to obtain the target text semantics of each cross-border document image;

[0142] The document category to which each cross-border document image belongs is determined based on the target text semantics to obtain a third image category.

[0143] In one embodiment, the image classification module 920 is specifically used to:

[0144] Perform OCR recognition on cross-border document images to obtain the target text content contained in each cross-border document image;

[0145] The target text content is recognized through the multimodal sub-model in the preset integrated model to obtain the text features, visual features and spatial features of each cross-border document image;

[0146] The document category to which each cross-border document image belongs is determined based on text features, visual features, and spatial features to obtain a fourth image category.

[0147] In one embodiment, the cross-border document image review device further includes:

[0148] A second acquisition module is used to acquire a training data set, where the training data set includes a plurality of cross-border document images;

[0149] A preprocessing module is used to preprocess the format of the training data set to obtain a preprocessed training data set;

[0150] An adjustment module, used to freeze all convolution layer parameters in a preset convolutional recurrent neural network model, and adjust the long short-term memory network and the fully connected layer to obtain an adjusted convolutional recurrent neural network model;

[0151] The training module is used to train the adjusted convolutional recurrent neural network model based on the preprocessed training data set to obtain a trained text recognition model.

[0152] In one embodiment, the audit module 950 is specifically used to:

[0153] Obtain key information of each document in the target cross-border business;

[0154] The key information of the document is stored through the short-term memory of the memory unit in the preset general large language model;

[0155] Based on the key information of the document, the intelligent agent is determined to be a document review expert role. The document review expert role is used to call the key information of the document and the preset document review database. The document review database is stored in the long-term memory of the memory unit;

[0156] The audit expert role calls the planning unit to perform task planning and obtains an audit plan, which includes multiple audit tasks to be executed in sequence.

[0157] The audit plan is executed by calling the execution unit through the audit expert role to determine whether the audit result of the target cross-border business is passed or failed.

[0158] See also Fig.10 The present disclosure also provides a hardware structure of an electronic device, the electronic device comprising:

[0159] The processor 1001 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0160] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the cross-border document image review method of the embodiment of the present disclosure;

[0161] Input / output interface 1003, used to implement information input and output;

[0162] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0163] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0164] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0165] The embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program, which implements the above-mentioned cross-border document image review method when executed by a processor.

[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The disclosed embodiment also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above cross-border document image review method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0168] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0169] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0170] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0171] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0172] In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0173] Finally, it should be noted that the above embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above embodiments, those skilled in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A cross-border document image review method, characterized in that: The method comprises: Obtain multiple cross-border document images of the target cross-border business; Classifying the cross-border document images through a preset integrated model to obtain a document image category of each cross-border document image; Performing text recognition on the cross-border document images through a preset text recognition model to obtain the document text content contained in each cross-border document image; Extracting information from the document text content contained in each cross-border document image based on the information extraction rule of each document image category to obtain key document information of each cross-border document image; Based on a preset general large language model, the key information of each document is reviewed to obtain the review result of the target cross-border business, and the review result includes passed and failed.

2. The cross-border document image review method according to claim 1, characterized in that: The cross-border document images are classified by a preset integrated model to obtain a document image category of each cross-border document image, including: Performing business feature recognition on the cross-border document image through a rule recognition sub-model in a preset integrated model, and classifying the cross-border document image based on the recognized target business features to obtain a first image category for each cross-border document image; Recognize the representation features of the cross-border document images through the image recognition sub-model in the preset integrated model, classify the cross-border document images based on the recognized target representation features, and obtain a second image category for each cross-border document image; Performing semantic recognition on the cross-border document image through a text recognition sub-model in a preset integrated model, and classifying the cross-border document image based on the recognized target text semantics to obtain a third image category of each cross-border document image; Performing text recognition, visual recognition, and spatial feature recognition on the cross-border document image simultaneously through a multimodal recognition sub-model in a preset integrated model, and classifying the cross-border document image based on the recognized text features, visual features, and spatial features to obtain a fourth image category for each cross-border document image; A weighted calculation is performed based on the first image category, the second image category, the third image category, and the fourth image category to determine the document image category of each cross-border document image.

3. The cross-border document image review method according to claim 2, characterized in that: The process of identifying business features of the cross-border document images by using a rule recognition sub-model in a preset integrated model, and classifying the cross-border document images based on the identified target business features to obtain a first image category of each cross-border document image includes: Performing OCR recognition on the cross-border document image to obtain multiple character strings contained in each cross-border document image; The character string is matched based on a rule recognition sub-model in a preset integrated model and a preset regular expression to obtain a target business feature of each cross-border document image, wherein the regular expression is used to identify the preset business feature; The document category to which each cross-border document image belongs is determined based on the target business feature to obtain a first image category.

4. The cross-border document image review method according to claim 2, characterized in that: The semantic recognition of the cross-border document image is performed by the text recognition sub-model in the preset integrated model, and the cross-border document image is classified based on the recognized target text semantics to obtain the third image category of each cross-border document image, including: Performing OCR recognition on the cross-border document image to obtain target text content contained in each cross-border document image; The target text content is semantically recognized by a text recognition sub-model in a preset integrated model to obtain the target text semantics of each cross-border document image; The document category to which each cross-border document image belongs is determined based on the target text semantics to obtain a third image category.

5. The cross-border document image review method according to claim 2, characterized in that: The method of simultaneously performing text recognition, visual recognition and spatial feature recognition on the cross-border document image through the multimodal recognition sub-model in the preset integrated model, and classifying the cross-border document image based on the recognized text features, visual features and spatial features to obtain a fourth image category of each cross-border document image includes: Performing OCR recognition on the cross-border document image to obtain target text content contained in each cross-border document image; The target text content is recognized by a multimodal sub-model in a preset integrated model to obtain text features, visual features, and spatial features of each cross-border document image; The document category to which each cross-border document image belongs is determined based on the text feature, the visual feature, and the spatial feature to obtain a fourth image category.

6. The cross-border document image review method according to claim 1, characterized in that: Before the text recognition is performed on the cross-border document image by using a preset text recognition model, a training process of the text recognition model is also included: Acquire a training data set, wherein the training data set includes a plurality of cross-border document images; Preprocessing the format of the training data set to obtain a preprocessed training data set; Freeze all convolutional layer parameters in the preset convolutional recurrent neural network model, and adjust the long short-term memory network and the fully connected layer to obtain an adjusted convolutional recurrent neural network model; The adjusted convolutional recurrent neural network model is trained based on the preprocessed training data set to obtain a trained text recognition model.

7. The cross-border document image review method according to claim 1, characterized in that: The key information of each document is reviewed based on the preset universal large language model to obtain the review result of the target cross-border business, and the review result includes passed review and failed review, including: Obtain key information of each document in the target cross-border business; The key information of the document is stored in the short-term memory of the memory unit in the preset universal large language model; Determining the agent as a document examination expert role based on the document key information, the document examination expert role is used to call the document key information and a preset document examination database, and the document examination database is stored in the long-term memory of the memory unit; The audit expert role calls the planning unit to perform task planning to obtain an audit plan, wherein the audit plan includes a plurality of audit tasks to be executed sequentially; The audit plan is executed by calling the execution unit through the audit expert role to determine whether the audit result of the target cross-border business is passed or failed.

8. A cross-border document image review device, characterized in that: The cross-border document image review device comprises: A first acquisition module is used to acquire multiple cross-border document images of a target cross-border business; An image classification module, used to classify the cross-border document images through a preset integrated model to obtain a document image category of each cross-border document image; A text recognition module, used to perform text recognition on the cross-border document image through a preset text recognition model to obtain the document text content contained in each cross-border document image; An information extraction module, configured to extract information from the document text content contained in each cross-border document image based on the information extraction rules of each document image category, and obtain key document information of each cross-border document image; The audit module is used to audit the key information of each document based on a preset general large language model to obtain the audit results of the target cross-border business, and the audit results include passed and failed.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the cross-border document image review method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-border document image review method according to any one of claims 1 to 7 is implemented.

Images