Certificate image processing method and device, electronic equipment and storage medium

Abstract

The invention provides a certificate image processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a certificate image; performing text recognition on the certificate image to obtain text information in the certificate image; performing standardization processing on the text information to obtain standardized text information; performing field verification on the standardized text information to obtain a verification result; and storing the certificate image and the standardized text information in response to the verification result representing that verification is passed. According to the method and the device, the certificate information of different formats can accord with a unified auditing standard, convenience is provided for subsequent verification, and thus the certificate auditing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for processing document images. Background Art

[0002] Taking the passport as an example, the relevant technology provides the following two solutions for passport review: Solution 1: mainly manual filling by users. When taking a photo and submitting a photocopy, the user manually fills in the key fields on the passport according to the system requirements, such as name, passport number, and date of birth; Solution 2: mainly manual review by the background. The user only needs to upload the passport photo. The background staff will view the uploaded passport photo and manually extract the passport information for review and information entry.

[0003] However, the manual filling method is cumbersome, and users need to manually input a large amount of information, which increases the burden on users. At the same time, users are prone to making mistakes when manually inputting, resulting in inaccurate information, affecting the audit results, and thus making the user experience poor, causing user disgust, and thus affecting service satisfaction. The manual review method in the background has high labor costs, requiring a large amount of manual review and information entry, which increases labor costs; secondly, manual review and information entry are slow, take a long time to process, and are overall inefficient; in addition, different auditors may have different judgment standards and input methods, which affects the consistency and accuracy of the information. Summary of the invention

[0004] The embodiments of the present application provide a method for processing certificate images, a method for training a language model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can make certificate information of different formats conform to a unified review standard, provide convenience for subsequent verification, and thus improve the efficiency and accuracy of certificate review.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for processing a document image, including:

[0007] Get the document image;

[0008] Performing text recognition on the certificate image to obtain text information in the certificate image;

[0009] Performing standardization processing on the text information to obtain standardized text information;

[0010] Performing field verification on the standardized text information to obtain a verification result;

[0011] In response to the verification result indicating that the verification is passed, the document image and the standardized text information are stored.

[0012] The present application embodiment provides a document image processing device, including:

[0013] An acquisition module, used to acquire a document image;

[0014] A text recognition module, used to perform text recognition on the document image to obtain text information in the document image;

[0015] A standardization processing module, used to perform standardization processing on the text information to obtain standardized text information;

[0016] A verification module, used to perform field verification on the standardized text information to obtain a verification result;

[0017] The storage module is used for storing the document image and the standardized text information in response to the verification result indicating that the verification is passed.

[0018] The present application provides a method for training a language model, including:

[0019] Acquire training data, wherein the training data includes certificate image samples of different formats and standardized text information annotated for the certificate image samples, wherein the standardized text information is obtained by standardizing the text information in the certificate image samples;

[0020] Dividing the training data into a training set and a validation set;

[0021] Training the pre-trained language model based on the training set to obtain a trained language model;

[0022] Verifying the trained language model based on the verification set to obtain a verification result;

[0023] In response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, the trained language model is used as a language model for standardization processing.

[0024] The present application provides a language model training device, including:

[0025] An acquisition module, used to acquire training data, wherein the training data includes certificate image samples of different formats and standardized text information annotated for the certificate image samples, wherein the standardized text information is obtained by standardizing the text information in the certificate image samples;

[0026] A partitioning module, used for partitioning the training data into a training set and a validation set;

[0027] A training module, used to train the pre-trained language model based on the training set to obtain a trained language model;

[0028] A verification module, used to verify the trained language model based on the verification set to obtain a verification result;

[0029] The determination module is configured to use the trained language model as a language model for standardization processing in response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold.

[0030] An embodiment of the present application provides an electronic device, including:

[0031] A memory for storing computer executable instructions;

[0032] The processor is used to implement the document image processing method or language model training method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0033] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the document image processing method or language model training method provided in the embodiment of the present application when executed by a processor.

[0034] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement the document image processing method or language model training method provided in the embodiment of the present application when executed by a processor.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] On the one hand, by performing text recognition on the document image, the text information in the document image can be obtained, thereby reducing the steps of manual input by users and manual review in the background, greatly improving the efficiency of information processing. On the other hand, by standardizing the recognized text information, it can ensure that the text information recognized from document images of different nationalities and different formats can be uniformly converted into a standard format, which provides convenience for subsequent verification, thereby improving the efficiency and accuracy of document review. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 1 is a schematic diagram of the architecture of a system 100 for processing a document image provided in an embodiment of the present application;

[0038] Figure 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0039] Figure 2B is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application;

[0040] Figure 3 This is a first flow chart of a method for processing a certificate image provided in an embodiment of the present application;

[0041] Figure 4 It is a second flow chart of the method for processing a certificate image provided in an embodiment of the present application;

[0042] Figure 5 is a third flow chart of the method for processing a certificate image provided in an embodiment of the present application;

[0043] Figure 6 It is a flowchart of the language model training method provided in the embodiment of the present application;

[0044] Figure 7 It is a process schematic diagram of the method for processing a certificate image provided in an embodiment of the present application;

[0045] Figure 8 It is a schematic diagram of the fine-tuning process of the large language model provided in the embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of the dual-run verification process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0048] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0050] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0053] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed may be executed.

[0054] 2) Optical Character Recognition (OCR): refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then uses character recognition methods to translate the shapes into computer text. That is, for printed characters, an optical method is used to convert the text in a paper document into a black and white dot matrix image file, and recognition software is used to convert the text in the image into text format for further editing and processing by word processing software.

[0055] 3) Large Language Model (LLM): refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence.

[0056] 4) Fine-tuning: It is an optimization method that makes the model adaptable to specific application scenarios by fine-tuning it on a specific task dataset.

[0057] 5) Supervised Fine-Tuning (SFT): refers to a fine-tuning method based on self-supervised learning, which introduces additional training data into the large language model to improve the model's targeting and generalization performance.

[0058] 6) Document image: refers to an image obtained by photographing a document, where the document may include a passport, identity card, campus card, employee card, and pass, etc.

[0059] 7) Application Programming Interface (API): It is a set of predefined functions that aims to provide applications and developers with the ability to access a set of routines based on certain software or hardware without having to access the source code or understand the details of the internal working mechanism.

[0060] The embodiments of the present application provide a method for processing certificate images, a method for training a language model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can make certificates of different formats conform to a unified review standard, provide convenience for subsequent verification, and thus improve the efficiency and accuracy of certificate review. The electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as a server, or as a terminal device, or can also be implemented in collaboration with a terminal device and a server. The following is an example of a method for processing certificate images provided by an embodiment of the present application being implemented in collaboration with a server and a terminal device.

[0061] For example, see Figure 1 , Figure 1 is a schematic diagram of the architecture of a system 100 for processing a certificate image provided in an embodiment of the present application. Figure 1 As shown, the document image processing system 100 includes: a server 200, a network 300 and a terminal device 400, wherein the network 300 can be a local area network or a wide area network, or a combination of the two, and the terminal device 400 is a terminal device associated with a user, and a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as a payment service client, an instant messaging client, and a browser.

[0062] In some embodiments, taking the client 410 as a payment service client as an example, when a user clicks on a scanning control displayed in a human-computer interaction interface provided by the payment service client, the camera of the terminal device 400 can be called to shoot the user's certificate (for example, a passport, an ID card, a student ID card, etc.) to obtain a certificate image, and then the terminal device 400 can send the certificate image to the server 200 through the network 300. After receiving the certificate image, the server 200 can extract the text information in the certificate image using OCR technology, and then standardize the extracted text information through a trained large language model (for example, the text information can be unified into a standard format), and then the server 200 can perform field verification on the standardized text information (for example, including field rationality verification and consistency verification) to obtain a verification result. When the verification result indicates that the verification is passed, the server 200 can also send the user's certificate image and the standardized text information to an external authoritative verification agency, so that the external authoritative verification agency can further verify the user's certificate image and the standardized text information. When the server 200 receives a notification message indicating that the verification has been passed, which is returned by an external authoritative verification agency, the user's certificate image and the standardized text information can be stored in the database.

[0063] In other embodiments, the document image processing method provided in the embodiments of the present application may also be implemented by a terminal device alone. Figure 1 Taking the terminal device 400 shown in as an example, after acquiring the user's ID image (such as a passport image), the terminal device 400 can use OCR technology to identify relevant text information (such as passport number, name, gender, date of birth, nationality, and passport issuance date, etc.) from the ID image, and then call the fine-tuned large language model to standardize the text information identified by OCR technology, for example, the identified text information can be unified into a standard, easy-to-process format. Subsequently, the terminal device 400 can perform a comprehensive field check (such as a rationality check and a consistency check) on the standardized text information to obtain a check result. When the check passes, the terminal device 400 can also send the user's ID image and the standardized text information to an external authoritative verification agency, so that the external authoritative verification agency can further verify the user's ID image and the standardized text information. When receiving a notification message indicating that the verification has passed from an external authoritative verification agency, the terminal device 400 can retain the user's ID image and standardized text information, for example, it can be stored locally on the terminal device 400, or it can also be stored in a secure database.

[0064] It should be noted that Figure 1 The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0065] In some embodiments, the terminal device or server can also implement the document image processing method or language model training method provided in the embodiments of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a payment service APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0066] The following is a description of the structure of an electronic device for executing a method for processing a document image provided in an embodiment of the present application. Figure 2A , Figure 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, Figure 2A The electronic device 500 shown includes: at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 530 is not described in detail. Figure 2A Various buses are labeled as bus system 530 .

[0067] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 540 may optionally include one or more storage devices that are physically remote from the processor 510.

[0069] The memory 540 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 540 described in the embodiments of the present application is intended to include any suitable type of memory.

[0070] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0071] Operating system 541, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0072] A network communication module 542, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0073] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2AThe document image processing device 543 stored in the memory 540 is shown, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5431, text recognition module 5432, standardization processing module 5433, verification module 5434, storage module 5435, evaluation module 5436, replacement module 5437, training module 5438, detection module 5439, deletion module 54310 and sending module 54311. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in Figure 2A For the sake of convenience, all the above modules are shown at once, but it should not be considered that the document image processing device 543 excludes the implementation of only including the acquisition module 5431, the text recognition module 5432, the standardization processing module 5433, the verification module 5434 and the storage module 5435. The functions of each module will be explained below.

[0074] The following is a description of the structure of an electronic device for executing a language model training method provided in an embodiment of the present application. Figure 2B , Figure 2B is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, Figure 2B The electronic device 600 shown includes: at least one processor 610, a memory 640, and at least one network interface 620. The various components in the electronic device 600 are coupled together via a bus system 630. Among them, the memory 640 includes an operating system 641, a network communication module 642, and a language model training device 643, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 6431, a division module 6432, a training module 6433, a verification module 6434, and a determination module 6435. The functions of each module will be described below.

[0075] The document image processing method provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0076] For example, see Figure 3 , Figure 3 is a first flow chart of the method for processing a certificate image provided in an embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0077] In step 101, a document image is acquired.

[0078] In some embodiments, taking the document image as a passport image (or passport image) as an example, when the user performs an account opening operation, the user needs to upload his passport image, wherein the passport image can be pre-photographed by the user and saved in the photo album, for example, the user can select the passport image from the photo album to upload; of course, the passport image can also be taken in real time, for example, when the user clicks the shooting button, the camera of the terminal device (such as the user's mobile phone or passport recognition device) can be called to shoot the user's passport to obtain the passport image, and then the captured passport image can be uploaded to the backend server through the network. For example, the user can use his own mobile phone to shoot the passport, and upload the captured passport image to the server; of course, the user can also place the passport in the recognition area specified by the passport recognition device, so that the passport recognition device can shoot the passport through the built-in or external camera, and upload the captured passport image to the server.

[0079] It should be noted that the document image in the embodiment of the present application may be other types of document images in addition to passport images, for example, it may be an identity card image (i.e., an image obtained by photographing the user's identity card), a student ID card image (i.e., an image obtained by photographing the user's student ID card), an employee ID card image (i.e., an image obtained by photographing the user's employee ID card), etc. The embodiment of the present application does not make any specific limitations on this.

[0080] In step 102, text recognition is performed on the document image to obtain text information in the document image.

[0081] In some embodiments, after receiving the document image uploaded by the terminal device (such as the user's mobile phone or account opening device), the server can use OCR technology to perform text recognition on the received document image to obtain the text information in the document image. For example, taking the document image as a passport image, the text information recognized from the passport image using OCR technology may include but is not limited to the following: passport number, name, gender, date of birth, nationality, passport issuance date / expiration date, former name, and parents' names, etc.

[0082] The following is a detailed description of the implementation process of using OCR technology to perform text recognition on the received document image.

[0083] For example, when the certificate image received by the server is a color image, since the information contained in the color image is too huge, before performing text recognition on the color certificate image, the certificate image can be binarized so that the certificate image only contains black foreground information and white background information, thereby improving the efficiency and accuracy of the recognition process; then, since the quality of the certificate image is limited by the input device, environment and printing quality of the certificate, before performing text recognition on the binary certificate image, the binary certificate image can also be denoised according to the characteristics of the noise to improve the accuracy of the subsequent text recognition process. In addition, since the scanning and shooting process involves manual operation, the certificate image received by the server may be tilted, so before performing text recognition on the certificate image, the image direction detection can also be performed and the image direction can be corrected. Subsequently, the layout of the corrected certificate image can be analyzed to separate the text blocks, images, tables and other elements in the certificate image, and the text blocks can be divided into separate text lines, and the text lines can be further divided into separate characters. After obtaining individual characters, feature extraction can be performed on each character to obtain corresponding character features. These features can be the shape, stroke width, direction, etc. of the character, which can be used in the subsequent pattern recognition process. After extracting the character features, a machine learning model or a deep learning network can be used to match the extracted character features with known character templates to identify each character. At the same time, natural language processing (NLP) technology can be used to further proofread the character recognition results based on contextual information to improve the accuracy. After obtaining the character recognition results output by OCR, tools such as dictionaries and grammar rules can be used for error correction to check errors in the OCR output. Finally, the recognized text information can be typeset and formatted.

[0084] It should be noted that in addition to using OCR technology to perform text recognition on certificate images, other methods can also be used to perform text recognition on certificate images, such as using a deep learning-based method to perform text recognition on certificate images (for example, a convolutional neural network or a recurrent neural network can be used to detect the text area of ​​the certificate image and recognize the characters in the text area), a feature-based method to perform text recognition on certificate images (for example, edge detection, texture analysis, color contrast and other methods can be used to extract text features in the certificate image, and then a machine learning algorithm can be used to recognize these features), and a region proposal network (RPN, Region Proposal Network) is used to identify text areas in certificate images, and further text detection and recognition are performed on the characters included in the text area. The embodiments of the present application do not specifically limit this.

[0085] In step 103, the text information is standardized to obtain standardized text information.

[0086] In some embodiments, step 103 can be implemented in the following manner: standardize the text information through a language model (e.g., a fine-tuned large language model) to obtain standardized text information, wherein the standardization process may include unifying the text information into a preset standard format, and the standard format may include standard field names and standard data structures, etc.

[0087] For example, taking the document image as a passport image, after using OCR technology or other text recognition methods to identify relevant text information from the passport image, the identified text information can also be standardized through a fine-tuned large language model. The purpose of standardization is to unify text information in different languages ​​or formats into a standard, easy-to-process format, thereby ensuring that passport information of different nationalities can be correctly processed.

[0088] The following is an explanation of the specific implementation process of standardizing the fine-tuned large language model.

[0089] For example, before the text information is standardized by the fine-tuned large language model, the text information can be cleaned first, for example, noise characters that may be generated during the recognition process can be deleted, such as misrecognition of special symbols, numbers 0 and letters O, etc., and spaces at both ends of the string and redundant spaces in the string can be removed. Then, according to the specified format of the passport information (for example, the passport number is usually a combination of a certain number of letters and numbers), it can be verified whether the recognition result conforms to the expected format, and the format of the date, name, passport number and other fields included in the text information can be unified (for example, the date can be unified into the format of YYYY-MM-DD). In addition, case conversion can be performed, for example, the characters included in the text information can be uniformly converted to uppercase or lowercase to meet the database storage standards. Of course, language conversion can also be performed, for example, information such as name and nationality that may be expressed in multiple languages ​​can be converted to a standard language expression. Alternatively, data mapping can also be performed, for example, for fields with standard codes such as country and gender, the recognized text information can be mapped to the corresponding standard code (for example, the country code ISO 3166-1). Finally, the standardized text information can also be stored in the database in a standard data structure (such as a dictionary structure of key-value pairs) to ensure data security and facilitate query.

[0090] It should be noted that the language model in the embodiment of the present application can be a fine-tuned large language model, which, after fine-tuning, can recognize, understand and process multiple languages ​​​​and complex text content, and can intelligently complete and correct missing or erroneous fields, thereby ensuring that different types of certificate information (such as passport information of different nationalities) can be processed correctly.

[0091] In other embodiments, before the text information is standardized through the language model, the following processing can also be performed: obtaining training data, wherein the training data includes document image samples of different formats and standardized text information annotated for the document image samples, and the standardized text information is obtained by standardizing the text information in the document image samples; dividing the training data into a training set and a verification set; training the pre-trained language model based on the training set to obtain a trained language model; verifying the trained language model based on the verification set to obtain a verification result; in response to the verification result indicating that the accuracy of the trained language model reaches (i.e., is greater than or equal to) an accuracy threshold, using the trained language model as the language model for standardization processing.

[0092] For example, taking the document image as a passport image, in the fine-tuning (or fine-tuning) process of the model, you can first collect passport image samples from multiple countries as training data, so as to ensure that the data set can cover the passport formats of different countries and regions, and then you can manually annotate these data, for example, you can manually standardize the text information in the passport image samples. Then you can divide the prepared training data into a training set and a validation set, so as to ensure that the model can be effectively verified and tuned during the training process. Then you can select a suitable pre-trained model (such as a large language model trained on a large data set), and fine-tune the model using the supervised fine-tuning (i.e., SFT) and full parameter fine-tuning (Full Parameter Fine Tuning) training mode on the training set. The fine-tuning process of the model can include multiple rounds of iterations. In each round of iteration, the model will adjust parameters according to the performance of the previous round to improve the accuracy of the model in the passport information formatting task.

[0093] The specific implementation process of supervised fine-tuning is described below.

[0094] For example, supervised fine-tuning is an important process in machine learning, especially in the field of deep learning. It involves further training on the basis of an existing pre-trained model using data from a specific task to improve the performance of the model on the task. Supervised fine-tuning can be understood as the second of two main stages, wherein the first stage is the pre-training stage, in which the model is trained on a large-scale unsupervised data set. The goal of pre-training is to allow the model to learn a wide range of language patterns and structures. The second stage, supervised fine-tuning, refers to training the model using annotated data (i.e., labeled data) to adapt to specific tasks, such as the standardized processing tasks in the embodiments of the present application.

[0095] Supervised fine-tuning mainly includes the following key steps and concepts:

[0096] 1. Data preparation: You need to prepare labeled data, which is usually manually labeled and includes input and expected output. For example, the input can be passport image samples from multiple countries, and the expected output can be standardized passport information labeled for passport image samples.

[0097] 2. Model architecture: In supervised fine-tuning, you can choose a model that has been pre-trained on a large-scale dataset and then fine-tune it based on it. Since the pre-trained model has learned rich language representations, it provides a good starting point for fine-tuning.

[0098] 3. Loss function: During fine-tuning, a loss function can be defined to measure the difference between the prediction results output by the model and the pre-labeled labels. Common loss functions include cross entropy loss (for classification tasks) and mean square error loss (for regression tasks);

[0099] 4. Optimization algorithm: You can choose a suitable optimization algorithm to update the model parameters so that the loss function is gradually reduced. For example, common optimization algorithms include stochastic gradient descent and Adam (an optimization algorithm based on adaptive learning rate, which can automatically adjust the learning rate during training to improve training efficiency and stability).

[0100] 5. Hyperparameter tuning: During fine-tuning, you can also adjust various hyperparameters, such as learning rate, batch size, training rounds, etc. Appropriate hyperparameter settings can significantly improve the performance of the model;

[0101] 6. Evaluation and verification: During the fine-tuning process, you can also regularly evaluate the performance of the model on the validation set (such as accuracy, precision, and mean square error) to prevent overfitting. In other words, the data in the validation set does not participate in training and is only used to evaluate the generalization ability of the model.

[0102] It should be noted that the SFT methods of large language models mainly include the following:

[0103] 1. Full parameter fine-tuning: Full parameter fine-tuning involves adjusting all weights of the model to fully adapt it to a specific domain or task. This method is suitable for situations where there is a large amount of high-quality training data that is highly relevant to the task, and all parameters are updated to maximize the model's understanding and performance of the specific task;

[0104] 2. Sparse Fine Tuning / Selective Fine Tuning: Sparse Fine Tuning / Selective Fine Tuning: Sparse Fine Tuning only selectively updates certain parameters in the model, especially when most of the pre-training data needs to be retained. This includes the following methods:

[0105] 1. Low-Rank Adaptation (LoRA): Fine-tuning is performed by adding a low-rank matrix to the model weight matrix, which allows the model to learn new task-specific patterns while retaining most of the pre-trained knowledge, thereby reducing the risk of overfitting and improving training efficiency.

[0106] 2. Quantized Low-Rank Adaptation (QLoRA): It combines low-rank adjustment with quantization technology to achieve efficient and resource-friendly fine-tuning.

[0107] 3. Frozen supervised fine-tuning: In this fine-tuning method, the weights of some or all pre-trained models are frozen (i.e., remain unchanged and no longer trained), that is, some layers of the model (such as the last layer or some intermediate layers) or newly added additional components (such as task-specific output layers or attention mechanisms) are trained. This prevents the pre-training knowledge from being over-covered, while allowing the model to learn specific decision boundaries for new tasks.

[0108] The following is a detailed description of the process of fine-tuning the pre-trained language model using the supervised fine-tuning and full parameter fine-tuning training modes.

[0109] In some embodiments, following the above example, the training set may include N document image samples, where N is a positive integer greater than 1. The above-mentioned training of the pre-trained language model based on the training set to obtain the trained language model can be implemented in the following manner: iterate n to perform the following processing, where 1≤n≤N: based on the nth language model to be trained, standardize the text information recognized from the nth document image sample to obtain standardized text information; determine the difference between the standardized text information output by the nth language model to be trained and the standardized text information pre-labeled for the nth document image sample; perform back propagation based on the difference, and update the parameters of the nth language model to be trained during the back propagation process to obtain the nth trained language model; where, when n=1, the 1st language model to be trained is the pre-trained language model, and when 2≤n≤N, the nth language model to be trained is the language model after the n-1th training.

[0110] For example, taking the document image samples as passport image samples, assuming that the training set includes 10 passport image samples, namely passport image sample 1 to passport image sample 10, for example, these 10 passport image samples can be obtained by scanning or photographing passports of 10 different countries, then, the pre-trained large language model can be trained using passport image sample 1 obtained by photographing the passport of country 1 to obtain the large language model after the first training, then the passport image sample 2 obtained by photographing the passport of country 2 can be used to train the large language model after the first training to obtain the large language model after the second training, and so on, finally, the passport image sample 10 obtained by photographing the passport of country 10 can be used to train the large language model after the ninth training to obtain the large language model after the tenth training. In other words, the model training process can have multiple rounds, and the current round of training is performed on the basis of the previous round, so as to improve the accuracy of the trained model in standardizing the passport information.

[0111] It should be noted that the specific process of using the passport image sample 1 to train the pre-trained large language model to obtain the large language model after the first training is as follows: the passport information (i.e., text information) identified from the passport image sample 1 is standardized using the pre-trained large language model to obtain the standardized passport information output by the pre-trained large language model, and then the standardized passport information output by the pre-trained large language model and the label pre-annotated for the passport image sample 1 (for example, the standardized passport information obtained by manually standardizing the passport information in the passport image sample 1) can be substituted into the loss function to obtain the difference between the two, and then back propagation can be performed based on the difference, and all parameters of the pre-trained large language model can be updated during the back propagation process to obtain the large language model after the first training. In addition, it should be noted that the training process from the second to the tenth time is similar to the first training process, and can be implemented with reference to the first training process, and the embodiments of the present application will not be repeated here.

[0112] The following describes the process of fine-tuning the model using low-rank adaptation.

[0113] In some embodiments, the above-mentioned updating of the parameters of the language model to be trained for the nth time during the back propagation process can be achieved in the following way: obtaining two low-rank matrices, and taking the product of the two low-rank matrices as the incremental parameter matrix included in the language model to be trained for the nth time; during the back propagation process, fixing the weight matrix included in the language model to be trained for the nth time, and updating the parameters of the incremental parameter matrix.

[0114] For example, during the back propagation process, the weight matrix (assuming it is marked as W0) included in the language model to be trained for the nth time can be frozen, that is, the parameters of the weight matrix are no longer subject to further adjustment, and at the same time, the trainable low-rank decomposition matrices A and B can be injected into the specific layer (such as the Transformer layer) of the language model to be trained for the nth time. These two matrices simulate the incremental parameter matrix (assuming it is marked as ΔW) in the full parameter fine-tuning through multiplication operations, thereby realizing the adaptation of the model to specific tasks. Specifically, assuming that the weight matrix included in the language model to be trained for the nth time is W0 and the incremental parameter matrix is ​​ΔW, ΔW can be expressed as the product of two low-rank matrices B and A, that is, ΔW≈BA, where the dimension of B is d×r, and the dimension of A is r×d, and r is much smaller than d. In this way, the large number of parameters that originally need to be adjusted (i.e., d×d) are simplified to the parameters of two smaller matrices (i.e., 2×r×d), thereby significantly reducing the computational complexity and storage requirements of fine-tuning. In other words, the trained A and B matrices are used to replace the corresponding parameters of the original pre-trained model, avoiding the extra computation and inference delay caused by increasing the network depth. In addition, by modifying the model's weight matrix, the internal representation and processing mechanism of the model are directly affected, which can have an impact at a deeper level and improve the learning and adaptability of the model. In addition, low-rank matrices can be introduced in different layers and model components, and fine-tuned according to specific tasks, providing greater flexibility.

[0115] The following will continue to explain the process of training the model using the training mode of fine-tuning some parameters.

[0116] In other embodiments, the above-mentioned updating of the parameters of the language model to be trained for the nth time during the back propagation process can also be achieved in the following way: during the back propagation process, some parameters of the language model to be trained for the nth time are updated, wherein the some parameters include weights, biases, and parameters that can control the behavior of the model.

[0117] For example, during the back propagation process, the performance of the model can be optimized by adjusting some parameters of the language model to be trained for the nth time. These parameters may include the weights and biases of the language model to be trained for the nth time, as well as other parameters that can control the behavior of the model. That is, during the back propagation process, only the weights, biases, and other parameters that can control the behavior of the language model to be trained for the nth time can be updated based on the error, which can reduce the training cost.

[0118] It should be noted that weights are one of the important parameters of the language model. They exist at each connection point of the model and are used to control how the input data affects the output of the model. Specifically, weights determine the extent to which the input data should be amplified or reduced before being passed to the next layer. Bias allows each neuron output to still have a non-zero value when there is no input or the input is zero. Bias can be seen as a "built-in threshold" for each neuron, which adjusts the activation threshold of the activation function and helps the model learn data patterns that do not only depend on the product of the input data weights. Weights and biases are updated during the training process of the model through the back propagation algorithm. At the beginning of training, weights and biases are usually initialized to small random values ​​(to avoid symmetry and zero gradient problems). Then, through forward propagation, the model predicts the input data and calculates the error between the predicted results and the true results. The back propagation process uses this error to calculate the gradient of the loss function relative to each weight and bias, that is, to find out how these parameters need to be changed to reduce the total error. After training is completed, the weights and biases of the model will be saved in the weight file for subsequent use and deployment of the model.

[0119] In some embodiments, continuing with the above example, when the trained language model is verified based on the verification set, the following processing can also be performed: anomaly detection is performed on multiple document image samples included in the verification set to obtain detection results; in response to the detection results indicating that there are abnormal document image samples in the verification set (for example, document image samples with stains, scratches, or handwritten marks), the abnormal document image samples are deleted from the verification set.

[0120] For example, taking the document image sample as a passport image sample, in the process of verifying the trained language model based on the data in the validation set (for example, evaluating the accuracy and response speed of the trained language model in actual applications), the noise data that may exist in the validation set can also be identified and processed. For example, through statistical analysis and anomaly detection algorithms, noise samples with incorrect labels, abnormal data, or abnormal model responses in the validation set can be identified, such as passport image samples with stains, scratches, or handwritten marks. The detected noise data can then be removed or necessary cleaning processing can be performed to prevent misleading the model.

[0121] In other embodiments, continuing with the above example, after the trained language model is verified based on the verification set, the following processing can also be performed: difficult samples in the verification set are obtained, wherein the difficult samples refer to document image samples that fail to be standardized by the trained language model. The above-mentioned response to the verification result indicating that the accuracy of the trained language model reaches the accuracy threshold, using the trained language model as the language model for standardization processing can be achieved in the following ways: in response to the verification result indicating that the accuracy of the trained language model reaches the accuracy threshold, the trained language model is re-trained based on the difficult samples to obtain a re-trained language model; and the re-trained language model is used as the language model for standardization processing.

[0122] For example, taking the document image samples as passport image samples, after verifying the trained language model based on the validation set, the passport image samples in the validation set that failed the standardization process can also be marked and sent for manual annotation and further analysis. Subsequently, these samples can be put back into the training set as difficult samples for subsequent training of the model. In other words, in the next training cycle, the focus is on training these difficult samples to enhance the model's ability to handle a variety of different situations, thereby further improving accuracy.

[0123] In some other embodiments, the above-mentioned language model may include a first language model and a second language model, wherein the first language model (e.g., an online real-time link model) is a language model currently running online, and the second language model (e.g., a dynamic fine-tuning model) is a language model to be launched online and continuously optimized based on the latest training data. The above-mentioned standardization of text information through language models can be achieved in the following manner: standardization of text information through the first language model. Subsequently, the following processing can also be performed: the accuracy of the first language model and the accuracy of the second language model are evaluated respectively; in response to the accuracy of the second language model being greater than the accuracy of the first language model, the first language model is replaced with the second language model.

[0124] For example, taking the first language model as an online real-time link model and the second language model as a dynamic fine-tuning model as an example, in the process of model development and deployment, in order to ensure that the dynamic fine-tuning model can effectively improve the performance of the online real-time link model, and at the same time avoid the degradation of the model capability caused by data pollution and other problems during the fine-tuning process, the embodiment of the present application designs a dual-run verification process. Specifically, the model can be divided into two categories, namely, the online real-time link model and the dynamic fine-tuning model, wherein the online real-time link model is the baseline version model currently running online (i.e., the model currently used to perform standardized processing), which can directly affect the processing results of real-time business; and the dynamic fine-tuning model is a large language model that is continuously optimized and fine-tuned based on the latest training data. It has not yet been launched and needs to undergo rigorous evaluation before it can be adopted. In the dual-run verification process, the two models can be processed using the same data source (i.e., the text information that needs to be standardized by the two models is identified from the same batch of certificate images) to ensure consistency. Through an asynchronous batch run script, batch data processing is performed on the online real-time link model and the dynamic fine-tuning model respectively, so as to compare the output results and corresponding accuracy indicators of the two models. Only when the accuracy of the dynamic fine-tuning model reaches the predetermined standard and remains stable compared to the accuracy of the online real-time link model, can the performance of the dynamic fine-tuning model be considered to meet the online conditions. At this time, the dynamic fine-tuning model can be used to replace the online real-time link model, that is, the dynamic fine-tuning model can be deployed to the real-time link to replace the original online real-time link model. After the dynamic fine-tuning model is deployed to the real-time link, that is, after the dynamic fine-tuning model is online, the performance of the dynamic fine-tuning model can be continuously tested, and dynamic fine-tuning and dual-run verification can be performed regularly to ensure that the performance of the model deployed in the real-time link is always in the optimal state. If the performance of the dynamic fine-tuning model is not as good as the online real-time link model (for example, the accuracy of the dynamic fine-tuning model is reduced due to data pollution and other reasons), the online real-time link model can be retained, and the dynamic fine-tuning model can continue to be trained and optimized until the performance of the dynamic fine-tuning model meets the online requirements to prevent affecting online business.

[0125] In some embodiments, the above-mentioned standardization processing can be implemented through a language model. When the standardization processing of text information through the language model fails, the following processing can also be performed: manually annotating the document image and re-training the language model based on the standardized document image.

[0126] For example, taking the ID image as a passport image, when the effect of the fine-tuned large language model reaches the expected goal in multiple evaluation processes, the fine-tuned large language model can be deployed to the production environment. In order to ensure the stability and continuous optimization of the model in practical applications, the embodiment of the present application can also set a detection mechanism to detect the operating status and performance of the model in real time. For example, when a new ID image that fails the standardization process is detected, the ID image that fails the standardization process can be manually annotated, and the fine-tuned large language model can be retrained based on the annotated ID image, thereby ensuring the continuous optimization of the model and being able to cope with new challenges.

[0127] In step 104, the standardized text information is subjected to field verification to obtain a verification result.

[0128] In some embodiments, see Figure 4 , Figure 4 is a second flow chart of the method for processing a certificate image provided in an embodiment of the present application, such as Figure 4 As shown, Figure 3 Step 104 shown may be performed by Figure 4 Steps 1041 and 1042 shown are implemented by combining Figure 4 The steps shown are explained.

[0129] In step 1041, a rationality check is performed on the fields included in the standardized text information to obtain a rationality check result.

[0130] Here, the rationality check may include at least one of the following: checking whether the length and character combination of the certificate number meet the preset standards, checking whether the end time of the validity period is after the current time, checking whether the date of birth is before the current time, and checking whether the name does not include invalid characters.

[0131] For example, taking the standardized text information as the standardized passport information, after the passport information identified from the passport image is standardized, the embodiment of the present application can also perform a comprehensive field check on the standardized passport information to ensure the accuracy of all information, for example, it can check whether the date is displayed in the correct format and is logically reasonable (for example, the date of birth is before the current date, the end date of the passport validity period is after the current date, etc.); check whether the length and character combination of the passport number meet the corresponding passport standards, for example, it should not contain special characters, etc.; check whether the name contains invalid characters, etc.

[0132] In step 1042, a consistency check is performed on the fields included in the standardized text information to obtain a consistency check result.

[0133] Here, the consistency check includes checking whether the field contents of corresponding fields in the machine readable zone and the visual inspection zone of the document image are consistent.

[0134] For example, taking the standardized text information as the standardized passport information, after the text information (i.e., the passport information) recognized from the passport image is standardized, it is also possible to check whether the corresponding fields of the machine readable zone and the visual readable zone have the same content, thereby ensuring the consistency of all information. That is, the embodiment of the present application can compare the information obtained from the machine readable zone of the document with the information obtained from the visual readable zone of the document, i.e., determine whether the two are consistent.

[0135] It should be noted that documents usually include a machine-readable zone and a visually readable zone, where machine-readable means that the information is designed to be automatically recognized and processed by machines (such as computers, scanners, etc.). In machine reading, information is usually encoded according to specific standards so that the machine can read it quickly and accurately. For example, documents such as passports, ID cards, and passes often contain a machine-readable zone (an area containing digital information that can be automatically read and verified by a machine. The machine-readable zone is usually located at the bottom of the information page of a passport or ID card). The machine can read the user's name, gender, date of birth, and other information from the machine-readable zone. Visual reading means that information is presented in a format that can be read by humans, usually in the form of text, images, etc., to facilitate human understanding and communication. The visual reading zone refers to the area in the document that can be read directly by the naked eye. This information is usually displayed intuitively in plain text.

[0136] In addition, it should be noted that in actual applications, step 1041 and step 1042 can be executed simultaneously or successively. For example, step 1041 can be executed first, and then step 1042. Of course, step 1042 can also be executed first, and then step 1041. The embodiment of the present application does not make specific limitations on this.

[0137] In step 105, in response to the verification result indicating that the verification is passed, the document image and the standardized text information are stored.

[0138] In some embodiments, see Figure 5 , Figure 5 is a third flow chart of the method for processing a certificate image provided in an embodiment of the present application, such as Figure 5 As shown, Figure 3 Step 105 shown may be performed by Figure 5 Steps 1051 and 1052 are shown to be implemented by combining Figure 5 The steps shown are explained.

[0139] In step 1051, in response to the verification result indicating that the verification is passed, the document image and the standardized text information are sent to the verification device of the verification agency.

[0140] In some embodiments, taking the document image as a passport image as an example, after the field verification of the standardized text information is passed, the user's passport image and the standardized text information can be sent to an external authoritative channel for further verification. These channels may include the National Immigration Bureau, the passport issuing agency or other international authoritative agencies to ensure the authenticity and legitimacy of the information. In other words, the main purpose of external verification is to prevent fraud and verify the legitimacy of the user's identity. If the external verification fails, the document image can be returned to manual annotation and review for further analysis and processing.

[0141] In other embodiments, when the verification result indicates that the verification has failed (for example, an error occurs in text recognition of the ID image), the following processing may also be performed: annotating the ID image and retraining the text recognition model used for text recognition based on the annotated ID image.

[0142] For example, if the verification result indicates a text recognition error, such as a problem caused by an OCR recognition error, the error data can be annotated and returned to the OCR stage to improve OCR technology and increase future recognition accuracy. For example, the text recognition model used in the OCR stage can be retrained based on the annotated data to improve the accuracy of text recognition.

[0143] In some embodiments, when the verification result indicates that the verification has failed (for example, an error occurs in the normalization processing of text information), the following processing may also be performed: the document image is used as a difficult sample, and the language model is retrained based on the difficult sample.

[0144] For example, if the problem with the verification result characterization is caused by an error in the standardization processing of text information, then this data can be returned to the large language model as a difficult sample for further training and optimization.

[0145] In step 1052, in response to receiving a notification message returned by the verification device indicating that the verification is successful, the document image and the standardized text information are stored.

[0146] In some embodiments, taking the document image as a passport image as an example, after the user's passport information passes external verification, the user's passport image and the standardized text information can be locally retained, for example, stored in a secure database and associated with the user's account opening record. In order to prevent unauthorized access, the embodiments of the present application can also ensure the secure storage of data and strict control of access rights. At the same time, the locally retained data (i.e., the user's passport image and standardized text information) can also be used for subsequent audits, user identity verification, compliance checks, and related management operations.

[0147] That is to say, through the local retention mechanism, the embodiment of the present application can retrieve the user's passport information at any time to provide support or conduct further verification when necessary. At the same time, in order to ensure that the user's data security and privacy are not violated, the embodiment of the present application can also ensure that the storage of data complies with the requirements of relevant laws and regulations, especially in terms of data privacy and information protection.

[0148] In some embodiments, after executing Figure 3 After step 105 shown, the following processing can also be performed: sending the ID image and standardized text information to the server of the payment service institution (such as a payment service provider) so that the server of the payment service institution performs the following processing: based on the ID image and the standardized text information, verifying the identity information of the object corresponding to the ID image (such as the holder of a passport); in response to the verification being successful, opening a corresponding payment account for the object.

[0149] For example, if the document image is a passport image, when a payment account needs to be opened for a user, the user's passport information and standardized text information (i.e., passport information) stored locally can be sent to the backend server of the payment service provider, so that the backend server of the payment service provider can verify the user's information based on the received passport information and standardized text information. When the verification is passed, the backend server of the payment service provider can open a corresponding payment account for the user.

[0150] The following is a description of the training method of the language model provided in the embodiment of the present application.

[0151] For example, see Figure 6 , Figure 6 is a flow chart of the language model training method provided in the embodiment of the present application, which will be combined with Figure 6 The steps shown are explained.

[0152] In step 201, training data is obtained.

[0153] In some embodiments, the training data may include certificate image samples of different formats and standardized text information annotated for the certificate image samples, wherein the standardized text information is obtained by standardizing text information identified from the certificate image samples.

[0154] For example, taking passport images as an example, during the fine-tuning (or fine-tuning) process of the model, you can first collect passport image samples from multiple countries as training data to ensure that the data set can cover passport formats of different countries and regions. Then you can manually annotate these data, for example, you can manually standardize the text information identified from the passport image samples.

[0155] In step 202, the training data is divided into a training set and a validation set.

[0156] In some embodiments, the training data may be divided into a training set and a validation set according to a preset ratio, for example, the training data may be divided into a training set and a validation set according to a ratio of 8:2, that is, 80% of the data in the training data is used as the training set, and the remaining 20% ​​of the data is used as the validation set. For example, taking the document image as a passport image as an example, assuming that the training data includes 100 pre-labeled passport image samples, 80 of the passport image samples may be divided into the training set, and the remaining 20 passport image samples may be divided into the validation set.

[0157] In step 203, the pre-trained language model is trained based on the training set to obtain a trained language model.

[0158] In some embodiments, a suitable pre-trained model (e.g., a large language model trained on a large data set) can be selected, and the model can be fine-tuned using supervised fine-tuning (i.e., SFT) and full parameter fine-tuning training modes on the training set. The fine-tuning process of the model can include multiple rounds of iterations, and in each round of iteration, the model will adjust parameters according to the performance of the previous round to improve the accuracy of the model in the passport information formatting task. The specific training process can refer to the description above, and the embodiments of the present application will not be repeated here.

[0159] In step 204, the trained language model is verified based on the verification set to obtain a verification result.

[0160] In some embodiments, the following processing can be performed for each ID image sample included in the verification set: the text information recognized from the ID image sample is standardized by the trained language model to obtain standardized text information; the standardized text information output by the trained language model is then compared with the standardized text information pre-labeled for the ID image sample to obtain a comparison result; then, based on multiple comparison results corresponding to multiple ID image samples included in the verification set, a final verification result can be determined. For example, assuming that the verification set includes 20 ID image samples, the standardized text information output by the trained language model for 15 of the ID image samples is the same as the pre-labeled standardized text information, which means that the accuracy of the trained language model is 75%.

[0161] In step 205, in response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, the trained language model is used as a language model for standardization processing.

[0162] In some embodiments, after the trained language model is verified based on the verification set and the verification result is obtained, the accuracy of the trained language model represented by the verification result can be compared with a preset accuracy threshold. When the accuracy of the trained language model is greater than or equal to the preset accuracy threshold (for example, 80%), the trained language model can be directly used as a language model for standardization processing, that is, the trained language model can be directly deployed online.

[0163] It should be noted that if the accuracy of the trained language model is less than the accuracy threshold, the trained language model may be trained again until the accuracy of the language model is greater than or equal to the accuracy threshold.

[0164] The method for processing certificate images provided in the embodiment of the present application, on the one hand, obtains text information in the certificate image by performing text recognition on the certificate image, thereby reducing the steps of manual input by users and manual review in the background, greatly improving the efficiency of information processing; on the other hand, the recognized text information is standardized by a fine-tuned large language model, thereby ensuring that the text information recognized from certificate images of different nationalities and different formats can be uniformly converted into a standard format, which facilitates subsequent verification, thereby improving the efficiency and accuracy of certificate review.

[0165] Below, taking the document image as a passport image as an example, an exemplary application of the embodiment of the present application in an actual application scenario is explained.

[0166] In the related art, passport review is generally based on manual filling by users or manual review by the backend. However, the manual filling method is cumbersome and prone to errors, resulting in inaccurate information, which in turn affects the review results; while the manual review method has high labor costs and requires a lot of manual review and information entry, which increases labor costs. At the same time, manual review and information entry are slow and take a long time to process, resulting in overall low efficiency.

[0167] In view of this, the embodiment of the present application provides a method for processing document images. First, through the automated processing of passport OCR, the steps of manual input by users and manual review in the background are reduced, greatly improving the efficiency of information processing. Secondly, by fine-tuning the large language model for specific tasks, the model can standardize the text information recognized by OCR, so as to ensure that passport information of different nationalities and different formats can be uniformly converted into a standard format, thereby solving the review problem caused by the differences in passport formats of multiple countries. In addition, the large language model can intelligently identify and convert various complex passport formats, significantly improving the accuracy of information processing, thereby reducing the error rate. In other words, through automated and intelligent processing, the technical solution provided by the embodiment of the present application greatly improves the efficiency of passport review, reduces the need for manual intervention, and thus speeds up the review.

[0168] In addition, it should be noted that the LLM solution provided by the relevant technology has a certain probability of "hallucination" problems in technical principles, and there is a certain uncertainty in data accuracy. If it is directly used in this compliance review scenario, there may be problems of audit omissions or erroneous interception. The technical solution provided in the embodiment of the present application combines the advantages of LLM and the programmatic detection method of the relevant technology, and performs a secondary verification on the standardized results, thereby avoiding the business risks that may be brought about by the model "hallucination" in advance, and at the same time does not incur a large amount of labor costs. It has the advantages of both the general iterative capability of LLM and the accuracy of the program logic in the relevant technology.

[0169] The following uses the payment account opening scenario as an example to specifically illustrate the document image processing method provided in the embodiment of the present application.

[0170] In order to improve the convenience of foreigners coming to China for consumption, the embodiment of the present application provides a solution that supports the use of passports for third-party payment account opening. During the account opening process, due to the need for anti-money laundering duties, the solution needs to collect the user's identity proof information (i.e., the passport information submitted by the user). The payment service provider can verify the user's information, retain it, and open a payment account based on the passport information submitted by the user. However, since the passport formats of different nationalities (for example, including field formats, field positions, languages, etc.) are different, and the passport as an identity document contains multiple fields (for example, including name, date of birth, passport number, etc.), this brings challenges to information collection.

[0171] In some embodiments, see Figure 7 , Figure 7 FIG. 1 is a schematic diagram of a method for processing a certificate image provided in an embodiment of the present application. Figure 7 As shown, the method for processing certificate images provided in the embodiment of the present application mainly includes processes such as uploading a photocopy, OCR recognition, text information standardization processing, field verification, external authoritative channel verification, and local retention. Each of the above processes is described in detail below.

[0172] 1. Upload a photocopy

[0173] In some embodiments, when opening an account, the user needs to upload a photocopy of his / her passport (ie, passport image).

[0174] 2. OCR Recognition

[0175] In some embodiments, after receiving the passport image uploaded by the user, the server may use OCR technology to extract relevant text information from the image. Since the passport text content may include multiple languages ​​and different formats, the text information recognized from the passport image using OCR technology may include but is not limited to the following:

[0176] Passport number: the number that uniquely identifies your passport;

[0177] Name: the full name of the licensee, including the first and last name, and sometimes the middle name or name prefix. The name field may be different in different countries and formats;

[0178] Gender: Usually marked as "M" (male) or "F" (female), although some countries may have other marking methods;

[0179] Date of birth: the date of birth of the license holder;

[0180] Nationality: the nationality of the licensee, which may be indicated by the full country name or the three-letter code of the ISO 3166-1 standard;

[0181] Passport issue date / expiration date: This indicates the legal period of use of the passport.

[0182] It should be noted that passports issued by some countries may also have other fields such as former names, parents' names, etc.

[0183] 3. Standardization of text information

[0184] In some embodiments, after extracting relevant text information from the passport image using OCR technology, the text information identified using OCR technology can be standardized using a fine-tuned large language model. This process is intended to unify text information in different languages ​​and formats into a standard, easy-to-process format. In addition, the large language model has been fine-tuned to recognize, understand and process multiple languages ​​and complex text content, and can intelligently complete and correct missing or erroneous fields, thereby ensuring that passport information of different nationalities can be processed correctly.

[0185] Combine the following Figure 8 The fine-tuning process of the large language model is explained.

[0186] For example, see Figure 8 , Figure 8 is a schematic diagram of the fine-tuning process of the large language model provided in the embodiment of the present application, such as Figure 8 As shown in the figure, during the model fine-tuning process, you can first collect passport image samples from multiple countries as training data to ensure that the data set can cover the passport formats of different countries and regions, and annotate and standardize these data. Then the prepared data set can be divided into a training set and a validation set to ensure that the model can be effectively verified and tuned during the training process. Next, you can select a suitable pre-trained model and fine-tune the model using the SFT and full parameter fine-tuning training modes on the training set. The fine-tuning process requires multiple rounds of iterations. In each round of iteration, the model parameters can be adjusted according to the performance of the previous round to improve its accuracy in the passport information formatting task.

[0187] After fine-tuning is completed, the model can be evaluated in batches by calling the API interface to test its performance on the validation set. For example, a large amount of input data can be obtained in batches from an actual production environment or a simulated environment through the API interface, and the data can be input into the model through the API interface. The system will automatically record the output results of the model and the corresponding performance indicators such as processing time and resource consumption. Batch evaluation can effectively test the accuracy and response speed of the model in practical applications. In addition, during the evaluation process, the system can also identify noise samples with incorrect labels, abnormal data or abnormal model response. For example, when scanning a passport, there may be stains, scratches, handwritten marks, etc. on the passport, and these noises will interfere with OCR recognition, resulting in incorrect information extraction. In view of this, the embodiment of the present application can send the detected noise samples to the manual annotation team for review and verification to decide whether to include or exclude them in the training data set. The noise samples confirmed by manual annotation will be eliminated or necessary cleaning will be performed to prevent misleading the model.

[0188] Continue to see Figure 8 After completing the model evaluation and denoising sample processing, the samples that failed the standardization process can also be marked and sent for manual annotation and further analysis. These samples can then be put back into the training dataset as difficult samples for subsequent model training. In other words, in the next training cycle, the focus is on training these difficult samples to enhance the model's ability to handle a variety of different situations, thereby improving accuracy.

[0189] It should be noted that when the effect of the model reaches the expected goal in multiple evaluation processes, the fine-tuned model can be deployed to the production environment. In order to ensure the stability and continuous optimization of the model in practical applications, the embodiment of the present application can also set a detection mechanism to detect the operating status and performance of the model in real time. If new standardized failure samples are detected, or the model performance begins to decline, these data will be used again for online learning and retraining, thereby ensuring the continuous optimization of the model and being able to cope with new challenges.

[0190] 4. Field Verification

[0191] In some embodiments, after the text information is standardized using the fine-tuned large language model, the embodiments of the present application may also perform a comprehensive field check on the standardized text information to ensure the accuracy and consistency of all information, wherein the field check can be divided into the following two main aspects:

[0192] Reasonableness check: Check whether the field content conforms to the expected format and range. For example, ensure that the date is displayed in the correct format and is logically reasonable (for example, the date of birth is before the current date, the passport validity period is in the future, etc.); the passport number length and character combination should conform to the passport standards of various countries and should not contain special characters; the name should not contain invalid characters, etc.

[0193] Consistency check: This ensures the consistency of information between different fields. For example, check whether the corresponding fields in the MRZ and VIZ have the same content.

[0194] It should be noted that if the field content is found to be unreasonable or inconsistent, it means that the verification fails. At this time, the user information can be transferred to manual review. Possible causes of error include:

[0195] OCR recognition errors: If the problem is due to OCR recognition errors, these errors are annotated and fed back to the OCR process to improve OCR technology and improve future recognition accuracy;

[0196] Errors in text information standardization: If the problem stems from errors in text information standardization, these data are returned to the large language model as difficult samples for further training and optimization.

[0197] 5. Verification by external authoritative channels

[0198] In some embodiments, after the field verification is passed, the user's passport image and standardized text information can be sent to external authoritative channels for verification. These channels may include national immigration bureaus, passport issuing agencies, or other international authoritative agencies to ensure the authenticity and legitimacy of the information. The main purpose of external verification is to prevent fraud and verify the legitimacy of the user's identity. If the external verification fails, the data can be returned to manual annotation and review for further analysis and processing.

[0199] 6. Local Retention

[0200] In some embodiments, after the user information passes external verification, the user's passport image and standardized text information can be locally retained, for example, in a secure database and associated with the user's account opening record. In addition, the secure storage of data and strict control of access rights can be ensured to prevent unauthorized access. At the same time, the locally retained data can be used for subsequent audits, user identity verification, compliance checks, and related management operations.

[0201] That is to say, the embodiment of the present application can retrieve the user's passport information at any time through the local retention mechanism, so as to provide support or conduct further verification when necessary. At the same time, the storage of data needs to comply with the requirements of relevant laws and regulations, especially in terms of data privacy and information protection, so as to ensure the security and privacy of user data.

[0202] In other embodiments, during the development and deployment of the model, in order to ensure that the dynamic fine-tuning model can effectively improve the performance of the online real-time link model, while avoiding the degradation of the model's capabilities due to problems such as data pollution during the fine-tuning process, the embodiments of the present application also provide a dual-run verification process.

[0203] For example, see Fig. 9 , Fig. 9 is a schematic diagram of the dual-run verification process provided by the embodiment of the present application, such as Fig. 9 As shown in the figure, first of all, the models can be divided into two categories: online real-time link model and dynamic fine-tuning model. Among them, the online real-time link model is the baseline version model currently running online, which can directly affect the processing results of real-time business. The dynamic fine-tuning model is a large language model that is continuously optimized and fine-tuned based on the latest training data. It has not yet been launched and needs to undergo rigorous evaluation before it can be adopted.

[0204] During the dual-run verification process, the two models will be processed using the same data source to ensure consistency. Through an asynchronous batch run script, batch data processing is performed on the online real-time link model and the dynamic fine-tuning model respectively, and the output results and corresponding accuracy indicators of the two models are compared. Only when the accuracy of the dynamic fine-tuning model reaches the predetermined standard and remains stable compared with the accuracy of the online real-time link model, the performance of the dynamic fine-tuning model can be considered to meet the online conditions. At this time, the dynamic fine-tuning model can be synchronized to a new baseline version and deployed to the real-time link to replace the original model (i.e., the online real-time link model). After the new model is launched, its performance can be continuously tested, and dynamic fine-tuning and dual-run verification can be performed regularly to ensure that the performance of the model is always in the best state. If the performance of the dynamic fine-tuning model is not as good as the baseline version (for example, due to data pollution and other reasons, the accuracy is reduced), the existing baseline version is retained, and the dynamic fine-tuning model is continuously trained and optimized until it meets the online requirements to prevent affecting online business.

[0205] It should be noted that, based on the passport information processing, the embodiment of the present application can also extend the application of the large language model to the information processing and format unification of other types of certificates (such as identity cards, Hong Kong and Macao passes, etc.), to achieve automation and standardization of information processing of multiple certificates. In addition, in the process of model fine-tuning, different fine-tuning methods can also be used, for example, LoRA or parameter fine-tuning (P-tuning, Parameter tuning) can be used to train and optimize the model.

[0206] Among them, LoRA, or low-rank adaptation, is a parameter-efficient fine-tuning method. Its core idea is to simulate the process of full parameter fine-tuning by introducing two low-rank matrices A and B, so as to reduce the required computing resources and the amount of fine-tuning parameters while maintaining the complexity and expressiveness of the pre-trained model. In LoRA, the weight matrix of the pre-trained model is frozen and no longer accepts further adjustments. Instead, trainable low-rank decomposition matrices A and B are injected into specific layers of the model (such as the Transformer layer). These two matrices simulate the incremental parameter matrix ΔW in full parameter fine-tuning through multiplication operations, thereby achieving the adaptation of the model to specific tasks. Specifically, assuming that the weight matrix of the pre-trained model is W0 and the incremental parameter matrix is ​​ΔW, LoRA represents ΔW as the product of two low-rank matrices B and A, that is, ΔW≈BA, where the dimension of B is d×r and the dimension of A is r×d, and r is much smaller than d. In this way, the large number of parameters (d×d) that originally needed to be adjusted are simplified to the parameters of two smaller matrices (i.e., 2×r×d), which significantly reduces the computational complexity and storage requirements of fine-tuning.

[0207] Among them, P-tuning is a technology widely used in the field of machine learning to improve model performance. Its core idea is to optimize the performance of the model by adjusting some parameters of the model during the training process. These parameters usually include the weights and biases of the neural network, as well as other parameters that control the behavior of the model. During the tuning process, it is necessary to select an optimal set of parameters so that the model has the smallest error on the test data. In practical applications, the process of selecting the best parameters is usually carried out through cross-validation. This method divides the data set into multiple parts, uses one part of the data for training, and uses another part of the data for validation. Then, the model parameters are adjusted so that the model performs best on the validation data. This process can be repeated many times to determine the best combination of parameters. When selecting the best parameters, some evaluation functions are usually used to evaluate the performance of the model. These evaluation functions can measure the error of the model on the training data and validation data, as well as other indicators, such as accuracy, recall rate, and F1 score. By comparing the evaluation results of different parameter combinations, the best parameters can be selected.

[0208] In summary, the document image processing method provided in the embodiment of the present application has the following beneficial effects:

[0209] With the help of the general understanding and generalization ability of the large language model for information, standardized processing of multiple passport formats is achieved, avoiding the problems of easy errors and poor experience caused by allowing users to directly input information, and also avoiding the situation where a large amount of manpower is required for back-end review. In other words, the embodiment of the present application greatly reduces labor costs by introducing a large language model to process information, and provides a reliable data basis for unified verification in subsequent processes, thereby improving the efficiency and accuracy of passport information collection and review.

[0210] The following is a description of an exemplary structure of the document image processing device 543 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2A As shown, the software modules stored in the document image processing device 543 of the memory 540 may include: an acquisition module 5431, a text recognition module 5432, a standardization processing module 5433, a verification module 5434 and a storage module 5435.

[0211] The acquisition module 5431 is used to acquire the document image; the text recognition module 5432 is used to perform text recognition on the document image to obtain the text information in the document image; the standardization processing module 5433 is used to perform standardization processing on the text information to obtain standardized text information; the verification module 5434 is used to perform field verification on the standardized text information to obtain a verification result; the storage module 5435 is used to store the document image and the standardized text information in response to the verification result indicating that the verification has passed.

[0212] In some embodiments, the standardization processing module 5433 is also used to perform standardization processing on text information through a language model to obtain standardized text information, wherein the standardization processing includes unifying the text information into a preset standard format, and the standard format includes standard field names and standard data structures.

[0213] In some embodiments, the language model includes a first language model and a second language model, wherein the first language model is the language model currently running online, and the second language model is a language model to be launched online and continuously optimized based on training data; the standardization processing module 5433 is also used to standardize text information through the first language model; the document image processing device 543 also includes an evaluation module 5436 and a replacement module 5437, wherein the evaluation module 5436 is used to evaluate the accuracy of the first language model and the second language model respectively; the replacement module 5437 is used to replace the first language model with the second language model in response to the accuracy of the second language model being greater than the accuracy of the first language model.

[0214] In some embodiments, the document image processing device 543 also includes a training module 5438, which is used to perform the following processing before the standardization processing module 5433 standardizes the text information through the language model: obtaining training data, wherein the training data includes document image samples of different formats, and standardized text information annotated for the document image samples, and the standardized text information is obtained by standardizing the text information in the document image samples; dividing the training data into a training set and a verification set; training the pre-trained language model based on the training set to obtain a trained language model; verifying the trained language model based on the verification set to obtain a verification result; in response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, using the trained language model as the language model for standardization processing.

[0215] In some embodiments, the training set includes N document image samples, where N is a positive integer greater than 1; the training module 5438 is also used to iterate n to perform the following processing, where 1≤n≤N: based on the nth language model to be trained, the text information recognized from the nth document image sample is standardized to obtain standardized text information; the difference between the standardized text information output by the nth language model to be trained and the standardized text information pre-labeled for the nth document image sample is determined; back propagation is performed based on the difference, and the parameters of the nth language model to be trained are updated during the back propagation process to obtain the nth trained language model; wherein, when n=1, the first language model to be trained is a pre-trained language model, and when 2≤n≤N, the nth language model to be trained is a language model after the n-1th training.

[0216] In some embodiments, the training module 5438 is also used to obtain two low-rank matrices, and use the product of the two low-rank matrices as the incremental parameter matrix included in the language model to be trained for the nth time; during the back propagation process, the weight matrix included in the language model to be trained for the nth time is fixed, and the parameters of the incremental parameter matrix are updated.

[0217] In some embodiments, the training module 5438 is also used to update some parameters of the language model to be trained for the nth time during the back propagation process, wherein the some parameters include weights, biases, and parameters that can control the behavior of the model.

[0218] In some embodiments, the document image processing device 543 also includes a detection module 5439 and a deletion module 54310, wherein the detection module 5439 is used to perform anomaly detection on multiple document image samples included in the verification set to obtain a detection result when the training module 5438 verifies the trained language model based on the verification set; the deletion module 54310 is used to delete the abnormal document image samples from the verification set in response to the detection result indicating that there are abnormal document image samples in the verification set.

[0219] In some embodiments, the acquisition module 5431 is further used to acquire difficult samples in the verification set after the training module 5438 verifies the trained language model based on the verification set, wherein the difficult samples are document image samples that fail to be standardized by the trained language model; the training module 5438 is further used to re-train the trained language model based on the difficult samples in response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, to obtain a re-trained language model; and use the re-trained language model as the language model for standardization processing.

[0220] In some embodiments, the verification module 5434 is also used to perform at least one of the following processes: performing a rationality check on the fields included in the standardized text information to obtain a rationality check result, wherein the rationality check includes at least one of the following: whether the length and character combination of the certificate number meet the preset standards, whether the end time of the validity period is after the current time, whether the date of birth is before the current time, and whether the name does not include invalid characters; performing a consistency check on the fields included in the standardized text information to obtain a consistency check result, wherein the consistency check includes checking whether the field contents of corresponding fields in the machine-readable zone and the visually readable zone are consistent.

[0221] In some embodiments, the standardization process is implemented through a language model; the training module 5438 is also used to perform field verification on the standardized text information in the verification module 5434, and after obtaining the verification result, perform at least one of the following processes: in response to the verification result characterizing a text recognition error for the ID image, annotating the ID image, and re-training the text recognition model used for text recognition based on the annotated ID image; in response to the verification result characterizing a standardization processing error for the text information, using the ID image as a difficult sample, wherein the difficult sample is used to re-train the language model.

[0222] In some embodiments, the document image processing device 543 also includes a sending module 54311, which is used to send the document image and standardized text information to the verification device of the verification agency in response to the verification result indicating that the verification is passed; the storage module 5435 is also used to store the document image and standardized text information in response to receiving a notification message indicating that the verification is passed returned by the verification device.

[0223] In some embodiments, the sending module 54311 is also used to send the ID image and standardized text information to the server of the payment service agency after the storage module 5435 stores the ID image and standardized text information, wherein the ID image and standardized text information are used for the server to perform the following processing: based on the ID image and the standardized text information, verifying the identity information of the object corresponding to the ID image; in response to the verification being passed, opening a corresponding payment account for the object.

[0224] In some embodiments, the standardization process is implemented through a language model; the training module 5438 is also used to manually annotate the document image in response to the failure of standardization of text information through the language model, and re-train the language model based on the annotated document image.

[0225] The following further describes the exemplary structure of the language model training device 643 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the language model training device 643 of the memory 640 may include: an acquisition module 6431, a division module 6432, a training module 6433, a verification module 6434 and a determination module 6435.

[0226] An acquisition module 6431 is used to acquire training data, wherein the training data includes document image samples of different formats and standardized text information annotated for the document image samples, wherein the standardized text information is obtained by standardizing the text information in the document image samples; a division module 6432 is used to divide the training data into a training set and a verification set; a training module 6433 is used to train a pre-trained language model based on the training set to obtain a trained language model; a verification module 6434 is used to verify the trained language model based on the verification set to obtain a verification result; a determination module 6435 is used to use the trained language model as a language model for standardization processing in response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold.

[0227] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 , Figure 4 , Figure 5 ,or Figure 6 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0228] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the computer device executes the document image processing method or language model training method described in the embodiment of the present application.

[0229] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the document image processing method provided by the present application embodiment, for example, Figure 3 , Figure 4 ,or Figure 5 The method for processing the document image shown, or Figure 6 The training method of the language model is shown.

[0230] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0231] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0232] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0233] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for processing a document image, characterized in that: The method comprises: Get the document image; Performing text recognition on the certificate image to obtain text information in the certificate image; Performing standardization processing on the text information to obtain standardized text information; Performing field verification on the standardized text information to obtain a verification result; In response to the verification result indicating that the verification is passed, the document image and the standardized text information are stored.

2. The method according to claim 1, characterized in that The step of performing standardization processing on the text information to obtain standardized text information includes: The text information is standardized by a language model to obtain standardized text information, wherein the standardization includes unifying the text information into a preset standard format, and the standard format includes a standard field name and a standard data structure.

3. The method according to claim 2, characterized in that The method further comprises: In response to failure of standardization processing of the text information by the language model, the document image is manually annotated, and the language model is retrained based on the annotated document image.

4. The method according to claim 2, characterized in that: After performing field verification on the standardized text information to obtain a verification result, the method further includes: Perform at least one of the following: In response to the verification result indicating a text recognition error for the document image, annotating the document image, and retraining a text recognition model used for the text recognition based on the annotated document image; In response to the verification result indicating a standardization processing error for the text information, the document image is used as a difficult sample, wherein the difficult sample is used to retrain the language model.

5. The method according to claim 2, characterized in that: The language model includes a first language model and a second language model, wherein the first language model is a language model currently running online, and the second language model is a language model to be launched online and continuously optimized based on training data; The standardizing of the text information by using a language model includes: Performing standardization processing on the text information by using the first language model; The method further comprises: Evaluating the accuracy of the first language model and the second language model respectively; In response to the accuracy of the second language model being greater than the accuracy of the first language model, the first language model is replaced by the second language model.

6. The method according to claim 2, characterized in that Before the text information is standardized by the language model, the method further includes: Acquire training data, wherein the training data includes certificate image samples of different formats and standardized text information annotated for the certificate image samples, wherein the standardized text information is obtained by standardizing the text information in the certificate image samples; Dividing the training data into a training set and a validation set; Training the pre-trained language model based on the training set to obtain a trained language model; Verifying the trained language model based on the verification set to obtain a verification result; In response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, the trained language model is used as a language model for performing the standardization process.

7. The method according to claim 6, characterized in that The training set includes N document image samples, where N is a positive integer greater than 1; The step of training the pre-trained language model based on the training set to obtain a trained language model includes: Iteration n performs the following processing, where 1≤n≤N: Based on the nth language model to be trained, standardizing the text information recognized from the nth document image sample to obtain standardized text information; Determining a difference between the standardized text information output by the nth language model to be trained and the standardized text information pre-labeled for the nth document image sample; Perform back propagation based on the difference, and update the parameters of the language model to be trained for the nth time during the back propagation process to obtain the language model after the nth training; When n=1, the language model to be trained for the first time is a pre-trained language model, and when 2≤n≤N, the language model to be trained for the nth time is a language model trained for the n-1th time.

8. The method according to claim 7, characterized in that The updating of the parameters of the language model to be trained for the nth time during the back propagation process includes: Obtain two low-rank matrices, and use the product of the two low-rank matrices as the incremental parameter matrix included in the language model to be trained for the nth time; During the back propagation process, the weight matrix included in the language model to be trained for the nth time is fixed, and the parameters of the incremental parameter matrix are updated.

9. The method according to claim 7, characterized in that: The updating of the parameters of the language model to be trained for the nth time during the back propagation process includes: During the back propagation process, some parameters of the language model to be trained for the nth time are updated, wherein the some parameters include weights, biases, and parameters capable of controlling the behavior of the model.

10. The method according to claim 6, characterized in that When verifying the trained language model based on the verification set, the method further includes: Performing anomaly detection on the plurality of certificate image samples included in the verification set to obtain a detection result; In response to the detection result indicating that there are abnormal document image samples in the verification set, the abnormal document image samples are deleted from the verification set.

11. The method according to claim 6, characterized in that After verifying the trained language model based on the verification set, the method further includes: Obtaining difficult samples in the verification set, wherein the difficult samples are document image samples that fail to be standardized by the trained language model; In response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, using the trained language model as a language model for performing the standardization process includes: In response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, retraining the trained language model based on the difficult sample to obtain a retrained language model; The retrained language model is used as the language model for performing the standardization process.

12. The method according to any one of claims 1 to 11, characterized in that: The performing field verification on the standardized text information to obtain a verification result includes: Perform at least one of the following: Performing a rationality check on the fields included in the standardized text information to obtain a rationality check result, wherein the rationality check includes at least one of the following: whether the length and character combination of the certificate number meet the preset standards, whether the end time of the validity period is after the current time, whether the date of birth is before the current time, and whether the name does not include invalid characters; A consistency check is performed on the fields included in the standardized text information to obtain a consistency check result, wherein the consistency check includes checking whether the field contents of corresponding fields in the machine readable zone and the visual readable zone are consistent.

13. The method according to any one of claims 1 to 11, characterized in that In response to the verification result indicating that the verification is passed, storing the document image and the standardized text information includes: In response to the verification result indicating that the verification is passed, sending the document image and the standardized text information to a verification device of a verification agency; In response to receiving a notification message returned by the verification device indicating that the verification is passed, the document image and the standardized text information are stored.

14. The method according to any one of claims 1 to 11, characterized in that After storing the document image and the standardized text information, the method further includes: The certificate image and the standardized text information are sent to a server of a payment service agency, wherein the certificate image and the standardized text information are used by the server to perform the following processing: based on the certificate image and the standardized text information, verifying the identity information of the object corresponding to the certificate image; in response to passing the verification, opening a corresponding payment account for the object.

15. A method for training a language model, characterized in that: The method comprises: Acquire training data, wherein the training data includes certificate image samples of different formats and standardized text information annotated for the certificate image samples, wherein the standardized text information is obtained by standardizing the text information in the certificate image samples; Dividing the training data into a training set and a validation set; Training the pre-trained language model based on the training set to obtain a trained language model; Verifying the trained language model based on the verification set to obtain a verification result; In response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold, the trained language model is used as a language model for standardization processing.

16. A document image processing device, characterized in that: The device comprises: An acquisition module, used to acquire a document image; A text recognition module, used to perform text recognition on the document image to obtain text information in the document image; A standardization processing module, used to perform standardization processing on the text information to obtain standardized text information; A verification module, used to perform field verification on the standardized text information to obtain a verification result; The storage module is used for storing the document image and the standardized text information in response to the verification result indicating that the verification is passed.

17. A language model training device, characterized in that: The device comprises: An acquisition module, used to acquire training data, wherein the training data includes certificate image samples of different formats and standardized text information annotated for the certificate image samples, wherein the standardized text information is obtained by standardizing the text information in the certificate image samples; A partitioning module, used for partitioning the training data into a training set and a validation set; A training module, used to train the pre-trained language model based on the training set to obtain a trained language model; A verification module, used to verify the trained language model based on the verification set to obtain a verification result; The determination module is configured to use the trained language model as a language model for standardization processing in response to the verification result indicating that the accuracy of the trained language model reaches an accuracy threshold.

18. An electronic device, characterized in that: include: A memory for storing computer executable instructions; A processor, configured to implement the document image processing method described in any one of claims 1 to 14, or the language model training method described in claim 15, when executing the computer executable instructions stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the document image processing method described in any one of claims 1 to 14 or the language model training method described in claim 15 is implemented.

20. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instructions are executed by a processor, the document image processing method described in any one of claims 1 to 14 or the language model training method described in claim 15 is implemented.

Images