Certificate identification model training method and device, storage medium and electronic equipment

Through the unified certificate identification model, the encoder, decoder and task-guided interpreter are used to solve the problems of large number of certificate identification models and high maintenance costs in the existing technology, and the high accuracy and low cost certificate identification effect is achieved.

CN119007218APending Publication Date: 2024-11-22ANT BLOCKCHAIN TECHNOLOGY (SHANGHAI) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410884183.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-18
Filing Date
2024-07-02
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing certificate identification technology requires hundreds of independent models to support various identification tasks of hundreds of certificates around the world, resulting in high maintenance and deployment costs and difficulty in uniform handling of different certificate identification tasks.

Method used

A unified document recognition model centered on document image is proposed. Through the combination of encoder, decoder and task-guided interpreter, self-supervised training and multi-modal converter, different document recognition tasks are uniformly processed to improve identification accuracy And reduce maintenance costs.

Benefits of technology

It realizes unified processing of different document identification tasks, improves recognition accuracy, reduces maintenance costs, and can flexibly adapt to the output of different document identification tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007218A_ABST
    Figure CN119007218A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a certificate identification model training method and device, a storage medium and electronic equipment, and the method comprises the steps: firstly obtaining a certificate image training sample, determining at least one certificate identification task associated with the certificate image training sample, and obtaining a task query vector corresponding to the at least one certificate identification task, inputting the certificate image training sample into the encoder for feature extraction to obtain image feature information, inputting the task query vector and the image feature information into the decoder to obtain task feature information, inputting the task feature information into the task guide interpreter, and outputting the task guide interpreter. And obtaining at least one output result output by the task guidance interpreter according to the task guidance interpreter, and training the certificate identification model by using a loss function according to the at least one output result to obtain a trained certificate identification model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a document recognition model training method, device, storage medium and electronic equipment. Background Art

[0002] There are many types of tasks in the document recognition process. Usually, there are hundreds of models supporting dozens of downstream tasks such as quality, compliance, and anti-counterfeiting of more than 100 types of documents worldwide, involving classification, regression, segmentation, etc. Due to the large number of models, the maintenance and deployment costs are very high. Summary of the invention

[0003] The embodiment of this specification provides a document recognition model training method, the document recognition model trained by the method is a unified model centered on the document image, and by unifying different types of tasks into the same model, the document recognition accuracy can be improved and the maintenance cost can be reduced. The document recognition model includes an encoder, a decoder and a task guidance interpreter, and the method includes:

[0004] Obtain document image training samples;

[0005] Determining at least one document recognition task associated with the document image training sample, and obtaining a task query vector corresponding to the at least one document recognition task;

[0006] Inputting the certificate image training sample into the encoder for feature extraction to obtain image feature information output by the encoder;

[0007] Inputting the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder;

[0008] Inputting the task feature information into the task guide interpreter to obtain at least one output result output by the task guide interpreter;

[0009] According to the at least one output result, the document recognition model is trained using a loss function to obtain a trained document recognition model.

[0010] Further, in some embodiments, the encoder is trained, and the method further comprises:

[0011] The encoder is pre-trained based on a self-supervised training method to obtain the pre-trained encoder.

[0012] Further, in some implementations, pre-training the encoder based on a self-supervised training method to obtain a pre-trained encoder includes:

[0013] Obtain pre-training document image samples;

[0014] Dividing the pre-trained document image sample into a plurality of non-overlapping sub-images, randomly determining at least one sub-image from the plurality of sub-images, and performing mask processing on the at least one sub-image;

[0015] Inputting one or more unmasked sub-images into the encoder to obtain a corresponding first sub-image feature sequence;

[0016] inserting the at least one masked sub-image into the first sub-image feature sequence according to the order of the at least one masked sub-image in the plurality of sub-images to obtain a corresponding second sub-image feature sequence;

[0017] The second sub-image feature sequence is input into a corresponding decoder for reconstruction to obtain a pre-trained encoder.

[0018] Furthermore, in some implementations, determining at least one document recognition task associated with the document image training sample includes:

[0019] The document image training sample is input into a trained document task classification model to obtain at least one document recognition task output by the document task classification model.

[0020] Furthermore, in some implementations, determining at least one document recognition task associated with the document image training sample includes:

[0021] At least one certificate recognition task associated with the certificate image training sample is determined according to the certificate type corresponding to the certificate image training sample.

[0022] Furthermore, in some implementations, obtaining the task query vector corresponding to the at least one document recognition task includes:

[0023] Obtaining a task query text corresponding to at least one document recognition task associated with the document image training sample;

[0024] For each of the at least one document recognition tasks, the task query text corresponding to the document recognition task and the associated information of the document image training sample about the document recognition task are input into the multimodal converter to obtain the task query vector corresponding to the document recognition task output by the multimodal converter; wherein the associated information includes at least one of associated image information, user attribute information, and user behavior information.

[0025] Furthermore, in some implementations, the associated information includes associated image information, and the method further includes:

[0026] For each certificate recognition task, the associated image information of the certificate image training sample with respect to the certificate recognition task is determined according to the task property of the certificate recognition task.

[0027] Furthermore, in some implementations, for each of the certificate recognition tasks, determining the associated image information of the certificate image training sample with respect to the certificate recognition task according to the task nature of the certificate recognition task includes:

[0028] For each of the certificate recognition tasks, if the task nature of the certificate recognition task is multi-frame, a flash frame image corresponding to the certificate image training sample is obtained, and the flash frame image is used as the associated image information.

[0029] In some implementations, for each of the certificate recognition tasks, determining, according to the nature of the certificate recognition task, the associated image information of the certificate image training sample with respect to the certificate recognition task includes:

[0030] For each of the document recognition tasks, if the task nature of the document recognition task is multi-frame, at least one angle image corresponding to the document image training sample is obtained, and the at least one angle image is used as the associated image information.

[0031] Furthermore, in some implementations, for each of the certificate recognition tasks, determining the associated image information of the certificate image training sample with respect to the certificate recognition task according to the task nature of the certificate recognition task includes:

[0032] For each certificate recognition task, if the task nature of the certificate recognition task is single frame, the associated image information of the certificate image training sample with respect to the certificate recognition task is set to empty.

[0033] Furthermore, in some embodiments, the task guidance interpreter includes a plurality of output units, and the output result corresponding to each certificate recognition task is output from the output unit corresponding to the certificate recognition task.

[0034] Furthermore, in some embodiments, the task guidance interpreter also includes a door control unit and a task guidance unit, the task guidance unit is used to control the opening and closing of the door control unit according to the task feature information, and the door control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the document recognition task.

[0035] Furthermore, in some embodiments, each output unit corresponds to a task type, and the gate control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the task type of the document recognition task. Further, in some embodiments, the task types include classification tasks, regression tasks, segmentation tasks, and detection tasks.

[0036] The embodiment of this specification also provides a document recognition method, including:

[0037] Inputting a target document image and a task query vector corresponding to at least one document recognition task associated with the target document image into a document recognition model, wherein the document recognition model is trained based on the document recognition model training method provided in an embodiment of this specification;

[0038] Obtaining at least one output result of the target document image output by the document recognition model with respect to the at least one document recognition task;

[0039] The document recognition result corresponding to the target document image is determined according to the at least one output result.

[0040] The embodiment of this specification also provides a document recognition model training device, the document recognition model includes an encoder, a decoder and a task guidance interpreter, and the device includes:

[0041] An acquisition module configured to acquire a training sample of a certificate image;

[0042] A determination module configured to determine at least one document recognition task associated with the document image training sample, and obtain a task query vector corresponding to the at least one document recognition task;

[0043] An encoding module configured to input the certificate image training sample into the encoder for feature extraction to obtain image feature information output by the encoder;

[0044] A decoding module, configured to input the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder;

[0045] an output module, configured to input the task feature information into the task guide interpreter to obtain at least one output result output by the task guide interpreter;

[0046] The training module is configured to train the document recognition model using a loss function according to the at least one output result to obtain a trained document recognition model.

[0047] The embodiment of this specification also provides a document recognition device, including:

[0048] An input module configured to input a target document image and a task query vector corresponding to at least one document recognition task associated with the target document image into a document recognition model, wherein the document recognition model is trained based on the document recognition model training method provided in an embodiment of this specification;

[0049] An output module, configured to obtain at least one output result of the target document image output by the document recognition model with respect to the at least one document recognition task;

[0050] The determination module is configured to determine the document recognition result corresponding to the target document image according to the at least one output result.

[0051] The embodiments of the present specification also provide a storage medium on which a computer program is stored, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0052] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0053] The embodiments of the present specification also provide a computer program product, wherein the computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the above method steps.

[0054] In an embodiment of the present specification, by obtaining a document image training sample and at least one document recognition task associated therewith, and obtaining a task query vector corresponding to the at least one document recognition task, the document image training sample is then input into the encoder for feature extraction to obtain image feature information output by the encoder, the task query vector and the image feature information are input into the decoder to obtain task feature information output by the decoder, the task feature information is input into the task guidance interpreter to obtain at least one output result output by the task guidance interpreter, and then according to the at least one output result, the document recognition model is trained using a loss function to obtain a trained document recognition model, thereby obtaining a unified document recognition model centered on the document image, which can be used to process different document recognition tasks, adapt to different document recognition task outputs, and can improve recognition accuracy with low maintenance costs; for different document recognition tasks, flexible task query vectors can be designed to learn corresponding knowledge, thereby the document recognition model in the embodiment of the present specification has good performance and potential in prompt training. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0056] Figure 1 A flowchart of a document recognition model training method provided in an embodiment of this specification;

[0057] Figure 2 A flowchart of a document recognition model training method provided in an embodiment of this specification;

[0058] Figure 3 A flowchart of a document recognition method provided in an embodiment of this specification;

[0059] Figure 4 A schematic diagram of the structural design of a certificate recognition model provided in the embodiments of this specification;

[0060] Figure 5 A schematic diagram of a multi-modal input provided in an embodiment of this specification;

[0061] Figure 6 A schematic diagram of the structure of a document recognition model training device provided in an embodiment of this specification;

[0062] Figure 7 A schematic diagram of the structure of a document recognition device provided in an embodiment of this specification;

[0063] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification.

[0064] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0065] To make the purpose, technical solution and advantages of this specification more clear, the technical solution of this specification will be clearly and completely described in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, rather than all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this specification. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart may describe each operation as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0066] Document recognition mainly refers to the identification of document information, the authenticity of documents, and other document-based recognition functions, which may appear in a variety of application scenarios, such as payment, security, etc. The document recognition process usually involves a variety of document recognition tasks (which may also be referred to as "downstream tasks" in this context), such as quality compliance, anti-counterfeiting, OCR (Optical Character Recognition) recognition, etc. Among them, quality compliance recognition tasks include but are not limited to classification, positioning, direction, corner point detection, integrity, clarity, reflection detection, signature detection and other subtasks, anti-counterfeiting recognition tasks include but are not limited to active light-based anti-material attack (such as anti-screen copying, anti-printing, anti-high imitation, etc.), information tampering anti-counterfeiting (such as anti-head portrait occlusion tampering, anti-text line tampering, anti-PS tampering, etc.) and other subtasks, OCR recognition tasks include but are not limited to text line detection, text line recognition, structuring and other subtasks. Document recognition tasks can also be divided according to task types, such as classification tasks (such as clarity, reflection, classification, direction judgment, etc. related to quality compliance recognition), regression tasks (such as angle detection), segmentation tasks (such as physical tampering, foreign object occlusion, text line detection), and detection tasks (such as face detection and signature detection). This application found that existing document recognition usually builds an independent model for each document recognition task. Therefore, document recognition requires hundreds of models to support various recognition tasks of hundreds of documents worldwide. As a result, the number of models is large and the maintenance and deployment costs are high. The purpose of this application is to provide a unified document recognition model centered on documents, so as to realize a unified model to handle different document recognition tasks.

[0067] Figure 1A flowchart of a document recognition model training method provided in an embodiment of this specification. In an embodiment of this specification, the document recognition model training method is applied to a document recognition model training device or an electronic device equipped with a document recognition model training device, or is applied to a document recognition device or an electronic device equipped with a document recognition device. In an embodiment of this specification, the document recognition model includes an encoder, a decoder, and a task-guided interpreter, the output of the encoder is input to the decoder, and the output of the decoder is input to the task-guided interpreter. The following will focus on Figure 1 The process shown is described in detail, and the document recognition model training method can specifically include the following steps:

[0068] S102, obtaining certificate image training samples.

[0069] In one or more embodiments of the present specification, the document image training samples are labeled samples. In some embodiments, for different document recognition tasks, a document image training sample set corresponding to the document recognition task is collected.

[0070] S104, determining at least one certificate recognition task associated with the certificate image training sample, and obtaining a task query vector corresponding to the at least one certificate recognition task.

[0071] In some embodiments, at least one certificate recognition task corresponding to the certificate image training sample is obtained from a label corresponding to the certificate image training sample, or at least one certificate recognition task associated with the certificate image training sample that is manually annotated is obtained.

[0072] In some embodiments, the determining of at least one certificate recognition task associated with the certificate image training sample includes: inputting the certificate image training sample into a trained certificate task classification model to obtain at least one certificate recognition task output by the certificate task classification model. In some embodiments, the certificate task classification model is trained by certificate image classification samples carrying certificate task classification labels, and the collected certificate image training samples are input into the trained certificate task classification model to obtain at least one certificate recognition task associated with the output certificate image training sample.

[0073] In some embodiments, the determining of at least one certificate recognition task associated with the certificate image training sample includes: determining at least one certificate recognition task associated with the certificate image training sample according to the certificate type corresponding to the certificate image training sample. In some embodiments, a mapping relationship between different certificate types and certificate recognition tasks is pre-established, and at least one certificate recognition task associated with the certificate image training sample is determined based on the certificate type corresponding to the certificate image training sample and the mapping relationship; for example, the certificate type corresponding to a certificate image training sample is an ID card, and based on the pre-set mapping relationship, at least one certificate recognition task associated with the ID card includes all subtasks in quality compliance recognition, anti-counterfeiting recognition, and OCR recognition.

[0074] In some embodiments, each document recognition task corresponds to a task query vector, and a task query vector includes one or more query tokens; for example, a classification task corresponds to N (N ≥ 1 and is a positive integer) query tokens, and the task query vector corresponding to the classification task can be expressed as N*256, where 256 is the length of each query token. In some embodiments, the task query vector can be obtained using a variety of encoding methods, for example, it can be encoded using a beit3 mixed input method, or it can be encoded using a double tower method. In some embodiments, the task query vector corresponding to each document recognition task is pre-obtained or pre-set.

[0075] S106, inputting the certificate image training sample into the encoder for feature extraction to obtain image feature information output by the encoder.

[0076] In some embodiments, the encoder uses ViT (Vision Transformer) as its encoder model. In some embodiments, a shared learnable positional embedding (Learnable PositionalEmbedding) is used, so that it can focus on the knowledge itself in the data of different document recognition tasks, rather than the task-specific spatiotemporal structure, which is conducive to learning the commonalities and internal connections of document knowledge. It should be noted that shared positional embedding can also reduce the number of parameters and computational load of the encoder, avoiding the learning of additional positional embeddings for different tasks and data sets, which brings troubles in downstream deployment.

[0077] In some embodiments, the encoder is pre-trained, and the method further comprises: pre-training the encoder based on a self-supervised training method to obtain the pre-trained encoder. It should be noted that the encoder is task-independent.

[0078] In some embodiments, the initialization of the entire encoder is implemented by an encoder based on a self-supervised training method. The specific self-supervised training method used includes but is not limited to MAE (Mask Auto Encoders), SIMCLR (an unsupervised learning method based on contrastive learning), etc., which is not limited in the embodiments of this specification.

[0079] In some embodiments, the pre-training of the encoder based on the self-supervised training method to obtain the pre-trained encoder includes: obtaining a pre-training document image sample; dividing the pre-training document image sample into a plurality of non-overlapping sub-images, randomly determining at least one sub-image among the plurality of sub-images, and masking the at least one sub-image; inputting one or more unmasked sub-images into the encoder to obtain a corresponding first sub-image feature sequence; inserting the at least one masked sub-image into the first sub-image feature sequence according to the order of the at least one masked sub-image in the plurality of sub-images to obtain a corresponding second sub-image feature sequence; and obtaining the pre-trained encoder by inputting the second sub-image feature sequence into the corresponding decoder for reconstruction.

[0080] In some embodiments, the pre-trained document image samples are unlabeled samples. In some embodiments, the pre-trained document image samples and the aforementioned document recognition image samples may be the same image or different images. In some embodiments, when the pre-trained document image samples are divided into a plurality of non-overlapping sub-images, the pre-trained document image samples may be divided into a plurality of non-overlapping sub-images of a predetermined size, and different segmentation algorithms or segmentation rules may be set for different document recognition tasks. As an example, the initialization process of the encoder includes: 1) obtaining unlabeled pre-trained image samples; 2) cutting the pre-trained image samples into non-overlapping patches (blocks), and randomly masking some patches; 3) inputting the visible patches into a pre-built encoder, and obtaining the features of each patch after encoding; 4) putting the masked patches back to their original positions, and inputting them into a decoder symmetrical to the encoder for reconstruction. On the document image data, the parameters of the encoder obtained by pre-training in a self-supervised training manner are used as the initialization parameters of the encoder in the document recognition model.

[0081] S108, inputting the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder.

[0082] In some embodiments, the task query vector is input to the decoder from outside the document recognition model. In some embodiments, the task query vector is input to the encoder together with the associated document image training sample, and the task query vector is input to the decoder via the encoder. In some embodiments, when training the document recognition model for a certain document recognition task, the task query vector corresponding to the document recognition task is pre-input to the decoder.

[0083] In some embodiments, the initialization of the decoder can be achieved by a pre-trained decoder, or by random initialization or by initialization based on a priori. In some embodiments, the decoder includes multiple layers of blocks, each of which includes a cross-attention layer, a self-attention layer, and an FFN (feed forward network) layer. Therefore, after receiving the image feature information from the encoder, the decoder can use the coordinates of the task query vector to obtain the position embedding corresponding to the image feature information through mapping. Each layer of block uses a task-specific task query vector to decode the required task feature information from the encoded image feature information. In some embodiments, for the t-th document recognition task, the corresponding task feature information It can be expressed as:

[0084]

[0085] Among them, k is a positive integer less than or equal to K, K is the number of blocks, D k represents the kth block, F represents the encoded image features, and F p is the position embedding corresponding to F obtained by mapping, denoted as F p =proj(A F ), A F Indicates the coordinates of the task feature vector of the document image, is a learnable position embedding, which is randomly initialized.

[0086] It should be noted that all parameters of the decoder are exactly the same between all tasks. The only difference between different data sets and different document recognition tasks is the task query vector used to query document knowledge. Different task query vectors only activate the relevant document knowledge in the encoder without changing its understanding of this knowledge. This makes it so that changes in the task query vector will not affect the document recognition model's expression of document knowledge. Therefore, the document recognition model can reuse and transfer the learned knowledge to the maximum extent between different task query vectors (or document recognition tasks).

[0087] S110, inputting the task feature information into the task guidance interpreter to obtain at least one output result output by the task guidance interpreter.

[0088] It should be noted that although the certificate knowledge between different certificate recognition tasks is shared, the outputs corresponding to different certificate recognition tasks are different. In the embodiments of this specification, in order to more efficiently utilize the certificate knowledge learned by the encoder, a task-guided interpreter is designed at the end of the decoder. The task-guided interpreter is used to reconstruct the outputs required for different certificate recognition tasks and find a way to connect the various certificate recognition tasks at a higher level.

[0089] In some embodiments, the task guidance interpreter includes multiple output units, and the output result corresponding to each certificate recognition task is output from the output unit corresponding to the certificate recognition task.

[0090] In some embodiments, for all document recognition tasks, a unified interpretation is given as a combination of four output units, which are represented as follows: feature representation Y f , global probability map Y p , local probability map Y m , bounding box coordinate Y bbox , where Y f Characterizes the overall feature of the input document image, Y p Y represents whether the semantics of the query exists in the input document image. m Represents the pixel-by-pixel semantic information of the input document image, Y bbox Represents the specific position of the queried document task in the input document image; each document recognition task and its subtask may correspond to one output unit, or may correspond to a combination of multiple output units. It should be noted that the guidance interpreter interprets the task guidance based on the association relationship between each output unit and the document recognition task, so as to output corresponding output results for different document recognition tasks. The association relationship may be pre-established or learned through a neural network model. The description of this association relationship in this specification is only for example.

[0091] In some embodiments, the task guidance interpreter further includes a door control unit and a task guidance unit, the task guidance unit is used to control the opening and closing of the door control unit according to the task characteristic information, and the door control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the document recognition task. In some embodiments, the door control unit includes a plurality of switches or switch components, the input of the door control unit is the task characteristic information output by the decoder, the task guidance unit includes at least one convolutional layer and at least one multilayer perceptron (MLP), and the task guidance unit is used to control the output of the corresponding output unit based on the task guidance.

[0092] In some embodiments, each output unit corresponds to a task type, and the gate control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the task type of the document recognition task. In some embodiments, the task types include classification tasks, regression tasks, segmentation tasks, and detection tasks. For example, the four output units of the task guidance interpreter are represented as follows: feature representation Y f , global probability map Y p , local probability map Y m , bounding box coordinate Y bbox, The detection task corresponds to the output unit Y f , the segmentation task corresponds to the output unit Y m , the output unit corresponding to the classification task is Y p , the regression task corresponds to the output unit Y bbox The door control unit includes 4 switches, of which switch g m For controlling output unit Y m , switch g p For controlling output unit Y p , switch g f For controlling output unit Y f , switch g bbox For controlling output unit Y bbox In some embodiments, each task type includes subtasks corresponding to one or more output units, for example, the feature retrieval subtask corresponds to output unit Y. f , the remake subtask corresponds to the output unit Y p , the text line detection subtask corresponds to the output unit Y m and Y p .

[0093] S112: According to the at least one output result, the document recognition model is trained using a loss function to obtain a trained document recognition model.

[0094] In some embodiments, according to the at least one output result, the loss of each document recognition task is calculated by back propagation using a loss function, and then the document recognition model is trained in an end-to-end manner to obtain a trained document recognition model. It should be noted that the document recognition model can also be trained in a phased manner, for example, the encoder is first trained, and after the training is completed, the decoder is trained using the trained encoder, and then the task-guided interpreter is trained using the trained encoder and the trained decoder. After obtaining the trained task-guided interpreter, the training of the document recognition model is completed.

[0095] It should be noted that in the entire document recognition model, only the task query vector input to the decoder is not shared among all tasks and datasets. Therefore, through the guidance of the task-guided interpreter, the document recognition model can learn document-related knowledge at different granularities while sharing 99% of the parameters.

[0096] It should be noted that, based on the document recognition model in the embodiments of this specification, for a scenario where multiple document recognition tasks are performed simultaneously, the encoder and decoder only need to run once to obtain the output results of all document recognition tasks.

[0097] In an embodiment of the present specification, by obtaining a document image training sample and at least one document recognition task associated therewith, and obtaining a task query vector corresponding to the at least one document recognition task, the document image training sample is then input into the encoder for feature extraction to obtain image feature information output by the encoder, the task query vector and the image feature information are input into the decoder to obtain task feature information output by the decoder, the task feature information is input into the task guidance interpreter to obtain at least one output result output by the task guidance interpreter, and then according to the at least one output result, the document recognition model is trained using a loss function to obtain a trained document recognition model, thereby obtaining a unified document recognition model centered on the document image, which can be used to process different document recognition tasks, adapt to different document recognition task outputs, and can improve recognition accuracy with low maintenance costs; for different document recognition tasks, flexible task query vectors can be designed to learn corresponding knowledge, thereby the document recognition model in the embodiment of the present specification has good performance and potential in prompt training. The certificate recognition model in the embodiments of this specification can learn the domain knowledge of the certificate. Compared with training models separately for different tasks, it has higher recognition accuracy and greatly reduces maintenance costs.

[0098] Figure 2A flowchart of a document recognition model training method provided in an embodiment of this specification specifically includes the following steps:

[0099] S202, obtaining certificate image training samples.

[0100] The implementation method of step S202 is the same as or similar to the implementation method of step S102 in the above embodiment, and will not be repeated here.

[0101] S204: Determine at least one certificate recognition task associated with the certificate image training sample.

[0102] The implementation of step S204 is the same or similar to the implementation of determining at least one document recognition task associated with the document image training sample in the aforementioned embodiment, and will not be described in detail here.

[0103] S206: Obtain a task query text corresponding to at least one certificate recognition task associated with the certificate image training sample.

[0104] In some embodiments, the task query text corresponding to each certificate recognition task can be preset. In some embodiments, the task query text corresponding to each certificate recognition task can be determined by the label corresponding to the certificate image training sample. In some embodiments, the task query text corresponding to each certificate recognition task can be obtained by prompt engineering, for example, by inputting prompt words for different certificate recognition tasks into the language model to obtain the task query text corresponding to each certificate recognition task output by the language model. In some embodiments, the task query text corresponding to each certificate recognition task can be determined in combination with the needs of the certificate recognition demander.

[0105] S208. For each of the at least one document recognition tasks, the task query text corresponding to the document recognition task and the associated information of the document image training sample about the document recognition task are input into the multimodal converter to obtain the task query vector corresponding to the document recognition task output by the multimodal converter; wherein the associated information includes at least one of associated image information, user attribute information, and user behavior information.

[0106] In some embodiments, the associated information of a document recognition task includes any information related to the document recognition task that can reflect the characteristics of the document. In some embodiments, the associated image information includes any associated images of the document image training sample with respect to the document recognition task. For example, the associated images of the document image training sample with respect to a document recognition task may be normal frame images, flash frame images, images at different angles, etc. corresponding to the document image training sample.

[0107] In some embodiments, the user attribute information includes information related to the document recognition task that can characterize the user's attributes, such as user identity type, user document authority, etc. For example, the security of a specific place can set multiple user identity types (such as ordinary visitors, internal ordinary staff, internal special staff, etc.) for their pass documents, and different user identity types may be associated with different document recognition tasks. In some embodiments, the user behavior information includes information used to characterize user and document-related behaviors, such as the user's document usage behavior, document application behavior, etc.

[0108] It should be noted that, for the document recognition task, the task query text associated with the document recognition task is designed into a multimodal model encoding to obtain a task query vector, which can realize task-specific queries based on document characteristics.

[0109] In some embodiments, the associated information includes associated image information, and the method further includes: for each of the certificate recognition tasks, according to the task nature of the certificate recognition task, determining the associated image information of the certificate image training sample with respect to the certificate recognition task. In some embodiments, the certificate recognition task has single-frame input and multi-frame input (that is, the number of input image frames is ≥ 2), that is, the task nature of the certificate recognition task may be single-frame or multi-frame (multi-frames include normal frames and flash frames). In some embodiments, a variety of methods can be used to determine the task nature of the certificate recognition task, for example, pre-setting the task nature corresponding to each certificate recognition task, and for example, judging whether its task nature is single-frame or multi-frame by frame changes in a specific area of ​​the certificate image training sample, if there is a laser pattern in a specific area, its task nature is multi-frame.

[0110] In some embodiments, for each of the certificate recognition tasks, according to the task nature of the certificate recognition task, determining the associated image information of the certificate image training sample with respect to the certificate recognition task includes: for each of the certificate recognition tasks, if the task nature of the certificate recognition task is multi-frame, obtaining the flash frame image corresponding to the certificate image training sample, and using the flash frame image as the associated image information. For example, for the physical tampering subtask in the regression task, firstly, it is determined that the task nature of the subtask is double-frame, and then the flash frame image corresponding to the certificate image training sample is used as the associated image information of the certificate image training sample, so as to use the flash frame image as one input in the multi-modal input to determine the task query vector corresponding to the subtask.

[0111] In some embodiments, for each of the document recognition tasks, according to the task nature of the document recognition task, determining the associated image information of the document image training sample about the document recognition task includes: for each of the document recognition tasks, if the task nature of the document recognition task is multi-frame, obtaining at least one angle image corresponding to the document image training sample, and using the at least one angle image as the associated image information. For example, for the foreign body occlusion subtask in the regression task, firstly, it is determined that the task nature of the subtask is multi-frame, and then the multiple angle images corresponding to the document image training sample (that is, the images corresponding to different angles) are used as the associated image information of the document image training sample, so as to use the multiple angle images as one input in the multimodal input to determine the task query vector corresponding to the subtask.

[0112] In some embodiments, for each of the certificate recognition tasks, according to the task nature of the certificate recognition task, determining the associated image information of the certificate image training sample with respect to the certificate recognition task includes: for each of the certificate recognition tasks, if the task nature of the certificate recognition task is a single frame, setting the associated image information of the certificate image training sample with respect to the certificate recognition task to null. For example, for the copy subtask in the classification task, if it is determined that the task nature of the subtask is a single frame, then setting the associated image information input in the multimodal input to null, that is, the task query vector corresponding to the subtask can be determined based only on the task query text.

[0113] S210, input the certificate image training sample into the encoder for feature extraction, and obtain image feature information output by the encoder. The implementation of step S210 is the same or similar to the implementation of step S106 in the above embodiment, and will not be repeated here.

[0114] S212, input the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder. The implementation of step S212 is the same or similar to the implementation of step S108 in the above embodiment, and will not be repeated here.

[0115] S214, inputting the task feature information into the task guide interpreter to obtain at least one output result output by the task guide interpreter. The implementation of step S214 is the same or similar to the implementation of step S110 in the above embodiment, and will not be repeated here.

[0116] S216, according to the at least one output result, the document recognition model is trained using a loss function to obtain a trained document recognition model. The implementation of step S216 is the same or similar to the implementation of step S112 in the above embodiment, and will not be repeated here.

[0117] It should be noted that in the embodiments of this specification, based on the task characteristics of the certificate, different certificate recognition tasks can be uniformly encoded into task query vectors by adopting a multimodal transformer structure, which can support single-frame and multi-frame input at the same time and is flexible and changeable.

[0118] Figure 3 A flowchart of a document recognition method provided in an embodiment of this specification, the specific process includes:

[0119] S302, inputting a target ID image and a task query vector corresponding to at least one ID recognition task associated with the target ID image into an ID recognition model, wherein the ID recognition model is trained based on the ID recognition model training method in the embodiment of this specification.

[0120] In some embodiments, there are multiple ways to obtain the target ID image; for example, the client device can obtain the target ID image by scanning or photographing, and send the target ID image to an electronic device deployed with the ID recognition model (after obtaining the target ID image, the client can first perform a client quality compliance recognition operation on the client, and after confirming that the client quality compliance recognition requirements are met, it is sent to the electronic device deployed with the ID recognition model for server-side ID recognition); for another example, the target ID image input by the user can be obtained; for another example, the target ID image can be determined from multiple obtained ID images.

[0121] In some embodiments, after obtaining the target document image, at least one document recognition task associated with the target document image is first determined, and a task query vector corresponding to the document recognition task is obtained. In some embodiments, a task query vector corresponding to at least one document recognition task associated with the target image is obtained together with the target document image. In some embodiments, the implementation method of determining at least one document recognition task associated with the target document image and obtaining the task query vector corresponding to the document recognition task is the same or similar to the implementation method of the corresponding operation in the aforementioned step S104, and is not repeated here.

[0122] S304, obtaining at least one output result of the target ID image output by the ID recognition model with respect to the at least one ID recognition task. The encoder in the ID recognition model outputs image feature information corresponding to the target ID image based on the input target ID image, and the decoder obtains task feature information corresponding to the target ID image based on the received image feature information and the task query vector decoding, and then the task guidance interpreter controls at least one output unit corresponding to the at least one ID recognition task associated with the target ID image to output a corresponding output result based on the task feature information from the encoder.

[0123] S306, determining the document recognition result corresponding to the target document image according to the at least one output result. In some embodiments, an algorithm decision (such as expert information verification, intelligent decision making, gang mining, etc.) is performed on the at least one output result to determine the document recognition result corresponding to the target document image.

[0124] It should be noted that document recognition based on the document recognition model of the embodiment of this specification has higher recognition accuracy and efficiency than the method of separately training recognition models for different tasks, and has low maintenance costs.

[0125] Figure 4 A schematic diagram of the structural design of a document recognition model provided in an embodiment of this specification. The document recognition model includes a decoder, an encoder, and a task-guided interpreter. The decoder inputs a document image (the document image input during the model training process is also the aforementioned document image training sample, and the document image input during the document recognition process after the model training is completed is also the aforementioned target document image). The task-guided interpreter has four output units, namely: feature representation Y f , global probability map Y p , local probability map Y m , bounding box coordinate Y bbox It should be noted that Figure 4 The inputs of the encoder and decoder, as well as the outputs of the task-guided interpreter shown are all 4-way. Those skilled in the art should understand that the number of inputs or outputs is only an example and not a limitation, and can actually be designed based on requirements.

[0126] It should be noted that Figure 4 The number of task query vectors and input paths shown are only examples. The task query vectors can be directly input to the decoder from the outside, or first input to the encoder and then input to the decoder via the encoder. As an example, the task query vector Q1 is "Quality:biur,expousure,classify...", the task query vector Q2 is "OCR-segment:line detect", the task query vector Q3 is "spoof:screen,color,color copy", and the task query vector Q4 is "Spoof-segment:tamper". Based on Figure 4 The document recognition model shown in the figure can realize the recognition of different document recognition tasks through the four output units shown in the figure, such as f Determine the recognition result of the feature retrieval task, based on Y m , Y p Determine the recognition results of corner point detection, text line detection and other element detection tasks based on Y p , Ybbox Determine the recognition results for remake tasks or quality compliance tasks.

[0127] Figure 5 This is a schematic diagram of a multi-modal input provided in the embodiment of this specification. For the structure of the document recognition model and its input and output, please refer to Figure 4 The embodiments shown are not described in detail here. Figure 5 As shown, a multimodal input encoding method is used to obtain a task query vector. Specifically, the task query text and associated information (such as flash frame image, at least one angle image, user attribute information, user behavior information, etc.) of the certificate image training sample about the certificate recognition task are input into the multimodal converter to obtain the task query vector corresponding to the certificate recognition task output by the multimodal converter. For example, for each certificate recognition task, if the certificate recognition task is multi-frame, the task query text and flash frame image corresponding to the certificate recognition task are input into the multimodal converter to obtain the output task query vector. If the certificate recognition task is a single frame, the task query text is input into the multimodal converter, and the associated information input is set to empty to obtain the output task query vector. By adopting a multimodal transformer structure, different certificate recognition tasks can be uniformly encoded into task query vectors, and single and multi-frame inputs are supported at the same time, which is flexible and changeable.

[0128] Figure 6 This is a structural diagram of a certificate recognition model training device provided in an embodiment of this specification. The certificate recognition model training device 1 includes an acquisition module 11, a determination module 12, an encoding module 13, a decoding module 14, an output module 15, and a training module 16.

[0129] The acquisition module 11 is used to acquire certificate image training samples.

[0130] The determination module 12 is used to determine at least one certificate recognition task associated with the certificate image training sample, and obtain a task query vector corresponding to the at least one certificate recognition task.

[0131] The encoding module 13 is used to input the document image training sample into the encoder for feature extraction to obtain image feature information output by the encoder.

[0132] The decoding module 14 is used to input the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder.

[0133] The output module 15 is used to input the task feature information into the task guidance interpreter to obtain at least one output result output by the task guidance interpreter.

[0134] The training module 16 is used to train the document recognition model using a loss function according to the at least one output result to obtain a trained document recognition model.

[0135] In some embodiments, the encoder is pre-trained, and the document recognition model training device 1 is further used to: pre-train the encoder based on a self-supervised training method to obtain the pre-trained encoder. In some embodiments, the pre-training of the encoder based on a self-supervised training method to obtain a pre-trained encoder includes: obtaining a pre-trained document image sample; dividing the pre-trained document image sample into a plurality of non-overlapping sub-images, randomly determining at least one sub-image among the plurality of sub-images, and performing masking on the at least one sub-image; inputting one or more unmasked sub-images into the encoder to obtain a corresponding first sub-image feature sequence; inserting the at least one masked sub-image into the first sub-image feature sequence according to the order of the at least one masked sub-image in the plurality of sub-images to obtain a corresponding second sub-image feature sequence; and reconstructing the second sub-image feature sequence by inputting the second sub-image feature sequence into the corresponding decoder to obtain a pre-trained encoder.

[0136] In some embodiments, determining at least one certificate recognition task associated with the certificate image training sample includes: inputting the certificate image training sample into a trained certificate task classification model to obtain at least one certificate recognition task output by the certificate task classification model.

[0137] In some embodiments, the determining of at least one certificate recognition task associated with the certificate image training sample includes: determining at least one certificate recognition task associated with the certificate image training sample according to the certificate type corresponding to the certificate image training sample.

[0138] In some embodiments, obtaining the task query vector corresponding to the at least one certificate recognition task includes: obtaining a task query text corresponding to at least one certificate recognition task associated with the certificate image training sample; for each certificate recognition task in the at least one certificate recognition task, inputting the task query text corresponding to the certificate recognition task and the associated information of the certificate image training sample about the certificate recognition task into the multimodal converter, and obtaining the task query vector corresponding to the certificate recognition task output by the multimodal converter; wherein the associated information includes at least one of associated image information, user attribute information, and user behavior information.

[0139] In some embodiments, the associated information includes associated image information, and the certificate recognition model training device 1 is also used to: for each certificate recognition task, determine the associated image information of the certificate image training sample regarding the certificate recognition task according to the task nature of the certificate recognition task.

[0140] In some embodiments, for each of the certificate recognition tasks, according to the task nature of the certificate recognition task, determining the associated image information of the certificate image training sample about the certificate recognition task includes: for each of the certificate recognition tasks, if the task nature of the certificate recognition task is multi-frame, obtaining the flash frame image corresponding to the certificate image training sample, and using the flash frame image as the associated image information.

[0141] In some embodiments, for each of the certificate recognition tasks, according to the task nature of the certificate recognition task, determining the associated image information of the certificate image training sample about the certificate recognition task includes: for each of the certificate recognition tasks, if the task nature of the certificate recognition task is multi-frame, obtaining at least one angle image corresponding to the certificate image training sample, and using the at least one angle image as the associated image information.

[0142] In some embodiments, for each of the certificate recognition tasks, according to the task nature of the certificate recognition task, determining the associated image information of the certificate image training sample regarding the certificate recognition task includes: for each of the certificate recognition tasks, if the task nature of the certificate recognition task is single frame, setting the associated image information of the certificate image training sample regarding the certificate recognition task to empty.

[0143] In some embodiments, the task guidance interpreter includes multiple output units, and the output result corresponding to each certificate recognition task is output from the output unit corresponding to the certificate recognition task.

[0144] In some embodiments, the task guidance interpreter also includes a door control unit and a task guidance unit, the task guidance unit is used to control the opening and closing of the door control unit according to the task feature information, and the door control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the document recognition task.

[0145] In some embodiments, each output unit corresponds to a task type, and the door control unit is used to control the output result corresponding to each certificate recognition task to be output from the output unit corresponding to the task type of the certificate recognition task.

[0146] Figure 7This is a schematic diagram of the structure of a certificate recognition device provided in an embodiment of this specification. The certificate recognition device 2 includes a certificate input module 21, a model output module 22 and a recognition result determination module 23.

[0147] The certificate input module 21 is used to input the target certificate image and the task query vector corresponding to at least one certificate recognition task associated with the target certificate image into the certificate recognition model, and the certificate recognition model is trained based on the certificate recognition model training method described in the embodiment of this specification.

[0148] The model output module 22 is used to obtain at least one output result of the target document image output by the document recognition model with respect to the at least one document recognition task.

[0149] The recognition result determination module 23 is used to determine the document recognition result corresponding to the target document image according to the at least one output result.

[0150] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments. The operations performed by the device and its modules are the same or similar to the implementation methods of the corresponding operations in the above method embodiments, and will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0151] The present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 to Figure 3 The method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 to Figure 3 The specific description of the illustrated embodiment will not be repeated here.

[0152] The present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1 to Figure 3 The method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 to Figure 3 The specific description of the illustrated embodiment will not be repeated here.

[0153] The embodiments of this specification also provide Figure 8 The structural diagram of the electronic device shown in FIG. Figure 8 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.

[0154] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0155] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0156] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0157] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0158] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0159] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0160] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0165] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0166] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0167] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0169] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A document recognition model training method, the document recognition model includes an encoder, a decoder and a task guidance interpreter, the method comprising: Obtain document image training samples; Determining at least one document recognition task associated with the document image training sample, and obtaining a task query vector corresponding to the at least one document recognition task; Inputting the certificate image training sample into the encoder for feature extraction to obtain image feature information output by the encoder; Inputting the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder; Inputting the task feature information into the task guide interpreter to obtain at least one output result output by the task guide interpreter; According to the at least one output result, the document recognition model is trained using a loss function to obtain a trained document recognition model.

2. The method according to claim 1, wherein the encoder is pre-trained, and the method further comprises: The encoder is pre-trained based on a self-supervised training method to obtain the pre-trained encoder.

3. The method according to claim 2, wherein the pre-training the encoder based on the self-supervised training method to obtain the pre-trained encoder comprises: Obtain pre-training document image samples; Dividing the pre-trained document image sample into a plurality of non-overlapping sub-images, randomly determining at least one sub-image from the plurality of sub-images, and performing mask processing on the at least one sub-image; Inputting one or more unmasked sub-images into the encoder to obtain a corresponding first sub-image feature sequence; inserting the at least one masked sub-image into the first sub-image feature sequence according to the order of the at least one masked sub-image in the plurality of sub-images to obtain a corresponding second sub-image feature sequence; The second sub-image feature sequence is input into a corresponding decoder for reconstruction to obtain a pre-trained encoder.

4. The method according to claim 1, wherein determining at least one document recognition task associated with the document image training sample comprises: The document image training sample is input into a trained document task classification model to obtain at least one document recognition task output by the document task classification model.

5. The method according to claim 1, wherein determining at least one document recognition task associated with the document image training sample comprises: At least one certificate recognition task associated with the certificate image training sample is determined according to the certificate type corresponding to the certificate image training sample.

6. According to the method of claim 1, obtaining the task query vector corresponding to the at least one document recognition task comprises: Obtaining a task query text corresponding to at least one document recognition task associated with the document image training sample; For each of the at least one document recognition tasks, the task query text corresponding to the document recognition task and the associated information of the document image training sample about the document recognition task are input into the multimodal converter to obtain the task query vector corresponding to the document recognition task output by the multimodal converter; wherein the associated information includes at least one of associated image information, user attribute information, and user behavior information.

7. The method according to claim 6, wherein the associated information includes associated image information, and the method further comprises: For each certificate recognition task, the associated image information of the certificate image training sample with respect to the certificate recognition task is determined according to the task property of the certificate recognition task.

8. The method according to claim 7, wherein for each of the certificate recognition tasks, determining the associated image information of the certificate image training sample with respect to the certificate recognition task according to the task nature of the certificate recognition task comprises: For each of the certificate recognition tasks, if the task nature of the certificate recognition task is multi-frame, a flash frame image corresponding to the certificate image training sample is obtained, and the flash frame image is used as the associated image information.

9. The method according to claim 7, wherein for each of the certificate recognition tasks, determining the associated image information of the certificate image training sample with respect to the certificate recognition task according to the task nature of the certificate recognition task comprises: For each of the document recognition tasks, if the task nature of the document recognition task is multi-frame, at least one angle image corresponding to the document image training sample is obtained, and the at least one angle image is used as the associated image information.

10. The method according to claim 7, wherein for each certificate recognition task, determining the associated image information of the certificate image training sample with respect to the certificate recognition task according to the task nature of the certificate recognition task comprises: For each certificate recognition task, if the task nature of the certificate recognition task is single frame, the associated image information of the certificate image training sample with respect to the certificate recognition task is set to empty.

11. According to the method of claim 1, the task-guided interpreter comprises a plurality of output units, and the output result corresponding to each certificate recognition task is output from the output unit corresponding to the certificate recognition task.

12. According to the method according to claim 11, the task guidance interpreter also includes a door control unit and a task guidance unit, the task guidance unit is used to control the opening and closing of the door control unit according to the task feature information, and the door control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the document recognition task.

13. According to the method of claim 12, each output unit corresponds to a task type, and the door control unit is used to control the output result corresponding to each document recognition task to be output from the output unit corresponding to the task type of the document recognition task.

14. According to the method of claim 13, the task types include classification tasks, regression tasks, segmentation tasks, and detection tasks.

15. A document recognition method, comprising: Inputting a target document image and a task query vector corresponding to at least one document recognition task associated with the target document image into a document recognition model, wherein the document recognition model is trained based on the method according to any one of claims 1 to 14; Obtaining at least one output result of the target document image output by the document recognition model with respect to the at least one document recognition task; The document recognition result corresponding to the target document image is determined according to the at least one output result.

16. A document recognition model training device, the document recognition model includes an encoder, a decoder and a task guidance interpreter, the device includes: An acquisition module configured to acquire a training sample of a certificate image; A determination module configured to determine at least one document recognition task associated with the document image training sample, and obtain a task query vector corresponding to the at least one document recognition task; An encoding module configured to input the certificate image training sample into the encoder for feature extraction to obtain image feature information output by the encoder; A decoding module, configured to input the task query vector and the image feature information into the decoder to obtain the task feature information output by the decoder; an output module, configured to input the task feature information into the task guide interpreter to obtain at least one output result output by the task guide interpreter; The training module is configured to train the document recognition model using a loss function according to the at least one output result to obtain a trained document recognition model.

17. A document recognition device, comprising: An image input module, configured to input a target document image and a task query vector corresponding to at least one document recognition task associated with the target document image into a document recognition model, wherein the document recognition model is trained based on the method according to any one of claims 1 to 14; A model output module, configured to obtain at least one output result of the target document image output by the document recognition model with respect to the at least one document recognition task; The recognition result determination module is configured to determine the document recognition result corresponding to the target document image according to the at least one output result.

18. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 or 15 are implemented.

19. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 14 or 15.

20. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 14 or 15 are implemented.

Citation Information

Patent Citations

  • Information code identification method and terminal equipment

    CN117612055A

  • Method and apparatus with multi-modal feature fusion

    EP4181079A1

  • Identity card information verification method and apparatus, device, and computer readable storage medium

    WO2020082610A1

  • Facial feature extraction method, apparatus and device

    WO2021184898A1