Text recognition method and device for certificate image, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a document image text recognition method and device, a storage medium and electronic equipment. According to the method, a pre-trained document image text recognition model is used for carrying out document image text recognition. In the training process of the document image text recognition model, a text mask image is extracted from a document image text strip sample through a pre-trained teacher branch to serve as a false label of a first student branch, character structure features are extracted to serve as a false label of a second student branch, and a false label of the second student branch is extracted between the teacher branch and the first student branch. According to the document image text recognition method, self-supervised training at the character strip level is carried out through the first loss function, and self-supervised training at the character level is carried out between the teacher branch and the second student branch through the second loss function, so that the document image text recognition model has relatively high text recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium and electronic device for document image text recognition. Background Art

[0002] In the field of identity authentication based on document image recognition, it is usually necessary to use a text recognition model to extract text information from document images for user identity identification. Traditional text recognition models are mostly trained using supervised algorithms, and their training process requires a large number of supervised training samples. However, it is very difficult to annotate a large number of document images, which results in low recognition accuracy of existing text recognition models. Summary of the invention

[0003] One or more embodiments of the present specification provide a method, device, storage medium and electronic device for document image text recognition, which can use unsupervised document image samples to train a text recognition model and improve the recognition accuracy of the document image text recognition model.

[0004] In a first aspect, a method for training a document image text recognition model is provided, the method comprising:

[0005] Obtain unsupervised examples of text strips in ID images;

[0006] Extracting a text mask image and character structure features from the document image text strip sample using a teacher branch;

[0007] Input the document image text strip sample into the first student branch to obtain predicted text sequence features;

[0008] Constructing a first pseudo label according to the text mask image, and determining a first loss function by adopting a contrastive learning strategy based on the first pseudo label and the text sequence feature;

[0009] Inputting the sample of the text strip of the certificate image into the second student branch to obtain predicted character structure features;

[0010] constructing a second pseudo label according to the character structure feature, and determining a second loss function by adopting a contrastive learning strategy based on the second pseudo label and the predicted character structure feature;

[0011] According to the first loss function and the second loss function, parameters of the first student branch and the second student branch are updated.

[0012] As an optional implementation of the method of the first aspect, the document image text strip sample is input into the first student branch to obtain the predicted text sequence feature, specifically including:

[0013] Randomly enhancing the sample of the text strip of the certificate image to obtain a first enhanced sample image;

[0014] The first enhanced sample image is input into the first student branch to obtain the predicted text sequence feature.

[0015] As an optional implementation of the method of the first aspect, the document image text strip sample is input into the second student branch to obtain the predicted character structure features, specifically including:

[0016] Randomly enhancing the sample of the text strip of the certificate image to obtain a second enhanced sample image;

[0017] The second enhanced sample image is input into the second student branch to obtain the predicted character structure feature.

[0018] As an optional implementation manner of the method described in the first aspect, the teacher branch includes:

[0019] A first text encoder, used to extract a first text feature from the document image text strip sample;

[0020] A character structure region extraction module, used to segment the text mask image from the first text feature, and extract character horizontal position information from the first text feature using an implicit attention alignment mechanism based on the text mask image;

[0021] The first detection head is used to generate the character structure feature according to the character horizontal position information and the first text feature.

[0022] As an optional implementation manner of the method according to the first aspect, the first student branch includes:

[0023] A second encoder, used for extracting sequential second text features from the document image text strip sample;

[0024] A projection head, used for projecting the second text feature into a unified feature space to obtain a sequence of sequential text projection frames;

[0025] The instance mapping module is used to select a number of projection frames from the sequential text projection frame sequence to generate instances as the predicted text sequence features.

[0026] As an optional implementation manner of the method according to the first aspect, the second student branch includes:

[0027] A third encoder, used for extracting a third text feature from the document image text strip sample;

[0028] A segmentation network, used to segment a foreground text image from the document image text strip sample;

[0029] The second detection head is used to generate the predicted character structure feature according to the third text feature and the foreground text image.

[0030] As an optional implementation manner of the method of the first aspect, the method further includes:

[0031] In response to a user modifying a certificate image recognition result output by the certificate image text recognition model, acquiring modified certificate text information;

[0032] Using the modified document text information as the true label of the corresponding document image to construct an optimized sample set;

[0033] The teacher branch is trained using the optimized sample set.

[0034] In a second aspect, a method for text recognition of a document image is provided, comprising:

[0035] Obtain the target document image;

[0036] Segmenting a text line image from the target document image;

[0037] Inputting the text line image of the target document image into the document image text recognition model trained by the training method of the document image text recognition model, using the first student branch of the document image text recognition model to obtain predicted text sequence features, and using the second student branch of the document image text recognition model to obtain predicted character structure features;

[0038] Inputting the predicted text sequence features and the predicted character structure features into a decoder of the document image text recognition model to obtain a text recognition result of the text line image of the target document image;

[0039] The text recognition result of the target document image is determined according to the text recognition results of all the text line images of the target document image.

[0040] In a third aspect, a training device for a document image text recognition model is provided, comprising:

[0041] A first data acquisition module configured to acquire unsupervised document image text strip samples;

[0042] The training module is configured to use a teacher branch to extract a text mask image and character structure features from the ID image text strip sample; input the ID image text strip sample into a first student branch to obtain a predicted text sequence feature; construct a first pseudo label according to the text mask image, and determine a first loss function based on the first pseudo label and the text sequence feature by using a contrastive learning strategy; input the ID image text strip sample into a second student branch to obtain a predicted character structure feature; construct a second pseudo label according to the character structure feature, and determine a second loss function based on the second pseudo label and the predicted character structure feature by using a contrastive learning strategy; and update the parameters of the first student branch and the second student branch according to the first loss function and the second loss function.

[0043] As an optional implementation manner of the device according to the third aspect, the device further includes:

[0044] A second data acquisition module is configured to acquire the modified document text information and the corresponding document image in response to the modification of the document image recognition result output by the document image text recognition model by the user;

[0045] The optimization module is configured to use the modified document text information as the true label of the corresponding document image to construct an optimized sample set, and use the optimized sample set to train the teacher branch.

[0046] In a fourth aspect, a text recognition device for a document image is provided, comprising:

[0047] A third data acquisition module is configured to acquire a target document image;

[0048] A preprocessing module configured to segment a text line image from the target document image;

[0049] The recognition module is configured to input the text line image of the target certificate image into the certificate image text recognition model trained based on the training method of the certificate image text recognition model, obtain the predicted text sequence feature by using the first student branch of the certificate image text recognition model, and obtain the predicted character structure feature by using the second student branch of the certificate image text recognition model; input the predicted text sequence feature and the predicted character structure feature into the decoder of the certificate image text recognition model to obtain the text recognition result of the text line image of the target certificate image;

[0050] The result generation module is configured to determine the text recognition result of the target document image according to the text recognition results of all the text line images of the target document image.

[0051] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is run on an electronic device, the electronic device executes the above-mentioned training method of the certificate image text recognition model, or executes the above-mentioned certificate image text recognition method.

[0052] In a sixth aspect, an electronic device is provided, including:

[0053] at least one memory for storing a program;

[0054] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned training method of the document image text recognition model, or execute the above-mentioned document image text recognition method.

[0055] The beneficial effect of the training method for the document image text recognition model described in the embodiments of this specification is that the method extracts a text mask image from the document image text strip samples as a pseudo label for the first student branch through a pre-trained teacher branch, extracts character structure features as pseudo labels for the second student branch, and performs self-supervised training at the text strip level between the teacher branch and the first student branch through a first loss function, and performs self-supervised training at the character level between the teacher branch and the second student branch through a second loss function, thereby improving the text recognition accuracy of the document image text recognition model.

[0056] The training device for the document image text recognition model, the document image text recognition method and device described in the embodiments of this specification also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 The structural diagram of a document image text recognition model described in the embodiments of this specification is exemplarily shown.

[0059] Figure 2 The structural diagram of a training framework of a document image text recognition model described in an embodiment of this specification is exemplarily shown.

[0060] Figure 3The flowchart of a training method for a document image text recognition model described in an embodiment of this specification is exemplarily shown.

[0061] Figure 4 A structural diagram of a teacher branch described in an embodiment of this specification is exemplarily shown.

[0062] Figure 5 A structural diagram of a first student branch described in an embodiment of this specification is exemplarily shown.

[0063] Figure 6 A schematic diagram of the structure of a second student branch described in an embodiment of this specification is exemplarily shown.

[0064] Figure 7 The structural diagram of a training device for a document image text recognition model described in an embodiment of this specification is exemplarily shown.

[0065] Figure 8 The flowchart of a method for text recognition of a document image described in an embodiment of this specification is exemplarily shown.

[0066] Fig. 9 The structural diagram of a text recognition device for a certificate image described in an embodiment of this specification is exemplarily shown.

[0067] Fig.10 The structural diagram of an electronic device described in an embodiment of this specification is exemplarily shown. DETAILED DESCRIPTION

[0068] First of all, it should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise.

[0069] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0070] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0071] In the field of identity authentication based on document image recognition, it is necessary to extract text information from the document image to verify the user's identity, which is usually achieved by a text recognition model. However, the current text recognition models are mostly trained using supervised training methods, and in the field of identity authentication based on document image recognition, due to the privacy characteristics of document images, it is difficult to obtain enough document images as training samples. Secondly, the workload of document image annotation is huge, and in the field of identity authentication based on document image recognition, there are a large number of low-quality document images, and the annotation of these low-quality document images is particularly difficult.

[0072] In view of this, one or more embodiments of the present specification propose a method, device, storage medium and electronic device for text recognition of certificate images, which can use an unsupervised approach to train a certificate image text recognition model with high recognition accuracy, thereby improving the accuracy of certificate image text recognition.

[0073] The following will further describe in detail the document image text recognition method, device, storage medium and electronic device described in one or more embodiments of this specification in conjunction with the drawings and specific embodiments of the specification, but this detailed description does not constitute a limitation on the embodiments of this specification.

[0074] Please refer to Figure 1 , Figure 1 The structure diagram of the document image text recognition model proposed in one or more embodiments of this specification is shown in FIG. The input of the document image text recognition model is the text line image of the document image, and the output of the document image text recognition model is the text recognition result of the text line image of the document image.

[0075] like Figure 1 As shown, the document image text recognition model mainly includes: a first student branch, a second student branch and a decoder. The first student branch is used to extract text line level predicted text sequence features from the text line image of the input document image; the second student branch is used to extract character level predicted character structure features from the text line image of the input document image; the decoder is used to generate text recognition results based on the predicted text sequence features and predicted character structure features.

[0076] In order to train the text recognition capability of the document image text recognition model, this paper proposes a training framework for the document image text recognition model. Figure 2 , Figure 2 Figure 2 shows a schematic diagram of the structure of the training framework. Figure 2 As shown, the training framework includes: a teacher branch, a first student branch and a second student branch.

[0077] The teacher branch is a pre-trained network branch whose parameters are frozen during the training process. The teacher branch is used to generate pseudo labels for the first student branch and the second student branch, respectively, so as to train the learning ability of the first student branch for text line level features and the learning ability of the second student branch for character level features.

[0078] Based on the above purpose, this paper proposes a training method for document image text recognition model. Figure 3 , Figure 3 A flowchart of a training method for a document image text recognition model is shown exemplarily.

[0079] like Figure 3 As shown, the training method includes steps S300 to S312:

[0080] S300: Obtain unsupervised document image text strip samples.

[0081] In some implementations, the document image may be first acquired, and then a text strip image may be extracted from the document image by text line detection as the document image text strip sample. This training method uses an unsupervised training method, so the document image text strip sample does not have a label.

[0082] S302: Using the teacher branch, extract the text mask image and character structure features from the document image text strip sample.

[0083] The teacher branch is mainly used to provide pseudo labels for the unsupervised learning of the first student branch and the second student branch. Based on the learning purpose of the first student branch for text level features, the teacher branch needs to have the ability to segment text masks to provide text mask images to construct pseudo labels for the self-supervised learning of the first student branch. Based on the learning purpose of the second student branch for character level features, the teacher branch also needs to have the ability to extract character structure features to provide character structure features to construct pseudo labels for the self-supervised learning of the second student branch.

[0084] Based on the above requirements, in some implementations, the teacher branch can adopt Figure 4 For the structure shown, please refer to Figure 4 , Figure 4The teacher branch shown includes a first text encoder, a character structure region extraction module and a first detection head.

[0085] The first text encoder is used to extract the first text feature from the document image text strip sample, and the encoder can be implemented using a network structure such as ResNet or ViT. Specifically, the document image text strip sample can be encoded into a set of feature sequences.

[0086] The character structure region extraction module is used to segment a text mask image from the first text feature, and extract character horizontal position information from the first text feature using an implicit attention alignment mechanism based on the text mask image.

[0087] In the character structure extraction module, a text mask pseudo-label Spl representing the foreground and background can be first obtained through a clustering algorithm. Then, upper-level features are extracted from the first text features extracted by the first text encoder, and the upper-level features are segmented through a text segmentation network to obtain a text mask image Sm. Here, the text segmentation network can be optimized by a binary cross entropy loss between the text mask pseudo-label Spl and the text mask image Sm.

[0088] Taking the first text encoder using ResNet as an example, the specific method for obtaining the text mask image Sm is as follows: the output P0 of ResNet network Conv 0, the output P1 of Block0 and the output P2 of Block 1 can be extracted, and then a top-down pyramid structure is constructed:

[0089]

[0090] in, represents the convolutional layer, Τ(Ο 2 ,s 1 ) represents the feature image O 2 The sampling frequency is s 1 Upsampling, Τ(Ο 1 ,s 0 ) represents the feature image O 1 The sampling frequency is s 0 Upsampling, feature image Ο 0 Used to generate the above-mentioned text mask image Sm.

[0091] In the character structure extraction module, implicit attention weights can be extracted from the feature sequence output by the first encoder through softmax, and the attention weights can be converted into the position information of the corresponding characters. In order to avoid alignment drift during the decoding process of the character structure extraction module, an orthogonal constraint can be applied to obtain sequence-aligned attention vectors. Specifically, these learnable attention weights can be used as vectors, and the alignment operation is performed by ensuring that they are orthogonal to each other and that each processed attention vector is aligned with the corresponding character of the text mask image Sm.

[0092] The first detection head is used to generate a character structure feature according to the character horizontal position information and the first text feature. Specifically, after the character structure extraction module obtains the aligned attention weight (the aligned attention weight represents the character horizontal position information), the character structure feature can be obtained by calculating the dot product between the aligned attention weight and the text mask image Sm.

[0093] S304: Input the document image text strip sample into the first student branch to obtain the predicted text sequence features.

[0094] In some embodiments, the document image text strip sample can be randomly enhanced (such as color jitter, color loss, grayscale conversion, affine transformation and perspective deformation, etc.) to obtain a first enhanced sample image. The first enhanced sample image is then input into the first student branch to obtain the predicted text sequence feature.

[0095] The first student branch is used to learn text features at the text line level. For this purpose, the first student branch can adopt the following methods: Figure 5 The structure shown includes a second encoder, a projection head and an instance mapping module.

[0096] The second encoder is used to extract a set of sequential representations, that is, sequential second text features, from the document image text strip samples.

[0097] The projection head can use a small auxiliary network to transform the representation to adapt to the processing requirements of feature sequences of different lengths. Specifically, the projection head is used to project the second text feature into a unified feature space to obtain a sequence of sequential text projection frames.

[0098] The instance mapping module is used to select a number of projection frames from the sequential text projection frame sequence to generate instances, and the generated instances are used as predictive text sequence features.

[0099] S306: Construct a first pseudo label according to the text mask image, and determine a first loss function by adopting a contrastive learning strategy based on the first pseudo label and text sequence features.

[0100] Specifically, the text mask image can be converted into a set of text sequence feature labels, which are the first pseudo labels mentioned above. Through the strategy of comparative learning, the first pseudo labels are used to construct the first loss function to train the self-supervised learning ability of the first student branch for text line level features.

[0101] S308: Input the document image text strip sample into the second student branch to obtain predicted character structure features.

[0102] The second student branch is used to learn character-level text features. For this purpose, the second student branch can adopt Figure 6 The structure shown includes: a third encoder, a segmentation network and a second detection head.

[0103] The third encoder is used to extract a third text feature from the document image text strip sample.

[0104] The segmentation network is used to segment the foreground text image from the document image text strip samples.

[0105] The second detection head is used to generate predicted character structure features according to the third text features and the foreground text image.

[0106] S310: construct a second pseudo label according to the character structure feature, and determine a second loss function by adopting a contrastive learning strategy based on the second pseudo label and the predicted character structure feature.

[0107] Specifically, the character structure feature can be used as the second pseudo label, and through the contrast learning strategy, the second pseudo label is used to construct the second loss function to train the self-supervised learning ability of the second student branch for character-level features.

[0108] S312: Update parameters of the first student branch and the second student branch according to the first loss function and the second loss function.

[0109] In some implementations, the first loss function and the second loss function may be weightedly summed to obtain a total loss function, and then the total loss function may be used to update the parameters of the first student branch and the second student branch.

[0110] As a preferred implementation of the training method of the above-mentioned document image text recognition model, the above-mentioned teacher branch can also be optimized as the document image text recognition model is used. Specifically:

[0111] When the user modifies the ID card image recognition result output by the ID card image text recognition model, it means that the text recognition result of the ID card image text recognition model is incorrect. At this time, the ID card text information modified by the user can be obtained as the true label of the corresponding ID card image to construct an optimized sample set.

[0112] By using this optimized sample set, the teacher branch can be supervised for training, thereby optimizing the parameters of the teacher branch. By optimizing the parameters of the teacher branch, the accuracy of the pseudo-label can be improved, thereby improving the accuracy of the first student branch and the second student branch for text feature extraction.

[0113] Based on the training method of the certificate image text recognition model, the learning ability of the certificate image text recognition model for text line level features and character level features can be trained, thereby improving the recognition accuracy of the certificate image text recognition model.

[0114] Corresponding to the training method of the document image text recognition model, this embodiment also provides a training device for the document image text recognition model. Figure 7 , Figure 7 The schematic diagram of the structure of a training device for a document image text recognition model is shown as an example. It should be noted that: Figure 3 The training method of the document image text recognition model shown can rely on Figure 7 The training device shown implements, but is not limited to, this training device.

[0115] like Figure 7 As shown, the training device of the document image text recognition model includes:

[0116] The first data acquisition module 701 is configured to acquire unsupervised document image text strip samples.

[0117] The training module 702 is configured to use a teacher branch to extract a text mask image and character structure features from a sample of a text strip in an ID image; input the sample of a text strip in an ID image into a first student branch to obtain a predicted text sequence feature; construct a first pseudo-label according to the text mask image, and determine a first loss function based on the first pseudo-label and the text sequence feature by adopting a contrastive learning strategy; input the sample of a text strip in an ID image into a second student branch to obtain a predicted character structure feature; construct a second pseudo-label according to the character structure feature, and determine a second loss function based on the second pseudo-label and the predicted character structure feature by adopting a contrastive learning strategy; and update the parameters of the first student branch and the second student branch according to the first loss function and the second loss function.

[0118] With respect to the first data acquisition module 701 , the first data acquisition module 701 may first acquire a document image, and then perform text line detection on the document image to acquire the above-mentioned document image text strip sample.

[0119] For the training module 702, the module is used to execute Figure 3 The training method of the document image text recognition model shown will not be repeated here.

[0120] Preferably, the training device may further include a second data acquisition module and an optimization module, wherein the second data acquisition module is configured to obtain the modified document text information and the corresponding document image in response to the modification of the document image recognition result output by the document image text recognition model by the user. The optimization module is configured to use the modified document text information as the true label of the corresponding document image to construct an optimized sample set, and use the optimized sample set to train the teacher branch to fine-tune the parameters of the teacher branch.

[0121] For the above-mentioned training device of a document image text recognition model, taking a module as an example of a software functional unit, the first data acquisition module 701 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first data acquisition module 701 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0122] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0123] As an example of a hardware functional unit, the first data acquisition module 701 may include at least one computing device, such as a server, etc. Alternatively, the first data acquisition module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0124] The multiple computing devices included in the first data acquisition module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the first data acquisition module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first data acquisition module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0125] In other embodiments, the first data acquisition module 701 can be used to execute any step in the training method of the above-mentioned document image text recognition model, and the training module 702 can be used to execute any step in the training method of the above-mentioned document image text recognition model. The steps that the first data acquisition module 701 and the training module 702 are responsible for implementing can be specified as needed, and the first data acquisition module 701 and the training module 702 respectively implement different steps in the training method of the above-mentioned document image text recognition model to achieve all the functions of the training device of the above-mentioned document image text recognition model.

[0126] In this implementation, the training device can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to realize the training function of the training device for the document image text recognition model.

[0127] Based on the certificate image text recognition model trained by the above-mentioned certificate image text recognition model training method, this embodiment further proposes a certificate image text recognition method.

[0128] like Figure 8As shown, in some embodiments, the document image text recognition method may include steps S800 to S806:

[0129] S800: Acquire a target document image.

[0130] Specifically, the target document image may be acquired by performing real-time image acquisition on the target document, or the target document image uploaded by the front-end system may be acquired. This embodiment does not limit the method for acquiring the target document image.

[0131] S802: Segment a text line image from the target document image.

[0132] In some implementations, a text line detection method may be used to segment a text line image from a target document image.

[0133] S804: Input the text line image of the target certificate image into the certificate image text recognition model to obtain the text recognition result of the text line image of the target certificate image.

[0134] Specifically, after the text line image of the target ID image is input into the ID image text recognition model, it will be input into the first student branch and the second student branch respectively. The first student branch extracts the predicted text sequence features from the text line image of the target ID image, and the second student branch extracts the predicted character structure features from the text line image of the target ID image. The predicted text sequence features and the predicted character structure features will be input into the decoder of the ID image text recognition model. The decoder performs feature fusion on the predicted character structure features and the predicted text sequence features, thereby accurately locating the regional structural features where the characters are located, and finally predicting the text recognition results of the text line image of the target ID image.

[0135] S806: Determine the text recognition result of the target document image according to the text recognition result of the text line image of the target document image.

[0136] Specifically, the text recognition results of all text line images of the target document image may be structurally analyzed, and the text recognition result of each text line image may be arranged according to the position information obtained by the analysis, so as to finally obtain the text recognition result of the target document image.

[0137] Corresponding to the above-mentioned document image text recognition method, this embodiment also provides a document image text recognition device. Fig. 9 , Fig. 9 An exemplary device for identifying text in a document image is shown, which can be used to implement the above-mentioned method for identifying text in a document image. It should be noted that the method for identifying text in a document image described in one or more embodiments of the present application can rely on Fig. 9The identification device shown implements, but is not limited to, this identification device.

[0138] like Fig. 9 As shown, the identification device comprises:

[0139] The third data acquisition module 901 is configured to acquire a target document image.

[0140] The pre-processing module 902 is configured to segment a text line image from the target document image.

[0141] The recognition module 903 is configured to input the text line image of the target certificate image into the certificate image text recognition model to obtain the text recognition result of the text line image of the target certificate image.

[0142] The result generation module 904 is configured to determine the text recognition result of the target document image according to the text recognition results of all text line images of the target document image.

[0143] Regarding the above-mentioned third data acquisition module 901, the third data acquisition module 901 can obtain the target document image by real-time image acquisition of the target document, or obtain the target document image uploaded by the front-end system. This embodiment does not limit the method for obtaining the target document image.

[0144] With respect to the above-mentioned preprocessing module 902, the preprocessing module 902 may segment a text line image from the target document image by means of text line detection.

[0145] With respect to the above-mentioned recognition module 903, the specific process of the recognition module 903 obtaining the text recognition result of the text line image of the target document image is as follows:

[0146] After the text line image of the target ID image is input into the ID image text recognition model, it will be input into the first student branch and the second student branch respectively. The first student branch extracts the predicted text sequence features from the text line image of the target ID image, and the second student branch extracts the predicted character structure features from the text line image of the target ID image. The predicted text sequence features and the predicted character structure features will be input into the decoder of the ID image text recognition model. The decoder will perform feature fusion on the predicted character structure features and the predicted text sequence features, so as to accurately locate the regional structural features where the characters are located, and finally predict the text recognition results of the text line image of the target ID image.

[0147] Regarding the above-mentioned result generation module 904, the result generation module 904 can perform structured analysis on the text recognition results of all text line images of the target document image, arrange the text recognition results of each text line image according to the position information obtained by the analysis, and finally obtain the text recognition result of the target document image.

[0148] For the above-mentioned device for text recognition of certificate images, taking a module as an example of a software functional unit, the third data acquisition module 901 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the third data acquisition module 901 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0149] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0150] As an example of a hardware functional unit, the third data acquisition module 901 may include at least one computing device, such as a server, etc. Alternatively, the third data acquisition module 901 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0151] The multiple computing devices included in the third data acquisition module 901 can be distributed in the same region or in different regions. The multiple computing devices included in the third data acquisition module 901 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the third data acquisition module 901 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0152] In other embodiments, the third data acquisition module 901 can be used to execute any step in the above-mentioned method for text recognition of certificate images, the preprocessing module 902 can be used to execute any step in the above-mentioned method for text recognition of certificate images, the recognition module 903 can be used to execute any step in the above-mentioned method for text recognition of certificate images, and the result generation module 904 can be used to execute any step in the above-mentioned method for text recognition of certificate images. The steps that the third data acquisition module 901, the preprocessing module 902, the recognition module 903, and the result generation module 904 are responsible for implementing can be specified as needed, and the third data acquisition module 901, the preprocessing module 902, the recognition module 903, and the result generation module 904 respectively implement different steps in the above-mentioned method for text recognition of certificate images to realize all the functions of the above-mentioned device for text recognition of certificate images.

[0153] In this implementation, the text recognition device for certificate images can also be applied to computing devices such as computers and servers, or to a computing device cluster including at least one computing device, to realize the text recognition function of certificate images.

[0154] In some embodiments, an electronic device is also provided. Fig.10 , the electronic device includes: a bus 1001, a processor 1002, a memory 1003 and a communication interface 1004. The processor 1002, the memory 1003 and the communication interface 1004 communicate with each other through the bus 1001. The electronic device can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the electronic device.

[0155] The bus 1001 may be a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10The bus 1001 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 1001 may include a path for transmitting information between various components of the electronic device (for example, the processor 1002, the memory 1003, and the communication interface 1004).

[0156] The processor 1002 may include any one or more processors such as a CPU, a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0157] The memory 1003 may include a volatile memory, such as a random access memory (RAM). The memory 1003 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0158] The memory 1003 stores executable program codes, and the processor 1002 executes the executable program codes to respectively implement the functions of the aforementioned first data acquisition module 701 and the training module 702, that is, to implement the functions of the aforementioned training device for the document image text recognition model, thereby implementing the aforementioned training method for the document image text recognition model. Alternatively, the processor 1002 executes the executable program codes to respectively implement the functions of the aforementioned third data acquisition module 901, the preprocessing module 902, the recognition module 903, and the result generation module 904, that is, to implement the functions of the aforementioned document image text recognition device, thereby implementing the aforementioned document image text recognition method.

[0159] The communication interface 1004 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the electronic device and other devices or a communication network.

[0160] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned training method of the certificate image text recognition model, or implements the above-mentioned certificate image text recognition method.

[0161] The computer-readable storage medium may be any available medium that can be stored in the electronic device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the electronic device to execute the above-mentioned training method for the document image text recognition model, or to execute the above-mentioned document image text recognition method.

[0162] It is to be understood that the structure illustrated in the embodiments of this specification does not constitute a specific limitation on the system of the embodiments of this specification. In other embodiments of the specification, the above system may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0163] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0164] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0165] It should be noted that the above examples are only specific embodiments of the present invention, and the present invention is obviously not limited to the above examples, and there are many similar variations. All variations directly derived or associated from the contents disclosed by the technicians in this field should fall within the protection scope of the present invention.

Claims

1. A training method for a document image text recognition model, comprising: Obtain unsupervised examples of text strips in ID images; Extracting a text mask image and character structure features from the document image text strip sample using a teacher branch; Input the document image text strip sample into the first student branch to obtain predicted text sequence features; Constructing a first pseudo label according to the text mask image, and determining a first loss function by adopting a contrastive learning strategy based on the first pseudo label and the text sequence feature; Inputting the sample of the text strip of the certificate image into the second student branch to obtain predicted character structure features; constructing a second pseudo label according to the character structure feature, and determining a second loss function by adopting a contrastive learning strategy based on the second pseudo label and the predicted character structure feature; According to the first loss function and the second loss function, parameters of the first student branch and the second student branch are updated.

2. The method according to claim 1, wherein the sample of the document image text strip is input into the first student branch to obtain the predicted text sequence feature, specifically comprising: Randomly enhancing the sample of the text strip of the certificate image to obtain a first enhanced sample image; The first enhanced sample image is input into the first student branch to obtain the predicted text sequence feature.

3. The method according to claim 1, wherein the sample of the document image text strip is input into the second student branch to obtain the predicted character structure features, specifically comprising: Randomly enhancing the sample of the text strip of the certificate image to obtain a second enhanced sample image; The second enhanced sample image is input into the second student branch to obtain the predicted character structure feature.

4. The method according to claim 1, wherein the teacher branch comprises: A first text encoder, used to extract a first text feature from the document image text strip sample; A character structure region extraction module, used to segment the text mask image from the first text feature, and extract character horizontal position information from the first text feature using an implicit attention alignment mechanism based on the text mask image; The first detection head is used to generate the character structure feature according to the character horizontal position information and the first text feature.

5. The method of claim 1, wherein the first student branch comprises: A second encoder, used for extracting sequential second text features from the document image text strip sample; A projection head, used for projecting the second text feature into a unified feature space to obtain a sequence of sequential text projection frames; The instance mapping module is used to select a number of projection frames from the sequential text projection frame sequence to generate instances as the predicted text sequence features.

6. The method of claim 1, wherein the second student branch comprises: A third encoder, used for extracting a third text feature from the document image text strip sample; A segmentation network, used to segment a foreground text image from the document image text strip sample; The second detection head is used to generate the predicted character structure feature according to the third text feature and the foreground text image.

7. The method of claim 1, further comprising: In response to a user modifying a certificate image recognition result output by the certificate image text recognition model, acquiring modified certificate text information; Using the modified document text information as the true label of the corresponding document image to construct an optimized sample set; The teacher branch is trained using the optimized sample set.

8. A method for text recognition of a document image, comprising: Obtain the target document image; Segmenting a text line image from the target document image; Inputting the text line image of the target document image into the document image text recognition model trained based on the method according to any one of claims 1 to 7, using the first student branch of the document image text recognition model to obtain predicted text sequence features, and using the second student branch of the document image text recognition model to obtain predicted character structure features; Inputting the predicted text sequence features and the predicted character structure features into a decoder of the document image text recognition model to obtain a text recognition result of the text line image of the target document image; The text recognition result of the target document image is determined according to the text recognition results of all the text line images of the target document image.

9. A training device for a document image text recognition model, comprising: A first data acquisition module configured to acquire unsupervised document image text strip samples; The training module is configured to use a teacher branch to extract a text mask image and character structure features from the ID image text strip sample; input the ID image text strip sample into a first student branch to obtain a predicted text sequence feature; construct a first pseudo label according to the text mask image, and determine a first loss function based on the first pseudo label and the text sequence feature by using a contrastive learning strategy; input the ID image text strip sample into a second student branch to obtain a predicted character structure feature; construct a second pseudo label according to the character structure feature, and determine a second loss function based on the second pseudo label and the predicted character structure feature by using a contrastive learning strategy; and update the parameters of the first student branch and the second student branch according to the first loss function and the second loss function.

10. The device of claim 9, further comprising: A second data acquisition module is configured to acquire the modified document text information and the corresponding document image in response to the modification of the document image recognition result output by the document image text recognition model by the user; The optimization module is configured to use the modified document text information as the true label of the corresponding document image to construct an optimized sample set, and use the optimized sample set to train the teacher branch.

11. A text recognition device for a document image, comprising: A third data acquisition module is configured to acquire a target document image; A preprocessing module configured to segment a text line image from the target document image; A recognition module, configured to input the text line image of the target certificate image into a certificate image text recognition model trained based on the method according to any one of claims 1 to 7, obtain predicted text sequence features using a first student branch of the certificate image text recognition model, and obtain predicted character structure features using a second student branch of the certificate image text recognition model; input the predicted text sequence features and the predicted character structure features into a decoder of the certificate image text recognition model to obtain a text recognition result of the text line image of the target certificate image; The result generation module is configured to determine the text recognition result of the target document image according to the text recognition results of all the text line images of the target document image.

12. A computer-readable storage medium storing a computer program, wherein when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7, or the method according to claim 8.

13. An electronic device, comprising: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7, or execute the method according to claim 8.

Images