Method, device, storage medium and electronic equipment for training general certificate base model

Abstract

Embodiments of the present specification disclose a method and device for training a general certificate base model, a storage medium and an electronic device. A plurality of certificate categories and a plurality of certificate image samples corresponding to each certificate category are obtained. Certificate image enhancement samples corresponding to the plurality of certificate image samples are obtained. According to the certificate categories, the certificate image samples, and the certificate image enhancement samples, a pre-set backbone network is trained by combining a supervised contrast learning framework to obtain a trained general certificate base model. The supervised contrast learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrast loss function. The supervised learning branch includes a supervised contrast loss function. The classification branch includes a cross-entropy loss function.

Description

Technical Field

[0001] This invention relates to computer technology, and more particularly to a method, apparatus, storage medium, and electronic device for training a universal document base model. Background Technology

[0002] With the development of mobile terminal technology and the widespread use of applications, more and more business scenarios require electronic identity authentication (Electronic Know Your Customer, eKYC). For example, various applications (such as financial applications) need to verify users' various documents for identity authentication. However, the global distribution of eKYC document data is uneven. Core documents have tens of thousands of copies each, while other documents have only a few copies or none at all. Traditional self-supervised algorithms treat each document as a separate category and perform self-supervised pre-training, without utilizing the category information of thousands of documents worldwide. On the other hand, direct CNN (Convolutional Neural Networks) classification faces the serious problem of class imbalance. Summary of the Invention

[0003] The purpose of the embodiments in this specification is to provide a method, apparatus, storage medium, and electronic device for training a universal document base model.

[0004] This specification provides a method for training a general document base model. The method combines a backbone network with a document-based supervised contrastive learning framework for training, resulting in a trained general document base model. By incorporating the characteristics of document scenarios (similar content and corresponding document categories for the same type of document), a supervised and self-supervised model training method based on documents is implemented. This method utilizes both a large amount of document data and its classification information. By combining data and classification information, the trained general document base model can be used in downstream tasks such as classification, quality compliance, and anti-counterfeiting in electronic identity authentication, improving the model's accuracy and training efficiency. The method includes:

[0005] Obtain multiple document categories and multiple document image samples corresponding to each document category;

[0006] Obtain the document image enhancement samples corresponding to the multiple document image samples;

[0007] Based on the document category, document image samples, and document image enhancement samples, a pre-defined backbone network is used to train a general document base model by combining a supervised contrastive learning framework. The supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function.

[0008] Furthermore, the self-supervised learning branch is connected to the first backbone network and the second backbone network. The input of the self-supervised contrastive loss function includes the output of the first backbone network and the output of the second backbone network. The input of the first backbone network is a document image sample, and the input of the second backbone network is a document image enhancement sample corresponding to the document image sample. The supervised learning branch is connected to the third backbone network and the fourth backbone network. The input of the supervised contrastive loss function includes the output of the third backbone network and the output of the fourth backbone network. The input of the third backbone network is the document image sample, and the input of the fourth backbone network is other document image samples belonging to the document category corresponding to the document image sample.

[0009] Furthermore, the input to the fourth backbone network is an enhanced image sample of the document corresponding to the document image sample or other document image samples belonging to the document category corresponding to the document image sample.

[0010] Furthermore, the input to the classification branch includes the output of the first backbone network, the output of the third backbone network, and the document category corresponding to the document image sample.

[0011] Furthermore, the self-supervised learning branch also includes two first projection networks, each of which takes the output of the first backbone network or the second backbone network as its input. Each first projection network is used to map the representation vector output by the backbone network connected to it into a vector for loss calculation. The supervised learning branch also includes two second projection networks, each of which takes the output of the third backbone network or the fourth backbone network as its input. Each second projection network is used to map the representation vector output by the backbone network connected to it into a vector for loss calculation.

[0012] Furthermore, the inputs to the classification branch include: the output of the first projection network corresponding to the first backbone network, the output of the second projection network corresponding to the third backbone network, and the document category corresponding to the document image sample.

[0013] Furthermore, the trained general document base model includes a trained backbone network and a single linear layer network connected to the trained backbone network.

[0014] The method further includes:

[0015] After training is completed, the first or second projection network connected to the trained backbone network is replaced with a single linear layer network.

[0016] Further, obtaining multiple enhanced document image samples corresponding to the multiple document image samples includes:

[0017] For each of the plurality of document image samples, at least one data augmentation method is used to perform image augmentation processing on the document image sample to obtain the corresponding document image augmentation sample.

[0018] Further, for each of the plurality of document image samples, image enhancement processing is performed on the document image sample using at least one data augmentation method to obtain the corresponding enhanced document image sample, including:

[0019] For each of the plurality of document image samples, the document image sample is randomly cropped to obtain a cropped document image sample. The resolution of the cropped document image sample is then adjusted so that the adjusted resolution is the same as the resolution of the original document image sample.

[0020] For each ID card image sample with adjusted resolution, at least one data augmentation method is used to perform image augmentation processing on the ID card image sample with adjusted resolution to obtain the corresponding ID card image augmentation sample.

[0021] This specification also provides a method for identifying document categories, the method comprising:

[0022] The first document image is input into the trained universal document base model to obtain the first representation vector output by the universal document base model, wherein the universal document base model is trained based on the method for training the universal document base model described in the embodiments of this specification;

[0023] The first representation vector is searched in the feature library. If a feature vector matching the first representation vector exists in the feature library, the document category corresponding to the feature vector is taken as the first document category corresponding to the first document image. The feature library includes multiple document categories and feature vectors corresponding to each document category.

[0024] Furthermore, the method also includes:

[0025] If there is no feature vector in the feature library that matches the first representation vector, the second document category corresponding to the first document image is obtained.

[0026] Add the second document category to the feature library, and add the first representation vector as the feature vector corresponding to the second document category to the feature library.

[0027] This specification also provides an embodiment of a method for training a downstream task model, the method comprising:

[0028] The model parameters corresponding to the trained general document base model are used as the training initialization parameters of the downstream task model. The general document base model is trained based on the method for training the general document base model described in the embodiments of this specification.

[0029] The downstream task model is trained according to the training initialization parameters to obtain the trained downstream task model.

[0030] This specification also provides an apparatus for training a universal document base model, comprising:

[0031] The document image sample acquisition module is used to obtain multiple document categories and multiple document image samples corresponding to each document category;

[0032] The document image enhancement sample acquisition module is used to obtain multiple document image enhancement samples corresponding to the multiple document image samples;

[0033] The general document base model training module is used to train a general document base model based on a preset backbone network, according to the document category, document image samples, and document image enhancement samples, by combining a supervised contrastive learning framework. The trained model is obtained by training the model using a supervised contrastive learning framework. The supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function.

[0034] This specification also provides an embodiment of a document category identification device, including:

[0035] The input module is used to input the first document image into the trained general document base model to obtain the first representation vector output by the general document base model, wherein the general document base model is trained based on the method for training the general document base model described in the embodiments of this specification;

[0036] The retrieval module is used to search the feature library for the first representation vector. If there is a feature vector in the feature library that matches the first representation vector, the document category corresponding to the feature vector is taken as the first document category corresponding to the first document image. The feature library includes multiple document categories and feature vectors corresponding to each document category.

[0037] This specification also provides an apparatus for training a downstream task model, comprising:

[0038] The parameter initialization module is used to use the model parameters corresponding to the trained general document base model as the training initialization parameters of the downstream task model, wherein the general document base model is trained based on the method for training the general document base model described in the embodiments of this specification.

[0039] The downstream task model training module is used to train the downstream task model according to the training initialization parameters to obtain the trained downstream task model.

[0040] This specification also provides a computer program product that stores at least one instruction adapted to be loaded by a processor and executed in accordance with the above-described method steps.

[0041] This specification also provides a storage medium storing a computer program adapted to be loaded by a processor and to execute the steps of the method described above.

[0042] This specification also provides an electronic device, including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method described above.

[0043] In the embodiments of this specification, the backbone network is trained using a document-based supervised contrastive learning framework to obtain a trained general document base model. By combining the scenario characteristics of documents (similar content of the same type of document and corresponding to the same document category), a document-based supervised + self-supervised model training method is implemented. This method utilizes both a large amount of document data and the classification information of the document data. By combining the data and classification information, the trained general document base model can be used in downstream tasks such as classification, quality compliance, and anti-counterfeiting in electronic identity authentication, thereby improving the accuracy and training efficiency of the model. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a method for training a general-purpose document base model, provided in an embodiment of this specification.

[0045] Figure 2A schematic diagram of a supervised contrastive learning framework provided as an example of an embodiment of this specification;

[0046] Figure 3 A flowchart illustrating a document category identification method provided in an embodiment of this specification;

[0047] Figure 4 A flowchart illustrating a method for training a downstream task model provided in an embodiment of this specification;

[0048] Figure 5 A schematic diagram of a device for training a universal document base model, provided as an embodiment of this specification;

[0049] Figure 6 A schematic diagram of the structure of a document category recognition device provided in the embodiments of this specification;

[0050] Figure 7 This is a schematic diagram of a device for training a downstream task model, provided as an embodiment of this specification.

[0051] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0053] Please see Figure 1 This is a flowchart illustrating a method for training a universal document base model provided in an embodiment of this specification. In this embodiment, the method for training a universal document base model is applied to a device for training a universal document base model (hereinafter referred to as a "first training device") or an electronic device equipped with a first training device. The following will focus on... Figure 1 The process shown will be explained in detail. The method for training the general document base model may specifically include the following steps:

[0054] S102, obtain multiple document categories and multiple document image samples corresponding to each document category.

[0055] In some embodiments, various global documents are classified according to document categories, resulting in multiple document categories and multiple document image samples corresponding to each category. In some embodiments, each document image sample is labeled with its corresponding document category, thereby constructing a training dataset based on multiple document categories and multiple document image samples corresponding to each category. In some embodiments, document categories include, but are not limited to, national ID cards, passports, etc. It should be noted that the above document categories are merely examples and not limitations. Those skilled in the art should understand that any document category can be included within the scope of protection of this specification, and this specification does not impose any limitations on it.

[0056] S104, obtain the document image enhancement samples corresponding to the plurality of document image samples.

[0057] In some embodiments, for a certain document image sample, a preset data augmentation method is used to perform image data augmentation processing on the document image sample to obtain one or more document image augmentation samples corresponding to the document image sample. Each document image augmentation sample represents a different viewpoint of its corresponding document image sample and contains some subsets of information in the document image sample. The data augmentation method includes, but is not limited to, data augmentation based on geometric transformation (such as flipping, rotating, cropping, translating, and adding noise), data augmentation based on color space transformation (such as adjusting brightness, contrast, saturation, channel separation, and grayscale conversion), and other data augmentation methods (such as random occlusion, automatic augmentation, random augmentation, sample generation adversarial network, local perturbation, image denoising, Sim augmentation (performing random color distortion and Gaussian blur in sequence, and possibly performing additional sparse image distortion operation at the end of the sequence)). It should be noted that the above data augmentation methods are only examples and not limitations. Those skilled in the art should understand that any data augmentation method can be included within the scope of protection of this specification, and this specification does not limit it.

[0058] S106, based on the document category, document image samples, and document image enhancement samples, a trained general document base model is obtained by training a pre-defined backbone network using a supervised contrastive learning framework. The supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function.

[0059] In some embodiments, multiple document categories, multiple document image samples corresponding to each document category, and one or more document image enhancement samples corresponding to each document image sample are used as training data. In some embodiments, a trained general document base model is obtained by training the training data based on a preset backbone network in conjunction with a supervised contrastive learning framework. The backbone network (also known as the basic network or core network) includes, but is not limited to, network structures such as ResNet50 and VIT base. The backbone network is used to extract features from the input training data (document image samples or document image enhancement samples). It should be noted that the above-described backbone network structure is only an example and not a limitation. Those skilled in the art should understand that any backbone network can be included within the scope of protection of this specification, and this specification does not impose any limitations on it.

[0060] In some embodiments, the input to the backbone network is a sample ID card image or an enhanced ID card image. During the training of the general ID card pedestal model, the backbone network is trained in conjunction with a supervised contrastive learning framework. This connects the backbone network to the supervised contrastive learning framework, and the feature vector output by the backbone network is used as the input to the supervised contrastive learning framework. In some embodiments, the input to the supervised contrastive learning framework includes, in addition to the feature vector output by the backbone network, the ID card category corresponding to the sample ID card image or the enhanced ID card image input to the backbone network. In some embodiments, the supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function and a softmax (normalization exponent) function. The self-supervised contrastive loss function is used to adjust the parameters of the backbone network by calculating a self-supervised contrastive loss based on two feature vectors output by the backbone network. The supervised contrastive loss function is used to adjust the parameters of the backbone network by calculating a supervised contrastive loss based on two feature vectors output by the backbone network. The softmax function is used to obtain the corresponding predicted classification information based on the feature vectors output by the backbone network. The predicted classification information includes at least one predicted classification and / or the probability value corresponding to each predicted classification. The cross-entropy loss function is used to adjust the parameters of the backbone network by calculating the loss based on the predicted classification information output by the softmax function and the document category corresponding to the document image sample.

[0061] In some embodiments, the types of classification branches include, but are not limited to, various classifiers such as linear classifiers (e.g., basic linear classifier, least squares linear classifier, perceptron, logistic regression classifier, etc.) and nonlinear classifiers (e.g., SVM (Support Vector Machine), decision tree, Naive Bayes, KNN (K-Nearest Neighbors algorithm), etc.). It should be noted that the above classifiers are merely examples and not limitations. Those skilled in the art should understand that any classifier can be included within the scope of protection of this specification, and this specification does not impose any limitations on it.

[0062] In some embodiments, the self-supervised learning branch can employ the self-supervised method of DINOv2 (a self-supervised computer vision model released by Meta), which is a combination of DINO (maximizing local information between visual Transformers) and iBOT (minimizing similarity between features) losses, and introduces the SwAV (sample centering) centering method. In addition, a regularization term is added to expand the features so that the features are more evenly distributed in the feature space.

[0063] In some embodiments, self-supervised learning branches include, but are not limited to, branches using different types of self-supervised learning algorithms, such as generative self-supervised learning branches, discriminative self-supervised learning branches, and contrastive-generative / adversarial self-supervised learning branches. If the self-supervised learning branch is a generative self-supervised learning branch, then its self-supervised contrastive loss function is a generative contrastive loss function; if the self-supervised learning branch is a discriminative self-supervised learning branch, then its self-supervised contrastive loss function is a discriminative contrastive loss function; if the self-supervised learning branch is a contrastive-generative / adversarial self-supervised learning branch, then its self-supervised contrastive loss function includes both generative and discriminative contrastive loss functions. It should be noted that the above self-supervised learning branches are merely examples and not limitations. Those skilled in the art should understand that self-supervised learning branches using any self-supervised learning algorithm can be included within the scope of protection of this specification, and this specification does not impose any limitations on them.

[0064] In some embodiments, supervised learning branches include, but are not limited to, branches using any type of supervised learning algorithm (e.g., classification algorithms (e.g., logistic regression, random forest, convolutional neural network CNN, multilayer perceptron MLP, etc.), regression algorithms (e.g., linear regression, multinomial regression, ridge regression, Lasso regression, etc.)). This example embodiment does not impose any particular limitation on this. In some embodiments, supervised contrastive loss functions include, but are not limited to, 0-1 loss functions, Hinge loss functions, exponential loss functions, squared loss functions, log-log loss functions, etc. This example embodiment does not impose any particular limitation on this.

[0065] In some embodiments, after training is complete, the trained backbone network is used as a general document base model, enabling the general document base model to learn not only the representation ability of document images but also the category information corresponding to the document images. In some embodiments, the feature vector output by the backbone network can be used as the representation vector output of the general document base model. A feature library is extracted for each type of document, and then features are extracted for new documents, along with the category in the feature library. In some embodiments, the model parameters of the general document base model can also be used as the initialization parameters for training downstream tasks (e.g., classification, quality compliance, anti-counterfeiting, OCR (Optical Character Recognition) tasks), which can improve the training efficiency and accuracy of downstream task models.

[0066] In some embodiments, the self-supervised learning branch is connected to a first backbone network and a second backbone network. The input to the self-supervised contrastive loss function includes the output of the first backbone network and the output of the second backbone network. The input of the first backbone network is a document image sample, and the input of the second backbone network is a document image enhancement sample corresponding to the document image sample. The supervised learning branch is connected to a third backbone network and a fourth backbone network. The input to the supervised contrastive loss function includes the output of the third backbone network and the output of the fourth backbone network. The input of the third backbone network is the document image sample, and the input of the fourth backbone network is other document image samples belonging to the document category corresponding to the document image sample. In some embodiments, a self-supervised learning branch is connected to a pair of backbone networks. The self-supervised contrastive loss function in the self-supervised learning branch calculates the contrastive loss based on a pair of feature vectors output by the pair of backbone networks, thereby adjusting the parameters of the backbone networks. The pair of backbone networks includes a first backbone network and a second backbone network. The first backbone network and the second backbone network have the same network structure and network parameters. Subsequently, during the training of the general document base model, the network parameters of the first backbone network and the second backbone network are adjusted synchronously based on the loss value calculated by the self-supervised contrastive loss function, the supervised contrastive loss function, or the cross-entropy loss function, so that the network parameters of the two are consistent (i.e., shared parameters). In some embodiments, the input to the first backbone network is an ID card image sample. If the current input to the first backbone network is one of the multiple ID card image samples, then the current input to the second backbone network is one of the one or more enhanced ID card image samples corresponding to that ID card image sample. For example, if the current input to the first backbone network is an ID card image of user X from country Y, then the current input to the second backbone network is the enhanced image corresponding to the ID card image of user X from country Y (for example, the enhanced image obtained by Gaussian blurring the ID card image of user X from country Y). In some embodiments, the supervised learning branch is connected to another pair of backbone networks. The supervised contrastive loss function in the supervised learning branch calculates the contrastive loss based on a pair of feature vectors output by the other pair of backbone networks, thereby adjusting the parameters of the backbone networks. The other pair of backbone networks includes a third backbone network and a fourth backbone network. The first backbone network, the second backbone network, the third backbone network, and the fourth backbone network have the same network structure and network parameters. Subsequently, during the training of the general document base model, the network parameters of the first backbone network, the second backbone network, the third backbone network, and the fourth backbone network are adjusted synchronously based on the loss values ​​calculated by the self-supervised contrastive loss function, the supervised contrastive loss function, or the cross-entropy loss function, so that the network parameters of these four backbone networks are consistent (i.e., shared parameters).In some embodiments, the current input of the third backbone network is the same as the current input of the first backbone network. If the current input of the first backbone network is one of the multiple document image samples, then the current input of the third backbone network is also that document image sample. The current input of the fourth backbone network is another document image sample belonging to the document category corresponding to that document image sample. For example, if the current input of the first backbone network is an image of a Y country ID card for user X, then the current input of the third backbone network is also an image of a Y country ID card for user X, and the current input of the fourth backbone network is an image of a Y country ID card for user Z.

[0067] In some embodiments, the input to the fourth backbone network is an enhanced image of the document corresponding to the document image sample or another document image sample belonging to the document category corresponding to the document image sample. In some embodiments, if the current input to the first backbone network or the third backbone network is one of the multiple document image samples, then the current input to the fourth backbone network can be, in addition to another document image sample belonging to the document category corresponding to the document image sample, one of the one or more enhanced image samples corresponding to the document image sample. For example, if the current input to the first backbone network is an image of a Y-country ID card for user X, then the current input to the third backbone network is also an image of a Y-country ID card for user X, and the current input to the fourth backbone network is an image of a Y-country ID card for user Z or an enhanced image corresponding to the Y-country ID card for user X (for example, an enhanced image obtained by Gaussian blurring the Y-country ID card for user X). In some embodiments, the current inputs of the second backbone network and the fourth backbone network are the same, that is, the current inputs of both are the same document image enhancement sample; or, the current inputs of the second backbone network and the fourth backbone network can be different, that is, the current inputs of both can be two different document image enhancement samples corresponding to a document image sample.

[0068] In some embodiments, the input to the classification branch includes the output of the first backbone network, the output of the third backbone network, and the document category corresponding to the document image sample. In some embodiments, the input to the classification branch includes a first feature vector output by the first backbone network, a second feature vector output by the third backbone network, and the document category corresponding to the document image sample input to the first backbone network corresponding to the first feature vector, or the document category corresponding to the document image sample input to the third backbone network corresponding to the second feature vector. In some embodiments, the softmax function in the classification branch is used to obtain the corresponding predicted classification information based on the first feature vector and the second feature vector. The predicted classification information includes at least one predicted classification and / or the probability value corresponding to each predicted classification. The cross-entropy loss function is used to adjust the parameters of the backbone network by calculating the loss based on the predicted classification information output by the softmax function and the document type input to the classification branch. Specifically, the first feature vector and the second feature vector can be synthesized into a single feature vector, and then the synthesized feature vector can be input into the softmax function to obtain the corresponding predicted classification information. Alternatively, the first feature vector and the second feature vector can be input into the softmax function separately to obtain the corresponding first classification information and second classification information, and then the first classification information and the second classification information can be synthesized into the corresponding predicted classification information.

[0069] In some embodiments, the self-supervised learning branch further includes two first projection networks, each of which takes the output of the first backbone network or the output of the second backbone network as its input. Each first projection network is used to map the representation vector output by the backbone network connected to it into a vector for loss calculation. The supervised learning branch further includes two second projection networks, each of which takes the output of the third backbone network or the output of the fourth backbone network as its input. Each second projection network is used to map the representation vector output by the backbone network connected to it into a vector for loss calculation. In some embodiments, the self-supervised learning branch further includes a pair of first projection networks, each of which is composed of a multilayer perceptron containing a hidden layer. One of the first projection networks is connected to a first backbone network, and the input of the first projection network is the feature vector (e.g., 2048-dimensional) output by the first backbone network. The other first projection network is connected to a second backbone network, and the input of the other first projection network is the feature vector (e.g., 2048-dimensional) output by the second backbone network. Each first projection network is used to map the representation vector output by the backbone network connected to it into a projection vector (e.g., 128-dimensional), that is, the output of each first projection network is a projection vector. The self-supervised contrastive loss function is used to calculate the loss based on the pair of projection vectors output by the pair of first projection networks respectively, wherein the two first projection networks correspond to the same network structure and network parameters. In some embodiments, the supervised learning branch further includes a pair of second projection networks, each consisting of a multilayer perceptron containing a hidden layer. One of the second projection networks is connected to a third backbone network, and its input is a feature vector (e.g., 2048-dimensional) output from the third backbone network. The other second projection network is connected to a fourth backbone network, and its input is a feature vector (e.g., 2048-dimensional) output from the fourth backbone network. Each second projection network maps the representation vector output from its connected backbone network into a projection vector (e.g., 128-dimensional), meaning each second projection network outputs a projection vector. A supervised contrastive loss function is used to calculate the loss based on the pair of projection vectors output by the two second projection networks, where the two second projection networks correspond to the same network structure and network parameters. In some embodiments, the first and second projection networks need to correspond to the same network structure and network parameters, or they can correspond to different network structures and / or different network parameters, respectively.

[0070] In some embodiments, the input to the classification branch includes: the output of the first projection network corresponding to the first backbone network, the output of the second projection network corresponding to the third backbone network, and the document category corresponding to the document image sample. In some embodiments, the feature vectors output by the first backbone network and the third backbone network can be directly input into the classification branch. Alternatively, the feature vector output by the first backbone network can first be input into the first projection network connected to it, and then the first projection network inputs the projection vector corresponding to the feature vector it outputs into the classification branch. Similarly, the feature vector output by the third backbone network can first be input into the second projection network connected to it, and then the second projection network inputs the projection vector corresponding to the feature vector it outputs into the classification branch.

[0071] In some embodiments, the trained general document base model includes a trained backbone network and a single linear layer network connected to the trained backbone network; wherein, the method further includes: after training is completed, replacing a first projection network or a second projection network connected to the trained backbone network with a single linear layer network. In some embodiments, after training is completed, the trained backbone network and the first projection network or the second projection network connected to the backbone network are used as the general document base model, and the backbone network can be any one of a first backbone network, a second backbone network, a third backbone network, and a fourth backbone network. In some embodiments, after training is completed, it is necessary to replace the first projection network or the second projection network connected to the backbone network in the general document base model with a single linear layer network. In some embodiments, the input of the single linear layer is a 2048-dimensional feature vector, and the output is a 128-dimensional feature vector.

[0072] In some embodiments, obtaining multiple enhanced document image samples corresponding to the multiple document image samples includes: for each document image sample in the multiple document image samples, performing image enhancement processing on the document image sample using at least one data enhancement method to obtain the corresponding enhanced document image sample. In some embodiments, for each of a plurality of document image samples, image enhancement processing is performed on the document image sample using at least one preset data enhancement method to obtain a document image enhanced sample corresponding to the document image sample. This can be achieved by selecting one of the preset at least one data enhancement methods for image enhancement processing, for example, randomly selecting one; or automatically selecting a data enhancement method that matches the attribute information (e.g., size information, color information, layout information, etc.) of the document image sample; or automatically selecting a data enhancement method that matches the document content of the preset at least one data enhancement method by performing image content recognition on the document image sample; or sequentially applying the preset at least one data enhancement method to the document image sample in a sequential order. The processing order of the at least one data enhancement method can be preset, randomly determined, or determined based on the attribute information or document content of the document image sample.

[0073] In some embodiments, the step of performing image enhancement processing on each of the plurality of document image samples using at least one data augmentation method to obtain a corresponding enhanced document image sample includes: randomly cropping each of the plurality of document image samples to obtain a cropped document image sample; adjusting the resolution of the cropped document image sample so that the adjusted resolution is the same as the resolution of the original document image sample; and performing image enhancement processing on each document image sample with adjusted resolution using at least one data augmentation method to obtain a corresponding enhanced document image sample. In some embodiments, it is necessary to first randomly crop the document image sample to obtain a cropped document image sample, then adjust the resolution of the cropped document image sample to the original resolution size of the original document image sample, and then perform image enhancement processing on the document image sample with adjusted resolution using at least one preset data augmentation method to obtain a corresponding enhanced document image sample.

[0074] Figure 2 This is a schematic diagram of an example of a supervised contrastive learning framework provided for an embodiment of this specification.

[0075] like Figure 2 As shown, the self-supervised learning branch is connected to a pair of backbone networks. The self-supervised contrastive loss function in the self-supervised learning branch calculates the contrastive loss based on a pair of feature vectors output by each of the backbone networks, thereby adjusting the parameters of the backbone networks. This pair of backbone networks includes a first backbone network and a second backbone network. The current input X1 of the first backbone network is one of the multiple document image samples, and the current input X2 of the second backbone network is one of the one or more document image enhancement samples corresponding to that document image sample. The supervised learning branch is connected to another pair of backbone networks. The supervised contrastive loss function in the supervised learning branch calculates the contrastive loss based on a pair of feature vectors output by the other pair of backbone networks, thereby adjusting the parameters of the backbone networks. The other backbone network consists of a third backbone network and a fourth backbone network. The input of the third backbone network is the same as the input of the first backbone network, that is, the current input X3 of the third backbone network is also the same document image sample. The current input X4 of the fourth backbone network is the document image enhancement sample corresponding to the document image sample or another document image belonging to the document category corresponding to the document image sample. The classification branch is connected to the first backbone network and the third backbone network. The current input X0 of the classification branch includes the first feature vector output by the first backbone network, the second feature vector output by the third backbone network, and the document category corresponding to the document image sample input to the first backbone network corresponding to the first feature vector or the document category corresponding to the document image sample input to the third backbone network corresponding to the second feature vector.

[0076] Figure 3 This is a flowchart illustrating a document category recognition method provided in an embodiment of this specification. In this embodiment, the document category recognition method is applied to a document category recognition device or an electronic device equipped with a document category recognition device. The following will focus on... Figure 3 The process shown will be explained in detail. The document category identification method may specifically include the following steps:

[0077] S302, the first document image is input into the trained universal document base model to obtain the first representation vector output by the universal document base model, wherein the universal document base model is trained based on the method for training the universal document base model described in the embodiments of this specification.

[0078] In some embodiments, the training method of the universal document base model has been described in detail above and will not be repeated here. In some embodiments, an image of a document is input into the trained universal document base model to obtain the first representation vector output by the universal document base model. In some embodiments, if the universal document base model only includes a trained backbone network, the feature vector output by the backbone network can be used as the first representation vector output by the universal document base model. If the universal document base model includes a trained backbone network and a single linear forming network connected to the backbone network (i.e., the feature vector output by the backbone network is input into a single linear layer network), the feature vector output by the single linear layer network can be used as the first representation vector output by the universal document base model.

[0079] S304, For the first representation vector, a search is performed in the feature library. If there is a feature vector in the feature library that matches the first representation vector, the document category corresponding to the feature vector is taken as the first document category corresponding to the first document image. The feature library includes multiple document categories and feature vectors corresponding to each document category.

[0080] In some embodiments, based on a trained universal document base model, a feature vector corresponding to each type of document is obtained, and the document category corresponding to that type of document and the feature vector are stored in a feature library. Furthermore, the mapping relationship (correspondence relationship) between the document category and the feature vector is stored in the feature library. In some embodiments, the first representation vector output by the universal document base model is searched in the feature library. If a feature vector matching the first representation vector exists in the feature library, the document category corresponding to that feature vector is taken as the first document category corresponding to the first document image. A feature vector matching the first representation vector can mean that the two vectors are identical, or it can mean that the similarity between the two vectors is greater than or equal to a preset similarity threshold. In some embodiments, a feature vector corresponding to a type of document exists in the feature library, thereby enabling the construction of a global document template library using this feature library.

[0081] In some embodiments, the method further includes: if no feature vector matching the first representation vector exists in the feature library, obtaining a second document category corresponding to the first document image; adding the second document category to the feature library, and adding the first representation vector as a feature vector corresponding to the second document category to the feature library. In some embodiments, if no feature vector matching the first representation vector exists in the feature library, obtaining the second document category corresponding to the first document image is not specifically limited in this example embodiment. For example, the second document category corresponding to the first document image input by the user can be obtained. Alternatively, image content recognition can be performed on the first document image, and the second document category corresponding to the first document image can be determined based on the recognition result. In some embodiments, adding the second document category to the feature library and adding the first representation vector as a feature vector corresponding to the second document category to the feature library means adding the second document category and the first representation vector to the feature library, and adding a mapping relationship (correspondence relationship) between the second document category and the first representation vector to the feature library. In some embodiments, adding a new document does not require retraining the model. Instead, the new document is simply input into a pre-trained general document base model, and the feature vector output by the general document base model is added to the feature library. This feature library can then be used to build a global document template library.

[0082] Figure 4 This is a flowchart illustrating a method for training a downstream task model provided in an embodiment of this specification. In this embodiment, the method for training a downstream task model is applied to a device for training a downstream task model (hereinafter referred to as a "second training device") or an electronic device equipped with a second training device. The following will focus on... Figure 4 The process shown will be described in detail. The method for training the downstream task model may specifically include the following steps:

[0083] S402, the model parameters corresponding to the trained general document base model are used as the training initialization parameters of the downstream task model, wherein the general document base model is trained based on the method for training the general document base model described in the embodiments of this specification.

[0084] In some embodiments, the training method of the general document base model has been described in detail above and will not be repeated here. In some embodiments, the model parameters (e.g., weights) corresponding to the trained general document base model can be directly used as the initialization parameters for training the downstream task model. That is, the downstream task model is pre-trained by training the general document base model. The downstream tasks include, but are not limited to, classification, quality compliance, anti-counterfeiting, OCR, etc. in document recognition tasks. It should be noted that the above-mentioned downstream tasks are only examples and not limitations. Those skilled in the art should understand that any downstream task can be included within the scope of protection of this specification, and this specification does not limit it.

[0085] S404, The downstream task model is trained according to the training initialization parameters to obtain a trained downstream task model. In some embodiments, training the downstream task model based on the training initialization parameters can improve the training efficiency and accuracy of the downstream task model. This example embodiment does not impose special limitations on the training process of the downstream task model. In some embodiments, the corresponding downstream task is executed based on the trained downstream task model.

[0086] Figure 5 This is a schematic diagram of a device for training a universal document base model, provided in an embodiment of this specification. This device (hereinafter referred to as "first training device 1") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the first training device 1 includes a document image sample acquisition module 11, a document image enhancement sample acquisition module 12, and a universal document base model training module 13.

[0087] The document image sample acquisition module 11 is used to acquire multiple document categories and multiple document image samples corresponding to each document category;

[0088] The document image enhancement sample acquisition module 12 is used to obtain multiple document image enhancement samples corresponding to the multiple document image samples;

[0089] The general document base model training module 13 is used to train a general document base model based on a preset backbone network according to the document category, document image samples, and document image enhancement samples, by combining a supervised contrastive learning framework. The supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function.

[0090] In some embodiments, the self-supervised learning branch is connected to a first backbone network and a second backbone network. The input to the self-supervised contrastive loss function includes the output of the first backbone network and the output of the second backbone network. The input of the first backbone network is a document image sample, and the input of the second backbone network is a document image enhancement sample corresponding to the document image sample. The supervised learning branch is connected to a third backbone network and a fourth backbone network. The input to the supervised contrastive loss function includes the output of the third backbone network and the output of the fourth backbone network. The input of the third backbone network is the document image sample, and the input of the fourth backbone network is other document image samples belonging to the document category corresponding to the document image sample.

[0091] In some embodiments, the input to the fourth backbone network is an enhanced image sample of the document corresponding to the document image sample or other document image samples belonging to the document category corresponding to the document image sample.

[0092] In some embodiments, the input to the classification branch includes the output of the first backbone network, the output of the third backbone network, and the document category corresponding to the document image sample.

[0093] In some embodiments, the self-supervised learning branch further includes two first projection networks, each of which takes the output of the first backbone network or the output of the second backbone network as its input. Each first projection network is used to map the representation vector output by the backbone network connected to it into a vector for loss calculation. The supervised learning branch further includes two second projection networks, each of which takes the output of the third backbone network or the output of the fourth backbone network as its input. Each second projection network is used to map the representation vector output by the backbone network connected to it into a vector for loss calculation.

[0094] In some embodiments, the input to the classification branch includes: the output of the first projection network corresponding to the first backbone network, the output of the second projection network corresponding to the third backbone network, and the document category corresponding to the document image sample.

[0095] In some embodiments, the trained general document base model includes a trained backbone network and a single linear layer network connected to the trained backbone network;

[0096] The first training device 1 is also used for:

[0097] After training is completed, the first or second projection network connected to the trained backbone network is replaced with a single linear layer network.

[0098] In some embodiments, obtaining multiple enhanced document image samples corresponding to the multiple document image samples includes:

[0099] For each of the plurality of document image samples, at least one data augmentation method is used to perform image augmentation processing on the document image sample to obtain the corresponding document image augmentation sample.

[0100] In some embodiments, the step of performing image enhancement processing on each of the plurality of document image samples using at least one data augmentation method to obtain a corresponding enhanced document image sample includes:

[0101] For each of the plurality of document image samples, the document image sample is randomly cropped to obtain a cropped document image sample. The resolution of the cropped document image sample is then adjusted so that the adjusted resolution is the same as the resolution of the original document image sample.

[0102] For each ID card image sample with adjusted resolution, at least one data augmentation method is used to perform image augmentation processing on the ID card image sample with adjusted resolution to obtain the corresponding ID card image augmentation sample.

[0103] Figure 6 This is a schematic diagram of a document category recognition device provided in an embodiment of this specification. This document category recognition device (hereinafter referred to as "document category recognition device 2") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the document category recognition device 2 includes an input module 21 and a retrieval module 22.

[0104] Input module 21 is used to input the first document image into the trained general document base model to obtain the first representation vector output by the general document base model, wherein the general document base model is trained based on the method for training the general document base model described in the embodiments of this specification;

[0105] The retrieval module 22 is used to search the feature library for the first representation vector. If there is a feature vector in the feature library that matches the first representation vector, the document category corresponding to the feature vector is taken as the first document category corresponding to the first document image. The feature library includes multiple document categories and feature vectors corresponding to each document category.

[0106] In some embodiments, the document category identification device 2 is further used for:

[0107] If there is no feature vector in the feature library that matches the first representation vector, the second document category corresponding to the first document image is obtained.

[0108] Add the second document category to the feature library, and add the first representation vector as the feature vector corresponding to the second document category to the feature library.

[0109] Figure 7 This is a schematic diagram of a device for training a downstream task model provided in an embodiment of this specification. This device (hereinafter referred to as "second training device 3") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the second training device 3 includes a parameter initialization module 31 and a downstream task model training module 32.

[0110] The parameter initialization module 31 is used to use the model parameters corresponding to the trained general document base model as the training initialization parameters of the downstream task model, wherein the general document base model is trained based on the method for training the general document base model described in the embodiments of this specification.

[0111] The downstream task model training module 32 is used to train the downstream task model according to the training initialization parameters to obtain the trained downstream task model.

[0112] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0113] This specification also provides a computer storage medium that can store multiple instructions adapted for loading by a processor and executing the methods described in this specification.

[0114] This specification also provides a computer program product that stores at least one instruction, which is loaded by the processor and executes the method described in this specification embodiment.

[0115] The embodiments in this specification also provide Figure 8 The diagram shows the structure of the electronic device. Figure 8 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned voice activity detection method.

[0116] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0117] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0118] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0119] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0120] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0121] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0127] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0129] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0131] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0132] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for training a general-purpose document base model, comprising: Obtain multiple document categories and multiple document image samples corresponding to each document category; Obtain the document image enhancement samples corresponding to the multiple document image samples; Based on the document category, document image samples, and document image enhancement samples, a pre-set backbone network is used to train a general document base model by combining a supervised contrastive learning framework. The supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function and a normalized exponential function. The self-supervised learning branch is connected to the first backbone network and the second backbone network. The input of the self-supervised contrastive loss function includes the output of the first backbone network and the output of the second backbone network. The input of the first backbone network is a document image sample, and the input of the second backbone network is the document image enhancement sample corresponding to the document image sample. The self-supervised contrastive loss function is used to adjust the parameters of the backbone network. The supervised learning branch is connected to the third backbone network and the fourth backbone network. The input of the supervised contrastive loss function includes the output of the third backbone network and the output of the fourth backbone network. The input of the third backbone network is the document image sample, and the input of the fourth backbone network is other document image samples belonging to the document category corresponding to the document image sample. The supervised contrastive loss function is used to adjust the parameters of the backbone network. The input to the classification branch includes the output of the first backbone network, the output of the third backbone network, and the document category corresponding to the document image sample. The normalized exponential function is used to obtain the corresponding predicted classification information based on the output of the first backbone network and the output of the third backbone network. The cross-entropy loss function is used to adjust the parameters of the backbone network by calculating the loss based on the predicted classification information and the document category.

2. The method according to claim 1, wherein the input of the fourth backbone network is an enhanced image sample of the document corresponding to the document image sample or other document image samples belonging to the document category corresponding to the document image sample.

3. The method according to claim 1, wherein the self-supervised learning branch further comprises two first projection networks, each first projection network receiving the output of the first backbone network or the output of the second backbone network as input, each first projection network being used to map the representation vector output by the backbone network connected to it into a vector for loss calculation; the supervised learning branch further comprises two second projection networks, each second projection network receiving the output of the third backbone network or the output of the fourth backbone network as input, each second projection network being used to map the representation vector output by the backbone network connected to it into a vector for loss calculation.

4. The method according to claim 3, wherein the input to the classification branch includes: The output of the first projection network corresponding to the first backbone network, the output of the second projection network corresponding to the third backbone network, and the document category corresponding to the document image sample.

5. The method according to claim 3 or 4, wherein the trained general document base model comprises a trained backbone network and a single linear layer network connected to the trained backbone network; in, The method further includes: After training is completed, the first or second projection network connected to the trained backbone network is replaced with a single linear layer network.

6. The method according to claim 1, wherein obtaining the plurality of enhanced document image samples corresponding to the plurality of document image samples comprises: For each of the plurality of document image samples, at least one data augmentation method is used to perform image augmentation processing on the document image sample to obtain the corresponding document image augmentation sample.

7. The method according to claim 6, wherein for each of the plurality of document image samples, image enhancement processing is performed on the document image sample using at least one data augmentation method to obtain a corresponding enhanced document image sample, comprising: For each of the plurality of document image samples, the document image sample is randomly cropped to obtain a cropped document image sample. The resolution of the cropped document image sample is then adjusted so that the adjusted resolution is the same as the resolution of the original document image sample. For each ID card image sample with adjusted resolution, at least one data augmentation method is used to perform image augmentation processing on the ID card image sample with adjusted resolution to obtain the corresponding ID card image augmentation sample.

8. A method for identifying document categories, comprising: The first document image is input into the trained universal document base model to obtain the first representation vector output by the universal document base model, wherein the universal document base model is trained based on the method of training the universal document base model according to any one of claims 1 to 7. The first representation vector is searched in the feature library. If a feature vector matching the first representation vector exists in the feature library, the document category corresponding to the feature vector is taken as the first document category corresponding to the first document image. The feature library includes multiple document categories and feature vectors corresponding to each document category.

9. The method according to claim 8, further comprising: If there is no feature vector in the feature library that matches the first representation vector, the second document category corresponding to the first document image is obtained. Add the second document category to the feature library, and add the first representation vector as the feature vector corresponding to the second document category to the feature library.

10. A method for training a downstream task model, comprising: The model parameters corresponding to the trained general document base model are used as the training initialization parameters of the downstream task model, wherein the general document base model is trained based on the method of training the general document base model according to any one of claims 1 to 7. The downstream task model is trained according to the parameters of the downstream task model initialization training to obtain the trained downstream task model.

11. An apparatus for training a universal document base model, comprising: The document image sample acquisition module is used to obtain multiple document categories and multiple document image samples corresponding to each document category; The document image enhancement sample acquisition module is used to obtain multiple document image enhancement samples corresponding to the multiple document image samples; The general document base model training module is used to train a general document base model based on a preset backbone network according to the document category, document image samples, and document image enhancement samples, by combining a supervised contrastive learning framework. The trained general document base model is obtained by training the model. The supervised contrastive learning framework includes a self-supervised learning branch, a supervised learning branch, and a classification branch. The self-supervised learning branch includes a self-supervised contrastive loss function, the supervised learning branch includes a supervised contrastive loss function, and the classification branch includes a cross-entropy loss function and a normalized exponential function. The self-supervised learning branch is connected to the first backbone network and the second backbone network. The input of the self-supervised contrastive loss function includes the output of the first backbone network and the output of the second backbone network. The input of the first backbone network is a document image sample, and the input of the second backbone network is the document image enhancement sample corresponding to the document image sample. The self-supervised contrastive loss function is used to adjust the parameters of the backbone network. The supervised learning branch is connected to the third backbone network and the fourth backbone network. The input of the supervised contrastive loss function includes the output of the third backbone network and the output of the fourth backbone network. The input of the third backbone network is the document image sample, and the input of the fourth backbone network is other document image samples belonging to the document category corresponding to the document image sample. The supervised contrastive loss function is used to adjust the parameters of the backbone network. The input to the classification branch includes the output of the first backbone network, the output of the third backbone network, and the document category corresponding to the document image sample. The normalized exponential function is used to obtain the corresponding predicted classification information based on the output of the first backbone network and the output of the third backbone network. The cross-entropy loss function is used to adjust the parameters of the backbone network by calculating the loss based on the predicted classification information and the document category.

12. A document category identification device, comprising: An input module is used to input a first document image into the trained universal document base model to obtain a first representation vector output by the universal document base model, wherein the universal document base model is trained based on the method for training a universal document base model according to any one of claims 1 to 7. The retrieval module is used to search the feature library for the first representation vector. If there is a feature vector in the feature library that matches the first representation vector, the document category corresponding to the feature vector is taken as the first document category corresponding to the first document image. The feature library includes multiple document categories and feature vectors corresponding to each document category.

13. An apparatus for training a downstream task model, comprising: The parameter initialization module is used to use the model parameters corresponding to the trained general document base model as the training initialization parameters of the downstream task model, wherein the general document base model is trained based on the method of training the general document base model according to any one of claims 1 to 7. The downstream task model training module is used to train the downstream task model according to the training initialization parameters to obtain the trained downstream task model.

14. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

15. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 10.

16. A computer program product having at least one instruction stored thereon, characterized in that, When the at least one instruction is executed by the processor, it implements the steps of the method according to any one of claims 1 to 10.

Images