A certificate type identification method, device, equipment and medium

Abstract

The present invention discloses a certificate type recognition method, device, equipment and medium, which relates to the field of image recognition. The technical scheme thereof mainly includes: obtaining a non-certificate image data set and a certificate image data set with multiple certificate types; taking a batch size of non-certificate images of the non-certificate image data set and two batch sizes of certificate images of the certificate image data set as training data each time, wherein the images of each batch of the certificate image data belong to different certificate categories, the categories of the certificate images at corresponding positions of the first batch and the second batch are the same, and the certificate images at corresponding positions of the first batch and the second batch are different; training a pre-constructed certificate type recognition model through the training data until the loss function converges, and obtaining a trained certificate type recognition model; recognizing the certificate image to be recognized based on the trained certificate type recognition model, and obtaining a recognition result of the certificate type.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and more specifically, to a certificate type recognition method, device, equipment and medium. Background Art

[0002] In modern electronic bidding, qualification verification and other fields, certificate images, as key verification materials, bear the responsibility of ensuring the legitimacy and integrity of documents. These certificate images are of various types and formats, including but not limited to corporate qualification certificates, qualification certificates, licenses and various compliance documents. With the popularization of electronic bidding, how to efficiently and accurately identify and classify these certificate images has become a technical problem that needs to be solved urgently.

[0003] Existing certificate image type recognition technologies mainly rely on two methods: OCR (Optical Character Recognition) technology and image feature analysis technology. OCR technology extracts text information from the certificate and then classifies or verifies it based on the text content. The recognition accuracy of this method is often affected by image quality and text fonts, and is prone to recognition errors. Image feature analysis methods attempt to classify based on image features, such as using simple edge detection, texture analysis and other methods to identify the type of certificate. Although this method can work in some cases, they mainly rely on simple visual features and cannot capture more complex key information in the certificate image, such as text, layout and other structured data in the image. In addition, classification algorithms that rely solely on image features have low classification accuracy for complex and diverse certificate image types.

[0004] In general, existing technologies mainly rely on a single modality (such as text or image features), and most of them lack effective feature fusion and optimization strategies, making it difficult to cope with the challenges of the diversity and complexity of certificate images, resulting in insufficient accuracy in certificate type recognition. Summary of the invention

[0005] The purpose of the present invention is to provide a certificate type identification method, device, equipment and medium, which solves the problem of insufficient accuracy in the certificate type identification process in the current prior art.

[0006] A first aspect of the present invention provides a method for identifying a certificate type, the method comprising:

[0007] Obtain a non-certificate image dataset and a certificate image dataset with multiple certificate types; wherein each certificate category of the certificate image data contains at least two certificate images;

[0008] Each time, a batch of non-certificate images of the non-certificate image data set and two batches of certificate images of the certificate image data set are taken as training data; wherein, the images of each batch of the certificate image data belong to different certificate categories, the categories of the certificate images at the corresponding positions of the first batch and the second batch are the same, and the certificate images at the corresponding positions of the first batch and the second batch are different;

[0009] The pre-built certificate type recognition model is trained by using training data until the loss function converges, thereby obtaining a trained certificate type recognition model; wherein the certificate type recognition model includes an image feature extraction model and a text feature extraction model;

[0010] The certificate image to be identified is identified based on the trained certificate type recognition model to obtain the identification result of the certificate type.

[0011] In one implementation, the loss function is expressed as:

[0012] , where Loss represents the loss function, N is the sample size of the batch; x i is the feature vector of the query sample in the first batch; x j is the feature vector of other certificate images in the first batch; y i is the feature vector of the certificate image at the same position as the query sample in the second batch; z j is the feature vector of the non-certificate image; τ is the temperature coefficient.

[0013] In one implementation, the pre-built certificate type recognition model is trained using training data until the loss function converges to obtain a trained certificate type recognition model, specifically:

[0014] Extracting a first image feature vector of training data through an image feature extraction model, calculating an image loss value of the first image feature vector through a loss function, and performing a first update on a network parameter of the image feature extraction model according to the image loss value using a reverse gradient propagation algorithm to obtain an image feature extraction model after the first update;

[0015] Extracting a first text feature vector of training data through a text feature extraction model, calculating a text loss value of the first text feature vector through a loss function, and performing a first update on a network parameter of the text feature extraction model according to the text loss value using a reverse gradient propagation algorithm to obtain a first updated text feature extraction model;

[0016] Extracting a second image feature vector and a second text feature vector of the training data respectively through the image feature extraction model and the text feature extraction model after the initial update;

[0017] The second image feature vector and the second text feature vector are fused to obtain a multimodal feature vector. The multimodal loss value of the multimodal feature vector is calculated through the loss function. The network parameters of the image feature extraction model and the text feature extraction model after the initial update are simultaneously updated for the second time according to the multimodal loss value using the back gradient propagation algorithm to obtain a trained certificate type recognition model.

[0018] In one implementation, the text feature extraction model is the ERNIE 3.0 model.

[0019] A second aspect of the present invention provides a certificate type identification device, the device comprising:

[0020] An image data acquisition module is used to acquire a non-certificate image data set and a certificate image data set having multiple certificate types; wherein each certificate type of the certificate image data contains at least two certificate images;

[0021] A training data generation module is used to take one batch size of non-certificate images of the non-certificate image data set and two batch sizes of certificate images of the certificate image data set as training data each time; wherein the images of each batch of the certificate image data belong to different certificate categories, the certificate images at corresponding positions of the first batch and the second batch have the same category, and the certificate images at corresponding positions of the first batch and the second batch are different;

[0022] A model training module, used to train a pre-built certificate type recognition model through training data until the loss function converges, thereby obtaining a trained certificate type recognition model; wherein the certificate type recognition model includes a parallel image feature extraction model and a text feature extraction model;

[0023] The type recognition module is used to recognize the certificate image to be recognized based on the trained certificate type recognition model to obtain the recognition result of the certificate type.

[0024] In one implementation, the loss function is expressed as:

[0025] , where Loss represents the loss function, N is the sample size of the batch; x i is the feature vector of the query sample in the first batch; x j is the feature vector of other certificate images in the first batch; yi is the feature vector of the certificate image at the same position as the query sample in the second batch; z j is the feature vector of the non-certificate image; τ is the temperature coefficient.

[0026] In one implementation, the model training module is specifically used to:

[0027] Extracting a first image feature vector of training data through an image feature extraction model, calculating an image loss value of the first image feature vector through a loss function, and performing a first update on a network parameter of the image feature extraction model according to the image loss value using a reverse gradient propagation algorithm to obtain an image feature extraction model after the first update;

[0028] Extracting a first text feature vector of training data through a text feature extraction model, calculating a text loss value of the first text feature vector through a loss function, and performing a first update on a network parameter of the text feature extraction model according to the text loss value using a reverse gradient propagation algorithm to obtain a first updated text feature extraction model;

[0029] Extracting a second image feature vector and a second text feature vector of the training data respectively through the image feature extraction model and the text feature extraction model after the initial update;

[0030] The second image feature vector and the second text feature vector are fused to obtain a multimodal feature vector. The multimodal loss value of the multimodal feature vector is calculated through the loss function. The network parameters of the image feature extraction model and the text feature extraction model after the initial update are simultaneously updated for the second time according to the multimodal loss value using the back gradient propagation algorithm to obtain a trained certificate type recognition model.

[0031] In one implementation, the text feature extraction model is the ERNIE 3.0 model.

[0032] According to a third aspect of the present invention, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of a certificate type identification method provided in the first aspect of the present invention are implemented.

[0033] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a certificate type identification method provided in the first aspect of the present invention are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] Aiming at the specific requirements of certificate images, the present invention designs an optimized training method for the image feature extraction model and the text feature extraction model to make them more adaptable to the feature distribution and classification requirements of certificate images. In particular, a method for inputting training data and a method for calculating the loss function are proposed, so that the optimized image feature extraction model and the text feature extraction model can more efficiently extract and characterize the key information in the certificate image, and effectively improve the expression ability of the feature vector. By jointly optimizing the image feature extraction model and the text feature extraction model, the image feature extraction model and the text feature extraction model can better utilize the correlation between the image and the text, reduce feature redundancy, and improve the accuracy and robustness of the model in complex certificate image judgment tasks, thereby quickly and accurately completing the type recognition task of the certificate image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0037] Figure 1 A flow chart of a certificate type identification method provided by an embodiment of the present invention;

[0038] Figure 2 A functional block diagram of a certificate type identification device provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0041] It should be noted that the terms "include" or "may include" used in various embodiments of the present application indicate the presence of the function, operation or element applied for, and do not limit the addition of one or more functions, operations or elements. In addition, as used in various embodiments of the present application, the terms "include", "have" and their cognates are only intended to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items.

[0042] It should be understood that terms such as "first" and "second" are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0043] Please refer to Figure 1 , Figure 1 A flow chart of a certificate type identification method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0044] S101, obtaining a non-certificate image dataset and a certificate image dataset having multiple certificate types; wherein each certificate type of the certificate image data contains at least two certificate images.

[0045] In this embodiment, the certificate image dataset is divided into categories according to whether the certificate format is the same. Certificates of the same category require the same format and type, and only the text on the certificate or the photo in the certificate is different. The certificate image dataset has 100,000 certificate images, 386 categories, and each category has at least 2 certificate images. The image format is unified in jpeg format, and the image is required to be clear and the text can be clearly recognized. The non-certificate image dataset is 50,000 non-certificate Internet images, and the image format is also jpeg.

[0046] S102, each time taking a batch size of non-certificate images of the non-certificate image data set and two batch sizes of certificate images of the certificate image data set as training data; wherein the images of each batch of the certificate image data belong to different certificate categories, the categories of the certificate images at corresponding positions of the first batch and the second batch are the same, and the certificate images at corresponding positions of the first batch and the second batch are different.

[0047] In this embodiment, when inputting data to the model during the training process, it is required to input 2 batches of certificate image data and 1 batch of non-image data each time, wherein the data in each certificate image batch belongs to different categories, but the data categories at corresponding positions in the two batches are the same. Therefore, the similar data is the data of the same category at the corresponding position in another batch, and the different data is the other data in the batch and 1 batch of non-certificate image data.

[0048] For example, each model training requires 2 batches of certified image data, denoted as X and Y, and 1 batch of non-certified image data, denoted as Z. For any position x in X, i , whose similar data is y i , the different data is X divided by xi Other data except z. Irrelevant data are those within Z except z i Other data outside. i With y i The certificates are of the same category and format, and only the text content on the certificates is different, so they are considered similar data. i Other certificate data other than x i The certificate types are different, and there are significant differences in the certificate formats, so they are treated as different data. The non-certificate image data in Z is very different from the certificate image, and as irrelevant data, it is necessary to remove z during use. i Data, in order to keep the data quantity consistent.

[0049] S103, training a pre-built certificate type recognition model through training data until the loss function converges, thereby obtaining a trained certificate type recognition model; wherein the certificate type recognition model includes an image feature extraction model and a text feature extraction model.

[0050] In this embodiment, AdamW is used to train and optimize the model, where the number of training rounds is set to 100.

[0051] Secondly, it should be noted that the expression of the loss function is:

[0052] , where Loss represents the loss function, N is the sample size of the batch; x i is the feature vector of the query sample in the first batch; x j is the feature vector of other certificate images in the first batch; y i is the feature vector of the certificate image at the same position as the query sample in the second batch; z j is the feature vector of the non-certificate image; τ is the temperature coefficient.

[0053] The simultaneous optimization and training of the image feature extraction model and the text feature extraction model will result in long model training cycles, large amounts of computation, and difficulty in convergence. To speed up model convergence, the model optimization training is divided into three stages, as follows:

[0054] The first stage is the parameter optimization of the image feature extraction model, specifically: extracting the first image feature vector of the training data through the image feature extraction model, calculating the image loss value of the first image feature vector through the loss function, and using the back gradient propagation algorithm to perform the initial update of the network parameters of the image feature extraction model according to the image loss value to obtain the image feature extraction model after the initial update.

[0055] It should be noted that the network parameters of the text extraction model are not involved in the calculation and parameter update at this time. Only the image feature vector of the extracted certificate is used to participate in the calculation of the loss function. This can achieve targeted optimization of the image feature extraction model.

[0056] The second stage is the parameter optimization of the text feature extraction model, specifically: extracting the first text feature vector of the training data through the text feature extraction model, calculating the text loss value of the first text feature vector through the loss function, and using the back gradient propagation algorithm to perform the initial update of the network parameters of the text feature extraction model according to the text loss value to obtain the text feature extraction model after the initial update.

[0057] It should be noted that at this time, the network parameters of the image feature extraction model do not participate in the calculation and parameter update. Only the extracted text feature vector is used to participate in the loss function calculation, so that the extraction of text feature vector has stronger semantic understanding ability, especially when dealing with terminology and formats in different types of certificates, it can provide more accurate representation.

[0058] The third stage is the joint optimization stage, specifically: extracting the second image feature vector and the second text feature vector of the training data respectively through the image feature extraction model and the text feature extraction model after the initial update; fusing the second image feature vector and the second text feature vector to obtain a multimodal feature vector, calculating the multimodal loss value of the multimodal feature vector through the loss function, and using the back gradient propagation algorithm to perform a second update on the network parameters of the image feature extraction model and the text feature extraction model after the initial update according to the multimodal loss value to obtain a trained certificate type recognition model.

[0059] Specifically, feature vector concatenation and standardization: The present invention forms a multimodal feature vector by standardizing and concatenating the image feature vector and the text feature vector. This feature vector fusion method not only effectively improves the representation ability of the model, but also improves the model's accuracy in determining the category of certificate images through deep learning optimization.

[0060] Model joint optimization: Through the joint optimization of image and text feature extraction modules, the system can better utilize the correlation between images and texts, reduce feature redundancy, and improve the accuracy and robustness of the model in complex certificate image determination tasks. Through the loss function, combined with the AdamW optimizer, the convergence speed and training effect of the model are further improved. In practical applications, after multiple rounds of optimization, the model can quickly and accurately complete the task of classifying certificate images.

[0061] It should be noted that in the third stage, the image feature extraction model optimized in the first stage and the text feature extraction model optimized in the second stage are both involved in the network parameter calculation and parameter update, and the model optimized in the first stage extracts the second image feature vector and the second text feature vector of the training data. At this time, the second image feature vector and the second text feature vector constitute a multimodal feature vector and participate in the calculation of the loss function. This stage is to simultaneously optimize the network parameters of the image feature extraction model and the text feature extraction model, improve the overall performance of the two models on multimodal data, and achieve the best fusion of image and text information.

[0062] As an example, the image feature extraction model provided in this embodiment uses vit-base-patch16-224-in21k in the Vision Transformer series model as the base model, and the text feature extraction model uses Ernie3.0-base-chinese as the base model. The ERNIE 3.0 series model (Large-Scale Knowledge Enhanced Pre-Training for Language Understanding And Generation) is a multi-paradigm unified pre-training framework based on knowledge enhancement, which aims to improve the performance of the model in natural language understanding and generation tasks by integrating autoregressive networks and autoencoder networks and introducing large-scale knowledge graphs. Ernie3.0-base-chinese, as a basic version of the series, also has these features and is specially optimized for the Chinese language. As those skilled in the art can understand, other base models can be used in addition to the above models.

[0063] S104, identifying the certificate image to be identified based on the trained certificate type identification model to obtain an identification result of the certificate type.

[0064] In this embodiment, regarding the identification of certificate types, first, the vector library is constructed: in the deployment and initialization stage, the multimodal feature vectors of known types of certificates (such as various versions of business licenses, qualification certificates, licenses, and license-related documents) are stored in the Faiss vector library in advance. For business licenses, feature vectors of business licenses of different years and formats are stored. Secondly, similarity calculation: For the multimodal feature vectors of the business license image to be determined, similarity retrieval is performed using cosine similarity through the Faiss vector library. In the high-dimensional vector space, the known category feature vectors that are most similar to the features of the certificate to be determined are quickly screened out. Finally, the category is determined: when the certificate category corresponding to the most similar feature vector is "corporate business license", it can be determined that the file is a corporate business license certificate.

[0065] In summary, in a certificate type identification method provided by an embodiment of the present invention, an optimized training method is designed for the image feature extraction model and the text feature extraction model in view of the specific needs of the certificate image, so that they are more adapted to the feature distribution and classification requirements of the certificate image. In particular, a method for inputting training data and a method for calculating the loss function are proposed, so that the optimized image feature extraction model and the text feature extraction model can more efficiently extract and characterize the key information in the certificate image, and effectively improve the expression ability of the feature vector. By jointly optimizing the image feature extraction model and the text feature extraction model, the image feature extraction model and the text feature extraction model can better utilize the correlation between the image and the text, reduce feature redundancy, and improve the accuracy and robustness of the model in complex certificate image determination tasks, thereby quickly and accurately completing the type identification task of the certificate image.

[0066] Please refer to Figure 2 , Figure 2 A functional block diagram of a certificate type identification device provided by an embodiment of the present invention, such as Figure 2 As shown, the device comprises:

[0067] An image data acquisition module 210 is used to acquire a non-certificate image data set and a certificate image data set having multiple certificate types, wherein each certificate type of the certificate image data includes at least two certificate images;

[0068] The training data generating module 220 is used to take one batch size of non-certificate images of the non-certificate image data set and two batch sizes of certificate images of the certificate image data set as training data each time; wherein the images of each batch of the certificate image data belong to different certificate categories, the certificate images at corresponding positions of the first batch and the second batch have the same category, and the certificate images at corresponding positions of the first batch and the second batch are different;

[0069] The model training module 230 is used to train the pre-built certificate type recognition model through training data until the loss function converges, thereby obtaining a trained certificate type recognition model; wherein the certificate type recognition model includes a parallel image feature extraction model and a text feature extraction model;

[0070] The type recognition module 240 is used to recognize the certificate image to be recognized based on the trained certificate type recognition model to obtain a recognition result of the certificate type.

[0071] A certificate type identification device in an embodiment of the present application is similar to the above Figure 1The certificate type identification method shown is a technical solution based on the same inventive concept. Through the detailed description of the certificate type identification method provided in the above embodiment, those skilled in the art can clearly understand the implementation process of a certificate type identification device in this embodiment, so for the sake of brevity of the specification, it will not be repeated here.

[0072] Accordingly, the present embodiment provides a certificate type recognition device, which designs an optimized training method for the image feature extraction model and the text feature extraction model in response to the specific needs of the certificate image, so that it is more suitable for the feature distribution and classification requirements of the certificate image. In particular, a method for inputting training data and a method for calculating the loss function are proposed, so that the optimized image feature extraction model and the text feature extraction model can more efficiently extract and characterize the key information in the certificate image, and effectively improve the expression ability of the feature vector. By jointly optimizing the image feature extraction model and the text feature extraction model, the image feature extraction model and the text feature extraction model can better utilize the correlation between the image and the text, reduce feature redundancy, and improve the accuracy and robustness of the model in complex certificate image determination tasks, thereby quickly and accurately completing the type recognition task of the certificate image.

[0073] In one embodiment, the loss function is expressed as:

[0074] , where Loss represents the loss function, N is the sample size of the batch; x i is the feature vector of the query sample in the first batch; x j is the feature vector of other certificate images in the first batch; y i is the feature vector of the certificate image at the same position as the query sample in the second batch; z j is the feature vector of the non-certificate image; τ is the temperature coefficient.

[0075] In one embodiment, the model training module 230 is specifically used to:

[0076] Extracting a first image feature vector of training data through an image feature extraction model, calculating an image loss value of the first image feature vector through a loss function, and performing a first update on a network parameter of the image feature extraction model according to the image loss value using a reverse gradient propagation algorithm to obtain an image feature extraction model after the first update;

[0077] Extracting a first text feature vector of training data through a text feature extraction model, calculating a text loss value of the first text feature vector through a loss function, and performing a first update on a network parameter of the text feature extraction model according to the text loss value using a reverse gradient propagation algorithm to obtain a first updated text feature extraction model;

[0078] Extracting a second image feature vector and a second text feature vector of the training data respectively through the image feature extraction model and the text feature extraction model after the initial update;

[0079] The second image feature vector and the second text feature vector are fused to obtain a multimodal feature vector. The multimodal loss value of the multimodal feature vector is calculated through the loss function. The network parameters of the image feature extraction model and the text feature extraction model after the initial update are simultaneously updated for the second time according to the multimodal loss value using the back gradient propagation algorithm to obtain a trained certificate type recognition model.

[0080] In one embodiment, the text feature extraction model is an OCR extraction model.

[0081] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 300 includes a processor 310, a memory 320, a communication interface 330, and at least one communication bus for connecting the processor 310, the memory 320, and the communication interface 330. The memory 320 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (PROM), or a portable CD-ROM, and the memory 320 is used for related instructions and data.

[0082] The communication interface 330 is used to receive and send data. The processor 310 may be one or more CPUs. When the processor 310 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor 310 in the electronic device 300 is used to read one or more programs 321 stored in the memory 320, and perform the following operations: obtaining a non-certificate image dataset and a certificate image dataset having multiple certificate types, wherein each certificate category of the certificate image data contains at least two certificate images; taking a batch size of non-certificate images of the non-certificate image dataset and two batch sizes of certificate images of the certificate image dataset as training data each time; wherein the images of each batch of the certificate image data belong to different certificate categories, the categories of the certificate images at the corresponding positions of the first batch and the second batch are the same, and the certificate images at the corresponding positions of the first batch and the second batch are different; training the pre-built certificate type recognition model through the training data until the loss function converges, and obtaining a trained certificate type recognition model; wherein the certificate type recognition model includes a parallel image feature extraction model and a text feature extraction model; recognizing the certificate image to be recognized based on the trained certificate type recognition model, and obtaining a recognition result of the certificate type.

[0083] It should be noted that the specific implementation of each operation can be Figure 1 The corresponding description of the method embodiment shown, the electronic device 300 can be used to execute a certificate type identification method of the above method embodiment of the present application, which will not be described in detail here.

[0084] In the embodiments of the present disclosure, a computer-readable storage medium is also provided, and the computer-readable storage medium is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the computer device and the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, and the storage space stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of a certificate type identification method in the above embodiment. Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0085] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A certificate type identification method, characterized in that: Methods include: Obtain a non-certificate image dataset and a certificate image dataset with multiple certificate types, wherein each certificate category of the certificate image data contains at least two certificate images; Each time, a batch of non-certificate images of the non-certificate image data set and two batches of certificate images of the certificate image data set are taken as training data; wherein, the images in each batch of the certificate image data belong to different certificate categories, the categories of the certificate images at the corresponding positions of the first batch and the second batch are the same, and the certificate images at the corresponding positions of the first batch and the second batch are different; The pre-built certificate type recognition model is trained by training data until the loss function converges to obtain a trained certificate type recognition model; wherein the certificate type recognition model includes a parallel image feature extraction model and a text feature extraction model; wherein the expression of the loss function is: , where Loss represents the loss function, N is the sample size of the batch; x i is the feature vector of the query sample in the first batch; x j is the feature vector of other certificate images in the first batch; y i is the feature vector of the certificate image at the same position as the query sample in the second batch; z j is the feature vector of the non-certificate image; τ is the temperature coefficient; the pre-built certificate type recognition model is trained through the training data until the loss function converges, and the trained certificate type recognition model is obtained, which is specifically: Extracting a first image feature vector of training data through an image feature extraction model, calculating an image loss value of the first image feature vector through a loss function, and performing a first update on a network parameter of the image feature extraction model according to the image loss value using a reverse gradient propagation algorithm to obtain an image feature extraction model after the first update; Extracting a first text feature vector of training data through a text feature extraction model, calculating a text loss value of the first text feature vector through a loss function, and performing a first update on a network parameter of the text feature extraction model according to the text loss value using a reverse gradient propagation algorithm to obtain a first updated text feature extraction model; Extracting a second image feature vector and a second text feature vector of the training data respectively through the image feature extraction model and the text feature extraction model after the initial update; The second image feature vector and the second text feature vector are fused to obtain a multimodal feature vector, a multimodal loss value of the multimodal feature vector is calculated by a loss function, and the network parameters of the image feature extraction model and the text feature extraction model after the initial update are updated for a second time according to the multimodal loss value by using a reverse gradient propagation algorithm to obtain a trained certificate type recognition model; The certificate image to be identified is identified based on the trained certificate type recognition model to obtain the identification result of the certificate type.

2. A certificate type identification method according to claim 1, characterized in that: The text feature extraction model is the ERNIE 3.0 model.

3. A certificate type identification device, characterized in that: The device includes: An image data acquisition module is used to acquire a non-certificate image data set and a certificate image data set having multiple certificate types; wherein each certificate type of the certificate image data contains at least two certificate images; A training data generation module is used to take one batch size of non-certificate images of the non-certificate image data set and two batch sizes of certificate images of the certificate image data set as training data each time; wherein the images in each batch of the certificate image data belong to different certificate categories, the certificate images at corresponding positions of the first batch and the second batch have the same category, and the certificate images at corresponding positions of the first batch and the second batch are different; The model training module is used to train the pre-built certificate type recognition model through training data until the loss function converges to obtain a trained certificate type recognition model; wherein the certificate type recognition model includes a parallel image feature extraction model and a text feature extraction model; wherein the expression of the loss function is: , where Loss represents the loss function, N is the sample size of the batch; x i is the feature vector of the query sample in the first batch; x j is the feature vector of other certificate images in the first batch; y i is the feature vector of the certificate image at the same position as the query sample in the second batch; z i is the feature vector of the non-certificate image; τ is the temperature coefficient; a model training module, specifically used to: extract the first image feature vector of the training data through the image feature extraction model, calculate the image loss value of the first image feature vector through the loss function, and use the reverse gradient propagation algorithm to perform the initial update of the network parameters of the image feature extraction model according to the image loss value to obtain the image feature extraction model after the initial update; extract the first text feature vector of the training data through the text feature extraction model, calculate the text loss value of the first text feature vector through the loss function, and use the reverse gradient propagation algorithm to perform the initial update of the network parameters of the text feature extraction model according to the text loss value to obtain the text feature extraction model after the initial update; respectively extract the second image feature vector and the second text feature vector of the training data through the image feature extraction model and the text feature extraction model after the initial update; fuse the second image feature vector and the second text feature vector to obtain a multimodal feature vector, calculate the multimodal loss value of the multimodal feature vector through the loss function, and use the reverse gradient propagation algorithm to perform the secondary update of the network parameters of the image feature extraction model and the text feature extraction model after the initial update according to the multimodal loss value to obtain a trained certificate type recognition model; The type recognition module is used to recognize the certificate image to be recognized based on the trained certificate type recognition model to obtain the recognition result of the certificate type.

4. A certificate type identification device according to claim 3, characterized in that: The text feature extraction model is the ERNIE 3.0 model.

5. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of a certificate type identification method as described in any one of claims 1 to 2 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of a certificate type identification method as claimed in any one of claims 1 to 2 are implemented.

Images