A zero-shot certificate authentication method based on a pre-trained large model

Abstract

The present application belongs to the technical field of certificate identification and forgery, and discloses a zero sample certificate identification and forgery method based on a pre-trained large model. First, the old version certificate anti-forgery feature description and a small amount of newly added feature description provided by the official are used to adaptively combine and construct a new version certificate anti-forgery description template library. Then, the text encoder of the pre-trained multi-modal large model Constrastive Language-Image Pre-training (CLIP) is used to extract the text features corresponding to the anti-forgery feature text description in the anti-forgery template library. In the identification and forgery stage, the pre-trained segmentation large model SegmentAnything Model (SAM) is used to segment the anti-forgery features from the certificate to be detected, and the image encoder of CLIP is used to encode the anti-forgery features to obtain the image features. By calculating the cosine similarity between the image features and the text features, it is determined whether the input certificate sample conforms to the anti-forgery features in the anti-forgery template library, thereby realizing the zero sample identification and forgery of the new type of certificate.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of certificate authentication, and particularly relates to a zero-shot certificate authentication method and system based on a pre-trained large model. BACKGROUND

[0002] Existing certificate authentication technologies mainly rely on supervised learning-based methods, which require a large number of labeled samples to train the model to ensure its effective identification of certificate anti-counterfeiting features. However, certificate designs vary from country to country and region to region, and the anti-counterfeiting features of certificates change constantly with version updates, making it very difficult to collect and label enough new certificate samples in practical applications. This high dependence on labeled data makes existing authentication models less adaptable to new certificates, severely affecting the accuracy and reliability of authentication.

[0003] Traditional certificate authentication technologies are usually based on visual feature matching, which compares watermarks, UV anti-counterfeiting patterns, and optically variable inks to determine the authenticity of certificates. These methods rely on explicit design and visibility of features, so they are not flexible enough to handle certificates with fast version updates and high feature diversity. Especially when faced with new certificate types that have not been seen before, these methods based on hand-designed features often lack sufficient generalization ability. In addition, the process of manually extracting and comparing anti-counterfeiting features is tedious and easily affected by human subjective factors, resulting in low efficiency and high error rates in actual operation.

[0004] In recent years, with the rapid development of deep learning, neural network-based certificate authentication methods have gradually replaced traditional hand-designed feature methods. These deep learning methods can automatically extract and identify complex anti-counterfeiting features by learning image features from a large number of samples. However, these methods still face the problem of insufficient sample data, especially when dealing with new certificates or rare types of certificates, the lack of labeled samples makes it difficult for the model to effectively learn the corresponding features, thereby reducing the accuracy of authentication. At the same time, some deep learning methods rely on specific certificate types and features, which makes them show poor generalization ability in the face of frequent changes in certificate versions.

[0005] With the development of globalization, the demand for cross-border travel and identity verification is increasing, and the requirements for certificate authentication in terms of security and real-time performance are also increasing. The existing authentication system is not able to cope with the challenges of new certificates, and there is an urgent need for a new method that can reduce the dependence on a large number of labeled samples and has high adaptability and generalization ability to cope with the changing certificate anti-counterfeiting feature design. This provides an opportunity for the development of new certificate authentication technology based on deep learning and large-scale pre-training models, thereby improving the automation and accuracy of certificate authentication and meeting the efficient and reliable needs in practical applications.

[0006] Through the above analysis, the problems and defects of the prior art are:

[0007] (1) The existing method still faces the problem of lack of sample data, especially when dealing with new certificates or rare types of certificates, the lack of labeled samples will make it difficult for the model to effectively learn the corresponding features, thereby reducing the accuracy of authentication.

[0008] (2) Some deep learning methods rely on specific certificate types and features, which makes them have poor generalization ability in the face of frequent changes in certificate versions. SUMMARY

[0009] In view of the problems existing in the prior art, the present application provides a zero-shot certificate authentication method based on a pre-trained large model.

[0010] The present application is implemented as follows: a zero-shot certificate authentication method based on a pre-trained large model comprises:

[0011] S1: using the adaptive combination of the anti-counterfeiting feature description of the old version certificate, combined with the small amount of description provided by the official, to construct a new version certificate anti-counterfeiting description template library;

[0012] S2: using the text encoder of the pre-trained multi-modal large model CLIP to extract the text features corresponding to the anti-counterfeiting feature text description in the anti-counterfeiting template library;

[0013] S3: using the pre-trained segmentation large model SAM to segment the certificate image to be detected to extract the anti-counterfeiting features;

[0014] S4: using the image encoder of the pre-trained multi-modal large model CLIP to encode the extracted anti-counterfeiting features to obtain the corresponding image features;

[0015] S5: by calculating the cosine similarity between the image features and the text features, it is judged whether the input certificate sample has the anti-counterfeiting features in the anti-counterfeiting template library, and when the number of anti-counterfeiting features that meet the target threshold is less than the target threshold, the input sample is judged as a fake certificate.

[0016] Further, the adaptive combination in step S1 includes analyzing the anti-fake feature description of the old version license and generating multiple combination modes to cover various anti-fake feature combination forms in the new license.

[0017] Further, the text features extracted by the text encoder in step S2 include feature vectors obtained through cross-modal contrast learning, which are used for feature matching with image features.

[0018] Further, in step S3, the segmentation large model SAM locates the key anti-fake region in the license through an automatic segmentation method to ensure the accuracy of the subsequent encoding process.

[0019] Further, the calculation of the cosine similarity in step S4 is used to measure the similarity between the text features in the template library and the image features of the input license sample. When the similarity between the text features and the image features in the anti-fake template library is less than the preset threshold, it is determined that the anti-fake feature in the template library does not exist.

[0020] Further, the zero-shot license authentication method based on the pre-trained large model is suitable for authenticating various licenses including but not limited to passports, identity cards, and visas.

[0021] Another object of the present application is to provide a zero-shot license authentication system based on a pre-trained large model, comprising:

[0022] The construction module is used to construct a new version license anti-fake description template library by utilizing adaptive combination of anti-fake feature descriptions of old version licenses and combining a small amount of descriptions provided by the official;

[0023] The extraction module is used to extract text features corresponding to anti-fake feature text descriptions in the anti-fake template library by using a pre-trained multi-modal large model CLIP text encoder;

[0024] The segmentation module is used to extract anti-fake features by segmenting the to-be-detected license image using a pre-trained segmentation large model SAM;

[0025] The encoding module is used to encode the extracted anti-fake features to obtain corresponding image features by using an image encoder of a pre-trained multi-modal large model CLIP;

[0026] The judgment module is used to judge whether the input license sample has anti-fake features in the anti-fake template library by calculating the cosine similarity between the image features and the text features. When the number of conforming anti-fake features is less than the target threshold, the input sample is determined to be a fake license.

[0027] Another object of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to perform the steps of the method for identifying a fake license based on a pre-trained large model.

[0028] Another object of the present application is to provide a computer-readable storage medium storing a computer program, the computer program being executed by a processor to cause the processor to perform the steps of the method for identifying a fake license based on a pre-trained large model.

[0029] Another object of the present application is to provide an information data processing terminal for implementing the system for identifying a fake license based on a pre-trained large model.

[0030] In combination with the above technical solutions and the technical problems solved, the technical solution of the present application has the following advantages and positive effects:

[0031] Firstly, the present application reduces the dependence on labeled data. By introducing a pre-trained large model, the present application effectively reduces the dependence on labeled samples. Compared with the high sample requirement of traditional supervised learning methods, the present application can achieve identification by combining existing anti-fake features and a small amount of official description information, significantly improving adaptability and flexibility. Improve the adaptability to new licenses: the present application can quickly respond to changes in new licenses by adaptively combining anti-fake features and generating an anti-fake template library. This method improves the generalization ability of the model to new licenses, reduces the difficulty of identification due to insufficient new license samples, and ensures that the identification system can still maintain high precision in constantly changing actual scenarios. Achieve efficient and automated identification: the present application realizes automatic segmentation and coding of anti-fake features through SAM model and CLIP model, and all key steps are automatically completed by the model, greatly reducing the need for human intervention. Compared with traditional methods that rely on manual feature extraction, the present application has obvious advantages in real-time and accuracy of identification, and can provide stable and reliable identification results in complex application environments.

[0032] Secondly, the technical solution of the present application can realize automatic identification of licenses under zero sample conditions by combining pre-trained multi-modal large models (such as CLIP) and segmentation models (such as SAM), filling the gap in the existing technology that lacks generalization ability for new licenses. In view of the limitation of existing automatic identification technology for licenses that relies on labeled samples, the present application proposes a technical solution based on a pre-trained large model, solving the identification difficulty caused by insufficient samples of new licenses. At the same time, by automatically extracting key anti-fake regions through a segmentation large model, the present application overcomes the shortcomings of traditional manually designed features in terms of diversity and flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a pre-training large model-based zero-shot certificate identification method flowchart provided by the embodiment of the present application.

[0034] Figure 2 is a pre-training large model-based zero-shot certificate identification system structure block diagram provided by the embodiment of the present application.

[0035] Figure 3 is a detailed process schematic diagram of the pre-training large model-based zero-shot certificate identification method provided by the embodiment of the present application.

[0036] Figure 4 is a pre-training large model-based zero-shot certificate identification method flowchart provided by the embodiment of the present application.

[0037] Figure 5 is a pre-training large model-based zero-shot certificate identification method flowchart provided by the embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0039] Embodiment one: new ID card anti-counterfeiting detection

[0040] 1) Step S1: Anti-counterfeiting feature template library construction

[0041] For the anti-counterfeiting features of the new ID card, including micro-text, watermark pattern, rainbow holographic anti-counterfeiting mark, etc., combined with the old ID card anti-counterfeiting feature description and a small amount of feature information provided by the official, an anti-counterfeiting description template library is constructed. The description features in the template library are as follows: "the front lower part has micro-text 'CHINA', which shows at a certain angle", "the color of the optically variable ink area changes with the viewing angle".

[0042] 2) Step S2: Text feature encoding

[0043] The CLIP text encoder is used to encode the description in the template library, and the text feature vector of each anti-counterfeiting feature is extracted.

[0044] 3) Step S3: To-be-detected ID card segmentation

[0045] The to-be-detected ID card image is input into the system, and the pre-training segmentation large model SAM is used to segment the ID card image, automatically extracting the key anti-counterfeiting regions, such as the micro-text region, the watermark region, and the optically variable region.

[0046] 4) Step S4: Image feature encoding

[0047] The anti-counterfeiting feature area extracted by the SAM is encoded by the CLIP image encoder to generate an image feature vector representing the visual features of each anti-counterfeiting area.

[0048] 5) Step S5: Feature matching and authenticity judgment

[0049] The cosine similarity between the image feature vector and the text feature vector is calculated to match the micro-text, watermark, and optically variable region one by one. If the number of matching features of the input ID card is less than the set threshold (e.g., less than 5), the ID card is determined to be a fake certificate; otherwise, it is determined to be a true ID card.

[0050] In the detection of the new version of the ID card, the embodiment can accurately identify fake ID cards under zero-sample training conditions, with a detection accuracy of 98%.

[0051] Example Two: New Passport Anti-Counterfeiting Detection

[0052] 1) Step S1: Construction of Anti-Counterfeiting Feature Template Library

[0053] The anti-counterfeiting features of the new passport are described, such as transparent watermark, anti-counterfeiting lines, ultraviolet fluorescent patterns, etc., and combined with the anti-counterfeiting description of the old passport, an anti-counterfeiting description template library is constructed. For example, the description includes: "displaying 'national emblem' pattern under ultraviolet light", "central area of the cover has a relief anti-counterfeiting line".

[0054] 2) Step S2: Text Feature Encoding

[0055] The text description in the anti-counterfeiting template library is encoded by the CLIP text encoder to generate the corresponding text feature vector.

[0056] 3) Step S3: Passport Image Segmentation

[0057] The passport image to be detected is input into the system, and the SAM model is used to automatically segment the image and locate the anti-counterfeiting areas, including ultraviolet fluorescent patterns, watermark areas, and relief lines.

[0058] 4) Step S4: Image Feature Encoding

[0059] The anti-counterfeiting area image segmented by the SAM is encoded by the CLIP image encoder to obtain the image feature vector, which is used for matching with the text features in the template library.

[0060] 5) Step S5: Similarity Calculation and Counterfeiting Judgment

[0061] The system calculates the cosine similarity between the image features and the text features to determine whether the anti-counterfeit features of each region meet the template description. If the number of anti-counterfeit features meets the target threshold (such as 4 features), the system determines that the passport is a fake passport; otherwise, it is a true passport.

[0062] In the passport anti-counterfeit detection, the system realizes efficient detection of the new version of the passport, can quickly identify the key anti-counterfeit features such as ultraviolet fluorescence and watermark, and the detection accuracy reaches more than 96%.

[0063] The above two embodiments respectively perform anti-counterfeit detection on the ID card and the passport, realize zero-shot anti-counterfeit detection through the combination of CLIP and SAM pre-training large models, and are suitable for true and false identification of various new versions of certificates.

[0064] As shown in Figure 1 , a zero-shot certificate identification method based on a pre-training large model provided by the embodiment of the application includes the following steps:

[0065] S1: Using the adaptive combination of the anti-counterfeit feature descriptions of the old version of the certificate, combined with a small amount of description provided by the official, a new version of the certificate anti-counterfeit description template library is constructed;

[0066] S2: The text encoder of the pre-trained multi-modal large model CLIP is used to extract the text features corresponding to the anti-counterfeit feature text descriptions in the anti-counterfeit template library;

[0067] S3: The pre-trained segmentation large model SAM is used to segment the image to be detected, and the anti-counterfeit features are extracted;

[0068] S4: The pre-trained multi-modal large model CLIP is used to encode the extracted anti-counterfeit features, and the corresponding image features are obtained;

[0069] S5: The cosine similarity between the image features and the text features is calculated to determine whether the input certificate sample meets the anti-counterfeit features in the anti-counterfeit template library. When the number of anti-counterfeit features meets the target threshold, the input sample is determined to be a fake certificate.

[0070] Firstly, the anti-counterfeit features of the old version of the certificate are analyzed and described, combined with a small amount of new version of the certificate anti-counterfeit feature text description provided by the official, and a new version of the certificate anti-counterfeit description template library is constructed. The anti-counterfeit features include various identifiable visual and texture features, such as watermark, embossing, optically variable pattern, microtext, etc. Through the adaptive combination, the old and new certificate feature descriptions are integrated, so that the template library covers more anti-counterfeit feature types, thereby meeting the zero-shot detection requirement. This template library provides standardized reference data for subsequent text encoding and feature matching.

[0071] For the constructed anti-counterfeiting template library, the text encoder of the pre-trained multi-modal large model CLIP is used to encode the anti-counterfeiting feature description. The text encoder of the CLIP model has the ability to map natural language descriptions to a high-dimensional feature space through large-scale training data. In this process, the input anti-counterfeiting feature text description is processed by the encoder to extract text feature vectors, which have high semantic expression ability and can be used as the basis for similarity calculation with image features.

[0072] For the input image to be detected, the pre-trained segmentation model SAM (Segment Anything Model) is used for image segmentation processing. The SAM model can perform high-precision region segmentation on the certificate image without the need for a large number of training samples, automatically identifying and extracting anti-counterfeiting feature regions such as watermarks, background textures, micro-texts, and other details. Through the segmentation results, the system can effectively extract the anti-counterfeiting feature regions of the certificate, providing accurate input data for subsequent image feature encoding, avoiding interference from irrelevant regions, and improving the accuracy of detection.

[0073] The image feature of the anti-counterfeiting feature region extracted by the SAM model is encoded using the pre-trained CLIP image encoder. The image encoder of CLIP has the ability to map image content to the same high-dimensional feature space as text features through large-scale image-text matching training. In this step, the extracted image anti-counterfeiting features are processed by the image encoder to obtain image feature vectors that can accurately express the visual information of the anti-counterfeiting features in the image, and cross-modal alignment with text features is achieved.

[0074] Cross-modal anti-counterfeiting feature matching is achieved by calculating the cosine similarity between the image feature vector and the text feature vector. Specifically, the system calculates the similarity between the image features of the input certificate sample and all anti-counterfeiting text features in the template library. When the similarity is higher than a certain threshold, it is considered that the anti-counterfeiting feature exists in the certificate; otherwise, it is considered that the feature is missing. The system counts the total number of anti-counterfeiting features that meet the template library in the input certificate and compares it with the set target threshold to determine the authenticity of the certificate.

[0075] After the similarity calculation is completed, the system integrates all the matching results of the anti-counterfeiting features to make a final judgment. When the number of anti-counterfeiting features that meet the template library in the input sample is lower than the target threshold, the system determines that the input sample is a fake certificate; otherwise, it is considered to be a true certificate. This method combines the powerful capabilities of the pre-trained large models CLIP and SAM to quickly and accurately identify the authenticity of the certificate under the condition of zero samples, and has high efficiency and universality, especially suitable for anti-counterfeiting verification and system updating of new version certificates.

[0076] In summary, the method provided by the application realizes cross-modal anti-fake feature matching and authenticity identification based on a pre-trained large model through text and image feature encoding, image segmentation, and cosine similarity calculation, solves the problem of zero-sample training in certificate anti-fake identification, and effectively improves the intelligent level and accuracy of anti-fake detection.

[0077] The adaptive combination in step S1 provided by the embodiment of the application includes analyzing the anti-fake feature description of the old version certificate and generating multiple combination modes to cover various anti-fake feature combination forms in the new certificate.

[0078] The text features extracted by the text encoder in step S2 provided by the embodiment of the application include feature vectors obtained through cross-modal contrast learning, which are used for feature matching with image features.

[0079] The segmentation large model SAM in step S3 provided by the embodiment of the application locates the key anti-fake area in the certificate through an automatic segmentation method to ensure the accuracy of the subsequent encoding process.

[0080] The calculation of the cosine similarity in step S4 provided by the embodiment of the application is used to measure the similarity between the text features in the template library and the image features of the input certificate sample. When the similarity between the text features and the image features in the anti-fake template library is less than a preset threshold, it is determined that the anti-fake feature in the template library does not exist.

[0081] The zero-sample certificate identification method based on the pre-trained large model provided by the embodiment of the application is applicable to the identification of various certificates including but not limited to passports, identity cards, and visas.

[0082] As shown in Figure 2 The zero-sample certificate identification system based on the pre-trained large model provided by the embodiment of the application includes:

[0083] The construction module is configured to construct a new version certificate anti-fake description template library by using adaptive combination of anti-fake feature descriptions of old version certificates and combining a small amount of descriptions provided by an official;

[0084] The extraction module is configured to extract text features corresponding to anti-fake feature text descriptions in the anti-fake template library by using a text encoder of a pre-trained multi-modal large model CLIP;

[0085] The segmentation module is configured to extract anti-fake features by segmenting a to-be-detected certificate image by using a pre-trained segmentation large model SAM;

[0086] The encoding module is configured to encode the extracted anti-fake features by using an image encoder of the pre-trained multi-modal large model CLIP to obtain corresponding image features;

[0087] A judgment module is configured to judge whether the input license sample has the anti-counterfeit features in the anti-counterfeit template library by calculating the cosine similarity between the image features and the text features, and when the number of the matched anti-counterfeit features is less than a target threshold, the input sample is determined as a fake license.

[0088] Another object of the present application is to provide a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to enable the processor to perform the steps of the zero-shot license authentication method based on a pre-trained large model.

[0089] Another object of the present application is to provide a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to enable the processor to perform the steps of the zero-shot license authentication method based on a pre-trained large model.

[0090] Another object of the present application is to provide an information data processing terminal for implementing the zero-shot license authentication system based on a pre-trained large model.

[0091] The present application is embodied as follows:

[0092] As shown in the accompanying drawings, Figure 3 The present application provides a zero-shot license authentication method based on a pre-trained large model, which reduces the dependence on labeled sample data by combining a pre-trained large model and an adaptive combination technique. The specific implementation steps are as follows:

[0093] S1: Using the adaptive combination of the anti-counterfeit feature descriptions of the old version of the license, combined with a small amount of description provided by the official, a new version of the license anti-counterfeit description template library is constructed.

[0094] Specifically, the anti-counterfeit features of the old version of the license are analyzed and summarized, and corresponding text descriptions are sorted out. Each anti-counterfeit feature such as watermark, optically variable ink, and ultraviolet spectrum pattern will be summarized as a text description. For example, the "ultraviolet pattern" in the old version of the passport is described as "stripes visible under ultraviolet light". Then, these text descriptions are split and combined to generate multiple possible anti-counterfeit feature descriptions. For example, for the newly added ultraviolet anti-counterfeit pattern in the new version of the passport, "ultraviolet wave stripes" and "plant pattern" can be split to form multiple possible anti-counterfeit feature descriptions. Finally, these split descriptions are used to construct the anti-counterfeit template library of the new version of the license.

[0095] S2: Using the text encoder of the pre-trained multi-modal large model CLIP, the text features corresponding to the anti-counterfeit feature text descriptions in the anti-counterfeit template library are extracted.

[0096] Specifically, the text encoder of the pre-trained multi-modal large model CLIP is used to encode the M pieces of anti-counterfeiting feature descriptions sorted out in step S1. The text encoder of the CLIP model can convert these text descriptions into feature vectors where each vector corresponds to an anti-counterfeiting feature description in the anti-counterfeiting template library.

[0097] S3: Use the pre-trained segmentation large model SAM to segment the to-be-detected certificate image and extract anti-counterfeiting features.

[0098] Specifically, as shown in Figure 4 , the pre-trained segmentation large model Segment Anything Model (SAM) is used to process the to-be-detected certificate image, and the key anti-counterfeiting feature region in the certificate is automatically segmented. The SAM model has zero-shot segmentation capability, which identifies and segments the region related to the anti-counterfeiting feature from the new version of the certificate sample, ensuring that the subsequent encoding process is not disturbed by the background.

[0099] S4: Use the image encoder of the pre-trained multi-modal large model CLIP to encode the extracted anti-counterfeiting features to obtain corresponding image features.

[0100] Specifically, the pre-trained CLIP image encoder is used to encode the N anti-counterfeiting features segmented from the SAM model. The image encoder of the CLIP converts the anti-counterfeiting feature region in the image into an image feature vector The CLIP model ensures that the text features and image features can be compared in the same feature space through cross-modal contrast learning. Therefore, after the image features are extracted, the image feature vector can be matched with the text feature vector across modalities.

[0101] S5: By calculating the cosine similarity between the image features and the text features, it is determined whether the input certificate sample has anti-counterfeiting features that match the anti-counterfeiting template library. When the number of anti-counterfeiting features that match is less than the target threshold, the input sample is determined to be a fake certificate.

[0102] Specifically, as shown in Figure 5 , for each anti-counterfeiting feature description, the cosine similarity between its corresponding text feature vector T m and all input image feature vectors I n is calculated, and the maximum cosine similarity corresponding to the anti-counterfeiting feature is taken as the probability p m that the anti-counterfeiting feature matches, and the calculation formula is as follows:

[0103]

[0104] A threshold value a (0 < a < 1) is set, and when the anti-counterfeiting feature probability pm When the number T exceeds the preset threshold a, it is considered that the input sample contains the anti-fake feature. Therefore, the number T of the anti-fake features that meet the condition is:

[0105]

[0106] A target threshold b (0 < b < 1) is set. If the number T of the anti-fake features that meet the condition is less than b x M (M is the total number of features in the template library), it is determined that the input sample is a fake certificate. Otherwise, it is determined to be a true certificate.

[0107] The present application can be applied to the field of certificate inspection, and the technology can be applied in airport border inspection stations and the like to realize the determination of the authenticity of new certificates.

[0108] The present application is specifically applied to the following embodiments:

[0109] Device requirements:

[0110] (1) A multi-spectral image acquisition device for acquiring multi-spectral images of certificates, such as visible light, ultraviolet light, infrared light, etc.

[0111] (2) A traditional recognition and identification system for initially determining the matching condition of the certificate and the old version certificate template.

[0112] (3) A zero-sample certificate identification system for running the identification method based on the pre-trained large model described in the present application to perform zero-sample identification of the certificate.

[0113] (4) A result feedback terminal for displaying the identification result.

[0114] Specific implementation steps:

[0115] S1: Obtain the full-spectrum image of the certificate through the multi-spectral image acquisition device, including the printed information under visible light, the anti-fake pattern under ultraviolet light, and the hidden watermark under infrared light. After the acquisition is completed, the image is transmitted to the traditional recognition and identification system for preliminary comparison.

[0116] S2: Analyze the collected certificate image using the traditional recognition and identification system, and determine whether the certificate meets the specific anti-fake features of the old version certificate through template comparison. If the certificate meets the existing certificate template, it is directly determined to be a true certificate; if the system cannot identify it, the collected image data is input into the zero-sample certificate identification system for further identification.

[0117] S3: Analyze the collected certificate image using the zero-sample certificate identification system to determine whether the certificate is a real new certificate.

[0118] S4: The authentication result is transmitted to the border inspection staff through the result feedback terminal. If the passport is determined to be true, the passenger is released; if the passport is determined to be fake, the system is marked as abnormal, prompting the border inspection personnel to manually review, and generating a detailed anti-fake feature detection report, including the feature matching similarity of each segmented region.

[0119] Related evidence of the technical effects achieved by the embodiments of the present application.

[0120] SAM is a pre-trained segmentation large model, which is trained by a large amount of images and labeled data, has strong generalization ability, and can complete the segmentation task of new images without labeled data. This feature is particularly suitable for extracting certificate anti-fake features, because the positions and shapes of anti-fake features in different certificates are diverse, but SAM can quickly identify regions with significant features based on image structure information. Without special training for anti-fake features, it can automatically segment key feature regions such as ultraviolet patterns and optically variable ink regions, ensuring that the segmentation results cover all possible anti-fake information.

[0121] CLIP is a pre-trained multi-modal large model, which uses cross-modal contrast learning technology to map text descriptions and image features into a unified feature space, so that text descriptions and corresponding image features can be directly calculated for similarity. This ability enables CLIP to use text descriptions in the anti-fake template library to identify corresponding features in new certificates without additional labeled data. This method not only has high computational efficiency, but also effectively quantifies the semantic relevance of image and text features, and is suitable for complex anti-fake feature matching tasks.

[0122] By combining the zero-shot ability of SAM and CLIP, the present application can complete the anti-fake task of new certificates in an efficient, flexible and accurate manner. The SAM model solves the problem of precise segmentation of key anti-fake regions, overcoming the limitations of traditional hand-designed feature segmentation, and providing high-quality input for subsequent recognition tasks. The CLIP model uses cross-modal contrast learning technology to quantify the similarity between text and image features through cosine similarity, making it possible to identify anti-fake features based on text descriptions, significantly improving the accuracy and efficiency of identification. This combination of technologies not only realizes efficient extraction and matching of anti-fake features under zero-shot conditions, but also has good scalability and adaptability, providing a practical and advanced solution to the anti-fake identification of new certificates.

[0123] It should be noted that embodiments of the present application can be realized by hardware, software, or a combination of software and hardware. The hardware portion can be realized by a special logic; the software portion can be stored in a memory and executed by a proper instruction execution system, such as a microprocessor or a specially designed hardware. A person of ordinary skill in the art can understand that the above-mentioned apparatus and method can be realized by computer executable instructions and / or included in processor control codes, such as a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The apparatus of the present application and its modules can be realized by a hardware circuit, such as a very large scale integrated circuit or a gate array, a semiconductor, such as a logic chip, a transistor, or a programmable hardware device, such as a field programmable gate array, a programmable logic device, or the like, by software executed by various types of processors, or by a combination of the above-mentioned hardware circuit and software, such as firmware.

[0124] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any modification, equivalent replacement, and improvement within the technical range disclosed by the present application, and within the spirit and principle of the present application, should be included in the protection scope of the present application.

Claims

1. A zero-shot authentication method for documents based on a pre-trained large model, characterized in that, Includes the following steps: S1: Utilize the adaptive combination of anti-counterfeiting feature descriptions of the old version of the certificate, combined with a small number of descriptions provided by the official authorities, to construct a new version of the certificate anti-counterfeiting description template library; S2: The text encoder of the pre-trained multimodal large model CLIP is used to extract the text features corresponding to the anti-counterfeiting feature text descriptions in the anti-counterfeiting template library; S3: Use the pre-trained segmentation model SAM to segment the certificate image to be detected and extract anti-counterfeiting features; S4: The extracted anti-counterfeiting features are encoded using a pre-trained multimodal large model CLIP image encoder to obtain the corresponding image features; S5: By calculating the cosine similarity between image features and text features, determine whether the input certificate sample matches the anti-counterfeiting features in the anti-counterfeiting template library. If the number of matching anti-counterfeiting features is less than the target threshold, the input sample is judged to be a fake certificate.

2. The zero-sample authentication method based on a pre-trained large model as described in claim 1, characterized in that, The adaptive combination in step S1 includes analyzing the anti-counterfeiting feature descriptions of the old version of the certificate and generating multiple combination methods to cover the various anti-counterfeiting feature combination forms in the new certificate.

3. The zero-sample authentication method based on a pre-trained large model as described in claim 1, characterized in that, The text features extracted by the text encoder in step S2 include feature vectors obtained through cross-modal contrastive learning, which are used for feature matching with image features.

4. The zero-sample authentication method based on a pre-trained large model as described in claim 1, characterized in that, In step S3, the segmentation model SAM uses an automatic segmentation method to locate key anti-counterfeiting areas in the certificate to ensure the accuracy of the subsequent encoding process.

5. The zero-sample authentication method based on a pre-trained large model as described in claim 1, characterized in that, The calculation of cosine similarity in step S5 is used to measure the similarity between text features in the template library and image features of the input certificate sample. When the similarity between text features and image features in the anti-counterfeiting template library is less than a preset threshold, it is determined that the anti-counterfeiting feature in the template library does not exist.

6. The zero-sample authentication method based on a pre-trained large model as described in claim 1, characterized in that, The zero-shot document authentication method based on pre-trained large models is applicable to the authentication of various documents, including passports, ID cards, and visas.

7. A zero-sample document authentication system based on a pre-trained large model, implementing the zero-sample document authentication method based on any one of claims 1-6, characterized in that, The zero-shot authentication system based on a pre-trained large model includes: The module is used to build a new version of the certificate anti-counterfeiting description template library by adaptively combining the anti-counterfeiting feature descriptions of the old version of the certificate with a small number of descriptions provided by the official. The extraction module is used to extract text features corresponding to anti-counterfeiting feature text descriptions from the anti-counterfeiting template library using a pre-trained multimodal large model CLIP text encoder. The segmentation module is used to segment the image of the certificate to be detected using the pre-trained segmentation model SAM and extract anti-counterfeiting features. The encoding module is used to encode the extracted anti-counterfeiting features using a pre-trained multimodal large model CLIP image encoder to obtain the corresponding image features; The judgment module is used to determine whether the input certificate sample matches the anti-counterfeiting features in the anti-counterfeiting template library by calculating the cosine similarity between image features and text features. When the number of matching anti-counterfeiting features is less than the target threshold, the input sample is judged to be a fake certificate.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the zero-sample authentication method based on a pre-trained large model as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the zero-sample authentication method based on a pre-trained large model as described in any one of claims 1-6.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the zero-sample certificate authentication system based on a pre-trained large model as described in claim 7.

Images