Certificate Anti-counterfeiting detection method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses an anti-counterfeiting detection method for a certificate, which comprises the following steps of: retrieving standard characteristics of a normal certificate image matched with multi-modal characteristics of a certificate image to be detected from a positive sample knowledge base, and retrieving counterfeit characteristics of a counterfeit certificate image matched with the multi-modal characteristics from a negative sample knowledge base; and generating enhanced prompt information according to the multi-modal features, the standard features and the forgery features, and performing anti-counterfeiting detection on the to-be-detected certificate image under the guidance of the enhanced prompt information through a pre-trained MLLM. Through the method, the MLLM can perform anti-counterfeiting detection on the certificate image to be detected by taking the similarity between the multi-modal feature of the certificate image to be detected and the standard feature of the positive sample and the similarity between the multi-modal feature of the certificate image to be detected and the forgery feature of the negative sample as the main basis under the guidance of the enhancement prompt information based on a retrieval enhancement generation method; and all tiny anomalies in the to-be-detected certificate image are not too sensitive, so that the accuracy of anti-counterfeiting detection can be effectively improved.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202610115638.2 (22) Application Date 2026.01.27 (71) Applicant Ant Blockchain Technology (Shanghai) Co., Ltd. Address Room 803, 8th Floor, No. 618 Waima Road, Huangpu District, Shanghai 200010 (72) Inventor He Wenwen (74) Patent Agency Shanghai Dongxin Patent & Trademark Agency (General Partnership) 31228 Patent Attorney Yang Danli (51) Int.Cl. G06V 10 / 764 (2022.01) G06V 10 / 82 (2022.01) G06F 16 / 334 (2025.01) G06V 10 / 40 (2022.01) G06V 10 / 74(2022.01) G06V 30 / 148(2022.01) G06N 3 / 09(2023.01) (54) Invention Title: A method, apparatus, storage medium and electronic device for anti-counterfeiting detection of a certificate (57) Abstract: This specification discloses an anti-counterfeiting detection method for a certificate. The method retrieves standard features of a normal certificate image that match the multimodal features of the certificate image to be detected from a positive sample knowledge base, retrieves forgery features of a forged certificate image that match the multimodal features from a negative sample knowledge base, and generates enhanced prompt information based on the multimodal features, standard features and forged features. Through a pre-trained MLLM, the certificate image to be detected is subjected to anti-counterfeiting detection under the guidance of the enhanced prompt information. Using the above method, MLLM can perform anti-counterfeiting detection on document images based on the method of retrieval enhancement and guided by enhanced prompts. It mainly relies on the similarity between the multimodal features of the document image to be detected and the standard features of positive samples and the counterfeit features of negative samples, without being overly sensitive to all minor anomalies in the document image, thus effectively improving the accuracy of anti-counterfeiting detection. Claims (2 pages), Description (8 pages), Drawings (2 pages), CN 121837790 A, 2026.04.10, CN 1 21 83 77 90 A. 1. A method for anti-counterfeiting detection of a document, the method comprising: acquiring an image of a document to be detected; identifying the document type based on the image of the document to be detected; extracting multimodal features from the image of the document to be detected, and based on the multimodal features, retrieving standard features of normal document images matching the multimodal features from a preset positive sample knowledge base corresponding to the document type, and retrieving forgery features of forged document images matching the multimodal features from a preset negative sample knowledge base corresponding to the document type; generating enhanced prompt information based on the multimodal features, standard features, and forgery features;The enhanced prompt information is input into a pre-trained multimodal large language model (MLLM) to perform anti-counterfeiting detection on the document image to be detected based on the enhanced prompt information. 2. The method as described in claim 1, further comprising extracting multimodal features from the document image to be detected, specifically including: determining, according to the document type, pre-defined key image regions in the document image to be detected for the document type; extracting local visual features from each key image region as local visual features to be detected; and determining, according to the document type, pre-defined key text regions in the document image to be detected for the document type; and extracting text features from each key text region in the document image to be detected. 3. The method as described in claim 2, wherein retrieving standard features of a normal document image matching the multimodal features from a preset positive sample knowledge base corresponding to the document type specifically includes: for each local visual feature to be detected, retrieving a specified number of standard local visual features with the highest similarity to the local visual feature to be detected from the preset positive sample knowledge base corresponding to the document type, all of which are used as standard features matching the local visual feature to be detected; and extracting visual feature anti-counterfeiting detection rules and text feature anti-counterfeiting detection rules corresponding to the document type from the positive sample knowledge base. 4. The method as described in claim 3, wherein generating enhanced prompt information based on the multimodal features, standard features, and counterfeiting features specifically includes: for each local visual feature to be detected, determining a first similarity between the local visual feature to be detected and the standard features matching the local visual feature to be detected; generating enhanced prompt information including at least the multimodal features, the counterfeiting features, the visual feature anti-counterfeiting detection rules, the text feature anti-counterfeiting detection rules, the standard features matching each local visual feature to be detected, and the first similarity determined for each local visual feature to be detected. 5. The method of claim 2, wherein retrieving forgery features of a forged document image matching the multimodal features from a preset negative sample knowledge base corresponding to the document type specifically includes: for each local visual feature to be detected, retrieving a specified number of forged local visual features with the highest similarity to the local visual feature to be detected from the preset negative sample knowledge base corresponding to the document type, as forged features matching the local visual feature to be detected; and extracting device information corresponding to the devices that historically uploaded each negative sample of the document type from the negative sample knowledge base, wherein the device information includes the geographical location of the device and / or the address information of the device. 6. The method of claim 5, wherein generating enhanced prompt information based on the multimodal features, standard features, and forged features specifically includes: for each local visual feature to be detected, determining the local visual feature to be detected and the forged features matching the local visual feature to be detected.The second similarity between the forged features of the visual features; and, determining the device information corresponding to the device that acquired the image of the document to be detected (Claim 1 / 2 page 2 CN 121837790 A); generating enhanced prompt information including at least the multimodal features, the standard features, the device information corresponding to the device that acquired the image of the document to be detected, the device information corresponding to the devices that historically uploaded each negative sample of the document type, the forged features matching each local visual feature to be detected, and the second similarity determined for each local visual feature to be detected. 7. A document anti-counterfeiting detection device, the device comprising: an acquisition module for acquiring a document image to be detected; a document type recognition module for identifying the document type of the document based on the document image to be detected; a feature extraction and retrieval module for extracting multimodal features from the document image to be detected, and retrieving standard features of normal document images matching the multimodal features from a preset positive sample knowledge base corresponding to the document type, and retrieving forgery features of forged document images matching the multimodal features from a preset negative sample knowledge base corresponding to the document type; a prompt generation module for generating enhanced prompt information based on the multimodal features, standard features, and forgery features; and a detection module for inputting the enhanced prompt information into a pre-trained multimodal large language model (MLLM) to perform anti-counterfeiting detection on the document image to be detected using the MLLM based on the enhanced prompt information. 8. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in any one of claims 1-6. 9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in any one of claims 1-6. 10. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method described in any one of claims 1-6. Claims 2 / 2 Page 3 CN 121837790 A Method, Apparatus, Storage Medium and Electronic Device for Anti-counterfeiting Detection of Certificates Technical Field

[0001] This specification relates to the field of computer technology, and particularly to a method, apparatus, storage medium and electronic device for anti-counterfeiting detection of certificates. Background Art

[0002] With the development of the times, electronic identity authentication (eKYC) technology has been widely used in various fields.

[0003] Through eKYC technology, users can directly use their terminals, such as mobile phones, tablets, and smartwatches, to collect images of their own identification documents and upload them to the server, allowing the server to verify the user's identity based on the document images.Verification greatly facilitates users' online business operations.

[0004] However, at the same time, with the development of image processing technology and artificial intelligence (AI) technology, the risk of attackers generating lifelike counterfeit documents through image processing technology or artificial intelligence generated content (AIGC) technology to break through identity verification is also increasing. Although there are currently methods for anti-counterfeiting detection of user-uploaded document images using end-to-end machine learning models, in order to capture the minor flaws of counterfeit documents, the model parameters are adjusted, making the model very sensitive to any minor anomalies, which makes it very easy to generate false alarms when detecting normal, slightly worn or reflective positive samples.

[0005] Therefore, how to perform anti-counterfeiting detection on user-uploaded document images to identify whether the document image is a counterfeit document image is an urgent problem to be solved. Summary of the Invention

[0006] Embodiments of this specification provide a method, device, storage medium and electronic device for anti-counterfeiting detection of documents to partially solve the problems existing in the prior art.

[0007] The embodiments of this specification adopt the following technical solution: This specification provides a method for anti-counterfeiting detection of documents, the method comprising: acquiring an image of a document to be detected collected for the document; identifying the document type of the document based on the image of the document to be detected; extracting multimodal features of the image of the document to be detected, and based on the multimodal features, retrieving standard features of normal document images that match the multimodal features from a preset positive sample knowledge base corresponding to the document type, and retrieving forgery features of forged document images that match the multimodal features from a preset negative sample knowledge base corresponding to the document type; generating enhanced prompt information based on the multimodal features, standard features, and forgery features; inputting the enhanced prompt information into a pre-trained multimodal large language model (MLLM) to perform anti-counterfeiting detection on the image of the document to be detected by the MLLM based on the enhanced prompt information.

[0008] This specification provides an anti-counterfeiting detection device for certificates. The device includes: an acquisition module for acquiring an image of a certificate to be detected; a certificate type recognition module for identifying the certificate type based on the image of the certificate to be detected; and a feature extraction and retrieval module for extracting multimodal features from the image of the certificate to be detected, and retrieving standard features of normal certificate images that match the multimodal features from a preset positive sample knowledge base corresponding to the certificate type, and retrieving forgery features of forged certificate images that match the multimodal features from a preset negative sample knowledge base corresponding to the certificate type.A prompt generation module is used to generate enhanced prompt information based on the multimodal features, standard features, and forgery features; a detection module is used to input the enhanced prompt information into a pre-trained multimodal large language model (MLLM) to perform anti-counterfeiting detection on the document image to be detected based on the enhanced prompt information.

[0009] This specification provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned document anti-counterfeiting detection method.

[0010] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, which, when executed by the processor, implements the above-mentioned document anti-counterfeiting detection method.

[0011] This specification provides a computer program product containing a computer program, which, when executed by a processor, implements the above-mentioned document anti-counterfeiting detection method.

[0012] The above-mentioned at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: The embodiments of this specification disclose a method for anti-counterfeiting detection of documents. This method retrieves standard features of normal document images that match the multimodal features of the document image to be detected from a positive sample knowledge base, retrieves forgery features of counterfeit document images that match the multimodal features from a negative sample knowledge base, and generates enhanced prompt information based on the multimodal features, standard features, and forgery features. Through a pre-trained MLLM, guided by the enhanced prompt information, anti-counterfeiting detection is performed on the document image to be detected. Through the above method, the MLLM can perform anti-counterfeiting detection on the document image to be detected based on the method of retrieval enhancement generation, guided by the enhanced prompt information, using the similarity between the multimodal features of the document image to be detected and the standard features of the positive samples and the forgery features of the negative samples as the main basis, without being overly sensitive to all minor anomalies in the document image to be detected, which can effectively improve the accuracy of anti-counterfeiting detection.

[0013] The accompanying drawings, which are included to provide a further understanding of this specification and constitute a part of this specification, illustrate exemplary embodiments of the present specification and are used to explain this specification, and do not constitute an improper limitation of this specification. In the drawings: Figure 1 is a schematic flowchart of an anti-counterfeiting detection method for a certificate provided in an embodiment of this specification; Figure 2 is a schematic diagram of an anti-counterfeiting detection device for a certificate provided in an embodiment of this specification; Figure 3 is a schematic structural diagram of an electronic device provided in an embodiment of this specification. Detailed Description

[0014] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments of this specification and corresponding drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, those skilled in the art...All other embodiments obtained by the author without creative effort are within the scope of protection of this specification.

[0015] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0016] Figure 1 is a schematic flowchart of the document anti-counterfeiting detection method provided by the embodiment of this specification, which specifically includes the following steps: Specification 2 / 8 Page 5 CN 121837790 A S100: Obtain the image of the document to be detected collected for the document.

[0017] In the embodiments of this specification, the device for document anti-counterfeiting detection using the method shown in Figure 1 can be any electronic device, including personal computers, mobile phones, tablets, servers, or server clusters composed of multiple servers. Those skilled in the art should understand that the specific device used to perform the anomaly detection method shown in Figure 1 does not constitute a limitation on this application. For ease of description, only the server is used as an example for the following description.

[0018] The server first obtains the image of the document to be detected collected by the user using the collection device. In the embodiments of this specification, the image of the document to be detected may be a normal document image directly collected by the user from a normal document, or it may be a forged document image collected by an attacker from a forged document, or a forged document image obtained by tampering with a normal document image. The acquisition device may include a terminal device with a built-in camera or an external camera, such as a mobile phone.

[0019] Regardless of whether the image of the document to be detected is a normal document image or a forged document image, the image of the document to be detected may be an image uploaded to the server by the user (or attacker) through a terminal device.

[0020] After the server obtains the image of the document to be detected, it may first preprocess the image of the document to be detected, and then perform anti-counterfeiting detection on the processed image of the document to be detected through subsequent steps. Among them, the preprocessing described in this specification includes aligning the face in the image of the document to be detected, performing perspective transformation correction on the image of the document to be detected, and normalizing the size of the image of the document to be detected to a fixed size.

[0021] S101: Identify the document type of the document based on the image of the document to be detected.

[0022] In the embodiments of this specification, the server can input the image of the document to be detected into a pre-trained Multimodal Large Language Model (MLLM) to identify the document type corresponding to the image of the document to be detected. Alternatively, the document type corresponding to the image of the document to be detected can be identified through pre-saved document templates. The specific method for identifying the document type is not limited in this specification.

[0023] S102: Extract the multimodal features of the image of the document to be detected, and based on the multimodal features, select from a preset...The standard features of a normal document image that match the multimodal features are retrieved from the positive sample knowledge base corresponding to the document type, and the forgery features of a forged document image that match the multimodal features are retrieved from the preset negative sample knowledge base corresponding to the document type.

[0024] After identifying the document type, the server can perform preliminary anti-counterfeiting detection on the document image to be detected through the MLLM. The structure of the MLLM described herein is not limited.

[0025] Specifically, the MLLM can first extract the multimodal features of the document image to be detected. The multimodal features of the document image to be detected may include at least the text features and visual features of the document image to be detected.

[0026] The text features of the document image to be detected may include the text features of the text contained in the document image to be detected. Specifically, the MLLM may include a text feature extractor based on an Optical Character Recognition (OCR) module. This text feature extractor first determines the key text regions in the image of the document to be detected, based on the document type, and then uses the OCR module to identify each field (i.e., each line of text) in each key text region of the image of the document to be detected, and then extracts the text features of each field.

[0027] The visual features of the image of the document to be detected may include the local visual features of each key image region in the image of the document to be detected. The MLLM may include a visual feature extractor (such as a visual feature extractor based on a convolutional neural network). This visual feature extractor first determines the key image regions in the image of the document to be detected, based on the document type, and then extracts the local visual features of each key image region as the local visual features to be detected.

[0028] The aforementioned key text areas and key image areas can be preset for the document type. For example, as per page 3 / 8 of the specification (CN 121837790 A), when the document type is an ID card, the preset key text areas may include the name area, gender area, date of birth area, ID number area, etc., and the preset key image areas may include the face area, national emblem area, Great Wall pattern area, and various anti-counterfeiting mark information areas, etc.

[0029] After extracting the multimodal features including the above-mentioned text features and the local visual features to be detected, the MLLM can first perform preliminary anti-counterfeiting detection on the document image to be detected. Specifically, it can detect whether the multimodal features of the document image to be detected contain at least one preset common counterfeit document image forgery features. These forgery features include at least one of screen capture features, print capture features, digital tampering features, and text semantic error features.

[0030] Among them, screen capture refers to an attacker displaying a normal document image on a screen and then using a capture device to capture the image displayed on the screen to forge a document image. Therefore, the preset screen capture features may include moiré pattern features, pixel grid features, screen border reflection features, etc.

[0031] Print capture refers to an attacker printing a normal document image using inkjet printing or laser printing, and then using a capture device to capture the printed image to forge a document image. Therefore, the preset print capture features may include color deviation features, color saturation features, and printer-specific CMYK ink dot features that are different from the real ink in a normal document image.

[0032] Digital tampering refers to an attacker forging a document image by using image editing techniques such as retouching a normal document image. Therefore, the preset digital tampering features may include pixel discontinuity features, blur features, and inconsistent lighting or shadow features at the edges of the document image to be detected or a specified image area in the document image to be detected.

[0033] Text semantics refers to the semantic relationship between the text in a normal document corresponding to a normal document image. For example, on a normal ID card, some digits of the ID number represent the user's registered residence, which may be associated with the user's address on the ID card; some digits represent the user's date of birth, which is associated with the user's birthday on the ID card; and some digits represent the user's gender, which is associated with the user's gender on the ID card. Text semantic fallacy, on the other hand, refers to the lack of semantic relationship between the text in the document corresponding to the document image to be detected, which is not present in the normal document. Therefore, the preset text semantic fallacy feature is used to indicate whether the text in the document image to be detected does not have the relationship that a normal document should have.

[0034] In the embodiments of this specification, the ability of the MLLM to detect the above-mentioned forgery features can be trained by supervised learning. Specifically, normal document images as positive samples and forged document images as negative samples can be obtained in advance, and the negative samples have at least one of the above-mentioned forgery features. The obtained samples are then input into the MLLM to be trained, so that the MLLM to be trained can perform preliminary anti-counterfeiting detection on the input samples and obtain preliminary detection results. The preliminary detection results include the confidence level of the input samples having each counterfeiting feature. Finally, based on the preliminary detection results and the pre-annotation of the input samples, the model parameters of the MLLM to be trained are fine-tuned. That is, with the training objective of reducing the difference between the preliminary detection results and the pre-annotation of the input samples, the model parameters of the generation network in the MLLM used to generate and output the preliminary detection results are adjusted, while the model parameters of other networks in the MLLM (such as text feature extractors and visual feature extractors) remain fixed.

[0035] After the MLLM is trained using the above method, the samples to be detected obtained in step S100 can be processed by the MLLM.The document image undergoes preliminary anti-counterfeiting detection to obtain preliminary detection results. Specifically, the MLLM can determine the confidence level of each of the aforementioned counterfeiting features in the document image to be detected, using this as the preliminary detection result. The preliminary detection result may include the confidence level of each counterfeiting feature in the document image to be detected and a preliminary detection conclusion regarding whether the document image to be detected is a counterfeit image.

[0036] Specifically, if the confidence level of each counterfeiting feature in the document image to be detected is less than a first confidence threshold (e.g., 0.10), it indicates that the MLLM is certain that the document image to be detected does not contain any of the aforementioned counterfeiting features, and the preliminary detection conclusion is that the document image to be detected is a normal document image. If, for any counterfeiting feature, the confidence level of the document image to be detected containing that counterfeiting feature is greater than a second confidence threshold (e.g., 0.90), it indicates that the MLLM is certain that the document image to be detected contains that counterfeiting feature, and the preliminary detection conclusion is that the document image to be detected is a counterfeit document image. The second confidence threshold is greater than the first confidence threshold.

[0037] If the MLLM is certain that the document image to be detected does not contain any of the aforementioned counterfeit features, or is certain that the document image to be detected contains one or more counterfeit features, then the deep anti-counterfeiting detection of the document image to be detected can be discontinued, and the preliminary detection result can be directly used as the final detection result.

[0038] However, if the first MLLM is not certain that the document image to be detected does not contain any of the aforementioned counterfeit features, then the deep anti-counterfeiting detection of the document image to be detected needs to be continued. That is, when the preliminary detection result obtained in step S102 includes the confidence level of the document image to be detected containing each counterfeit feature, if the confidence level of the document image to be detected containing at least one counterfeit feature falls into the preset confidence interval, then it is determined that the document to be detected needs to be deep detected. The preset confidence interval is the confidence interval from the first confidence threshold to the second confidence threshold, such as a closed interval from 0.10 to 0.90.

[0039] When performing depth detection on an image of a document to be detected, the MLLM can perform anti-counterfeiting detection based on Retrieval-Augmented Generation (RAG) using a preset positive sample knowledge base and a negative sample knowledge base. Specifically, the MLLM can retrieve standard features of a normal document image that match the multimodal features of the document image to be detected from a preset positive sample knowledge base corresponding to the document type of the document. At the same time, it can retrieve forgery features of a counterfeit document image that match the multimodal features of the document image to be detected from a preset negative sample knowledge base corresponding to the document type of the document.

[0040] Specifically, the aforementioned positive sample knowledge base can pre-store standard local visual features of normal ID card images corresponding to this ID card type, as well as visual feature anti-counterfeiting detection rules and text anti-counterfeiting detection rules corresponding to this ID card type. When pre-constructing the positive sample knowledge base, the visual feature extractor in the aforementioned MLLM can be used to extract visual features from each key image region in the normal ID card image of this ID card type. Each extracted visual feature is used as a standard local visual feature. Furthermore, the pre-set visual feature anti-counterfeiting detection rules and text feature anti-counterfeiting detection rules corresponding to this ID card type can be stored in the positive sample knowledge base. Visual feature anti-counterfeiting detection rules may include authoritative visual feature anti-counterfeiting detection rules such as a certain key image region showing a fluorescent pattern under ultraviolet light. Text feature anti-counterfeiting detection rules may include authoritative text feature anti-counterfeiting detection rules such as certain digits of the ID card number being associated with the date of birth.

[0041] When retrieving standard features that match the multimodal features of the document image to be detected from the positive sample knowledge base, specifically for each local visual feature to be detected extracted from the document image to be detected, a specified number of standard local visual features with the highest similarity to the local visual feature to be detected can be retrieved from the preset positive sample knowledge base corresponding to the document type. These standard features are all used as standard features that match the local visual feature to be detected. The visual feature anti-counterfeiting detection rules and text feature anti-counterfeiting detection rules corresponding to the document type are extracted from the positive sample knowledge base. The specified number can be set as needed.

[0042] Correspondingly, the negative sample knowledge base can pre-store the counterfeit local visual features of counterfeit document images corresponding to the document type as negative samples. These counterfeit document images as negative samples can all be counterfeit document images that have been detected in the past. The negative sample knowledge base can also store the device information corresponding to the devices that uploaded these counterfeit document images in the past. The device information described in this specification includes, but is not limited to, at least one of the geographical location of the device and the address information of the device. The address information of the device may include the IP address information and / or MAC address information of the device, etc. This is because, generally speaking, the location and device used by attackers when launching attacks by forging document images are relatively fixed. Therefore, this device information can serve as a strong basis for subsequent in-depth anti-counterfeiting detection.

[0043] When retrieving forged features that match the multimodal features of the document image to be detected from the negative sample knowledge base, specifically for each local visual feature to be detected, a specified number of forged local visual features with the highest similarity to the local visual feature to be detected can be retrieved from the preset negative sample knowledge base corresponding to the document type for each local visual feature to be detected.For the counterfeit features that match the local visual features to be detected, the device information corresponding to the devices of each negative sample uploaded in history for this type of document is extracted from the negative sample knowledge base. The specified number of counterfeit features extracted can be preset as needed; this specified number can be the same as or different from the specified number of standard features extracted, and this specification does not impose any restrictions on this.

[0044] S103: Generate enhanced prompt information based on the multimodal features, standard features, and counterfeit features.

[0045] In the embodiments of this specification, when performing deep anti-counterfeiting detection on the document image to be detected, after extracting standard features, visual feature anti-counterfeiting detection rules, and text feature anti-counterfeiting detection rules from the positive sample knowledge base in step S102, and extracting counterfeit features and device information corresponding to the devices of each negative sample uploaded in history from the negative sample knowledge base, enhanced prompt information can be generated based on this retrieved prior knowledge.

[0046] Specifically, for prior knowledge retrieved from the positive sample knowledge base, for each local visual feature to be detected, a first similarity can be determined between the local visual feature to be detected and the standard feature matching the local visual feature to be detected, and an enhanced prompt information is generated that includes at least the multimodal features of the document image to be detected, counterfeit features, visual feature anti-counterfeiting detection rules, text feature anti-counterfeiting detection rules, standard features matching each local visual feature to be detected, and the first similarity determined for each local visual feature to be detected.

[0047] For the prior knowledge retrieved from the negative sample knowledge base, for each local visual feature to be detected, a second similarity can be determined between the local visual feature to be detected and the forged feature matched with the local visual feature to be detected, and the device information corresponding to the device that acquired the image of the document to be detected can be determined. Then, an enhanced prompt information is generated that includes at least the multimodal features of the image of the document to be detected, the standard features, the device information corresponding to the device that acquired the image of the document to be detected, the device information corresponding to the devices that uploaded each negative sample of the document type in history, the forged features matched with each local visual feature to be detected, and the second similarity determined for each local visual feature to be detected.

[0048] Combining the prior knowledge retrieved from the positive sample knowledge base and the negative sample knowledge base, the generated enhanced prompt information may include: the multimodal features of the document image to be detected, visual feature anti-counterfeiting detection rules, text feature anti-counterfeiting detection rules, standard features matching each local visual feature to be detected, a first similarity determined for each local visual feature to be detected, device information corresponding to the device that acquired the document image to be detected, device information corresponding to each negative sample of this document type uploaded in the past, counterfeit features matching each local visual feature to be detected, and a second similarity determined for each local visual feature to be detected.

[0049] For example, the enhanced prompt information could be: You are a world-class document authentication expert. Here is a picture that claims to be a "Chinese Resident Identity Card, 2013 version, front". Please determine its authenticity based on the background information I provide, and give a detailed, step-by-step reasoning process. The background information is as follows: Multimodal features of the document image to be detected; Standard features; Counterfeit features; Anti-counterfeiting detection rules for visual features and text features extracted from the positive sample knowledge base; First similarity; Second similarity; Device information corresponding to the device that collected the document image to be detected and device information corresponding to the devices that uploaded each negative sample of this document type in the past; Please begin your analysis.

[0050] S104: Input the enhanced prompt information into the pre-trained multimodal large language model MLLM, so that the MLLM can perform anti-counterfeiting detection on the document image to be detected based on the enhanced prompt information.

[0051] After generating the enhanced prompt information through the above step S103, the MLLM can be called again, and the enhanced prompt information can be input into the MLLM, so that the MLLM can perform deep anti-counterfeiting detection on the document image to be detected under the guidance of the enhanced prompt information, and obtain the result of the deep anti-counterfeiting detection.

[0052] Specifically, under the guidance of the enhanced prompt information, the MLLM can determine whether the multimodal features of the document image to be detected are closer to the standard features in the positive sample knowledge base or closer to the counterfeit features in the negative sample knowledge base, and obtain the first detection result. It can also use the visual feature anti-counterfeiting detection rules and text feature anti-counterfeiting detection rules contained in the enhanced prompt information to perform rule-based anti-counterfeiting detection on the multimodal features of the document image to be detected, and obtain the second detection result. It can also detect the source risk of the document image to be detected based on the device information corresponding to the device that collected the document image to be detected and the device information corresponding to the devices that uploaded each negative sample in history, and obtain the third detection result. Finally, based on the above first to third detection results, the final deep detection result is obtained.

[0053] The enhanced prompt information may also include prompt information to guide the MLLM to output the reason for obtaining the depth detection result. In this way, in addition to outputting the final depth detection result, the MLLM can also output the reason for obtaining the depth detection result, making the depth detection result interpretable.

[0054] Through the above method, when performing depth anti-counterfeiting detection, the MLLM can, based on the retrieval enhancement generation method and guided by the enhanced prompt information, perform anti-counterfeiting detection on the document image to be detected, primarily based on the similarity between the multimodal features of the document image to be detected and the standard features of the positive samples and the counterfeit features of the negative samples. Since the MLLM does not target depth...The anti-counterfeiting detection method uses counterfeit document images as negative samples to fine-tune the model parameters, thus avoiding contamination of the model parameters and preventing oversensitivity to all minor anomalies in the document images to be detected. This does not reduce the recognition performance of massive amounts of white samples in real-world application scenarios, and can effectively improve the accuracy of anti-counterfeiting detection.

[0055] The above is an anti-counterfeiting detection method for documents provided by the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media, and electronic devices.

[0056] Figure 2 is a schematic diagram of an anti-counterfeiting detection device for a document provided in an embodiment of this specification. The device includes: an acquisition module 200, used to acquire an image of a document to be detected collected for the document; a document type recognition module 201, used to identify the document type of the document based on the image of the document to be detected; a feature extraction and retrieval module 202, used to extract multimodal features of the image of the document to be detected, and based on the multimodal features, retrieve standard features of normal document images that match the multimodal features from a preset positive sample knowledge base corresponding to the document type, and retrieve forgery features of forged document images that match the multimodal features from a preset negative sample knowledge base corresponding to the document type; a prompt generation module 203, used to generate enhanced prompt information based on the multimodal features, standard features, and forgery features; and a detection module 204, used to input the enhanced prompt information into a pre-trained multimodal large language model (MLLM) to perform anti-counterfeiting detection on the image of the document to be detected by the MLLM based on the enhanced prompt information. Instruction manual, pages 7 / 8, 10 CN 121837790 A

[0057] Optionally, the feature extraction and retrieval module 202 is specifically used to: determine, according to the document type, each key image region preset for the document type in the image of the document to be detected; extract local visual features from each key image region as local visual features to be detected; and, according to the document type, determine each key text region preset for the document type in the image of the document to be detected; extract each text feature from each key text region in the image of the document to be detected.

[0058] Optionally, the feature extraction and retrieval module 202 is specifically used to: for each local visual feature to be detected, retrieve a specified number of standard local visual features with the highest similarity to the local visual feature to be detected from a preset positive sample knowledge base corresponding to the document type; and, extract visual feature anti-counterfeiting detection rules and text feature anti-counterfeiting detection rules corresponding to the document type from the positive sample knowledge base.

[0059] Optionally, the prompt generation module 203 is specifically used to determine, for each local visual feature to be detected, the...The first similarity between the local visual feature to be detected and the standard feature matching the local visual feature to be detected; generating enhanced prompt information including at least the multimodal feature, the forgery feature, the visual feature anti-counterfeiting detection rule, the text feature anti-counterfeiting detection rule, the standard feature matching each local visual feature to be detected, and the first similarity determined for each local visual feature to be detected.

[0060] Optionally, the feature extraction and retrieval module 202 is specifically used to, for each local visual feature to be detected, retrieve a specified number of forged local visual features with the highest similarity to the local visual feature to be detected from a preset negative sample knowledge base corresponding to the document type, as forged features matching the local visual feature to be detected; and extract device information corresponding to the devices that have historically uploaded the document type from the negative sample knowledge base, the device information including the geographical location of the device and / or the address information of the device.

[0061] Optionally, the prompt generation module 203 is specifically used to: for each local visual feature to be detected, determine a second similarity between the local visual feature to be detected and a forged feature matching the local visual feature to be detected; and determine the device information corresponding to the device that acquired the image of the document to be detected; and generate enhanced prompt information including at least the multimodal features, the standard features, the device information corresponding to the device that acquired the image of the document to be detected, the device information corresponding to the devices that historically uploaded each negative sample of the document type, the forged features matching each local visual feature to be detected, and the second similarity determined for each local visual feature to be detected.

[0062] This specification also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can be used to execute the above-described document anti-counterfeiting detection method.

[0063] This specification also provides a computer program product comprising a computer program, which, when executed by a processor, implements the above-described document anti-counterfeiting detection method.

[0064] Based on the document anti-counterfeiting detection method shown in Figure 1, this specification also provides a structural schematic diagram of the electronic device shown in Figure 3. As shown in Figure 3, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned document anti-counterfeiting detection method.

[0065] The above descriptions are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any changes or variations made within the spirit and principles of this specification are not permitted.Modifications, equivalent substitutions, improvements, etc., should all be included within the scope of the claims in this specification. Description of the specification 8 / 8 page 11 CN 121837790 A Figure 1 Description of the drawings 1 / 2 page 12 CN 121837790 A Figure 2 Figure 3 Description of the drawings 2 / 2 page 13 CN 121837790 A Abstract The embodiment of the invention discloses an anti-counterfeiting detection method for a certificate, which consists of the following steps of: retrieving standard characteristics of a normal certificate image matched with multi-modal characteristics of a certificate image to be detected from a positive sample knowledge base, and retrieving counterfeit characteristics of a counterfeit certificate image matched with the multi-modal characteristics from a negative sample knowledge base; and generating enhanced prompt information according to the multi-modal features, the standard features and the forgery features, and performing anti-counterfeiting detection on the to-be-detected certificate image under the guidance of the enhanced prompt information through a pre-trained MLLM. Through the method, the MLLM can perform anti-counterfeitingdetection on the certificate image to be detected by taking the similarity between the multi-modal feature of the certificate image to be detected and the standard feature of the positive sample and the similarity between the multi-modal feature of the certificate image to be detected and the forgery feature of the negative sample as the main basis under the guidance of the enhancement prompt information based on a retrieval enhancement generation method; and all tiny anomalies in the to-be-detected certificate image are not too sensitive, so that the accuracy of anti-counterfeiting detection can be effectively improved.

Claims

1. A method for detecting the anti-counterfeiting features of a certificate, the method comprising: Acquire the image of the document to be inspected, based on the document collection process; Based on the image of the document to be detected, identify the document type; Extract the multimodal features of the document image to be detected, and based on the multimodal features, retrieve the standard features of normal document images that match the multimodal features from a preset positive sample knowledge base corresponding to the document type, and retrieve the forgery features of forged document images that match the multimodal features from a preset negative sample knowledge base corresponding to the document type. Enhanced prompt information is generated based on the multimodal features, standard features, and forgery features; The enhanced prompt information is input into a pre-trained multimodal large language model (MLLM) to perform anti-counterfeiting detection on the document image to be detected based on the enhanced prompt information.

2. The method as described in claim 1, extracting the multimodal features of the image of the document to be detected, specifically includes: Based on the document type, each key image region preset for the document type is determined in the document image to be detected, and local visual features of each key image region are extracted as local visual features to be detected; and based on the document type, each key text region preset for the document type is determined in the document image to be detected, and text features of each key text region are extracted from the document image to be detected.

3. The method as described in claim 2, wherein retrieving standard features of a normal document image matching the multimodal features from a preset positive sample knowledge base corresponding to the document type specifically includes: For each local visual feature to be detected, a specified number of standard local visual features with the highest similarity to the local visual feature to be detected are retrieved from the pre-set positive sample knowledge base corresponding to the document type. These standard features are all used as standard features that match the local visual feature to be detected. Furthermore, the visual feature anti-counterfeiting detection rules and text feature anti-counterfeiting detection rules corresponding to the document type are extracted from the positive sample knowledge base.

4. The method as described in claim 3, wherein enhanced prompt information is generated based on the multimodal features, standard features, and forgery features, specifically including: For each local visual feature to be detected, determine the first similarity between the local visual feature to be detected and the standard feature that matches the local visual feature to be detected; The generated information includes at least the multimodal features, the forgery features, the visual feature anti-counterfeiting detection rules, the text feature anti-counterfeiting detection rules, standard features that match each local visual feature to be detected, and enhanced prompt information with a first similarity determined for each local visual feature to be detected.

5. The method as described in claim 2, wherein retrieving forgery features of a forged document image matching the multimodal features from a preset negative sample knowledge base corresponding to the document type specifically includes: For each local visual feature to be detected, a specified number of forged local visual features with the highest similarity to the local visual feature to be detected are retrieved from a pre-set negative sample knowledge base corresponding to the document type, and these are used as forged features that match the local visual feature to be detected; and, device information corresponding to the devices that have historically uploaded the document type is extracted from the negative sample knowledge base, the device information including the geographical location of the device and / or the address information of the device.

6. The method as described in claim 5, wherein enhanced prompt information is generated based on the multimodal features, standard features, and forgery features, specifically including: For each local visual feature to be detected, a second similarity is determined between the local visual feature to be detected and the forged feature that matches the local visual feature to be detected; and the device information corresponding to the device that acquired the image of the document to be detected is determined. The generated information includes at least the multimodal features, the standard features, device information corresponding to the device that acquired the image of the document to be detected, device information corresponding to the device that uploaded each negative sample of the document type in history, forgery features that match each local visual feature to be detected, and enhanced prompt information with a second similarity determined for each local visual feature to be detected.

7. An anti-counterfeiting detection device for a certificate, the device comprising: The acquisition module is used to acquire the image of the document to be detected for document collection. The document type recognition module is used to identify the document type of the document based on the image of the document to be detected; The feature extraction and retrieval module is used to extract the multimodal features of the document image to be detected, and based on the multimodal features, retrieve standard features of normal document images that match the multimodal features from a preset positive sample knowledge base corresponding to the document type, and retrieve forgery features of forged document images that match the multimodal features from a preset negative sample knowledge base corresponding to the document type. The prompt generation module is used to generate enhanced prompt information based on the multimodal features, standard features, and forgery features; The detection module is used to input the enhanced prompt information into a pre-trained multimodal large language model (MLLM) so that the MLLM can perform anti-counterfeiting detection on the document image to be detected based on the enhanced prompt information.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images