Certificate image anti-counterfeiting identification method and device

By building an anti-counterfeiting knowledge base and using the fusion vector of the document image and the problem text for feature enhancement, the problem of reliability of document image recognition and model fine-tuning cost in the prior art is solved, and more efficient document image authenticity recognition and matching are achieved. Quick response of new tampering methods.

CN118968636APending Publication Date: 2024-11-15ANT BLOCKCHAIN TECHNOLOGY (SHANGHAI) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411019492.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing anti-counterfeiting identification methods of document images mainly rely on text content comparison and lack reliability. Especially in the absence of an official information database, it is difficult to effectively identify forged document images. Due to the continuous changes in tampering methods, frequent models are required. Fine-tuning increases training costs.

Method used

A pre-trained anti-counterfeiting knowledge base is built, a feature vector of multiple tampered image elements is stored, and the knowledge base is queried through the fusion vector of the certificate image and its problem text, and feature enhancement is performed, and finally the anti-counterfeiting recognition model is used to identify the authenticity of the certificate image.

Benefits of technology

It improves the authenticity recognition ability of document images, avoids frequent model fine-tuning, enhances the rapid response ability to new tampering methods, and ensures the accuracy and flexibility of identification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968636A_ABST
    Figure CN118968636A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an anti-counterfeiting identification method and device for a certificate image, and the method comprises the steps: obtaining the certificate image and a problem text corresponding to the certificate image; and inputting the certificate image and the problem text into a pre-trained representation model to carry out multi-modal data fusion processing to obtain a target fusion vector. And based on the target fusion vector, querying an anti-counterfeiting knowledge base to obtain a plurality of target feature vectors matched with the target fusion vector. Respective feature vectors of a plurality of tampered image elements are recorded in the anti-counterfeiting knowledge base, and the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating style characteristics of the tampered image element. And performing feature enhancement on the target fusion vector based on the plurality of target feature vectors, and inputting the enhanced target fusion vector into the anti-counterfeiting identification model to obtain an identification result indicating whether the certificate image is a forged image or not.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of anti-counterfeiting identification, and in particular, to a method and device for anti-counterfeiting identification of document images. Background Art

[0002] In the field of financial business, user authentication is often performed by checking the ID image. For example, in the electronic identity authentication EKYC (Electron ic Know Your Customer) scenario, a key step is to verify the user's true identity based on the user's ID image. However, some lawbreakers, who specialize in black market activities, deceive the financial system by submitting forged ID images in order to achieve illegal profit. In addition, some technical bloggers in the open source community share technical methods for making ID images on social media platforms for scientific research purposes. The disclosure of this information may be exploited by lawbreakers, who will use these technologies to conduct fraudulent attacks on the current financial system, thereby causing economic losses to the institution. Therefore, it is necessary to perform anti-counterfeiting identification on the user's ID image.

[0003] At present, anti-counterfeiting identification is mainly performed by identifying the text content on the document image and then comparing the text content with the official information database. However, most countries cannot provide such an official information database. Therefore, a more reliable anti-counterfeiting identification solution for document images is needed. Summary of the invention

[0004] One or more embodiments of the present specification describe a method and device for anti-counterfeiting identification of certificate images, which can improve the ability to identify the authenticity of certificate images.

[0005] In a first aspect, a method for anti-counterfeiting identification of a document image is provided, comprising:

[0006] Get the document image and its corresponding question text;

[0007] Inputting the document image and question text into a pre-trained representation model to perform multimodal data fusion processing to obtain a target fusion vector;

[0008] Based on the target fusion vector, query the anti-counterfeiting knowledge base to obtain a number of target feature vectors matching the target fusion vector; the anti-counterfeiting knowledge base records the feature vectors of a plurality of tampered image elements; wherein the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating its style characteristics;

[0009] The target fusion vector is feature enhanced based on the plurality of target feature vectors, and the enhanced target fusion vector is input into an anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a forged image.

[0010] In a second aspect, an anti-counterfeiting identification device for a certificate image is provided, comprising:

[0011] An acquisition unit, used for acquiring a certificate image and a corresponding question text;

[0012] An input unit, used for inputting the document image and the question text into a pre-trained representation model to perform multi-modal data fusion processing to obtain a target fusion vector;

[0013] A query unit is used to query an anti-counterfeiting knowledge base based on the target fusion vector to obtain a plurality of target feature vectors matching the target fusion vector; the anti-counterfeiting knowledge base records the feature vectors of a plurality of tampered image elements; wherein the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating its style characteristics;

[0014] The enhancement unit is used to perform feature enhancement on the target fusion vector based on the plurality of target feature vectors, and input the enhanced target fusion vector into the anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a forged image.

[0015] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0016] In a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0017] The anti-counterfeiting identification method and device for certificate images provided in one or more embodiments of the present specification pre-builds an anti-counterfeiting knowledge base, which stores multiple feature vectors of tampered image elements. Afterwards, when performing anti-counterfeiting identification on the certificate image, the anti-counterfeiting knowledge base is queried based on the fusion vector of the certificate image and its question text, and the fusion vector is enhanced using the queried feature vector, and the anti-counterfeiting identification model is used to identify whether the certificate image is a forged image based on the enhanced fusion vector, thereby improving the ability to identify the authenticity of the certificate image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0020] Figure 2 A flow chart of an anti-counterfeiting identification method for a certificate image according to an embodiment is shown;

[0021] Figure 3 A schematic diagram of a characterization model in one example is shown;

[0022] Figure 4 A schematic diagram of an anti-counterfeiting knowledge base in an example is shown;

[0023] Figure 5 A schematic diagram of answer text in one example is shown;

[0024] Figure 6 A schematic diagram of an anti-counterfeiting identification device for a document image according to an embodiment is shown. DETAILED DESCRIPTION

[0025] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0026] As mentioned above, the method of anti-counterfeiting identification based on the text content in the document image is not generally applicable. For this reason, some solutions propose to perform anti-counterfeiting identification based on the document image itself. For example, a large number of negative samples (i.e., forged document images) are collected in advance, and the large model is fine-tuned based on these negative samples, and then the authenticity of the document image is identified based on the fine-tuned large model. However, since the tampering methods are constantly changing, in order to ensure that the large model can identify the document images tampered with based on new tampering methods, the large model needs to be fine-tuned frequently, which will increase the training cost.

[0027] To this end, the embodiment of this specification proposes an improved solution, that is, pre-building an anti-counterfeiting knowledge base, which stores multiple feature vectors of tampered image elements. Afterwards, when performing anti-counterfeiting identification on the document image, based on the fusion vector of the document image and its question text, the anti-counterfeiting knowledge base is queried, and the fusion vector is enhanced using the queried feature vector, and the anti-counterfeiting identification model is used to identify whether the document image is a forged image based on the enhanced fusion vector, thereby realizing the enhancement of the recognition ability of the anti-counterfeiting identification model by knowledge injection, thereby avoiding frequent fine-tuning of the model, and the solution is more universal.

[0028] In addition, the above-mentioned anti-counterfeiting knowledge base can be updated based on new tampering methods collected in real time, that is, the feature vectors of image elements tampered with based on new tampering methods are updated to the anti-counterfeiting knowledge base. This not only improves the anti-counterfeiting recognition ability of the anti-counterfeiting recognition model, but also improves the rapid response capability to new tampering methods, thereby better preventing new tampering risks.

[0029] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification. Figure 1 In the method, the document image and its corresponding question text can be input into the pre-trained representation model, and the fusion vector of the document image and the question text can be obtained through the representation model. Then, based on the fusion vector, the anti-counterfeiting knowledge base is queried to obtain several feature vectors matching it, and the fusion vector is enhanced by using the several feature vectors. Finally, the enhanced feature vector is input into the anti-counterfeiting recognition model to obtain the recognition result of whether the document image is a forged image.

[0030] Figure 2 The flowchart of the anti-counterfeiting identification method of the certificate image according to one embodiment is shown. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the method may include the following steps:

[0031] Step S202, obtaining the certificate image and its corresponding question text.

[0032] The above-mentioned certificate image can also be called an identity document, which can include several of the following image elements: face, text, anti-counterfeiting mark, logo, etc. In addition, it can also include a chip, etc.

[0033] In one example, the above question text is used to ask whether the document image is a forged image. Of course, in practice, the question text is also used to ask about the tampering position of the forged image, etc.

[0034] Step S204: input the document image and question text into a pre-trained representation model to perform multi-modal data fusion processing to obtain a target fusion vector.

[0035] In one embodiment, the above representation model is a multimodal model, which is used to fuse multimodal data such as text and images. It should be noted that the number of parameters of the representation model is much smaller than that of the Large Multimodal Models (LMM).

[0036] Figure 3 A schematic diagram of a characterization model in one example is shown. Figure 3In the above, the representation model includes first and second encoders and a mapping layer. Specifically, the first encoder is used to process the document image to obtain an image representation of the document image, and the second encoder is used to process the question text to obtain a text representation of the question text. The mapping layer is used to fuse the image representation and the text representation to obtain a target fusion vector.

[0037] The first encoder mentioned above can be implemented as a convolutional neural network, such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), FCN (Fully Convolutional Networks), etc. Of course, it can also be implemented as other neural networks, such as MLP (MultiLayer Perceptron Network) network, etc.

[0038] The second encoder mentioned above may be a Transformer encoder, that is, it encodes the question text based on an attention mechanism.

[0039] The above mapping layer is also called a linear mapping layer, and the above fusion processing specifically includes using a mapping function to perform multiple linear mapping processes on the combination of the image representation and the text representation. The mapping function here can be, for example, a GELU function.

[0040] Of course, in practice, the above-mentioned representation model can also be implemented based on other networks or models that can perform fusion processing on multimodal data, and this specification does not limit this.

[0041] It should be noted that in addition to the above-mentioned target fusion vector, the representation model can also output the image representation of the document image and the text representation of the question text.

[0042] Step S206: Based on the target fusion vector, query the anti-counterfeiting knowledge base to obtain a number of target feature vectors that match the target fusion vector.

[0043] The anti-counterfeiting knowledge base records the feature vectors of multiple tampered image elements. Any tampered image element can be classified into one of the following element categories: face, text, anti-counterfeiting mark, logo, etc. Of course, in practice, the element category can also include chips, etc.

[0044] In one embodiment, the feature vector of any tampered image element (hereinafter referred to as the first tampered image element) is obtained by the following steps:

[0045] The first tampered image element and its first description text are input into the above representation model to obtain a first fusion vector, and the first fusion vector is used as the first feature vector of the first tampered image element.

[0046] The first description text indicates the style characteristics of the first tampered image element, such as raised edges, red background, local pixel disharmony, etc.

[0047] In addition, the process of obtaining the first fusion vector and the target fusion vector is similar, which will not be repeated here.

[0048] In another embodiment, the feature vector of any tampered image element may also be an image representation extracted from the tampered image element using a convolutional neural network such as CNN.

[0049] The query of the anti-counterfeiting knowledge base specifically includes calculating the similarity between the target fusion vector and the feature vectors of each tampered image element in the anti-counterfeiting knowledge base, and determining each feature vector with a similarity greater than a preset threshold as each target feature vector. The similarity here includes but is not limited to cosine similarity, Euclidean distance, etc.

[0050] It should be noted that, in practice, for any tampered image element, the corresponding fusion vector and image representation can be recorded simultaneously, so that when the query result based on the fusion vector in the anti-counterfeiting knowledge base is not ideal, the query result can be obtained based on the image representation in the anti-counterfeiting knowledge base, that is, the query result is obtained by calculating the similarity between the target fusion vector and the image representation of each tampered image element.

[0051] Figure 4 A schematic diagram of an anti-counterfeiting knowledge base in one example is shown. Figure 4 In the anti-counterfeiting knowledge base, feature vectors of n tampered image elements are recorded, and each tampered image element belongs to one of the following element categories: face, text, anti-counterfeiting mark, and logo, etc. In addition, the feature vector of each tampered image element corresponds to an image-text pair, which includes the tampered image element and its description text.

[0052] It should be noted that in order to ensure the effectiveness of the anti-counterfeiting knowledge base, it is usually necessary to collect new tampering methods in real time, and tamper with the corresponding image elements according to the new tampering methods to obtain new tampered image elements. After that, the new feature vector determined based on the new tampered image element and its description text is updated to the anti-counterfeiting knowledge base.

[0053] Step S208, feature enhancement is performed on the target fusion vector based on a number of target feature vectors, and the enhanced target fusion vector is input into an anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a forged image.

[0054] Among them, the above-mentioned anti-counterfeiting recognition model can be obtained by fine-tuning the multimodal large model based on the sample document image (the specific fine-tuning process will be explained later); it can also be obtained by training a neural network model (such as DNN) based on the sample document image.

[0055] It should be understood that each of the above-mentioned target feature vectors corresponds to a tampered image element, and each tampered image element belongs to an element category, so each target feature vector belongs to an element category. In this step, the target fusion vector can be feature enhanced based on the contribution of the element category to which each target feature vector belongs. Among them, the contribution of each element category can be set according to expert experience or actual needs.

[0056] The following is an explanation of the specific process of feature enhancement:

[0057] In one embodiment, each target feature vector may be sorted from high to low according to the similarity between each target feature vector and the target fusion vector, to obtain each target feature vector in a first sort. For each target feature vector in the first sort, each target feature vector is corrected according to the contribution of the element category to which it belongs, to obtain each corrected feature vector in the first sort. The target fusion vector is combined with each corrected feature vector in the first sort to obtain an enhanced target fusion vector.

[0058] The similarity on which the target feature vectors are sorted is the similarity calculated when the target feature vectors are selected.

[0059] In a more specific embodiment, for any target feature vector, the product of the contribution of the element category to which it belongs and the target feature vector is determined as the corresponding modified feature vector.

[0060] Of course, in practice, the corresponding modified feature vector may also be determined by performing other operations on the target feature vector and the contribution of the element category to which it belongs, and this specification does not limit this.

[0061] It should be noted that the target feature vectors under the first sorting are the target feature vectors with the highest similarity first, followed by the target feature vectors with the second highest similarity, and so on, and finally the target feature vectors with the lowest similarity. The modified feature vectors under the first sorting are the modified feature vectors arranged in the order of the corresponding target feature vectors.

[0062] In addition, the above-mentioned combination of the target fusion vector and each modified feature vector in the first sorting may specifically include splicing the target fusion vector to the front end of each modified feature vector in the first sorting. That is, the target fusion vector is arranged at the front, followed by each modified feature vector in the first sorting.

[0063] Of course, in practice, the target fusion vector may also be spliced ​​to the back end of each modified feature vector in the first sorting order, and this specification does not limit this.

[0064] For example, suppose there are four element categories: face, text, anti-counterfeiting mark and logo, and the contribution of each of the four element categories is 0.3, 0.1, 0.5 and 0.1 respectively. It is also assumed that five target feature vectors are queried: x1, x2, x3, x4 and x5, and their similarities with the target fusion vector are 0.75, 0.8, 0.77, 0.9 and 0.82 respectively. Then after sorting the target feature vectors from high to low according to the similarity, the target feature vectors in the first sorting are obtained: x4, x5, x2, x3 and x1. If x4 and x3 belong to faces, and x5, x2 and x1 belong to logos, then the corrected feature vectors in the first sorting can be obtained: 0.3·x4, 0.1·x5, 0.1·x2, 0.3·x3 and 0.1·x1.

[0065] In another embodiment, each target feature vector may be sorted by combining similarity and contribution at the same time. For example, each target feature vector may be sorted from high to low according to the product of the similarity and contribution corresponding to each target feature vector. It should be understood that in this sorting method, the target feature vector may be corrected or not (for the specific correction process, see one of the above embodiments), and this specification does not limit this.

[0066] As mentioned above, the output of the representation model may also include text features of the text question. In the case where the text features are also included, the enhanced target fusion vector and the text features may be input into the anti-counterfeiting identification model together. More specifically, the text features may be spliced ​​at the back end of the enhanced target fusion vector, and then the spliced ​​result may be input into the anti-counterfeiting identification model.

[0067] In practice, the above recognition results can not only indicate whether the document image is a forged image, but also indicate the tampering location and the content and style of the tampered document image, etc. The style here includes but is not limited to the content form (such as text or pictures, etc.), color and shape, etc.

[0068] It should be noted that when the above-mentioned anti-counterfeiting recognition model is obtained by fine-tuning the pre-trained multimodal large model, the above-mentioned recognition result is output in the form of answer text.

[0069] Figure 5 A schematic diagram of answer text in one example is shown. Figure 5The answer text also indicates the following: 1. Document type: C1 document from country A. 2. Conclusion: The document has been tampered with. 3. Tampered location: (10,200), (400,210), (400,260) and (10,260), the four groups of numbers represent the coordinates of the four vertices. 4. Tampered content and style: A white paper with the words 'LIM CHEE MENG'.

[0070] The following is an explanation of the fine-tuning process of the multimodal large model:

[0071] Specifically, the sample document image and its sample question text can be input into the representation model to obtain a sample fusion vector. Based on the sample fusion vector, the anti-counterfeiting knowledge base is queried to obtain a number of sample feature vectors that match the sample fusion vector. The sample fusion vector is feature enhanced based on the several sample feature vectors, and the enhanced sample fusion vector is input into the multimodal large model to obtain a predicted answer text. According to the difference between the predicted answer text and the calibrated answer text of the sample document image, the parameters of the multimodal large model are adjusted; after multiple rounds of parameter adjustment, the final multimodal large model can be obtained, and the final multimodal large model is used as the above-mentioned anti-counterfeiting recognition model.

[0072] In this solution, the text difference loss between the predicted answer text and the calibrated answer text can be used to measure the difference between the two.

[0073] In one embodiment, the text difference loss can be determined based on the text edit distance between the predicted answer text and the calibrated answer text, wherein the larger the text edit distance, the larger the text difference loss, that is, the text difference loss is positively correlated with the text edit distance. Specifically, the text edit distance (Edit Di stance), also known as the Levenshtein distance, refers to the minimum number of editing operations required to convert one character string (text) into another, and the larger the distance, the more different they are. The editing operation may include replacing one character with another character, inserting a character, and deleting a character.

[0074] In another embodiment, the predicted answer text and the calibrated answer text may be respectively converted into a first vector and a second vector by word embedding, and the text similarity between the predicted answer text and the calibrated answer sample may be determined based on the first vector and the second vector; further, the text difference loss may be determined based on the text similarity. The higher the text similarity, the smaller the text difference loss. In other embodiments, there are other ways to determine the text difference loss, which are not listed here one by one.

[0075] In other words, this scheme can fine-tune the pre-trained multimodal large model based on the difference between the predicted answer text and the calibrated answer text of the sample document image.

[0076] In summary, the anti-counterfeiting identification method for certificate images provided in the embodiments of this specification uses the feature vectors of tampered image elements queried from the anti-counterfeiting knowledge base to enhance the target fusion vector, which can greatly improve the authenticity identification ability of certificate images. In addition, this solution constructs an anti-counterfeiting knowledge base and updates it based on new tampering methods collected in real time, which can realize the injection of new risks and new domain knowledge into the input of the large model in the form of features, that is, the update of anti-counterfeiting domain knowledge is realized without fine-tuning the large model. Finally, since the actual operation of knowledge injection is essentially to enhance the Prompt, it can ensure the flexibility and timeliness of the Prompt, and can also ensure the advancement of knowledge.

[0077] In summary, this solution has the following advantages:

[0078] 1. Storage at the image element level can achieve targeted enhancement of features, thereby improving the accuracy of recognition results.

[0079] 2. No additional training required: The representation model used in this solution is pre-trained and does not require additional training, which can improve response time.

[0080] 3. Prompt modification is more flexible: This solution converts the large model fine-tuning task into a Prompt design task, which involves higher timeliness of Prompt.

[0081] 4. High Prompt Diversity: There can be a variety of flexible combinations in Prompt, which makes it easier for the model to accurately identify forged images.

[0082] 5. Fast knowledge update: The knowledge base is constantly updated to ensure the advancement of knowledge.

[0083] 6. The multimodal large model has richer features: When the anti-counterfeiting recognition model is implemented based on the multimodal large model, it can receive richer multimodal inputs and the recognition results are more accurate.

[0084] Corresponding to the above-mentioned document image anti-counterfeiting identification method, an embodiment of the present specification also provides a document image anti-counterfeiting identification device, such as Figure 6 As shown, the device may include:

[0085] The acquisition unit 602 is used to acquire the certificate image and its corresponding question text.

[0086] The input unit 604 is used to input the document image and the question text into a pre-trained representation model to perform multi-modal data fusion processing to obtain a target fusion vector.

[0087] The query unit 606 is used to query the anti-counterfeiting knowledge base based on the target fusion vector to obtain a number of target feature vectors matching the target fusion vector. The anti-counterfeiting knowledge base records the feature vectors of multiple tampered image elements, wherein the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating its style characteristics.

[0088] The enhancement unit 608 is used to perform feature enhancement on the target fusion vector based on a number of target feature vectors, and input the enhanced target fusion vector into the anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a forged image.

[0089] Wherein, any of the above tampered image elements belongs to one of the following element categories:

[0090] Faces, text, anti-counterfeiting marks and logos.

[0091] In one embodiment, the apparatus further includes: a determining unit 610;

[0092] The input unit 604 is further used to input the first tampered image element and its first description text into the representation model to obtain a first fusion vector;

[0093] The determining unit 610 is configured to use the first fusion vector as a first feature vector of a first tampered image element.

[0094] In one embodiment, the enhancement unit 608 includes:

[0095] A sorting submodule 6082 is used to sort each target feature vector according to the similarity between each target feature vector and the target fusion vector from high to low, so as to obtain each target feature vector in a first sorting;

[0096] The correction submodule 6084 is used to correct each target feature vector in the first sorting according to the contribution of the element category to which it belongs, so as to obtain each corrected feature vector in the first sorting;

[0097] The combining submodule 6086 is used to combine the target fusion vector with each modified feature vector under the first sorting to obtain an enhanced target fusion vector.

[0098] In a more specific embodiment, the correction submodule 6084 is specifically used to:

[0099] For any target feature vector, the product of the contribution of the element category to which it belongs and the target feature vector is determined as the corresponding modified feature vector.

[0100] In another more specific embodiment, the combination submodule 6086 is specifically used for:

[0101] The target fusion vector is concatenated to the front end of each corrected feature vector in the first sorting order to obtain an enhanced target fusion vector.

[0102] In one embodiment, the output of the representation model further includes: a text representation of the question text;

[0103] The input unit 604 is specifically used for:

[0104] The enhanced target fusion vector and text representation are input into the anti-counterfeiting recognition model.

[0105] In one embodiment, the representation model includes: first and second encoders and a mapping layer; the input unit 604 includes:

[0106] A processing submodule 6042 is used to process the document image and the question text respectively using the first and second encoders to obtain an image representation and a text representation;

[0107] The fusion submodule 6044 is used to utilize the mapping layer to fuse the image representation and the text representation to obtain a target fusion vector.

[0108] In one embodiment, the above recognition result also indicates the tampering location and the content and style of the tampered document image.

[0109] In one embodiment, the anti-counterfeiting recognition model is obtained by fine-tuning a pre-trained multimodal large model based on the difference between the predicted answer text and the calibrated answer text of the sample document image.

[0110] In one embodiment, any tampered image element is obtained by tampering with the corresponding original image element based on a tampering method collected in real time.

[0111] The functions of the functional units of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0112] An anti-counterfeiting identification device for a certificate image provided by an embodiment of the present specification can improve the ability to identify the authenticity of a certificate image.

[0113] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.

[0114] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 2 The method described.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the medium or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this specification. It should be understood that the above description is only the specific implementation method of this specification and is not intended to limit the protection scope of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of this specification should be included in the protection scope of this specification.

Claims

1. A method for anti-counterfeiting identification of a document image, comprising: Get the document image and its corresponding question text; Inputting the document image and question text into a pre-trained representation model to perform multimodal data fusion processing to obtain a target fusion vector; Based on the target fusion vector, query the anti-counterfeiting knowledge base to obtain a number of target feature vectors matching the target fusion vector; the anti-counterfeiting knowledge base records the feature vectors of a plurality of tampered image elements; wherein the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating its style characteristics; The target fusion vector is feature enhanced based on the plurality of target feature vectors, and the enhanced target fusion vector is input into an anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a forged image.

2. The method according to claim 1, wherein: Any tampered image element belongs to one of the following element categories: Faces, text, anti-counterfeiting marks and logos.

3. The method according to claim 1, wherein: The first feature vector of any first tampered image element is obtained by the following steps: Inputting the first tampered image element and its first description text into the representation model to obtain a first fusion vector; The first fusion vector is used as a first feature vector of the first tampered image element.

4. The method according to claim 1, wherein: The step of performing feature enhancement on the target fusion vector includes: According to the similarity between each target feature vector and the target fusion vector, the target feature vectors are sorted from high to low to obtain each target feature vector in a first sorting order; For each target feature vector in the first sorting, according to the contribution of the element category to which it belongs, the target feature vector is corrected to obtain each corrected feature vector in the first sorting; The target fusion vector is combined with each modified feature vector under the first sorting to obtain an enhanced target fusion vector.

5. The method according to claim 4, wherein: The modifying of each target feature vector comprises: For any target feature vector, the product of the contribution of the element category to which it belongs and the target feature vector is determined as the corresponding modified feature vector.

6. The method according to claim 4, wherein: The combining of the target fusion vector and each corrected feature vector in the first sorting includes: The target fusion vector is concatenated to the front end of each modified feature vector in the first sorting order to obtain an enhanced target fusion vector.

7. The method according to claim 1, wherein: The output of the representation model also includes: a text representation of the question text; The step of inputting the enhanced target fusion vector into the anti-counterfeiting recognition model comprises: The enhanced target fusion vector and text representation are input into the anti-counterfeiting recognition model.

8. The method according to claim 1, wherein: The representation model includes: first and second encoders and a mapping layer; the fusion process includes: Processing the document image and the question text respectively using the first and second encoders to obtain an image representation and a text representation; The image representation and the text representation are fused using the mapping layer to obtain the target fusion vector.

9. The method according to claim 1, wherein: The recognition result also indicates the tampering location and the content and style of the tampered document image.

10. The method according to claim 1, wherein: The anti-counterfeiting recognition model is obtained by fine-tuning a pre-trained multimodal large model based on the difference between the predicted answer text and the calibrated answer text of the sample document image.

11. The method according to claim 1, wherein: Any tampered image element is obtained by tampering with the corresponding original image element based on the tampering method collected in real time.

12. An anti-counterfeiting identification device for a certificate image, comprising: An acquisition unit, used for acquiring a certificate image and a corresponding question text; An input unit, used for inputting the document image and the question text into a pre-trained representation model to perform multi-modal data fusion processing to obtain a target fusion vector; A query unit, configured to query an anti-counterfeiting knowledge base based on the target fusion vector to obtain a plurality of target feature vectors matching the target fusion vector; the anti-counterfeiting knowledge base records the feature vectors of each of a plurality of tampered image elements; wherein a feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating its style characteristics; The enhancement unit is used to perform feature enhancement on the target fusion vector based on the plurality of target feature vectors, and input the enhanced target fusion vector into the anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a forged image.

13. The device according to claim 12, wherein: Any tampered image element belongs to one of the following element categories: Faces, text, anti-counterfeiting marks and logos.

14. The apparatus according to claim 12, further comprising: Identify the unit; The input unit is further used to input the first tampered image element and its first description text into the representation model to obtain a first fusion vector; The determining unit is configured to use the first fusion vector as a first feature vector of the first tampered image element.

15. The device according to claim 12, wherein: The enhancement unit comprises: A sorting submodule, used to sort the target feature vectors according to the similarity between the target feature vectors and the target fusion vector from high to low, so as to obtain the target feature vectors in a first sorting order; A correction submodule, used for correcting each target feature vector in the first sorting according to the contribution of the element category to which it belongs, to obtain each corrected feature vector in the first sorting; The combination submodule is used to combine the target fusion vector with each modified feature vector in the first sorting order to obtain an enhanced target fusion vector.

16. The device according to claim 15, wherein: The correction submodule is specifically used for: For any target feature vector, the product of the contribution of the element category to which it belongs and the target feature vector is determined as the corresponding modified feature vector.

17. The device according to claim 15, wherein: The combined submodule is specifically used for: The target fusion vector is concatenated to the front end of each modified feature vector in the first sorting order to obtain an enhanced target fusion vector.

18. The device according to claim 12, wherein: The output of the representation model also includes: a text representation of the question text; The input unit is specifically used for: The enhanced target fusion vector and text representation are input into the anti-counterfeiting recognition model.

19. The device according to claim 12, wherein: The representation model includes: first and second encoders and a mapping layer; the input unit includes: A processing submodule, configured to process the document image and the question text respectively using the first and second encoders to obtain an image representation and a text representation; The fusion submodule is used to utilize the mapping layer to fuse the image representation and the text representation to obtain the target fusion vector.

20. The device according to claim 12, wherein: The recognition result also indicates the tampering location and the content and style of the tampered document image.

21. The device according to claim 12, wherein: The anti-counterfeiting recognition model is obtained by fine-tuning a pre-trained multimodal large model based on the difference between the predicted answer text and the calibrated answer text of the sample document image.

22. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

23. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.

Citation Information

Cited By

  • Identity document recognition

    WO2026184106A1