Anti-counterfeiting identification method and apparatus for document images

Abstract

Provided in the embodiments of the present description are an anti-counterfeiting identification method and apparatus for document images. The identification method comprises: acquiring a document image and question text corresponding thereto; inputting the document image and the question text into a pre-trained representation model for fusion processing of multi-modal data, in order to obtain a target fusion vector; on the basis of the target fusion vector, querying an anti-counterfeiting knowledge base to obtain several target feature vectors matching the target fusion vector, wherein the anti-counterfeiting knowledge base records respective feature vectors of a plurality of image elements that have been tampered with, and a feature vector of a single image element that has been tampered with is determined on the basis of the image element that has been tampered with and description text indicating style features of the image element that has been tampered with; and on the basis of the several target feature vectors, performing feature enhancement on the target fusion vector, and inputting the enhanced target fusion vector into an anti-counterfeiting identification model, in order obtain an identification result indicating whether the document image is a counterfeit image.

Description

Anti-fake identification method and device for certificate image

[0001] The present application claims priority to the Chinese patent application No. 202411019492.9, filed on July 26, 2024, and entitled "Anti-fake identification method and device for certificate image", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] One or more embodiments of the present specification relate to the field of anti-fake identification, in particular to an anti-fake identification method and device for certificate image. BACKGROUND

[0003] In the field of financial services, user authentication is often performed by examining the certificate image. For example, in the Electronic Know Your Customer (EKYC) scenario, a key step is to verify the user's true identity based on the user's certificate image. However, some unscrupulous individuals who specialize in black production activities submit counterfeit certificate images to cheat the financial system in order to achieve the purpose of illegal profit. In addition, some technical bloggers in open source communities share technical methods for making certificate images on social media platforms for research purposes. The disclosure of this information may be used by unscrupulous individuals who will use these technologies to conduct fraud attacks on existing financial systems, thereby causing economic losses to institutions. Therefore, it is necessary to perform anti-fake identification on the user's certificate image.

[0004] Currently, anti-fake identification is mainly performed by identifying the text content on the certificate image and then comparing the text content with an official information library. However, most countries cannot provide such an official information library. Therefore, a more reliable anti-fake identification solution for certificate images is needed.

[0005] SUMMARY

[0006] One or more embodiments of the present specification describe an anti-fake identification method and device for certificate image, which can improve the identification ability of the authenticity of the certificate image.

[0007] In a first aspect, an anti-fake identification method for certificate image is provided, comprising:

[0008] obtaining a certificate image and a corresponding question text thereof;

[0009] inputting the certificate image and the question text into a pre-trained representation model to perform fusion processing of multi-modal data, to obtain a target fusion vector;

[0010] query, based on the target fusion vector, a forgery prevention knowledge base to obtain a plurality of target feature vectors matched with the target fusion vector; the forgery prevention knowledge base records feature vectors of a plurality of tampered image elements respectively; wherein the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating style characteristics thereof;

[0011] perform feature enhancement on the target fusion vector based on the plurality of target feature vectors, and input the enhanced target fusion vector into a forgery prevention identification model to obtain an identification result indicating whether the certificate image is a forged image.

[0012] In a second aspect, a forgery prevention identification device for a certificate image is provided, comprising:

[0013] an acquisition unit configured to acquire a certificate image and a corresponding question text thereof;

[0014] an input unit configured to input the certificate image and the question text into a pre-trained representation model for fusion processing of multi-modal data to obtain a target fusion vector;

[0015] a query unit configured to query, based on the target fusion vector, a forgery prevention knowledge base to obtain a plurality of target feature vectors matched with the target fusion vector; the forgery prevention knowledge base records feature vectors of a plurality of tampered image elements respectively; wherein the feature vector of a single tampered image element is determined based on the tampered image element and a description text indicating style characteristics thereof;

[0016] an enhancement unit configured to perform feature enhancement on the target fusion vector based on the plurality of target feature vectors, and input the enhanced target fusion vector into a forgery prevention identification model to obtain an identification result indicating whether the certificate image is a forged image.

[0017] In a third aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed in a computer, the computer executes the method of the first aspect.

[0018] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0019] The anti-counterfeiting identification method and device of the certificate image provided by one or more embodiments of the specification pre-construct an anti-counterfeiting knowledge base, wherein a plurality of feature vectors of tampered image elements are stored. Then, when performing anti-counterfeiting identification on the certificate image, the anti-counterfeiting knowledge base is queried based on the fusion vector of the certificate image and its question text, the feature vector queried is used to enhance the feature of the fusion vector, and the anti-counterfeiting identification model is used to identify whether the certificate image is a counterfeit image based on the enhanced fusion vector, thereby improving the identification ability of the certificate image. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the specification, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the specification, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] FIG. 1 is a schematic diagram of an implementation scenario of one embodiment disclosed by the specification;

[0022] FIG. 2 shows a flowchart of an anti-counterfeiting identification method of a certificate image according to one embodiment;

[0023] FIG. 3 shows a schematic diagram of a representation model in one example;

[0024] FIG. 4 shows a schematic diagram of an anti-counterfeiting knowledge base in one example;

[0025] FIG. 5 shows a schematic diagram of an answer text in one example;

[0026] FIG. 6 shows a schematic diagram of an anti-counterfeiting identification device of a certificate image according to one embodiment. DETAILED DESCRIPTION

[0027] The solutions provided by the specification will be described below with reference to the drawings.

[0028] As described above, the method of performing anti-counterfeiting identification based on the text content in the certificate image does not have general applicability. Therefore, some solutions propose to perform anti-counterfeiting identification based on the certificate image itself. For example, a large number of negative samples (i.e. counterfeit certificate images) are collected in advance, and the large model is fine-tuned based on these negative samples. Then, the fine-tuned large model is used to identify the authenticity of the certificate image. However, since the tampering method is constantly changing, in order to ensure that the large model can identify the certificate image tampered based on the new tampering method, the large model needs to be fine-tuned frequently, which will increase the training cost.

[0029] To this end, an improved scheme is proposed in the embodiments of the present specification, that is, an anti-counterfeiting knowledge base is constructed in advance, in which a plurality of feature vectors of tampered image elements are stored. Then, when performing anti-counterfeiting identification on a certificate image, the anti-counterfeiting knowledge base is queried based on a fusion vector of the certificate image and its question text, the feature vector queried is used to enhance the fusion vector, and an anti-counterfeiting identification model is used to identify whether the certificate image is a counterfeit image based on the enhanced fusion vector, thereby realizing the enhancement of the identification capability of the anti-counterfeiting identification model through knowledge injection, and thus the model can be fine-tuned frequently, and the scheme is more general.

[0030] In addition, for the anti-counterfeiting knowledge base described above, it can be updated based on a new tampering method collected in real time, that is, the feature vector of the image element tampered based on the new tampering method is updated to the anti-counterfeiting knowledge base, thereby not only improving the anti-counterfeiting identification capability of the anti-counterfeiting identification model, but also improving the rapid response capability to the new tampering method, and thus the new tampering risk can be better prevented.

[0031] FIG. 1 is a schematic diagram of an implementation scenario of one embodiment disclosed in the present specification. In FIG. 1, the certificate image and its corresponding question text can be input into a pre-trained representation model, and the fusion vector of the certificate image and the question text can be obtained through the representation model. Then, based on the fusion vector, the anti-counterfeiting knowledge base is queried to obtain a plurality of feature vectors matched therewith, and the plurality of feature vectors are used to enhance the fusion vector. Finally, the enhanced feature vector is input into an anti-counterfeiting identification model to obtain the identification result of whether the certificate image is a counterfeit image.

[0032] FIG. 2 shows a flowchart of an anti-counterfeiting identification method of a certificate image according to one embodiment. The method can be executed by any device, equipment, platform, cluster of equipment having computing and processing capabilities. As shown in FIG. 2, the method can include the following steps:

[0033] Step S202, obtaining a certificate image and its corresponding question text.

[0034] The certificate image described above can also be referred to as an identity document, which can include a plurality of image elements such as face, text, anti-counterfeiting identifier, and logo, etc. In addition, it can also include a chip, etc.

[0035] In one example, the question text described above is used to ask whether the certificate image is a counterfeit image. Of course, in practice, the question text is also used to ask the tampering position of the counterfeit image, etc.

[0036] Step S204, inputting the certificate image and the question text into a pre-trained representation model for fusion processing of multi-modal data to obtain a target fusion vector.

[0037] In an embodiment, the above-mentioned representation model is a multi-modal model, which is used to perform fusion processing on multi-modal data such as text and image. It should be noted that the parameter amount of the representation model is much smaller than that of a large multi-modal model (LMM).

[0038] FIG. 3 shows a schematic diagram of a representation model in one example. In FIG. 3, the representation model includes a first and second encoder and a mapping layer. Specifically, the first encoder is used to process the certificate image to obtain an image representation of the certificate image, and the second encoder is used to process the question text to obtain a text representation of the question text. The mapping layer is used to perform fusion processing on the image representation and the text representation to obtain a target fusion vector.

[0039] The first encoder mentioned above can be implemented as a convolutional neural network, such as a CNN (Convolutional Neural Networks), an RNN (Recurrent Neural Network), an FCN (Fully Convolutional Networks), and the like. Of course, it can also be implemented as other neural networks, such as an MLP (MultiLayer Perceptron Network) network, and the like.

[0040] The second encoder mentioned above can be a Transformer encoder, i.e., it encodes the question text based on an attention mechanism.

[0041] The mapping layer mentioned above is also called a linear mapping layer, and the above-mentioned fusion processing specifically includes performing multiple linear mapping processing on the combination of the image representation and the text representation using a mapping function. The mapping function here can be, for example, a GELU function, and the like.

[0042] Of course, in practice, the above-mentioned representation model can also be implemented based on other networks or models capable of performing fusion processing on multi-modal data, which are not limited in the present specification.

[0043] It should be noted that in addition to the above-mentioned target fusion vector, the representation model can also output the image representation of the certificate image and the text representation of the question text.

[0044] Step S206: Based on the target fusion vector, querying the anti-fake knowledge base to obtain a plurality of target feature vectors matched with the target fusion vector.

[0045] The anti-counterfeiting knowledge base records the feature vectors of a plurality of tampered image elements. Any tampered image element can belong to one of the following element categories: face, text, anti-counterfeiting mark, and logo, etc. Of course, in practice, the element category can also include a chip, etc.

[0046] In one embodiment, the feature vector of any tampered image element (hereinafter referred to as a first tampered image element) is obtained by the following steps:

[0047] The first tampered image element and its first description text are input into the above-mentioned representation model to obtain a first fusion vector. The first fusion vector is taken as the first feature vector of the first tampered image element.

[0048] The first description text indicates the style characteristics of the first tampered image element, such as edge protrusion, red background, local pixel discordance, etc.

[0049] In addition, the first fusion vector is similar to the process of obtaining the target fusion vector, which will not be repeated here.

[0050] In another embodiment, the feature vector of any tampered image element can also be an image representation extracted from the tampered image element using a convolutional neural network such as CNN.

[0051] The above-mentioned query anti-counterfeiting knowledge base specifically includes calculating the similarity between the target fusion vector and the feature vectors of each tampered image element in the anti-counterfeiting knowledge base, and determining each feature vector with a similarity greater than a preset threshold as each target feature vector. Here, the similarity includes but is not limited to cosine similarity, Euclidean distance, etc.

[0052] It should be noted that in practice, for any tampered image element, the corresponding fusion vector and image representation can be recorded at the same time, so that in the case that the query result based on the fusion vector in the anti-counterfeiting knowledge base is not ideal, the query result can be obtained based on the image representation in the anti-counterfeiting knowledge base, i.e. by calculating the similarity between the target fusion vector and the image representation of each tampered image element to obtain the query result.

[0053] Figure 4 shows an anti-counterfeiting knowledge base diagram in one example. In Figure 4, the anti-counterfeiting knowledge base records the feature vectors of n tampered image elements, and each tampered image element belongs to one of the following element categories: face, text, anti-counterfeiting mark, and logo, etc. In addition, the feature vector of each tampered image element corresponds to a text-image pair, which includes the tampered image element and its description text.

[0054] It should be noted that, in order to ensure the effectiveness of the anti-counterfeiting knowledge base, it is usually necessary to collect new tampering methods in real time, and tamper with the corresponding image elements according to the new tampering method to obtain new tampered image elements. Then, the new feature vector determined based on the new tampered image elements and the description text thereof is updated into the anti-counterfeiting knowledge base.

[0055] In step S208, the target fusion vector is enhanced based on the plurality of target feature vectors, and the enhanced target fusion vector is input into the anti-counterfeiting identification model to obtain an identification result indicating whether the certificate image is a counterfeit image.

[0056] The anti-counterfeiting identification model can be obtained by fine-tuning a multi-modal large model based on sample certificate images (the specific fine-tuning process will be described later); or it can be obtained by training a neural network model (such as a DNN) based on sample certificate images.

[0057] It should be understood that each of the plurality of target feature vectors corresponds to a tampered image element, and each tampered image element belongs to an element category, so each target feature vector belongs to an element category. In this step, the target fusion vector can be enhanced based on the contribution of the element category to which each target feature vector belongs. The contribution of each element category can be set according to expert experience or actual needs.

[0058] The specific process of feature enhancement is described below:

[0059] In one embodiment, the target feature vectors can be sorted in descending order of similarity between the target feature vectors and the target fusion vector to obtain the target feature vectors in the first order. For the target feature vectors in the first order, the contribution of the element category to which each target feature vector belongs is corrected to obtain the modified feature vectors in the first order. The target fusion vector is combined with the modified feature vectors in the first order to obtain the enhanced target fusion vector.

[0060] The similarity according to which the target feature vectors are sorted is the similarity calculated when the target feature vectors are selected.

[0061] In a more specific embodiment, for any target feature vector, the product of the contribution of the element category to which the target feature vector belongs and the target feature vector is determined as the corresponding modified feature vector.

[0062] Of course, in practice, the corresponding modified feature vector can also be determined by performing other operations on the target feature vector and the contribution of the element category to which it belongs, which is not limited in this specification.

[0063] It should be noted that the target feature vectors in the first ranking refer to the target feature vector with the highest similarity first, followed by the target feature vector with the second highest similarity, and so on, and finally the target feature vector with the lowest similarity. The modified feature vectors in the first ranking refer to the modified feature vectors arranged in the order of the corresponding target feature vectors.

[0064] In addition, the combination of the target fusion vector and the modified feature vectors in the first ranking can specifically include splicing the target fusion vector at the front end of the modified feature vectors in the first ranking. That is, the target fusion vector is arranged at the front, followed by the modified feature vectors in the first ranking.

[0065] Of course, in practice, the target fusion vector can also be spliced at the rear end of the modified feature vectors in the first ranking, which is not limited in the present specification.

[0066] For example, assume that there are four element categories: face, text, anti-fake mark, and logo, and the contribution degrees of the four element categories are 0.3, 0.1, 0.5, and 0.1, respectively. It is also assumed that five target feature vectors x1, x2, x3, x4, and x5 are found, and the similarities between them and the target fusion vector are 0.75, 0.8, 0.77, 0.9, and 0.82, respectively. Then, after sorting the target feature vectors according to the similarity from high to low, the target feature vectors in the first ranking are obtained: x4, x5, x2, x3, and x1. If x4 and x3 belong to the face, and x5, x2, and x1 belong to the logo, then the modified feature vectors in the first ranking can be obtained: 0.3x4, 0.1x5, 0.1x2, 0.3x3, and 0.1x1.

[0067] In another embodiment, the target feature vectors can be sorted by combining the similarity and the contribution degree. For example, the target feature vectors can be sorted according to the product of the similarity and the contribution degree corresponding to each target feature vector from high to low. It should be understood that in this sorting manner, the target feature vectors can be modified or not modified (for details of the modification process, see the above embodiment), which is not limited in the present specification.

[0068] As mentioned earlier, the output of the representation model can also include the text feature of the text question. In the case of also including the text feature, the enhanced target fusion vector and the text feature can be input into the anti-fake identification model together. More specifically, the text feature can be spliced at the rear end of the enhanced target fusion vector, and then the spliced result is input into the anti-fake identification model.

[0069] In practice, the recognition result can not only indicate whether the certificate image is a fake image, but also indicate the tampering position, the content and style of the tampered certificate image, and the like. Here, the style includes but is not limited to content form (such as text or picture, etc.), color, shape, and the like.

[0070] It should be noted that when the above anti-fake recognition model is obtained by fine-tuning the pre-trained multi-modal large model, the recognition result is output in the form of an answer text.

[0071] FIG. 5 shows an answer text diagram in one example. In FIG. 5, the answer text indicates the following aspects: 1. certificate type: C1 certificate of country A; 2. conclusion content: the certificate is tampered; 3. tampering position: (10, 200), (400, 210), (400, 260), and (10, 260), which represent four vertex coordinates respectively; and 4. tampered content and style: white paper with the text 'LIM CHEE MENG'.

[0072] The fine-tuning process of the multi-modal large model is described as follows:

[0073] Specifically, the sample certificate image and the sample question text thereof can be input into the representation model to obtain a sample fusion vector. Based on the sample fusion vector, the anti-fake knowledge base is queried to obtain a plurality of sample feature vectors matched with the sample fusion vector. The sample fusion vector is feature-enhanced based on the plurality of sample feature vectors, and the enhanced sample fusion vector is input into the multi-modal large model to obtain a predicted answer text. The parameters of the multi-modal large model are adjusted according to the difference between the predicted answer text and the labeled answer text of the sample certificate image. After multiple rounds of parameter adjustment, the final multi-modal large model is obtained, which is used as the above anti-fake recognition model.

[0074] In the present scheme, the text difference loss between the predicted answer text and the labeled answer text can be used to measure the difference between the two.

[0075] In one embodiment, the text difference loss can be determined according to the text edit distance between the predicted answer text and the labeled answer text, wherein the greater the text edit distance, the greater the text difference loss, i.e., the text difference loss is positively correlated with the text edit distance. Specifically, the text edit distance (Edit Distance), also known as Levenshtein distance, is the minimum number of editing operations required to convert one string (text) to another. The greater the distance, the more different they are. The editing operation can include replacing one character with another, inserting a character, and deleting a character.

[0076] In another embodiment, the predicted answer text and the labeled answer text can also be respectively converted into a first vector and a second vector in the manner of word embedding, and based on the first vector and the second vector, a text similarity of the predicted answer text and the labeled answer sample is determined; and then, the text difference loss can be determined according to the text similarity. The higher the text similarity is, the smaller the text difference loss is. In other embodiments, there are other ways to determine the text difference loss, which are not enumerated here.

[0077] That is, the present scheme can fine-tune the pre-trained multi-modal large model based on the difference between the predicted answer text and the labeled answer text of the sample certificate image.

[0078] In summary, the certificate image anti-counterfeiting method provided by the embodiments of the present specification can greatly improve the authenticity of the certificate image recognition by using the feature vector of the tampered image element queried from the anti-counterfeiting knowledge base to enhance the target fusion vector. In addition, the present scheme constructs an anti-counterfeiting knowledge base and updates it based on the newly collected tampering methods, which can realize the injection of new risks and new field knowledge into the input of the large model in the form of features, that is, to update the anti-counterfeiting field knowledge without fine-tuning the large model. Finally, since the actual operation of knowledge injection is essentially an enhancement of Prompt, it can guarantee the flexibility and timeliness of Prompt, and also ensure the advancement of knowledge.

[0079] In summary, the present scheme has the following advantages:

[0080] 1. Image element level storage can realize targeted enhancement of features, thereby improving the accuracy of the recognition result.

[0081] 2. No additional training: The representation model used in the present scheme is pre-trained and does not need additional training, thereby improving the response timeliness.

[0082] 3. Prompt modification is more flexible: The present scheme converts the large model fine-tuning task into a Prompt design task, which involves higher timeliness of Prompt.

[0083] 4. High diversity of Prompt: There can be various flexible combinations in Prompt, which facilitates accurate identification of forged images by the model.

[0084] 5. Fast knowledge update: The knowledge base is constantly updated, which can ensure the advancement of knowledge.

[0085] 6. Multi-modal large model features are more abundant: When the anti-counterfeiting identification model is implemented based on a multi-modal large model, it can receive more abundant multi-modal input, and the recognition result is more accurate.

[0086] Corresponding to the above-mentioned anti-fake identification method of the certificate image, an embodiment of the present specification also provides an anti-fake identification device of a certificate image, as shown in FIG. 6, which can include:

[0087] The acquisition unit 602 is configured to acquire the certificate image and the corresponding question text.

[0088] The input unit 604 is configured to input the certificate image and the question text into the pre-trained representation model for fusion processing of multi-modal data to obtain a target fusion vector.

[0089] The query unit 606 is configured to query an anti-fake knowledge base based on the target fusion vector to obtain a plurality of target feature vectors matched with the target fusion vector, wherein the anti-fake knowledge base records a plurality of feature vectors of tampered image elements, and the feature vector of a single tampered image element is determined based on the tampered image element and the description text indicating the style characteristics thereof.

[0090] The enhancement unit 608 is configured to perform feature enhancement on the target fusion vector based on the plurality of target feature vectors, and input the enhanced target fusion vector into an anti-fake identification model to obtain an identification result indicating whether the certificate image is a fake image.

[0091] Any of the above-mentioned tampered image elements belongs to one of the following element categories:

[0092] Face, text, anti-fake mark, and logo.

[0093] In an embodiment, the device further includes a determination unit 610.

[0094] The input unit 604 is further configured to input the first tampered image element and the first description text thereof into the representation model to obtain a first fusion vector.

[0095] The determination unit 610 is configured to determine the first fusion vector as the first feature vector of the first tampered image element.

[0096] In an embodiment, the enhancement unit 608 includes:

[0097] The sorting sub-module 6082 is configured to sort each target feature vector in descending order of similarity between each target feature vector and the target fusion vector to obtain each target feature vector in the first sorting;

[0098] The correction sub-module 6084 is configured to correct each target feature vector in the first sorting according to the contribution degree of the element category to which the target feature vector belongs to obtain each corrected feature vector in the first sorting;

[0099] The combination submodule 6086 is configured to combine the target fusion vector and each modified feature vector in the first ranking to obtain an enhanced target fusion vector.

[0100] In a more specific embodiment, the modification submodule 6084 is specifically configured to:

[0101] For any target feature vector, the product of the contribution degree of the element category to which the target feature vector belongs and the target feature vector is determined as the corresponding modified feature vector.

[0102] In another more specific embodiment, the combination submodule 6086 is specifically configured to:

[0103] The target fusion vector is spliced at the front end of each modified feature vector in the first ranking to obtain an enhanced target fusion vector.

[0104] In an embodiment, the output of the representation model further includes a text representation of the question text.

[0105] The input unit 604 is specifically configured to:

[0106] The enhanced target fusion vector and the text representation are input into the anti-fake identification model.

[0107] In an embodiment, the representation model includes a first and second encoder and a mapping layer, and the input unit 604 includes:

[0108] The processing submodule 6042 is configured to process the certificate image and the question text using the first and second encoders respectively to obtain an image representation and a text representation.

[0109] The fusion submodule 6044 is configured to fuse the image representation and the text representation using the mapping layer to obtain a target fusion vector.

[0110] In an embodiment, the identification result further indicates a tampering position and content and style of the certificate image after tampering.

[0111] In an embodiment, the anti-fake identification model is obtained by fine-tuning a pre-trained multi-modal large model based on the difference between the predicted answer text and the labeled answer text of the sample certificate image.

[0112] In an embodiment, any tampered image element is obtained by tampering a corresponding original image element based on a real-time collected tampering method.

[0113] The functions of each functional unit of the above-mentioned embodiment device can be realized by the steps of the above-mentioned method embodiment, and therefore, the specific working process of the device provided by an embodiment of the present specification will not be repeated here.

[0114] This specification provides an embodiment of an anti-counterfeiting identification device for document images, which can improve the ability to identify the authenticity of document images.

[0115] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the method described in conjunction with FIG2.

[0116] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein executable code is stored in the memory, and the processor, when executing the executable code, implements the method described in conjunction with FIG2.

[0117] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the medium or device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0118] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0119] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this specification. It should be understood that the above description is only a specific embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this specification should be included within the scope of protection of this specification.

Claims

1. A method for anti-counterfeiting identification of document images, comprising: Obtain the image of the document and its corresponding question text; The document image and the question text are input into a pre-trained representation model for multimodal data fusion processing to obtain the target fusion vector; Based on the target fusion vector, the anti-counterfeiting knowledge base is queried to obtain several target feature vectors that match the target fusion vector; the anti-counterfeiting knowledge base records the feature vectors of multiple tampered image elements; the feature vector of a single tampered image element is determined based on the tampered image element and the descriptive text indicating its style characteristics; Based on the aforementioned target feature vectors, the target fusion vector is enhanced, and the enhanced target fusion vector is input into the anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a counterfeit image.

2. The method according to claim 1, wherein, Any tampered image element belongs to one of the following element categories: Faces, text, anti-counterfeiting marks, and logos.

3. The method according to claim 1, wherein, The first feature vector of any first tampered image element is obtained through the following steps: The first tampered image element and its first descriptive text are input into the representation model to obtain the first fusion vector; The first fusion vector is used as the first feature vector of the first tampered image element.

4. The method according to claim 1, wherein, The feature enhancement of the target fusion vector includes: The target feature vectors are sorted from high to low according to their similarity to the target fusion vector, resulting in the target feature vectors in the first sort. For each target feature vector in the first ranking, the target feature vector is modified according to the contribution of its element category to obtain each modified feature vector in the first ranking. The target fusion vector is combined with each of the modified feature vectors under the first sorting to obtain the enhanced target fusion vector.

5. The method according to claim 4, wherein, The modification of each target feature vector includes: For any target feature vector, the product of the contribution of its element category and the target feature vector is determined as the corresponding modified feature vector.

6. The method according to claim 4, wherein, The step of combining the target fusion vector with each of the corrected feature vectors under the first sorting includes: The target fusion vector is concatenated to the front of each of the modified feature vectors in the first sorting to obtain the enhanced target fusion vector.

7. The method according to claim 1, wherein, The output of the representation model also includes: the textual representation of the question text; The step of inputting the enhanced target fusion vector into the anti-counterfeiting recognition model includes: The enhanced target fusion vector and text representation are input into the anti-counterfeiting recognition model.

8. The method according to claim 1, wherein, The representation model includes: first and second encoders and a mapping layer; the fusion process includes: The first and second encoders are used to process the document image and the question text respectively to obtain image representation and text representation; The image representation and text representation are fused using the mapping layer to obtain the target fusion vector.

9. The method according to claim 1, wherein, The identification results also indicate the location of the alteration and the content and style of the altered document image.

10. The method according to claim 1, wherein, The anti-counterfeiting recognition model is obtained by fine-tuning a pre-trained multimodal large model based on the difference between the predicted answer text and the labeled answer text of the sample document image.

11. The method according to claim 1, wherein, Any tampered image element is obtained by tampering with the corresponding original image element based on a real-time collected tampering method.

12. An anti-counterfeiting identification device for an ID card image, comprising: The acquisition unit is used to acquire the document image and its corresponding question text; The input unit is used to input the document image and the question text into a pre-trained representation model for multimodal data fusion processing to obtain a target fusion vector; The query unit is used to query the anti-counterfeiting knowledge base based on the target fusion vector to obtain several target feature vectors that match the target fusion vector; the anti-counterfeiting knowledge base records the feature vectors of multiple tampered image elements. The feature vector of a single tampered image element is determined based on the tampered image element and the descriptive text indicating its style characteristics; The enhancement unit is used to enhance the target fusion vector based on the plurality of target feature vectors, and input the enhanced target fusion vector into the anti-counterfeiting recognition model to obtain a recognition result indicating whether the document image is a counterfeit image.

13. The apparatus according to claim 12, wherein, Any tampered image element belongs to one of the following element categories: Faces, text, anti-counterfeiting marks, and logos.

14. The apparatus of claim 12, further comprising: Define the unit; The input unit is further configured to input the first tampered image element and its first descriptive text into the representation model to obtain the first fusion vector; The determining unit is used to use the first fusion vector as the first feature vector of the first tampered image element.

15. The apparatus according to claim 12, wherein, The enhancement unit includes: The sorting submodule is used to sort the target feature vectors from high to low according to the similarity between each target feature vector and the target fusion vector, so as to obtain the target feature vectors under the first sorting. The correction submodule is used to correct each target feature vector under the first sorting according to the contribution of its element category, so as to obtain each corrected feature vector under the first sorting. The combination submodule is used to combine the target fusion vector with each of the modified feature vectors under the first sorting to obtain the enhanced target fusion vector.

16. The apparatus according to claim 15, wherein, The correction submodule is specifically used for: For any target feature vector, the product of the contribution of its element category and the target feature vector is determined as the corresponding modified feature vector.

17. The apparatus according to claim 15, wherein, The combined submodule is specifically used for: The target fusion vector is concatenated to the front of each of the modified feature vectors in the first sorting to obtain the enhanced target fusion vector.

18. The apparatus according to claim 12, wherein, The output of the representation model also includes: the textual representation of the question text; The input unit is specifically used for: The enhanced target fusion vector and text representation are input into the anti-counterfeiting recognition model.

19. The apparatus according to claim 12, wherein, The representation model includes: first and second encoders and a mapping layer; the input unit includes: The processing submodule is used to process the document image and the question text using the first and second encoders respectively, to obtain image representation and text representation; The fusion submodule is used to perform fusion processing on the image representation and text representation using the mapping layer to obtain the target fusion vector.

20. The apparatus according to claim 12, wherein, The identification results also indicate the location of the alteration and the content and style of the altered document image.

21. The apparatus according to claim 12, wherein, The anti-counterfeiting recognition model is obtained by fine-tuning a pre-trained multimodal large model based on the difference between the predicted answer text and the labeled answer text of the sample document image.

22. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in a computer, it causes the computer to perform the method according to any one of claims 1-11.

23. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-11.

Images