Material authenticity detection method and device in certificate anti-counterfeiting detection

Abstract

The embodiment of the invention provides a material authenticity detection method and device in certificate anti-counterfeiting detection. The method comprises the following steps: acquiring image data of a to-be-detected card; the image data are input into a multi-modal model for model processing, the multi-modal model comprises a first branch network, a second branch network, a third branch network and a fusion network, and the model processing comprises the steps that differential features of the image data are extracted through the first branch network, and the differential features of the image data are extracted through the second branch network; extracting a frequency domain feature of the image data through the second branch network, extracting an enhanced feature of the image data through the third branch network, and performing fusion processing on the differential feature, the frequency domain feature and the enhanced feature through the fusion network to obtain a fusion feature; and determining a prediction result of the to-be-detected card about the authenticity of the material according to the fusion features by using a classifier. The accuracy of material authenticity detection can be improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the computer field, and in particular, to a material authenticity detection method and device in document anti-counterfeiting detection. Background Art

[0002] At present, electronic identity authentication is widely used. Document anti-counterfeiting is a key link in identity authentication. A common way to forge documents is to print out the document in color or black and white, and then impersonate the user's real document for identity authentication. This type of document forgery needs to be identified in document anti-counterfeiting detection.

[0003] In the prior art, a large number of real certificates and their corresponding printed images are often used to train the detection model, which is used to detect the authenticity of the material corresponding to the input card data. Among them, the user's certificate information belongs to the user's private data, and the private data needs to be protected from being leaked. Due to the influence of various environmental factors, it is difficult to obtain sufficient number and types of printed images in reality, which affects the detection effect of the detection model and the accuracy of material authenticity detection is low. Summary of the invention

[0004] One or more embodiments of the present specification describe a material authenticity detection method and device in document anti-counterfeiting detection, which can improve the accuracy of material authenticity detection.

[0005] In a first aspect, a method for detecting the authenticity of a material in document anti-counterfeiting detection is provided, the method comprising:

[0006] Obtain image data of the card to be tested;

[0007] Inputting the image data into a multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting differential features of the image data through the first branch network, extracting frequency domain features of the image data through the second branch network, extracting enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fused features;

[0008] A classifier is used to determine a prediction result of the authenticity of the material of the card to be tested according to the fusion features.

[0009] In a possible implementation, the first branch network is a convolutional neural network; and extracting the differential features of the image data through the first branch network includes:

[0010] The convolutional neural network is used to extract differential features of the image data by using a central difference convolution method.

[0011] In a possible implementation, extracting the frequency domain features of the image data through the second branch network includes:

[0012] The second branch network is used to obtain a frequency spectrum of the image data by discrete cosine transform, and the frequency spectrum is used as a frequency domain feature of the image data.

[0013] In a possible implementation, extracting enhanced features of the image data through the third branch network includes:

[0014] The image data is mixed with other image data of different card types through the third branch network to obtain cross-domain enhanced features of the image data.

[0015] In a possible implementation, the multimodal model determines the prediction loss in the following manner during model training:

[0016] Inputting sample image data of any sample detection card in the sample set into the multimodal model and classifier to obtain a first prediction category of the sample detection card regarding the authenticity of the material;

[0017] Determine the main loss based on the difference between the first predicted category and the label category corresponding to the sample detection card;

[0018] Based on the fusion features of the sample test cards output by the multimodal model, the similarity between the sample test cards in the sample set is determined; the sample test cards whose label categories represent that the material is true are used as cluster centers, and the sample test cards are clustered according to the similarity;

[0019] The auxiliary loss is determined according to the distance between any sample detection card and the cluster center corresponding to its clustering result, and the label category corresponding to the sample detection card; the main loss and the auxiliary loss together constitute the prediction loss.

[0020] In a possible implementation, the multimodal model is trained in the following manner:

[0021] Inputting sample image data of the sample detection card into the multimodal model, and obtaining fusion features of the sample detection card through the multimodal model;

[0022] Extracting material style sub-features and content sub-features from the fused features through a first extraction network and a second extraction network respectively;

[0023] Inputting the material style sub-feature into a style branch network, and outputting a first prediction result of the sample detection card regarding the authenticity of the material through the style branch network;

[0024] Inputting the content sub-feature into a content branch network, and outputting a second prediction result about the card type of the sample detection card through the content branch network;

[0025] Based on the first prediction result and the second prediction result, model parameters of the multimodal model are updated by adversarial training.

[0026] Furthermore, the step of inputting the material style sub-feature into the style branch network includes:

[0027] The material style sub-feature is subjected to random interpolation processing using the content sub-feature to obtain an interpolation feature;

[0028] The interpolated features are input into the style branch network.

[0029] Furthermore, the updating of the model parameters of the multimodal model by adversarial training includes:

[0030] Based on the first prediction result, adjusting model parameters of the multimodal model and the style branch network with the goal of minimizing the prediction loss regarding the authenticity of the material;

[0031] Based on the second prediction result, adjusting the model parameters of the content branch network with the goal of minimizing the prediction loss regarding the card type;

[0032] Based on the second prediction result, the model parameters of the multimodal model are adjusted with the goal of minimizing the cross entropy between the second prediction result and the uniform distribution.

[0033] In a second aspect, a material authenticity detection device for document anti-counterfeiting detection is provided, the device comprising:

[0034] An acquisition unit, used to acquire image data of a card to be detected;

[0035] a processing unit, configured to input the image data acquired by the acquisition unit into a multimodal model for model processing, wherein the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, wherein the model processing includes extracting differential features of the image data through the first branch network, extracting frequency domain features of the image data through the second branch network, extracting enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fused features;

[0036] The prediction unit is used to use a classifier to determine the prediction result of the material authenticity of the card to be detected according to the fusion features obtained by the processing unit.

[0037] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0038] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0039] Through the method and device provided in the embodiment of this specification, firstly, the image data of the card to be detected is obtained; then the image data is input into the multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting the differential features of the image data through the first branch network, extracting the frequency domain features of the image data through the second branch network, extracting the enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fusion features; finally, the classifier is used to determine the prediction result of the authenticity of the material of the card to be detected according to the fusion features. As can be seen from the above, the embodiment of this specification adopts a multimodal fusion detection scheme, in which the differential features are conducive to characterizing the style rendering details of the material, the frequency domain features are conducive to characterizing the printing clues of high-frequency noise points, and the enhanced features are conducive to enriching the features of different fields, so that the generalization of the model is stronger, and the prediction result of the authenticity of the material of the card to be detected is determined by fusing the differential features, frequency domain features and enhanced features, which can improve the accuracy of material authenticity detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0041] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0042] Figure 2 A flow chart of a material authenticity detection method in document anti-counterfeiting detection according to an embodiment is shown;

[0043] Figure 3A schematic diagram of a central difference convolution processing method according to an embodiment is shown;

[0044] Figure 4 A schematic diagram showing a characteristic mixing method according to an embodiment;

[0045] Figure 5 A schematic diagram of a multimodal model training method according to an embodiment is shown;

[0046] Figure 6 A schematic diagram showing a method for determining auxiliary loss according to an embodiment;

[0047] Figure 7 A schematic diagram of a multimodal model training method according to another embodiment is shown;

[0048] Figure 8 A schematic block diagram of a material authenticity detection device in document anti-counterfeiting detection according to an embodiment is shown. DETAILED DESCRIPTION

[0049] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0050] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves material authenticity detection in document anti-counterfeiting detection. It can be understood that a typical form of counterfeiting in document anti-counterfeiting detection is material counterfeiting. For example, a printed image of a real document is a fake document. The paper used for the printed image is usually significantly different from the material of the real document. This type of document counterfeiting can be identified through material authenticity detection. Figure 1 , which shows a real card and three fake certificates corresponding to the real card. The real card and the fake certificate have the same text content and pictures, but the materials are different. There are various clues of material forgery in the fake certificates. For example, the part marked by the oval dotted line frame in fake certificate A has uneven color distribution, the part marked by the oval dotted line frame in fake certificate B has color deviation and smudge problem, and the part marked by the oval dotted line frame in fake certificate C has particle noise problem. These problems may be caused by the quality of the printer and inkjet. In addition, there are also problems such as uneven surface and curling deformation caused by thin printing paper. These are all breakthroughs in finding clues of material forgery.

[0051] In the embodiment of this specification, for the task of detecting the authenticity of card materials, it is necessary to design a model to capture the style rendering details of the material and the printing clues of high-frequency noise, so as to identify material counterfeiting, and it is expected to improve the accuracy of material authenticity detection.

[0052] Figure 2A flow chart of a material authenticity detection method in document anti-counterfeiting detection according to an embodiment is shown. The method can be based on Figure 1 The implementation scenario shown in Figure 1 is as follows. Figure 2 As shown, the material authenticity detection method in the document anti-counterfeiting detection in this embodiment includes the following steps: Step 21, obtaining the image data of the card to be detected; Step 22, inputting the image data into the multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting the differential features of the image data through the first branch network, extracting the frequency domain features of the image data through the second branch network, extracting the enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fusion features; Step 23, using a classifier, determining the prediction result of the material authenticity of the card to be detected according to the fusion features. The specific execution methods of the above steps are described below.

[0053] First, in step 21, image data of the card to be detected is obtained. It is understandable that the card to be detected can be in the form of a card, and its image data can be acquired by a camera, for example, by taking a photo with a camera built into a mobile phone to acquire its image data.

[0054] In the embodiments of this specification, the card to be detected may be a real certificate, or it may be a fake certificate with a printed image. It is understandable that the material of a fake certificate is different from that of a real certificate. A fake certificate can be identified by capturing the forgery clues in its image data.

[0055] Then in step 22, the image data is input into a multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting the differential features of the image data through the first branch network, extracting the frequency domain features of the image data through the second branch network, extracting the enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fused features. It can be understood that by further processing the image data, including extracting differential features, frequency domain features and enhanced features, and fusing the above features, material forgery clues such as noise points and deformations can be stably highlighted, which can improve the accuracy of material authenticity detection.

[0056] In one example, the first branch network is a convolutional neural network; and extracting the differential features of the image data through the first branch network includes:

[0057] The convolutional neural network is used to extract differential features of the image data by using a central difference convolution method.

[0058] In this example, the center difference convolution is used to replace the ordinary convolution. Ordinary convolution usually directly aggregates the information of local pixel brightness, which is easily affected by environmental factors such as external lighting. It is also difficult to characterize fine-grained material fraud features, which easily leads to weak generalization ability and stability of the model. In contrast, the center difference convolution uses both brightness and gradient information. These spatial difference features have stronger lighting invariance and can also capture finer-grained material fraud clues, such as color cast details, grid effects, local deformation, etc.

[0059] Figure 3 FIG. 4 is a schematic diagram showing a central difference convolution processing method according to an embodiment. Figure 3 , two-dimensional spatial convolution is the basic operation used for visual tasks in convolutional neural networks. In convolutional neural networks, the input and output data of the convolution layer are sometimes called feature maps. Among them, the input data of the convolution layer is called the input feature map, and the output data is called the output feature map. Center difference convolution is different from ordinary convolution. Let's compare the similarities and differences between center difference convolution and ordinary convolution. For ordinary convolution, two-dimensional convolution has two main steps: first, sample the local receptive field area R on the input feature map x; then aggregate the sampled values ​​through weighted summation. Therefore, the output feature map y can be expressed as Among them, p0 represents the current position on the input feature map and the output feature map, and p n Enumerate the positions in R. For the central difference convolution, the central difference operation is introduced in the basic convolution to enhance its representation and generalization capabilities. Similarly, the central difference convolution also includes two steps, namely sampling and aggregation. The sampling step is similar to the basic convolution, but the aggregation step is different. The central difference convolution is more inclined to aggregate the center gradient of the sampled value. The output feature map y can be expressed as

[0060]

[0061] In one example, extracting the frequency domain features of the image data through the second branch network includes:

[0062] The second branch network is used to obtain a frequency spectrum of the image data by discrete cosine transform, and the frequency spectrum is used as a frequency domain feature of the image data.

[0063] In this example, the spectrum of the image is extracted by discrete cosine transform and used as a separate frequency domain branch input to guide the model to learn more high-frequency attack clues that can express material fraud. For a two-dimensional image, discrete cosine transform can be performed based on the following formula:

[0064]

[0065] Among them, f(i,j) is the pixel value of the input image, and F(u,v) is the result after discrete cosine transform, that is, the coefficient of discrete cosine transform. The low-order part with small change amplitude (i.e. low-frequency information) is reflected in the upper left of the spectrum graph, and the high-order part with large change amplitude (i.e. high-frequency information) is reflected in the lower right of the spectrum graph.

[0066] In one example, extracting enhanced features of the image data through the third branch network includes:

[0067] The image data is mixed with other image data of different card types through the third branch network to obtain cross-domain enhanced features of the image data.

[0068] In this example, features of card types from different fields can be mixed to create a richer cross-domain feature space, making the model more generalizable to unknown card types.

[0069] Figure 4 FIG. 2 is a schematic diagram showing a characteristic mixing method according to an embodiment. Figure 4 , for the original image data of the card to be detected, the domain generalization problem of the model for unknown card types is considered, that is, some factors that affect the generalization of the model to new fields are explored. In the embodiments of this specification, the domain generalization problem is solved through data operation, and the cross-domain nature of the training data or features is enhanced by enhancing, changing and generating data or features, which can be modularly embedded into the existing model structure, and the amount of engineering transformation is small. When there are large differences in feature distribution between data in different domains, feature mixing can well match the high-order information in these cross-domain features, accept features that belong to more complex distribution situations, and use precise histogram matching to perform high-order matching and alignment of the empirical cumulative distribution function of the image features. After the original image data and the style image are feature mixed, the image data after feature mixing is obtained, which can collide with a richer cross-domain feature space. Among them, the above-mentioned style image and the original image data have different card types.

[0070] Finally, in step 23, a classifier is used to determine the prediction result of the authenticity of the material of the card to be tested based on the fused features. It can be understood that the above classifier can map the fused features to one of two categories, that is, to achieve binary classification, the two categories include a category where the material is authentic and a category where the material is fake.

[0071] Figure 5 FIG. 1 is a schematic diagram showing a multimodal model training method according to an embodiment. Figure 5 , the multimodal model includes the first branch network, the second branch network, the third branch network and the fusion network. The three branch networks focus on extracting differential features, frequency domain features and enhanced features respectively. These three features are fused and used for binary classification training of the authenticity of card materials. In terms of training strategy, considering the imbalance of the black and white ratio of different card training data, resampling optimization can be performed for different card domains according to the magnitude difference of each field. In terms of loss function, considering that card material forgery will encounter more unknown attack methods in the future, a single center constraint is introduced as an auxiliary loss function based on the binary cross entropy loss function. Its basic principle is to cluster known real cards as a single center, and learn more unknown various material attacks as divergent distributions away from a single center. When new material attacks appear in the future, they will be intercepted by the model with a high probability as long as they are far away from the center of the real card.

[0072] In one example, the multimodal model determines the prediction loss in the following manner during model training:

[0073] Inputting sample image data of any sample detection card in the sample set into the multimodal model and classifier to obtain a first prediction category of the sample detection card regarding the authenticity of the material;

[0074] Determine the main loss based on the difference between the first predicted category and the label category corresponding to the sample detection card;

[0075] Based on the fusion features of the sample test cards output by the multimodal model, the similarity between the sample test cards in the sample set is determined; the sample test cards whose label categories represent that the material is true are used as cluster centers, and the sample test cards are clustered according to the similarity;

[0076] The auxiliary loss is determined according to the distance between any sample detection card and the cluster center corresponding to its clustering result, and the label category corresponding to the sample detection card; the main loss and the auxiliary loss together constitute the prediction loss.

[0077] In this example, the prediction loss includes not only the main loss based on binary classification, but also the auxiliary loss based on clustering results, which is helpful for the model to identify new types of material attacks that will appear in the future. As long as the distance from the true center is far, it will be intercepted by the model with a high probability, thereby improving the accuracy of detecting material counterfeiting.

[0078] Figure 6 FIG. 1 is a schematic diagram showing a method for determining auxiliary loss according to an embodiment. Figure 6 , the figure on the left corresponds to the clustering result, the star represents a fake certificate, the circle represents a real certificate, the auxiliary loss helps the model achieve the following training goals, the distance between the fake certificate and its cluster center is greater than the preset edge distance, and the distance between the real certificate and its cluster center is less than the preset edge distance, so that the card to be tested can be judged whether it is a fake certificate based on the distance between the card to be tested and the cluster center. The two curves on the right represent the auxiliary loss of the real certificate and the auxiliary loss of the fake certificate respectively. When determining the auxiliary loss LOSS, for the real certificate, if the distance d between the real certificate and its cluster center is larger, the auxiliary loss is larger; for the fake certificate, if the distance d between the fake certificate and its cluster center is larger, the auxiliary loss is smaller, especially when the distance d is larger than the edge distance, the auxiliary loss is close to 0.

[0079] In the embodiments of this specification, when faced with hundreds of card types, the model needs to more accurately extract material-related anti-counterfeiting features, while reducing the interference of different card formats and text information on material features. From the perspective of material anti-counterfeiting tasks, for different card types, their feature space is modeled into two subspaces with decoupled features: content subspace and style subspace. The modeling of the content subspace mainly focuses on the content information and layout design of the card, such as content features such as text, avatars, and icons; while the modeling of the style subspace mainly focuses on the material style features that are more important for the identification of card color printing attacks, such as style features such as presentation material and rendering style.

[0080] Figure 7 FIG. 2 is a schematic diagram showing a multimodal model training method according to another embodiment. Figure 7 , based on the multimodal model, the feature decoupling of content and style is achieved by designing an additional network structure. The fusion features extracted by the multimodal model will be input into the first extraction network and the second extraction network respectively. The first extraction network outputs the material style sub-features, which are then input into the style branch network. Its learning goal is the binary classification task of the authenticity of the material, which is also the main task concerned by the embodiments of this specification; correspondingly, the second extraction network outputs the content sub-features, which are input into the content branch network. Its learning goal is the multi-classification task of the card type, which is an auxiliary task introduced to achieve feature decoupling.

[0081] In one example, the multimodal model is trained in the following manner:

[0082] Inputting sample image data of the sample detection card into the multimodal model, and obtaining fusion features of the sample detection card through the multimodal model;

[0083] Extracting material style sub-features and content sub-features from the fused features through a first extraction network and a second extraction network respectively;

[0084] Inputting the material style sub-feature into a style branch network, and outputting a first prediction result of the sample detection card regarding the authenticity of the material through the style branch network;

[0085] Inputting the content sub-feature into a content branch network, and outputting a second prediction result about the card type of the sample detection card through the content branch network;

[0086] Based on the first prediction result and the second prediction result, model parameters of the multimodal model are updated by adversarial training.

[0087] In this example, there is both a main task about the authenticity of the material and an auxiliary task about the card type. The two are combined for adversarial training to achieve feature decoupling of content and style, thereby improving the accuracy of detecting material forgery.

[0088] Furthermore, the step of inputting the material style sub-feature into the style branch network includes:

[0089] The material style sub-feature is subjected to random interpolation processing using the content sub-feature to obtain an interpolation feature;

[0090] The interpolated features are input into the style branch network.

[0091] In this example, random interpolation processing is used to stimulate the stability of the material anti-counterfeiting task by enhancing the material style sub-features with randomly different card content layout features.

[0092] Furthermore, the updating of the model parameters of the multimodal model by adversarial training includes:

[0093] Based on the first prediction result, adjusting model parameters of the multimodal model and the style branch network with the goal of minimizing the prediction loss regarding the authenticity of the material;

[0094] Based on the second prediction result, adjusting the model parameters of the content branch network with the goal of minimizing the prediction loss regarding the card type;

[0095] Based on the second prediction result, the model parameters of the multimodal model are adjusted with the goal of minimizing the cross entropy between the second prediction result and the uniform distribution.

[0096] In this example, the multimodal model can be suppressed from extracting fewer content features from the source, which is beneficial to improving the accuracy of material anti-counterfeiting tasks.

[0097] For example, in order to highlight the diversity and importance of style features, a three-stage adversarial training mechanism is used to suppress the feature strength related to content layout in the content branch network. The adversarial training process for the multimodal model, style branch network and content branch network is summarized into the following three stages: Stage 1: Prioritize updating the parameters G of the style branch network s and the parameters G of the multimodal model f ; Phase 2, then only update the parameters G of the content branch network c , the goal is to classify the card types; in stage 3, adversarial training is used to update only the parameters G of the multimodal model f , the goal is to minimize the cross entropy between the output of the content branch network and the uniform distribution, thereby suppressing the multimodal model from extracting fewer content features from the source. Among them, the goal of stage 3 can be expressed as follows:

[0098]

[0099] According to the method provided in the embodiment of this specification, the image data of the card to be detected is first obtained; then the image data is input into the multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting the differential features of the image data through the first branch network, extracting the frequency domain features of the image data through the second branch network, extracting the enhanced features of the image data through the third branch network, fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fusion features; finally, using a classifier, the prediction result of the authenticity of the material of the card to be detected is determined according to the fusion features. As can be seen from the above, the embodiment of this specification adopts a multimodal fusion detection scheme, wherein the differential features are conducive to characterizing the style rendering details of the material, the frequency domain features are conducive to characterizing the printing clues of high-frequency noise points, and the enhanced features are conducive to enriching the features of different fields, making the model more generalized, and by fusing the differential features, frequency domain features and enhanced features, the prediction result of the authenticity of the material of the card to be detected is determined, which can improve the accuracy of material authenticity detection.

[0100] According to another aspect of the embodiment, there is also provided a material authenticity detection device in document anti-counterfeiting detection, which is used to execute the method provided in the embodiment of this specification. Figure 8FIG. 1 is a schematic block diagram of a material authenticity detection device in document anti-counterfeiting detection according to an embodiment. Figure 8 As shown, the device 800 includes:

[0101] An acquisition unit 81 is used to acquire image data of a card to be detected;

[0102] The processing unit 82 is used to input the image data acquired by the acquisition unit 81 into a multimodal model for model processing, wherein the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, and the model processing includes extracting differential features of the image data through the first branch network, extracting frequency domain features of the image data through the second branch network, extracting enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fused features;

[0103] The prediction unit 83 is used to use a classifier to determine the prediction result of the material authenticity of the card to be detected according to the fusion features obtained by the processing unit 82.

[0104] Optionally, as an embodiment, the first branch network is a convolutional neural network; the processing unit 82 is specifically used to extract differential features of the image data through the convolutional neural network by adopting a central difference convolution method.

[0105] Optionally, as an embodiment, the processing unit 82 is specifically configured to obtain a frequency spectrum of the image data by means of discrete cosine transform through the second branch network, and use the frequency spectrum as a frequency domain feature of the image data.

[0106] Optionally, as an embodiment, the processing unit 82 is specifically configured to perform feature mixing on the image data with other image data of different card types through the third branch network to obtain cross-domain enhanced features of the image data.

[0107] Optionally, as an embodiment, the multimodal model determines the prediction loss in the following manner when performing model training:

[0108] Inputting sample image data of any sample detection card in the sample set into the multimodal model and classifier to obtain a first prediction category of the sample detection card regarding the authenticity of the material;

[0109] Determine the main loss based on the difference between the first predicted category and the label category corresponding to the sample detection card;

[0110] Based on the fusion features of the sample test cards output by the multimodal model, the similarity between the sample test cards in the sample set is determined; the sample test cards whose label categories represent that the material is true are used as cluster centers, and the sample test cards are clustered according to the similarity;

[0111] The auxiliary loss is determined according to the distance between any sample detection card and the cluster center corresponding to its clustering result, and the label category corresponding to the sample detection card; the main loss and the auxiliary loss together constitute the prediction loss.

[0112] Optionally, as an embodiment, the multimodal model is trained in the following manner:

[0113] Inputting sample image data of the sample detection card into the multimodal model, and obtaining fusion features of the sample detection card through the multimodal model;

[0114] Extracting material style sub-features and content sub-features from the fused features through a first extraction network and a second extraction network respectively;

[0115] Inputting the material style sub-feature into a style branch network, and outputting a first prediction result of the sample detection card regarding the authenticity of the material through the style branch network;

[0116] Inputting the content sub-feature into a content branch network, and outputting a second prediction result about the card type of the sample detection card through the content branch network;

[0117] Based on the first prediction result and the second prediction result, model parameters of the multimodal model are updated by adversarial training.

[0118] Furthermore, the step of inputting the material style sub-feature into the style branch network includes:

[0119] The material style sub-feature is subjected to random interpolation processing using the content sub-feature to obtain an interpolation feature;

[0120] The interpolated features are input into the style branch network.

[0121] Furthermore, the updating of the model parameters of the multimodal model by adversarial training includes:

[0122] Based on the first prediction result, adjusting model parameters of the multimodal model and the style branch network with the goal of minimizing the prediction loss regarding the authenticity of the material;

[0123] Based on the second prediction result, adjusting the model parameters of the content branch network with the goal of minimizing the prediction loss regarding the card type;

[0124] Based on the second prediction result, the model parameters of the multimodal model are adjusted with the goal of minimizing the cross entropy between the second prediction result and the uniform distribution.

[0125] Through the device provided in the embodiment of this specification, first, the acquisition unit 81 acquires the image data of the card to be detected; then the processing unit 82 inputs the image data into the multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting the differential features of the image data through the first branch network, extracting the frequency domain features of the image data through the second branch network, extracting the enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fusion features; finally, the prediction unit 83 uses a classifier to determine the prediction result of the material authenticity of the card to be detected according to the fusion features. As can be seen from the above, the embodiment of this specification adopts a multimodal fusion detection solution, in which differential features are conducive to characterizing the style rendering details of the material, frequency domain features are conducive to characterizing the printing clues of high-frequency noise, and enhanced features are conducive to enriching the features of different fields, making the model more generalizable. By fusing differential features, frequency domain features and enhanced features, the prediction results of the authenticity of the material of the card to be tested are determined, which can improve the accuracy of material authenticity detection.

[0126] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.

[0127] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 2 The method described.

[0128] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0129] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A material authenticity detection method in document anti-counterfeiting detection, the method comprising: Obtain image data of the card to be tested; Inputting the image data into a multimodal model for model processing, the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, the model processing includes extracting differential features of the image data through the first branch network, extracting frequency domain features of the image data through the second branch network, extracting enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fused features; A classifier is used to determine a prediction result of the authenticity of the material of the card to be tested according to the fusion features.

2. The method of claim 1, wherein: The first branch network is a convolutional neural network; and extracting the differential features of the image data through the first branch network includes: The convolutional neural network is used to extract differential features of the image data by using a central difference convolution method.

3. The method of claim 1, wherein: The extracting the frequency domain features of the image data by the second branch network includes: The second branch network is used to obtain a frequency spectrum of the image data by discrete cosine transform, and the frequency spectrum is used as a frequency domain feature of the image data.

4. The method of claim 1, wherein: The step of extracting enhanced features of the image data through the third branch network includes: The image data is mixed with other image data of different card types through the third branch network to obtain cross-domain enhanced features of the image data.

5. The method of claim 1, wherein: The multimodal model determines the prediction loss in the following manner during model training: Inputting sample image data of any sample detection card in the sample set into the multimodal model and classifier to obtain a first prediction category of the sample detection card regarding the authenticity of the material; Determine the main loss based on the difference between the first predicted category and the label category corresponding to the sample detection card; Based on the fusion features of the sample test cards output by the multimodal model, the similarity between the sample test cards in the sample set is determined; the sample test cards whose label categories represent that the material is true are used as cluster centers, and the sample test cards are clustered according to the similarity; The auxiliary loss is determined according to the distance between any sample detection card and the cluster center corresponding to its clustering result, and the label category corresponding to the sample detection card; the main loss and the auxiliary loss together constitute the prediction loss.

6. The method of claim 1, wherein: The multimodal model is trained in the following manner: Inputting sample image data of the sample detection card into the multimodal model, and obtaining fusion features of the sample detection card through the multimodal model; Extracting material style sub-features and content sub-features from the fused features through a first extraction network and a second extraction network respectively; Inputting the material style sub-feature into a style branch network, and outputting a first prediction result of the sample detection card regarding the authenticity of the material through the style branch network; Inputting the content sub-feature into a content branch network, and outputting a second prediction result about the card type of the sample detection card through the content branch network; Based on the first prediction result and the second prediction result, model parameters of the multimodal model are updated by adversarial training.

7. The method of claim 6, wherein: The step of inputting the material style sub-feature into the style branch network comprises: The material style sub-feature is subjected to random interpolation processing using the content sub-feature to obtain an interpolation feature; The interpolated features are input into the style branch network.

8. The method of claim 6, wherein: The method of updating the model parameters of the multimodal model by adversarial training includes: Based on the first prediction result, adjusting model parameters of the multimodal model and the style branch network with the goal of minimizing the prediction loss regarding the authenticity of the material; Based on the second prediction result, adjusting the model parameters of the content branch network with the goal of minimizing the prediction loss regarding the card type; Based on the second prediction result, the model parameters of the multimodal model are adjusted with the goal of minimizing the cross entropy between the second prediction result and the uniform distribution.

9. A material authenticity detection device for document anti-counterfeiting detection, the device comprising: An acquisition unit, used to acquire image data of a card to be detected; a processing unit, configured to input the image data acquired by the acquisition unit into a multimodal model for model processing, wherein the multimodal model includes a first branch network, a second branch network, a third branch network and a fusion network, wherein the model processing includes extracting differential features of the image data through the first branch network, extracting frequency domain features of the image data through the second branch network, extracting enhanced features of the image data through the third branch network, and fusing the differential features, frequency domain features and enhanced features through the fusion network to obtain fused features; The prediction unit is used to use a classifier to determine the prediction result of the material authenticity of the card to be detected according to the fusion features obtained by the processing unit.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

11. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 8 is implemented.

Images