Certificate anti-counterfeiting detection and model training method and device, storage medium and product

Abstract

One or more embodiments of the invention provide a certificate anti-counterfeiting detection and model training method and device, a storage medium and a product. The certificate anti-counterfeiting detection method comprises the following steps: acquiring a to-be-detected target certificate image; preprocessing the target certificate image to generate image data of at least two modes; wherein the image data of the at least two modes comprises a texture feature map generated based on an image difference between the target certificate image and a reference certificate image corresponding to the certificate type of the target certificate image; the reference certificate image is an image obtained by averaging pixel values of the same pixel positions in a plurality of certificate images corresponding to the certificate type of the target certificate image; and carrying out certificate anti-counterfeiting detection on the target certificate image based on the image data of the at least two modes.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and in particular, to a document anti-counterfeiting detection method, a document anti-counterfeiting detection model training method, an image detection method, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In various scenarios, document anti-counterfeiting detection has become increasingly critical. For example, financial applications often need to verify electronic ID cards for identity authentication or account opening processes; another example is that companies need to verify the validity of employees' work cards. During the electronic ID card account opening authentication process, key information such as the face image, ID number, and name on the ID card is often tampered with, which may lead to deceiving the application system and successfully opening a false account. Therefore, ensuring the authenticity of the document information has become a technical problem to be solved by technicians in this field. Summary of the invention

[0003] In view of this, one or more embodiments of the present specification provide a document anti-counterfeiting detection method, a document anti-counterfeiting detection model training method, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a document anti-counterfeiting detection method is proposed, comprising:

[0006] Obtain the target document image to be detected;

[0007] Preprocessing the target document image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on an image difference between the target document image and a reference document image corresponding to the document type of the target document image; the reference document image is an image obtained by averaging pixel values ​​at the same pixel position in a plurality of document images corresponding to the document type of the target document image;

[0008] Based on the image data of the at least two modalities, document anti-counterfeiting detection is performed on the target document image.

[0009] According to a second aspect of the embodiments of this specification, a method for training a document anti-counterfeiting detection model is provided, comprising:

[0010] Acquire a training sample set, the training sample set comprising a number of certificate image samples marked with anti-counterfeiting labels, the anti-counterfeiting labels being used to indicate the authenticity of the certificate image samples;

[0011] Preprocessing each of the document image samples to obtain image data of at least two modes corresponding to each of the document image samples; wherein the image data of at least two modes includes a texture feature map generated based on an image difference between the document image sample and a reference document image corresponding to the document type of the document image sample; the reference document image is an image obtained by averaging pixel values ​​at the same pixel position in a plurality of document images corresponding to the document type of the document image sample;

[0012] Based on the image data of the at least two modalities corresponding to each of the document image samples and the anti-counterfeiting labels of the document image samples, supervised training is performed on the document anti-counterfeiting detection model to be trained.

[0013] According to a third aspect of an embodiment of this specification, there is provided an image detection method, including:

[0014] Acquire the target image to be detected;

[0015] Preprocessing the target image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on an image difference between the target image and a reference image corresponding to the type of the target image; the reference image is an image obtained by averaging pixel values ​​at the same pixel position in a plurality of images corresponding to the type of the target image;

[0016] Image detection is performed on the target image based on the image data of the at least two modalities.

[0017] According to a fourth aspect of the embodiments of this specification, there is provided an electronic device, including:

[0018] processor;

[0019] a memory for storing processor-executable instructions;

[0020] When the processor executes the executable instructions, it is used to implement the method described in the first aspect, the second aspect or the third aspect.

[0021] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect, the second aspect or the third aspect are implemented.

[0022] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect, the second aspect or the third aspect when executed by a processor.

[0023] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0024] In the embodiments of this specification, at least two different visual representations or features are generated from the target document image to increase the robustness and accuracy of subsequent document anti-counterfeiting detection. The at least two modal image data include texture modal image data. Further, this embodiment provides a method for obtaining texture modal image data (i.e., texture feature map), which generates a texture feature map based on the image difference between the target document image and the reference document image, and can effectively separate the document content itself and the document texture. The texture feature map can help detect forged textures or added watermarks. Forgery usually causes the texture of the document image to be different from the normal reference image. This difference can be clearly displayed in the texture feature map, which helps to improve the accuracy of document anti-counterfeiting detection.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the architecture of a certificate anti-counterfeiting detection service system provided by an exemplary embodiment.

[0027] Figure 2 It is a flowchart of a method for training a document anti-counterfeiting detection model provided by an exemplary embodiment.

[0028] Figure 3 It is a schematic diagram of obtaining a texture feature map provided by an exemplary embodiment.

[0029] Figure 4 It is a schematic diagram of obtaining a spectrum diagram provided by an exemplary embodiment.

[0030] Figure 5 It is a schematic diagram of obtaining a second document image provided by an exemplary embodiment.

[0031] Figure 6 It is a schematic diagram of obtaining a noise characteristic diagram provided by an exemplary embodiment.

[0032] Figure 7 It is a structural schematic diagram of a certificate anti-counterfeiting detection model provided by an exemplary embodiment.

[0033] Figure 8 It is a flow chart of a document anti-counterfeiting detection method provided by an exemplary embodiment.

[0034] Fig. 9 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0036] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0037] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] Figure 1 FIG. 1 is a schematic diagram of the architecture of a document anti-counterfeiting detection service system provided by an exemplary embodiment. Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, including a PC (Personal Computer) 13, a mobile phone 14, and the like.

[0039] The server 11 may be a physical server including an independent host, or the server 11 may be a virtual server carried by a host cluster. During operation, the server 11 may run a server-side program of a certain application to implement the relevant functions of the application. For example, when the server 11 runs a program of a certificate anti-counterfeiting detection service, it may be implemented as a corresponding certificate anti-counterfeiting detection server.

[0040] PC13 and mobile phone 14 are only some types of electronic devices that users can use. In fact, users can obviously also use electronic devices such as the following types: tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of this specification do not limit this. During operation, the electronic device can run a program on the client side of a certain application to implement the relevant functions of the application. For example, when the electronic device runs a program for a certificate anti-counterfeiting detection service, it can be implemented as a client of the certificate anti-counterfeiting detection service. Among them, the application program of the client of the above-mentioned certificate anti-counterfeiting detection service can be started and run on the electronic device. The program on the client side can be a native application installed on the electronic device, or the program on the client side can be a small program, a quick application or other similar forms. Of course, when using web page technologies such as HTML5 or similar, the relevant functions can be implemented through the page displayed by the browser. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0041] As for the network 12 for interaction between electronic devices such as PC 13 and mobile phone 14 and server 11, it is possible to select a wired or wireless network to achieve communication based on the communication mode supported by the corresponding electronic device, and this specification does not limit this. For example, PC 13 can support both wired and wireless communication, so it can use a wired or wireless network to achieve communication as needed, while mobile phone 14 usually only supports wireless communication, so it can use a wireless network to achieve communication.

[0042] Exemplarily, the document anti-counterfeiting detection method provided in the embodiments of this specification can be executed by any one of the above-mentioned client or the above-mentioned server. For example, the server can receive the target document image to be detected uploaded by the client, and then pre-process the target document image to be detected to generate image data of at least two modalities, and the image data of at least two modalities at least include a texture feature map, so as to perform document anti-counterfeiting detection on the target document image based on the image data of at least two modalities, thereby realizing the capture of multiple aspects of the target document image based on the features reflected by the image data of multiple modalities (especially the texture features reflected by the texture feature map), so as to more reliably detect whether the document has been tampered with or forged, and improve the accuracy of anti-counterfeiting detection.

[0043] Alternatively, the document anti-counterfeiting detection method provided in the embodiments of this specification may also be executed in part by the client and in another part by the server; for example, the client obtains the target document image to be detected, pre-processes the target document image to generate at least two modal image data, and then sends the at least two modal image data to the server, so that the server performs document anti-counterfeiting detection on the target document image based on the at least two modal image data. This embodiment does not impose any restrictions on this.

[0044] For example, considering the development and application of neural network technology, it can be applied to the document anti-counterfeiting detection process to improve the accuracy and efficiency of detection. For example, a document anti-counterfeiting detection model can be trained for the document anti-counterfeiting detection process.

[0045] The following is an example of the training process of the document anti-counterfeiting detection model:

[0046] See also Figure 2 , Figure 2 A flowchart of a method for training a certificate anti-counterfeiting detection model is shown, and the training method can be executed by a server or other electronic devices. It is understandable that the device used to execute the training method and the device used to execute the certificate anti-counterfeiting detection method can be different devices or the same device, and this embodiment does not impose any restrictions on this. For example, the training method for the certificate anti-counterfeiting detection model can be executed by a first device to obtain a trained certificate anti-counterfeiting detection model, and then the trained certificate anti-counterfeiting detection model is deployed to a second device used to execute the certificate anti-counterfeiting detection method. Figure 2 As shown, the training method of the document anti-counterfeiting detection model includes:

[0047] In S201, a training sample set is obtained, where the training sample set includes a number of document image samples marked with anti-counterfeiting labels, where the anti-counterfeiting labels are used to indicate the authenticity of the document image samples.

[0048] For example, in order to implement anti-counterfeiting detection for different certificates, the training sample set may include certificate image samples corresponding to various certificate types, including but not limited to identity cards, passports, driver's licenses, or work permits.

[0049] This embodiment does not impose any restrictions on the source of the ID image samples, and can be obtained according to the actual application scenario, such as the images of genuine IDs authorized by the institution, and the samples of forged IDs generated by various forgery techniques known in the market, to ensure that the model can learn the characteristics of real IDs and identify various forgery techniques. Each ID image sample needs to be labeled as "real" or "fake", and this labeling process can be performed through manual review to ensure the accuracy and consistency of the labels.

[0050] The preparation of document image samples and anti-counterfeiting labels is the basis for training an effective document anti-counterfeiting detection model. By learning the features and patterns in these document image samples, the neural network can accurately identify and classify new document images in practical applications, thereby helping to improve the ability and efficiency of document anti-counterfeiting.

[0051] In S202, each ID image sample is preprocessed to obtain image data of at least two modalities corresponding to each ID image sample; wherein the image data of at least two modalities includes a texture feature map generated based on the image difference between the ID image sample and a reference ID image corresponding to the ID type of the ID image sample; the reference ID image is an image obtained by taking the average value of pixel values ​​at the same pixel position in multiple ID images corresponding to the ID type of the ID image sample.

[0052] This embodiment is based on preprocessing the document image samples to obtain image data of multiple modalities for use in the model training process, so that the model can learn multiple aspects of the document image information, thereby improving the accuracy of anti-counterfeiting detection.

[0053] The image data of at least two modalities include image data of texture modality. Furthermore, the present embodiment provides a method for obtaining image data of texture modality (i.e., texture feature map), and generates a texture feature map based on the image difference between the document image sample and the reference document image. The document content itself and the document texture can be effectively separated. The texture feature map can help detect forged textures or added watermarks. Forgery usually causes the texture of the document image to be different from that of a normal reference image. This difference can be clearly displayed in the texture feature map, which helps to improve the accuracy of document anti-counterfeiting detection.

[0054] Here, the process of acquiring image data of texture modality (ie, texture feature map) is exemplarily described:

[0055] First, for the multiple document images collected in advance belonging to each document type, each document image can be aligned with a preset first template image, the first template image is of a first size and contains only the document content. For example, a preset feature point detection algorithm can be used to detect feature points from the document image and the first template image respectively, and then a matching feature point pair is found in the document image and the first template image based on a preset feature point matching algorithm, and a homography matrix is ​​calculated based on the matching feature point pair. The homography matrix describes the transformation relationship from one image to another image, and then the document image is transformed using the homography matrix, including transformation operations such as rotation, scaling, and translation, so as to realize alignment of each document image with the preset first template image, and further the size of the aligned document image can be adjusted to match the size of the first template image. The multiple document images obtained through the above processing process are all of the first size and contain only the document content, which can avoid interference caused by non-document content (such as background content). Among them, corresponding first template images can be set for different document types, and this implementation does not impose any restrictions on this.

[0056] For the reference document image corresponding to each document type, the pixel values ​​at the same pixel position in multiple document images corresponding to the document type may be averaged. Exemplarily, the multiple document images corresponding to the document type are document images of a first size obtained through the above processing process, and the reference document image corresponding to each document type obtained is also of the first size and contains only information about the document content.

[0057] For each of the above-mentioned certificate image samples, each certificate image sample can be aligned with the reference certificate image. For example, a preset feature point detection algorithm can be used to detect feature points from the certificate image sample and from the reference certificate image respectively, and then matching feature point pairs are found in the certificate image sample and the reference certificate image based on the preset feature point matching algorithm, and a homography matrix is ​​calculated based on the matching feature point pairs. The certificate image is then transformed using the homography matrix, including rotation, scaling, translation and other transformation operations, so as to achieve alignment of each certificate image sample with the reference certificate image. The size of the aligned certificate image sample can be further adjusted to a first size to obtain a first certificate image. Exemplarily, the size of the first certificate image is the first size and only includes certificate content.

[0058] See also Figure 3After obtaining the first document image corresponding to each document image sample, a texture feature map can be generated based on the image difference between the first document image and the reference document image. Specifically, the pixel value at each pixel position in the texture feature map is the difference between the pixel value at the same pixel position in the first document image and the pixel value at the same pixel position in the reference document image. For example, the pixel value of the first pixel in the first row of the texture feature map is the difference between the pixel value of the first pixel in the first row of the first document image and the pixel value of the first pixel in the first row of the reference document image. The pixel values ​​of other pixel positions in the texture feature map can be obtained based on a similar method, which will not be repeated here.

[0059] After being processed in the above manner, the size of the texture feature map corresponding to each certificate image sample is also the first size and only includes the texture feature information of the certificate content, which can eliminate the interference caused by the texture information of non-certificate content (such as background content). Authenticity identification usually relies on the unique security features on the certificate, such as printing details, watermarks or micro-text, focusing on the texture information of these certificate contents, so that the certificate anti-counterfeiting detection model can more accurately analyze and compare these features, thereby improving the anti-counterfeiting detection capability.

[0060] In some embodiments, the image data of the at least two modalities further includes at least one of the following: image data of a color modality, image data of a frequency domain modality, and image data of a noise modality.

[0061] Here, the acquisition process of image data (i.e., spectrum diagram) in frequency domain mode is described as follows:

[0062] In a possible implementation, each document image sample may be transformed from the time domain to the frequency domain to obtain a frequency spectrum, and the size of the frequency spectrum is the same as the size of the document image sample.

[0063] In another possible implementation, in order to avoid interference caused by frequency information of non-document content, the first document image corresponding to each document image sample can be transformed from the time domain to the frequency domain, thereby obtaining a spectrum diagram. Figure 4, the DCT (Discrete Cosine Transform) algorithm can be used to transform the first document image from the time domain to the frequency domain to obtain a spectrum diagram. The size of the spectrum diagram is the same as the size of the first document image, that is, the spectrum diagram is also of the first size and only contains the frequency information of the document content. Discrete cosine transform is a mathematical transformation method widely used in the field of image processing and compression. It can convert a two-dimensional image matrix into a frequency domain representation. One of its main advantages is that most of the image information is concentrated in the transformed low-frequency coefficients, while the high-frequency coefficients usually correspond to subtle textures or noise in the image. In this embodiment, since the spectrum diagram mainly captures the frequency information of the document content in the first document image, this method is more resistant to interference, can eliminate the interference caused by frequency information other than the document content, and improve the accuracy and stability of anti-counterfeiting detection.

[0064] Of course, other transformation algorithms may also be used, such as Fourier Transform, Wavelet Transform, or Haar Transform, etc., and this embodiment does not impose any limitation on this.

[0065] Here is an exemplary description of the process of acquiring image data in color mode:

[0066] In a possible implementation, the size of each document image sample can be adjusted to a predefined second size to obtain a second document image that can be used as image data of a color mode. For example, a second template image can be pre-set, the second template image is of a second size and includes document content and background content other than the document content. A preset feature point detection algorithm can be used to detect feature points from the document image sample and from the preset second template image respectively, and then a matching feature point pair is found in the document image sample and the second template image based on a preset feature point matching algorithm, and a homography matrix is ​​calculated based on the matching feature point pair, the homography matrix describes the transformation relationship from one image to another, and then the document image is transformed using the homography matrix, including transformation operations such as rotation, scaling, and translation, so as to achieve alignment of each document image sample with the preset second template image, and further the size of the aligned document image sample can be adjusted to match the size of the second template image, and if the size is insufficient, it is padded with pixels of a preset color to obtain a second document image with a second size. In this embodiment, by uniformly processing the sizes of the various certificate image samples, the second certificate images corresponding to the various certificate image samples obtained have the same size, which helps the certificate anti-counterfeiting detection model to learn and analyze the color mode. Figure 5As shown, after alignment and resizing in the above manner, the second document image corresponding to each document image sample includes the document content and background content in addition to the document content. In this embodiment, including the background content can provide complete context information of the document image, which helps to further improve the accuracy and reliability of document verification. The background content can help distinguish a real document image from a possible forged or tampered image. Some forgery techniques may ignore or change the document background, so including the background can provide an additional layer of security and reduce the possibility of fraud and forgery.

[0067] In another possible implementation, considering that the difference between a real certificate and a forged certificate mainly lies in the difference in the certificate content, the location of the certificate content is the main forgery location in the certificate, and the certificate content includes text content and / or image content (such as a face image), etc. Therefore, the certificate content in the second certificate image can be masked to enhance generalization, and the masked image is further converted into a grayscale image, thereby reducing the influence of light or shadow and improving detection accuracy.

[0068] Here is an exemplary description of the process of acquiring image data of the noise mode:

[0069] This embodiment uses an SRM filter to extract image data in a noise mode. The SRM (Steganalysis Rich Model) filter is a method for steganalysis. Steganalysis is a technique that aims to detect and analyze hidden information in digital media, which may be embedded through steganography, and the purpose may be to conceal communication or information transmission. For example, the SRM filter may use a variety of features, including but not limited to color histograms, texture features, statistical features, etc., to capture traces of hidden information in an image.

[0070] In a possible implementation, the SRM filter can be used to extract local noise from each document image sample to obtain a noise feature map, the size of which is the same as the size of the document image sample. This embodiment can analyze the noise distribution in different areas of the image by extracting the local noise feature map. These noise features can include various statistical features such as mean, variance, standard deviation, etc., which provide a strong basis for identifying and distinguishing true and false documents.

[0071] In another possible implementation, see Figure 6, the SRM filter can be used to extract local noise from the second document image to obtain a noise feature map, the size of the noise feature map is the same as the size of the second document image, that is, the size of the noise feature map is the second size and the noise feature map includes the local noise of the document content and the local noise of the background content. In this embodiment, since the noise feature map covers the local noise of the document content and the background content, the noise characteristics of the entire image can be comprehensively analyzed. This comprehensive analysis can help determine the noise distribution pattern of the entire image, rather than just being limited to a certain part. The local noise of a real document is usually related to the specific production process and materials of the document itself, while a forged image may have inconsistencies in the local noise of the document content and the background content. Extracting the overall noise feature map can help detect forged documents more comprehensively, because it is often difficult for a forger to maintain consistent noise characteristics across the entire image.

[0072] In S203, supervised training is performed on the document anti-counterfeiting detection model to be trained based on the image data of at least two modalities corresponding to each document image sample and the anti-counterfeiting label of the document image sample.

[0073] Exemplarily, image data of at least two modalities corresponding to each certificate image sample can be input into the certificate anti-counterfeiting detection model to be trained for processing to obtain the anti-counterfeiting detection result, and the certificate anti-counterfeiting detection model to be trained is trained with minimizing the difference between the anti-counterfeiting detection result and the anti-counterfeiting label as the optimization goal.

[0074] In this embodiment, the use of image data of multiple modalities can help the model effectively learn features related to anti-counterfeiting. These features may involve the microstructure, color distribution, texture features or noise distribution on the document. These features may show different physical properties in images of different modalities. Through training, the model can better understand and utilize this information. Supervised training allows the model to directly learn the relationship between the document image and its anti-counterfeiting label. With accurate label information, the model can more accurately distinguish between authentic and fake documents, reduce the possibility of false alarms and missed detections, and thus improve the overall detection accuracy.

[0075] In the supervised training process, the following iterative process may be executed cyclically: based on the document anti-counterfeiting detection model to be trained, image data of at least two modalities corresponding to each document image sample are processed to obtain a document anti-counterfeiting detection result.

[0076] In the case where the iteration end condition is not met, the model parameters of the certificate anti-counterfeiting detection model to be trained are adjusted based on the difference between the certificate anti-counterfeiting detection result and the anti-counterfeiting label of the certificate image sample; wherein the adjusted certificate anti-counterfeiting detection model to be trained is used as the certificate anti-counterfeiting detection model to be trained in the next round of iteration. Exemplarily, the difference between the model prediction result and the real anti-counterfeiting label can be measured by a predefined loss function. The loss function can be a cross-entropy loss function (Cross-Entropy Loss) or a mean squared error loss function (Mean Squared Error Loss), but is not limited to this. The value of the loss function is calculated based on the certificate anti-counterfeiting detection result and the anti-counterfeiting label of the certificate image sample. For example, the smaller the value of the loss function, the closer the prediction result of the model is to the real label. Once the value of the loss function is obtained, it is necessary to calculate the gradient of the loss function with respect to the model parameters. The gradient is a vector of the rate of change of the loss function, indicating the speed of the change of the loss function in each direction under the current model parameters. The gradient descent algorithm can be used to adjust the model parameters.

[0077] When the iteration end condition is met, the document anti-counterfeiting detection model to be trained used in the current round of iteration process is used as the final document anti-counterfeiting detection model that has been trained.

[0078] It is understandable that the embodiments of this specification do not impose any restrictions on this iteration end condition, and can be specifically set according to the actual application scenario. For example, the iteration end condition is: reaching a preset number of iterations, or the loss function converges to a sufficiently small value.

[0079] In some embodiments, based on the above description, it can be known that the embodiments of this specification use an SRM filter to obtain image data of the noise mode of each document image sample, and the SRM filter includes a filter kernel, and the filter kernel in the SRM filter is a preset size (such as: 5×5×3) and the initial value of the parameter in the filter kernel is a random value. In each round of iteration, step S202 is used to obtain image data of at least two modes corresponding to each document image sample, for example, the image data of at least two modes includes image data of the noise mode.

[0080] After executing each round of iteration except the last round of iteration, in addition to adjusting the model parameters of the document anti-counterfeiting detection model, the parameters in the filter kernel in the SRM filter can also be adjusted based on the difference between the document anti-counterfeiting detection result and the anti-counterfeiting label of the document image sample. The purpose of the adjustment is to optimize the extraction of noise features to improve the accuracy and stability of anti-counterfeiting detection. Among them, the adjusted SRM filter is used to obtain the noise feature map for the next round of iteration. In this way, the ability to analyze the noise features of the document image can be gradually improved, thereby effectively improving the overall performance of the document anti-counterfeiting detection model.

[0081] In some embodiments, the training sample set used to train the document anti-counterfeiting detection model includes several document image samples that can be divided into positive document image samples (real documents) and negative document image samples (fake documents) according to anti-counterfeiting labels. The negative document image samples in the training document image sample set can be generated based on at least two generation methods.

[0082] In one possible generation method, real documents, including driver's licenses, ID cards, passports, etc., are first used to make the basic layer of the PSD template. Using image editing software (such as Photoshop), the collected document images are analyzed and layered. Each layer may contain different information parts such as personal information, photos, signatures, etc., to obtain a basic PSD template. Then, different false documents can be customized based on specific personal information (such as name, address, birthday, etc.) and corresponding portraits. This personalized information will be embedded in the corresponding layer of the PSD template to generate a seemingly legitimate document, that is, a negative document image sample.

[0083] In another possible generation method, first use real documents, including driver's licenses, ID cards, passports, etc., use OCR (Optical Character Recognition) to identify the fields and positions in the documents, erase each field, and then write other people's information, overwrite it, and harmonize the written fields to generate a document that appears to be legitimate on the surface, that is, a negative document image sample.

[0084] Among them, the negative document image samples used in the first round of iteration process are generated based on a preset negative document image sample generation strategy, and the negative document image sample generation strategy is at least used to indicate at least one of the total number of negative document image samples generated and the number of negative document image samples generated by each generation method.

[0085] After executing each round of iteration except the last round of iteration, the prediction accuracy of the negative document image samples in this round of iteration can be further counted; wherein, the prediction accuracy includes: at least one of the prediction accuracy for all negative document image samples and the prediction accuracy for the negative document image samples generated by each generation method. Based on the statistical prediction accuracy, the negative document image sample generation strategy corresponding to the negative document image samples used in this round of iteration can be adjusted; the purpose of the adjustment is to improve the recognition ability and generalization ability of the model for negative documents, thereby improving the overall training effect; and then according to the adjusted negative document image sample generation strategy, the negative document image samples for the next round of iteration are generated, and the generated negative document image samples for the next round of iteration and the positive document image samples included in the training sample set can be composed of the training sample set for the next round of iteration, and steps S202 and S203 are repeatedly executed, thereby gradually improving the performance of the model on the negative document image samples. With the progress of each round of iteration, the model can more effectively adapt to the diversity and complexity of various negative document image samples. This embodiment repeatedly adjusts and optimizes the negative document sample generation strategy in multiple rounds of iterations to continuously improve the prediction accuracy of the model in processing negative documents, thereby enhancing the overall training effect and the generalization ability of the model.

[0086] For example, assume that the negative document image samples used in the first round of iterations are generated based on the following negative document image sample generation strategy: a total of 1,000 negative document image samples, of which 600 are generated by generation method A and 400 are generated by generation method B. And the total number of positive document image samples in the first round of iterations is 1,000. The training samples used in the first round of iterations include 2,000 document image samples marked with anti-counterfeiting labels.

[0087] After the first round of iterations, the prediction accuracy of the document anti-counterfeiting detection model for the negative document image samples in this round of iteration is statistically analyzed. It is assumed that the prediction accuracy of the document anti-counterfeiting detection model for all 1,000 negative document image samples is 65%, the prediction accuracy for the negative document image samples generated by generation method A is 70%, and the prediction accuracy for the negative document image samples generated by generation method B is 60%.

[0088] Based on the statistical results of the first round of iterations, it was found that the prediction accuracy for all 1000 negative ID image samples was low, and the prediction accuracy of the negative ID image samples generated by generation method B was lower than that of the negative ID image samples generated by generation method A. The adjusted negative ID image sample generation strategy was adopted: the total number of negative ID image samples in the next round of iterations increased by 200, and the number of negative ID image samples generated by generation method B was greater than the number of negative ID image samples generated by generation method A, so as to improve the prediction accuracy for the negative ID image samples generated by generation method B.

[0089] The following negative samples are generated according to the adjusted negative document image sample generation strategy: 1200 negative document image samples are generated, of which 500 are generated by generation method A and 700 are generated by generation method B. The generated negative document image samples are combined with the above 1000 positive document image samples to form a complete training sample set for the next round of iteration.

[0090] After the second round of iterations, the prediction accuracy of the document anti-counterfeiting detection model for the negative document image samples in this round of iteration is statistically analyzed. It is assumed that the prediction accuracy of the document anti-counterfeiting detection model for all 1,200 negative document image samples is 85%, the prediction accuracy for the negative document image samples generated by generation method A is 88%, and the prediction accuracy for the negative document image samples generated by generation method B is 80%.

[0091] Based on the statistical results of the second round of iterations, it is found that the prediction accuracy for all 1,200 negative document images is improved, but the prediction accuracy of the negative document image samples generated by generation method B is still lower than that of the negative document image samples generated by generation method A. The adjusted negative document image sample generation strategy can be: the total number of negative document image samples in the next round of iterations is reduced by 300, and the number of negative document image samples generated by generation method B is still greater than the number of negative document image samples generated by generation method A, so as to improve the prediction accuracy for the negative document image samples generated by generation method B.

[0092] The following negative samples are generated according to the adjusted negative document image sample generation strategy: 900 negative document images are generated, of which 400 are generated by generation method A and 500 are generated by generation method B. The generated negative document image samples are combined with the above 1000 positive document image samples to form a complete training sample set for the next round of iteration.

[0093] By analogy, with each round of iteration, the negative document image sample generation strategy can be gradually optimized to improve the model's recognition and generalization capabilities for negative document image samples. The ultimate goal is that at the end of the overall training, the model can effectively distinguish various types of negative document image samples, thereby improving overall accuracy and security. Through such an iterative process, we can continuously learn and adapt to different types of negative document image samples, and ultimately achieve the goal of improving recognition accuracy and reducing misjudgment rate.

[0094] In some embodiments, the image data of at least two modalities of each document image sample includes a texture feature map and at least one of the following: a spectrum map, a second document image, a grayscale map, and a noise feature map. Then the document anti-counterfeiting detection model includes a color extraction network, a frequency extraction network, and a noise extraction network, and a texture extraction network; the color extraction network is used to extract color features from the second document image and / or the grayscale map; the frequency extraction network is used to extract frequency features from the spectrum map; the noise extraction network is used to extract noise features from the noise feature map; and the texture extraction network is used to extract texture features from the texture feature map.

[0095] It is understandable that this embodiment does not impose any restrictions on the model structures of the color extraction network, the frequency extraction network, the noise extraction network, and the texture extraction network, and specific settings can be made according to the actual application scenario. For example, the color extraction network can adopt a convolutional neural network (CNN) structure, such as ResNet, VGG, etc., to extract color distribution and features from the second document image. The frequency extraction network can use a transformation-based method, such as a fast Fourier transform (FFT) combined with a convolution layer or a recurrent neural network (RNN) to capture frequency features. The noise extraction network may adopt an autoencoder structure to learn and extract the representation of noise from the noise feature map, or it may be a specially designed convolutional neural network. The texture extraction network may be a specially designed convolutional neural network, such as a network based on Gabor filters, or a pre-trained texture feature extractor (such as TextureNet) may be used to extract texture features. The selection of each network depends on the specific task requirements and data characteristics. For example, if it is necessary to process complex color distribution, a deeper CNN structure may be selected; while for capturing frequency features, FFT combined with a simple neural network may be an effective choice. In practical applications, these networks may be used in combination to obtain more comprehensive and accurate feature extraction of multiple aspects of document images.

[0096] In some embodiments, the at least two modalities of image data of each document image sample include a texture feature map, a spectrum map, a noise feature map, and at least one of the following: a second document image or a grayscale map. Figure 7 The document anti-counterfeiting detection model includes a color extraction network, a frequency extraction network, a noise extraction network, a texture extraction network, two bilinear pooling layers, and a classifier. The introduction of the bilinear pooling layer not only makes the information between different features more closely integrated, but also improves the expression ability of the final feature vector, thereby enhancing the accuracy and robustness of the classifier in the document anti-counterfeiting detection task. This feature-level fusion helps to fully understand and analyze document images from multiple perspectives, allowing the model to more reliably distinguish between real and forged documents.

[0097] In a first possible implementation, the size of the second document image, the size of the grayscale image, and the size of the noise feature map are all the second size, and the second document image includes the document content and background content other than the document content. The noise feature map includes local noise of the document content and local noise of the background content. Figure 7 , where a bilinear pooling layer is used to fuse the color features obtained by the color extraction network and the noise features obtained by the noise extraction network to obtain the first fused feature. The first bilinear pooling layer fuses the color features and the noise features. These two features can provide the color information and local noise information of the document respectively. Through bilinear pooling, the association and complementarity between them can be effectively captured, and the performance of these features in detection and classification tasks can be enhanced. It can also effectively improve the ability to distinguish between document content and background content. For example, color information can help distinguish specific parts of a document, while noise information can help detect possible signs of forgery or tampering in the image.

[0098] The size of the spectrum map and the size of the texture feature map are both the first size. The spectrum map and the texture feature map are obtained based on the first document image. The spectrum map only contains the frequency information of the document content, and the texture feature map only contains the texture information of the document content. Please refer to Figure 7 , another bilinear pooling layer is used to fuse the frequency features and texture features to obtain the second fused features. The second bilinear pooling layer fuses the frequency information and texture information. The frequency features contain the frequency distribution of the content of the certificate, while the texture features capture the texture details of the surface of the certificate. Through bilinear pooling, these features can be combined to provide a richer and more comprehensive description of the certificate features, enhancing the ability to judge the authenticity of the certificate. Frequency information can help detect printing or collage forgeries, while texture information helps detect possible duplication or photo replacement in the image.

[0099] The classifier is used to perform document anti-counterfeiting detection based on the first fusion feature and the second fusion feature, and output the document anti-counterfeiting detection result. Exemplarily, the classifier includes a fully connected layer, which can splice the first fusion feature and the second fusion feature, input the spliced ​​result into the classifier for document anti-counterfeiting detection, and output the document anti-counterfeiting detection result.

[0100] In a second possible implementation, based on the above description of obtaining image data of various modalities, it can be seen that there may be a situation where the sizes of image data of different modalities are inconsistent. In this case, the color features, noise features, frequency features, and texture features corresponding to the image data of the four modalities can be arbitrarily combined in pairs, and one of the two bilinear pooling layers and the classifier are used to fuse two of the features, and the other bilinear pooling layer and the classifier are used to fuse the remaining two features. Finally, the classifier performs document anti-counterfeiting detection based on the fused features output by the two bilinear pooling layers.

[0101] In other embodiments, the image data of at least two modalities of each document image sample includes a texture feature map and at least one of the following: a spectrum map, a second document image, a grayscale map, and a noise feature map, and the document anti-counterfeiting detection model includes a color extraction network, at least one of a frequency extraction network and a noise extraction network, a texture extraction network, and a classifier. After obtaining at least one of a color feature, a noise feature, and a frequency feature and a texture feature, the image data of at least two modalities obtained can be spliced, and then the spliced ​​result can be input into the classifier for document anti-counterfeiting detection, and the document anti-counterfeiting detection result can be output.

[0102] After training, the document anti-counterfeiting detection model can be used in a variety of practical applications and tasks, including but not limited to:

[0103] In the first application scenario, the model is deployed to actual document detection equipment for real-time identification and verification of various types of documents, such as ID cards, passports, driver's licenses, etc. This can achieve a fast and automatic verification process in banks, airports, border checkpoints, etc. to ensure the authenticity and legitimacy of the documents.

[0104] In the second application scenario, the document anti-counterfeiting detection model is deployed on the cloud platform and services are provided through APIs or other interfaces. This approach can support large-scale document anti-counterfeiting detection requests, such as batch processing.

[0105] In the third application scenario, the document anti-counterfeiting detection model is embedded in the mobile application for real-time document anti-counterfeiting detection on the user's mobile phone or tablet. For example, in law enforcement, security review or investigation activities, the document anti-counterfeiting detection model is embedded in the mobile application, which can be used as a tool to help analyze and evaluate the authenticity of the documents involved in the case, providing objective basis and support for decision-making.

[0106] Regardless of the device it is deployed on, the document anti-counterfeiting detection model can play an important role in protecting data security and improving efficiency. When deploying, factors such as real-time requirements, security, resource consumption, and user experience need to be considered in order to meet the needs of specific application scenarios to the greatest extent possible.

[0107] In some embodiments, see Figure 8 The embodiment of this specification provides a document anti-counterfeiting detection method, which is applied to a server or other electronic devices. The method includes:

[0108] In S801, a target document image to be detected is obtained.

[0109] Exemplarily, the document type of the target document image may be any type, including but not limited to an identity card, a passport, a driver's license, or a work permit.

[0110] In S802, the target ID image is preprocessed to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on the image difference between the target ID image and a reference ID image corresponding to the ID type of the target ID image; the reference ID image is an image obtained by averaging pixel values ​​at the same pixel position in multiple ID images corresponding to the ID type of the target ID image.

[0111] In this embodiment, at least two different visual representations or features are generated from the target document image to increase the robustness and accuracy of subsequent detection. The at least two modal image data include texture modal image data. Further, this embodiment provides a method for obtaining texture modal image data (i.e., texture feature map), and generates a texture feature map based on the image difference between the target document image and the reference document image, which can effectively separate the document content itself and the document texture. The texture feature map can help detect forged textures or added watermarks. Forgery usually causes the texture of the document image to be different from the normal reference image. This difference can be clearly displayed in the texture feature map, which helps to improve the accuracy of document anti-counterfeiting detection.

[0112] Here, the process of acquiring image data of texture modality (ie, texture feature map) is exemplarily described:

[0113] Based on the document type of the target document image, a reference document image corresponding to the document type of the target document image can be obtained. The reference document image is an image obtained by taking the average value of pixel values ​​at the same pixel position in multiple document images corresponding to the document type of the target document image. The specific process of obtaining the reference image can be found in the above description and will not be repeated here.

[0114] The target ID image and the reference ID image may be aligned. For example, a preset feature point detection algorithm may be used to detect feature points from the target ID image and feature points from the reference ID image, respectively. Then, based on a preset feature point matching algorithm, matching feature point pairs are found in the target ID image and the reference ID image. A homography matrix is ​​calculated based on the matching feature point pairs. The ID image is then transformed using the homography matrix, including rotation, scaling, translation and other transformation operations, so as to align the target ID image with the reference ID image. Further, the size of the aligned target ID image may be adjusted to a first size to obtain a first ID image. Exemplarily, the size of the first ID image is the first size and includes only the ID content.

[0115] After obtaining the first document image corresponding to the target document image, a texture feature map can be generated based on the image difference between the first document image and the reference document image. Specifically, the pixel value at each pixel position in the texture feature map is the difference between the pixel value at the same pixel position in the first document image and the pixel value at the same pixel position in the reference document image. For example, the pixel value of the first pixel in the first row of the texture feature map is the difference between the pixel value of the first pixel in the first row of the first document image and the pixel value of the first pixel in the first row of the reference document image. The pixel values ​​of other pixel positions in the texture feature map can be obtained based on a similar method, which will not be repeated here.

[0116] Exemplarily, after being processed in the above manner, the size of the texture feature map corresponding to the target document image is also the first size and only includes texture feature information of the document content, which can eliminate interference caused by texture information of non-document content (such as background content). Authenticity identification usually relies on unique security features on the document, such as printed details, watermarks or micro-text. Focusing on the texture information of these document contents can more accurately analyze and compare these features, thereby improving anti-counterfeiting detection capabilities.

[0117] In some embodiments, the image data of the at least two modalities further includes at least one of the following: image data of a color modality, image data of a frequency domain modality, and image data of a noise modality.

[0118] Here, the acquisition process of image data (i.e., spectrum diagram) in frequency domain mode is described as follows:

[0119] In a possible implementation, the target document image may be transformed from the time domain to the frequency domain to obtain a frequency spectrum diagram, and the size of the frequency spectrum diagram is the same as the size of the target document image.

[0120] In another possible implementation, in order to avoid interference caused by frequency information of non-document content, the first document image corresponding to the target document image can be transformed from the time domain to the frequency domain, thereby obtaining a spectrum diagram, the size of which is the same as the size of the first document image, that is, the spectrum diagram is also of the first size and only contains the frequency information of the document content. Exemplarily, the DCT (Discrete Cosine Transform) algorithm can be used to transform the first document image from the time domain to the frequency domain to obtain a spectrum diagram. In this embodiment, since the spectrum diagram mainly captures the frequency information of the document content in the first document image, this method is more resistant to interference, can eliminate interference caused by frequency information of non-document content, and improve the accuracy and stability of anti-counterfeiting detection.

[0121] Of course, other transformation algorithms may also be used, such as Fourier Transform, Wavelet Transform, or Haar Transform, etc., and this embodiment does not impose any limitation on this.

[0122] Here is an exemplary description of the process of acquiring image data in color mode:

[0123] In a possible implementation, the size of the target document image can be adjusted to a predefined second size to obtain a second document image that can be used as image data of a color mode. For example, a second template image can be pre-set, the second template image is of a second size and includes document content and background content other than the document content. A preset feature point detection algorithm can be used to detect feature points from the target document image and from a preset second template image, respectively, and then a matching feature point pair is found in the target document image and the second template image based on a preset feature point matching algorithm, and a homography matrix is ​​calculated based on the matching feature point pair, the homography matrix describes the transformation relationship from one image to another, and then the homography matrix is ​​used to transform the document image, including transformation operations such as rotation, scaling, and translation, so as to realize alignment of the target document image with the preset second template image, and further the size of the aligned target document image can be adjusted to match the size of the second template image, and if the size is insufficient, it is padded with pixels of a preset color to obtain a second document image with a second size. After alignment and resizing in the above manner, the second document image corresponding to the target document image includes both the document content and the background content in addition to the document content. In this embodiment, including the background content can provide complete contextual information of the document image, which helps to further improve the accuracy and reliability of document verification. The background content can help distinguish a real document image from a possible forged or tampered image. Some forgery techniques may ignore or change the document background, so including the background can provide an additional layer of security and reduce the possibility of fraud and forgery.

[0124] In another possible implementation, considering that the difference between a real certificate and a forged certificate mainly lies in the difference in the certificate content, the location of the certificate content is the main forgery location in the certificate, and the certificate content includes text content and / or image content (such as a face image), etc. Therefore, the certificate content in the second certificate image can be masked to enhance generalization, and the masked image is further converted into a grayscale image, thereby reducing the influence of light or shadow and improving detection accuracy.

[0125] Here is an exemplary description of the process of acquiring image data of the noise mode:

[0126] This embodiment uses an SRM filter to extract image data in a noise mode. The SRM (Steganalysis Rich Model) filter is a method for steganalysis. Steganalysis is a technique that aims to detect and analyze hidden information in digital media, which may be embedded through steganography, and the purpose may be to conceal communication or information transmission. For example, the SRM filter may use a variety of features, including but not limited to color histograms, texture features, statistical features, etc., to capture traces of hidden information in an image.

[0127] In a possible implementation, the SRM filter can be used to extract local noise from the target document image to obtain a noise feature map, the size of which is the same as the size of the target document image. This embodiment can analyze the noise distribution in different areas of the image by extracting the local noise feature map. These noise features can include various statistical features such as mean, variance, standard deviation, etc., which provide a strong basis for identifying and distinguishing true and false documents.

[0128] In another possible implementation, the SRM filter can be used to extract local noise from the second document image to obtain a noise feature map, the size of which is the same as the size of the second document image, that is, the size of the noise feature map is the second size and the noise feature map includes the local noise of the document content and the local noise of the background content. In this embodiment, since the noise feature map covers the local noise of the document content and the background content, the noise characteristics of the entire image can be comprehensively analyzed. This comprehensive analysis can help determine the noise distribution pattern of the entire image, rather than just being limited to a certain part. The local noise of a real document is usually related to the specific production process and materials of the document itself, while a forged image may have inconsistencies in the local noise of the document content and the background content. Extracting the overall noise feature map can help detect forged documents more comprehensively, because it is often difficult for a forger to maintain consistent noise characteristics throughout the image.

[0129] In S803, document anti-counterfeiting detection is performed on the target document image based on image data of at least two modalities.

[0130] This embodiment performs document anti-counterfeiting detection by acquiring image data of at least two modes of the target document image, thereby capturing multiple aspects of the target document image information based on the features reflected by the image data of multiple modes (especially the texture features reflected by the texture feature map). This can more reliably detect whether the document has been tampered with or forged, thereby improving the accuracy of anti-counterfeiting detection.

[0131] In a possible implementation, a large number of data sets of real legal documents can be used to construct a legal document model. This legal document model can be a feature space composed of feature vectors of known legal documents. After obtaining image data of at least two modalities, feature vectors can be extracted from the image data of at least two modalities, and then the distance between the feature vector of the target document image and the legal document model is calculated. If the distance exceeds a preset distance threshold, the target document image is determined to be forged or abnormal.

[0132] In another possible implementation, the document anti-counterfeiting detection model trained in the above manner can be used to perform document anti-counterfeiting detection. Image data of at least two modalities of the target document image can be input into the pre-trained document anti-counterfeiting detection model to perform document anti-counterfeiting detection, and obtain the document anti-counterfeiting detection result for the target document image; wherein the document anti-counterfeiting detection model is a model obtained by supervised training based on a number of document image samples marked with anti-counterfeiting labels, and the anti-counterfeiting labels are used to indicate the authenticity of the document image samples.

[0133] In some embodiments, the image data of at least two modalities of the target document image includes a texture feature map and at least one of the following: a spectrum map, a second document image, a grayscale map, and a noise feature map. Then the document anti-counterfeiting detection model includes at least one of a color extraction network, a frequency extraction network, and a noise extraction network, and a texture extraction network; the color extraction network is used to extract color features from the second document image and / or the grayscale map; the frequency extraction network is used to extract frequency features from the spectrum map; the noise extraction network is used to extract noise features from the noise feature map; and the texture extraction network is used to extract texture features from the texture feature map.

[0134] It is understandable that this embodiment does not impose any restrictions on the model structures of the color extraction network, the frequency extraction network, the noise extraction network, and the texture extraction network, and specific settings can be made according to actual application scenarios.

[0135] In some embodiments, the at least two modalities of the target document image image data include a texture feature map, a spectrum map, a noise feature map, and at least one of the following: a second document image and a grayscale map. Figure 7 The document anti-counterfeiting detection model includes a color extraction network, a frequency extraction network, a noise extraction network, a texture extraction network, two bilinear pooling layers, and a classifier. The introduction of the bilinear pooling layer not only makes the information between different features more closely integrated, but also improves the expression ability of the final feature vector, thereby enhancing the accuracy and robustness of the classifier in the document anti-counterfeiting detection task. This feature-level fusion helps to fully understand and analyze document images from multiple perspectives, allowing the model to more reliably distinguish between real and forged documents.

[0136] Exemplarily, the size of the second document image, the size of the grayscale image, and the size of the noise feature map are all the second size, and the second document image includes the document content and the background content other than the document content. The noise feature map includes the local noise of the document content and the local noise of the background content. Figure 7 , where a bilinear pooling layer is used to fuse the color features obtained by the color extraction network and the noise features obtained by the noise extraction network to obtain the first fused feature. The first bilinear pooling layer fuses the color features and the noise features. These two features can provide the color information and local noise information of the document respectively. Through bilinear pooling, the association and complementarity between them can be effectively captured, and the performance of these features in detection and classification tasks can be enhanced. It can also effectively improve the ability to distinguish between document content and background content. For example, color information can help distinguish specific parts of a document, while noise information can help detect possible signs of forgery or tampering in the image.

[0137] The size of the spectrum map and the size of the texture feature map are both the first size. The spectrum map and the texture feature map are obtained based on the first document image. The spectrum map only contains the frequency information of the document content, and the texture feature map only contains the texture information of the document content. Please refer to Figure 7 , another bilinear pooling layer is used to fuse the frequency features and texture features to obtain the second fused features. The second bilinear pooling layer fuses the frequency information and texture information. The frequency features contain the frequency distribution of the content of the certificate, while the texture features capture the texture details of the surface of the certificate. Through bilinear pooling, these features can be combined to provide a richer and more comprehensive description of the certificate features, enhancing the ability to judge the authenticity of the certificate. Frequency information can help detect printing or collage forgeries, while texture information helps detect possible duplication or photo replacement in the image.

[0138] The classifier is used to perform document anti-counterfeiting detection based on the first fusion feature and the second fusion feature, and output the document anti-counterfeiting detection result. Exemplarily, the classifier includes a fully connected layer, which can splice the first fusion feature and the second fusion feature, input the spliced ​​result into the classifier for document anti-counterfeiting detection, and output the document anti-counterfeiting detection result.

[0139] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0140] In some embodiments, considering that the method of acquiring image data of multiple modalities of a single image is not limited to application in document anti-counterfeiting detection scenarios, but can also be applied in other image detection scenarios, therefore, based on the same concept as the above method, the embodiment of this specification further provides an image detection method, including:

[0141] Acquire the target image to be detected;

[0142] Preprocessing the target image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on an image difference between the target image and a reference image corresponding to the type of the target image; the reference image is an image obtained by taking the average value of pixel values ​​at the same pixel position in multiple images corresponding to the type of the target image;

[0143] Image detection is performed on the target image based on image data of at least two modalities.

[0144] For example, in industrial quality inspection and product inspection scenarios, the target image may be a product image, and defects on the product surface may be detected based on at least two modal image data of the product image. Multimodal image data may be used to improve detection accuracy, and texture feature maps may be used to identify and classify different types of defects.

[0145] For example, in a document and printing quality inspection scenario, the target image may be a printed product image, and quality problems of the printed product, such as text clarity, color accuracy, or image integrity, may be detected based on image data of at least two modalities of the printed product image to improve product quality control and inspection efficiency.

[0146] In one implementation, the sizes of the multiple images and the reference image are all predefined first sizes. Preprocessing the target image includes: aligning the target image with the reference image, and adjusting the size of the aligned target image to the first size to obtain a first image; wherein the texture feature map is generated based on the image difference between the first image and the reference image.

[0147] Exemplarily, the first image is an image including only content.

[0148] Exemplarily, the pixel value at each pixel position in the texture feature map is the difference between the pixel value at the same pixel position in the first image and the pixel value at the same pixel position in the reference image.

[0149] In one implementation, the image data of at least two modalities also includes at least one of the following: (1) a spectrum diagram obtained by transforming the first image from the time domain to the frequency domain; (2) a second image obtained by adjusting the size of the target image to a predefined second size; wherein the second image includes content and background content other than the content; (3) masking the document content in the second image and converting the masked image into a grayscale image; (4) a noise feature map obtained by extracting local noise from the second image using an SRM filter.

[0150] In one implementation, image detection is performed on a target image based on image data of at least two modalities, including: inputting the image data of at least two modalities into a pre-trained image detection model for image detection, and obtaining an image detection result for the target image; wherein the image detection model is a model obtained by supervised training based on a number of labeled image samples, and the specific content of the label can be set according to the specific image detection scenario. For example, in industrial quality inspection and product inspection scenarios, the label is used to describe the defects of the product in the product image.

[0151] In one implementation, the image data of at least two modalities includes at least one of a second image, a grayscale image, a spectrum image, and a noise feature image, and a texture feature image. The image detection model includes at least one of a color extraction network, a frequency extraction network, and a noise extraction network, and a texture extraction network. The color extraction network is used to extract color features from the second image and / or grayscale image; the frequency extraction network is used to extract frequency features from the spectrum image; the noise extraction network is used to extract noise features from the noise feature image; and the texture extraction network is used to extract texture features from the texture feature image.

[0152] In one implementation, the image detection model includes a color extraction network, a frequency extraction network, a noise extraction network, a texture extraction network, two bilinear pooling layers and a classifier; one of the bilinear pooling layers is used to fuse the color features and the noise features to obtain a first fused feature; wherein the size of the second image and the size of the noise feature map are both the second size; the other bilinear pooling layer is used to fuse the frequency features and the texture features to obtain a second fused feature; wherein the size of the spectrum map and the size of the texture feature map are both the first size; the classifier is used to perform image detection based on the first fused feature and the second fused feature, and output the image detection result. The training process of the image detection model is similar to the training process of the above-mentioned document anti-counterfeiting detection model. For details, please refer to the above description and will not be repeated here.

[0153] Based on the same concept as the above method, an embodiment of this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0154] Fig. 9 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Fig. 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 902 reading the corresponding computer program from the non-volatile memory 910 into the memory 908 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0155] For example, the document anti-counterfeiting detection device can be applied to Fig. 9 The device shown in the figure is used to implement the technical solution of this specification. The document anti-counterfeiting detection device may include:

[0156] A target document image module is used to obtain a target document image to be detected;

[0157] A preprocessing module, configured to preprocess the target document image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on an image difference between the target document image and a reference document image corresponding to the document type of the target document image; the reference document image is an image obtained by taking the average value of pixel values ​​at the same pixel position in a plurality of document images corresponding to the document type of the target document image;

[0158] The document anti-counterfeiting detection module is used to perform document anti-counterfeiting detection on the target document image based on the image data of the at least two modalities.

[0159] Exemplarily, the training device of the document anti-counterfeiting detection model can be applied to Fig. 9 The device shown in the figure is used to implement the technical solution of this specification. Among them, the training device of the certificate anti-counterfeiting detection model may include:

[0160] A training sample set acquisition module, used to acquire a training sample set, wherein the training sample set includes a number of certificate image samples marked with anti-counterfeiting labels, wherein the anti-counterfeiting labels are used to indicate the authenticity of the certificate image samples;

[0161] A preprocessing module, used to preprocess each of the document image samples to obtain image data of at least two modes corresponding to each of the document image samples; wherein the image data of at least two modes includes a texture feature map generated based on the image difference between the document image sample and a reference document image corresponding to the document type of the document image sample; the reference document image is an image obtained by taking the average value of pixel values ​​at the same pixel position in multiple document images corresponding to the document type of the document image sample;

[0162] The model training module is used to perform supervised training on the document anti-counterfeiting detection model to be trained based on the image data of the at least two modalities corresponding to each of the document image samples and the anti-counterfeiting labels of the document image samples.

[0163] Exemplarily, the image detection device can be applied to Fig. 9 The device shown in the figure is used to implement the technical solution of this specification. The image detection device includes:

[0164] The target image module is used to obtain the target image to be detected.

[0165] A preprocessing module is used to preprocess the target image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on the image difference between the target image and a reference image corresponding to the type of the target image; the reference image is an image obtained by taking the average value of pixel values ​​at the same pixel position in multiple images corresponding to the type of the target image.

[0166] The image detection module is used to perform image detection on a target image based on image data of at least two modalities.

[0167] The specific implementation of each module in the above device can be found in the description of the relevant parts of the above method, which will not be repeated here.

[0168] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0169] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0170] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0171] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A document anti-counterfeiting detection method, include: Obtain the target document image to be detected; Preprocessing the target document image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on an image difference between the target document image and a reference document image corresponding to the document type of the target document image; the reference document image is an image obtained by averaging pixel values ​​at the same pixel position in a plurality of document images corresponding to the document type of the target document image; Based on the image data of the at least two modalities, document anti-counterfeiting detection is performed on the target document image.

2. The method according to claim 1, wherein the sizes of the plurality of document images and the reference document image are both predefined first sizes; Preprocessing the target document image, include: Aligning the target document image with the reference document image, and adjusting the size of the aligned target document image to the first size to obtain a first document image; The texture feature map is generated based on the image difference between the first document image and the reference document image.

3. The method according to claim 2, wherein the first certificate image is an image including only certificate content; and / or The pixel value at each pixel position in the texture feature map is the difference between the pixel value at the same pixel position in the first document image and the pixel value at the same pixel position in the reference document image.

4. The method according to claim 2, wherein the image data of at least two modalities further comprises at least one of the following: A spectrum obtained by transforming the first document image from the time domain to the frequency domain; A second document image obtained by adjusting the size of the target document image to a predefined second size; in, The second certificate image includes certificate content and background content other than the certificate content; Performing mask processing on the certificate content in the second certificate image, and converting the masked image into a grayscale image; and A noise feature map is obtained by extracting local noise from the second document image using an SRM filter.

5. The method according to any one of claims 1 to 4, wherein the document anti-counterfeiting detection is performed on the target document image based on the image data of the at least two modalities, include: The image data of the at least two modalities are input into a pre-trained document anti-counterfeiting detection model to perform document anti-counterfeiting detection, and obtain a document anti-counterfeiting detection result for the target document image; wherein the document anti-counterfeiting detection model is a model obtained by supervised training based on a number of document image samples marked with anti-counterfeiting labels, and the anti-counterfeiting labels are used to indicate the authenticity of the document image samples.

6. The method according to claim 5, wherein the image data of the at least two modalities comprises at least one of a second document image, a grayscale image, a spectrum image, and a noise feature image, and the texture feature image; The document anti-counterfeiting detection model includes at least one of a color extraction network, a frequency extraction network and a noise extraction network, and a texture extraction network; The color extraction network is used to extract color features from the second document image and / or the grayscale image; The frequency extraction network is used to extract frequency features from the spectrum graph; The noise extraction network is used to extract noise features from the noise feature map; The texture extraction network is used to extract texture features from the texture feature map.

7. According to the method of claim 6, the document anti-counterfeiting detection model includes the color extraction network, the frequency extraction network, the noise extraction network, the texture extraction network, two bilinear pooling layers and a classifier; One of the bilinear pooling layers is used to fuse the color feature and the noise feature to obtain a first fused feature; in, The size of the second document image and the size of the noise feature map are both the second size; Another bilinear pooling layer is used to fuse the frequency feature and the texture feature to obtain a second fused feature; wherein the size of the spectrum map and the size of the texture feature map are both the first size; The classifier is used to perform document anti-counterfeiting detection based on the first fusion feature and the second fusion feature, and output the document anti-counterfeiting detection result.

8. A training method for a document anti-counterfeiting detection model, include: Acquire a training sample set, the training sample set comprising a number of certificate image samples marked with anti-counterfeiting labels, the anti-counterfeiting labels being used to indicate the authenticity of the certificate image samples; Preprocessing each of the document image samples to obtain image data of at least two modes corresponding to each of the document image samples; wherein the image data of at least two modes includes a texture feature map generated based on an image difference between the document image sample and a reference document image corresponding to the document type of the document image sample; the reference document image is an image obtained by averaging pixel values ​​at the same pixel position in a plurality of document images corresponding to the document type of the document image sample; Based on the image data of at least two modalities corresponding to each of the certificate image samples and the anti-counterfeiting labels of the certificate image samples, supervised training is performed on the certificate anti-counterfeiting detection model to be trained.

9. The method according to claim 8, wherein the document anti-counterfeiting detection model to be trained is subjected to supervised training based on the image data of at least two modalities corresponding to each of the document image samples and the anti-counterfeiting label of the document image sample, include: The following iterative process is executed in a loop: based on the document anti-counterfeiting detection model to be trained, the image data of the at least two modalities corresponding to each document image sample is processed to obtain a document anti-counterfeiting detection result; When the iteration end condition is not met, the model parameters of the certificate anti-counterfeiting detection model to be trained are adjusted based on the difference between the certificate anti-counterfeiting detection result and the anti-counterfeiting label of the certificate image sample; wherein the adjusted certificate anti-counterfeiting detection model to be trained is used as the certificate anti-counterfeiting detection model to be trained in the next round of iteration.

10. The method according to claim 9, wherein the plurality of certificate image samples included in the training sample set can be divided into positive certificate image samples and negative certificate image samples according to the anti-counterfeiting labels; in, The negative document image samples used in the first round of iteration are generated based on a preset negative document image sample generation strategy; The method further comprises: After executing each round of iteration process except the last round of iteration process, the prediction accuracy rate of the negative document image sample in the current round of iteration process is counted; Based on the prediction accuracy, adjusting the negative document image sample generation strategy corresponding to the negative document image sample used in the current round of iteration process; According to the adjusted negative document image sample generation strategy, negative document image samples are generated for the next round of iterative process.

11. The method according to claim 10, wherein the negative document image samples in the training document image sample set are generated based on at least two generation methods; The prediction accuracy include: Prediction accuracy for all negative document image samples, and / or prediction accuracy for negative document image samples generated by each generation method; The negative document image sample generation strategy is used to at least indicate the total number of negative document image samples generated and / or the number of negative document image samples generated by each generation method.

12. The method according to any one of claims 9 to 11, wherein the sizes of the plurality of document images and the reference document image are both predefined first sizes; Preprocessing each of the document image samples, include: Aligning each of the certificate image samples with the reference certificate image, and adjusting the size of the certificate image samples after the alignment to the first size to obtain a first certificate image; wherein the first certificate image only includes certificate content; The texture feature map is generated based on the image difference between the first document image and the reference document image.

13. The method according to claim 12, wherein the image data of at least two modalities further comprises at least one of the following: A spectrum obtained by transforming the first document image from the time domain to the frequency domain; A second document image obtained by adjusting the size of the document image sample to a predefined second size; in, The second certificate image includes certificate content and background content other than the certificate content; Performing mask processing on the certificate content in the second certificate image, and converting the masked image into a grayscale image; and A noise feature map is obtained by extracting local noise from the second document image using an SRM filter.

14. The method according to claim 13, wherein the filter kernel in the SRM filter is of a preset size and the initial values ​​of the parameters in the filter kernel are random values; The method also include: After executing each round of iteration process except the last round of iteration process, the parameters in the filter kernel in the SRM filter are adjusted based on the difference between the document anti-counterfeiting detection result and the anti-counterfeiting label of the document image sample; wherein the adjusted SRM filter is used to obtain a noise feature map for the next round of iteration process.

15. The method according to claim 13, wherein the document anti-counterfeiting detection model comprises at least one of a color extraction network, a frequency extraction network and a noise extraction network, and a texture extraction network; The color extraction network is used to extract color features from the second document image and / or the grayscale image; The frequency extraction network is used to extract frequency features from the spectrum graph; The noise extraction network is used to extract noise features from the noise feature map; The texture extraction network is used to extract texture features from the texture feature map.

16. According to the method of claim 15, the document anti-counterfeiting detection model comprises the color extraction network, the frequency extraction network, the noise extraction network, the texture extraction network, two bilinear pooling layers and a classifier; One of the bilinear pooling layers is used to fuse the color feature and the noise feature to obtain a first fused feature; in, The size of the second document image and the size of the noise feature map are both the second size; Another bilinear pooling layer is used to fuse the frequency feature and the texture feature to obtain a second fused feature; wherein the size of the spectrum map and the size of the texture feature map are both the first size; The classifier is used to perform document anti-counterfeiting detection based on the first fusion feature and the second fusion feature, and output the document anti-counterfeiting detection result.

17. An image detection method, include: Acquire the target image to be detected; Preprocessing the target image to generate image data of at least two modalities; wherein the image data of at least two modalities includes a texture feature map generated based on an image difference between the target image and a reference image corresponding to the type of the target image; the reference image is an image obtained by averaging pixel values ​​at the same pixel position in a plurality of images corresponding to the type of the target image; Image detection is performed on the target image based on the image data of the at least two modalities.

18. An electronic device, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 17 by executing the executable instructions.

19. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

20. A computer program product comprising a computer program / instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 17.

Images