An Adversarial Watermark Generation Method and System Based on an OCR Black-Box Model

Abstract

The present invention discloses a method and system for generating adversarial watermarks for an OCR black-box model. The method includes: obtaining a watermark image and randomly generating a watermark feature variable; generating an adversarial watermark sample according to the watermark feature variable; generating an individual and a population of adversarial watermarks composed of adversarial watermark samples with different watermark feature variables; obtaining an original image, and combining the original image and the individual of the adversarial watermark population to form an adversarial enhanced image; using the OCR black-box model to recognize the adversarial enhanced image, and calculating the misclassification rate of the OCR black-box model as the population fitness value; generating a guiding vector operator according to the differential evolution algorithm, and performing mutation and crossover selection operations on the population individuals until the population meets the condition that the population fitness no longer changes or the algorithm iteration number threshold, and then outputting the optimal population individuals and population.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement, and particularly relates to a method and system for generating adversarial watermarks for an OCR black-box model Background Art

[0002] Currently, with the development of OCR technology (Optical Character Recognition), it has been widely applied to the banking field, such as for the recognition of documents like identity cards, bank cards, household registers, driving licenses, and also for the recognition of bills such as business licenses and VAT invoices. Currently, the vast majority of advanced OCR systems adopt deep learning models, which bring high performance but also face similar problems to most models. Among them, the vulnerability of being easily attacked by adversarial samples has a fatal impact on OCR systems. Due to the semantic characteristics of text, a single-character difference can lead to a huge deviation. Especially in scenarios such as important documents, contracts, identity recognition, and handwritten character recognition, attacks by adversarial samples can cause huge losses

[0003] In the banking scenario, many data that require OCR tasks are watermarked. Watermarks are very common in documents and are often used to indicate the intellectual property rights, confidentiality levels, and company logos of documents. In Asian cultures, there are also seals used to indicate document authorization. When people read the text content of a document, they will automatically ignore the influence of watermarks or seals. If perturbations are hidden in the watermarks, adversarial attacks can be achieved without being visually perceived by the human eye, thereby further improving the robustness and generalization ability of the OCR recognition model. For the recognition of a single image, the size, position, rotation angle, etc. of the watermark have a great impact on the desired model attack effect. Therefore, how to obtain the optimal image watermark is of great research value

[0004] According to the degree of understanding of the target model by the attacker, it can be divided into white-box attack (White-box Attack), gray-box attack (Gray-box Attack), and black-box attack (Black-box Attack). In a white-box attack, the attacker knows all the parameter information of the model, the input during the training phase, and the labels; in a gray-box attack, the attacker only knows part of the information of the model, such as training labels, and can use a trainable surrogate model to estimate data; in a black-box attack, the attacker knows nothing about the model structure, parameters, etc., and can only interact with the model through the output Summary of the Invention

[0005] One of the invention objectives of the present invention is to provide a method and system for generating adversarial watermarks for an OCR black-box model. The method and system optimize the OCR black-box model and design a guiding operator to accelerate the convergence of the OCR black-box model optimization algorithm, thereby quickly and effectively detecting the robustness of the OCR model

[0006] Another object of the present invention is to provide a method and a system for generating adversarial watermarks for an OCR black-box model. The method and the system use the differential evolution algorithm as the guiding algorithm for the OCR black-box model to construct an image population, and obtain the natural gradient direction through the guiding algorithm, thereby accelerating the convergence of the population. Therefore, the black-box attack effect can be achieved under limited resource conditions.

[0007] Another object of the present invention is to provide a method and a system for generating adversarial watermarks for an OCR black-box model. The method and the system can efficiently, low-cost, and purposefully generate adversarial watermarks through the optimized OCR black-box model.

[0008] To achieve at least one of the above-mentioned objects of the invention, the present invention further provides a method for generating adversarial watermarks for an OCR black-box model, the method comprising:

[0009] Obtain a watermark image and randomly generate a watermark feature variable; generate an adversarial watermark sample according to the watermark feature variable;

[0010] Generate an individual and a population of adversarial watermark populations composed of adversarial watermark samples with different watermark feature variables;

[0011] Obtain an original image, and combine the original image and an individual of the adversarial watermark population to form an adversarial enhanced image;

[0012] Use the OCR black-box model to identify the adversarial enhanced image, and calculate the misclassification rate of the OCR black-box model as the population fitness value;

[0013] Generate a guiding vector operator according to the differential evolution algorithm, and perform mutation and crossover selection operations on the population individuals until the population meets the condition that the population fitness no longer changes or the algorithm iteration number threshold, and then output the optimal population individuals and population.

[0014] According to one preferred embodiment of the present invention, the method for obtaining the adversarial enhanced image includes: after obtaining the original image and the adversarial watermark image, the original image and the adversarial watermark image are superimposed through an adversarial enhanced image constraint model to obtain the adversarial enhanced constraint model, where the adversarial enhanced image constraint model is:

[0015] min f(X+g(Ψ));

[0016]

[0017] where f is the accuracy function of the original image input into the OCR black-box model, g is the relationship function between the watermark feature variable and the watermark image, Ψ is a strengthening adversarial watermark perturbation picture of the same size as X, lb i and ubi They are the range values of different characteristic variables of the watermark respectively, and s.t represents the constraint model.

[0018] According to another preferred embodiment of the present invention, the watermark characteristic variables include: the width of a single watermark; the height of a single watermark; the x coordinate of the upper left corner point of the initial watermark in the image; the y coordinate of the upper left corner point of the initial watermark in the image; the distance between every two watermark widths along the width direction of the initial watermark; the distance between every two watermark heights along the width direction of the initial watermark; the rotation angle of all watermarks; the transparency (gray value) of all watermarks.

[0019] According to another preferred embodiment of the present invention, the method for calculating and generating the guiding vector operator by the differential evolution algorithm includes: calculating the fitness value of each individual in the population, where the fitness value is the accuracy value calculated after the OCR black box model of each adversarial enhanced image is recognized. The lower the accuracy calculated after recognition, the higher the corresponding fitness value. After sorting the population individuals from high to low according to the fitness value, the guiding vector operator is calculated:

[0020]

[0021] where u is the layering size, N is the total number of individuals in the population, Q k is a certain population individual, where μ ∈ [1, 0.5N]. According to another preferred embodiment of the present invention, the method for the differential evolution algorithm to execute population mutation includes:

[0022] v i，G+1 = X best，G + F(X r1，G + X r2，G - X r3，G - X r4，G ) + F(g);

[0023] where v i，G+1 is the i-th individual in the next generation, X best，G is the best individual in the current generation, X r1，G , X r2，G , X r3，G , X r4，G are 4 randomly selected individuals in the current generation respectively, F is the scaling coefficient, and n individuals are generated by repeated calculation n times as the mutant offspring in the current iteration.

[0024] According to another preferred embodiment of the present invention, the method for the differential evolution algorithm to execute population crossover includes: t i，G+1 = v i，G+1 ; t i，G+1 [j] = X i，G [j];

[0025] where t i，G+1 is the i-th individual of the next generation, v i，G+1 is the i-th mutated individual generated, X i，G is the i-th parent of the current generation, CR is the mutation coefficient, randn(0, 1) is the standard normal distribution of the population, and n individuals are generated by repeating the calculation n times as the offspring in the current iteration.

[0026] According to another preferred embodiment of the present invention, the adversarial watermark generation method includes: selecting the individual with the best fitness from the current population as the parent individual, and finding another parent individual from the remaining population by using the roulette wheel method. Search for the parent individuals of two different populations in a loop. If the two parent individuals are different, select the parent individual with greater fitness to further compare with other parent individuals of different populations, and increment the loop count by one. Set the number of loop rounds, and exit when the loop count exceeds the number of loop rounds.

[0027] According to another preferred embodiment of the present invention, the parent individual obtained by the roulette wheel method is input into the Gaussian regression model. The mean m and standard deviation σ of the fitness function value of the target of the currently generated offspring individual are predicted through the Gaussian regression model, and the fitness function value f(off) of the actual target of the current offspring is calculated.

[0028] To achieve at least one of the above invention objectives, the present invention further provides an adversarial watermark generation system for an OCR black box model, and the system executes the above-mentioned adversarial watermark generation method for an OCR black box model.

[0029] The present invention further provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to execute the above-mentioned adversarial watermark generation method for an OCR black box model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It shows a schematic flowchart of an adversarial watermark generation method for an OCR black box model of the present invention.

[0031] Figure 2 It shows a schematic diagram of the generated adversarial watermark in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles defined in the following description can be applied to other implementation schemes, variant schemes, improvement schemes, equivalent schemes, and other technical schemes that do not depart from the spirit and scope of the present invention.

[0033] It is understood that the term "one" should be understood as "at least one" or "one or more". That is, in one embodiment, the number of an element can be one, while in other embodiments, the number of the element can be multiple. The term "one" should not be understood as a limitation on the quantity.

[0034] Please combine Figure 1 - Figure 2 , the present invention discloses a method and system for generating adversarial watermarks for an OCR black-box model, wherein the method includes: First, it is necessary to generate adversarial watermark samples, where the adversarial watermark samples include a watermark image and watermark feature variables. The watermark image is directly generated by a related watermark generation device, and the watermark feature variables are related information such as the layout, position, and transparency of the watermark itself. By randomly changing the watermark feature variables, different adversarial watermark information can be generated, and thus watermark samples with multiple adversarial watermark information can be generated.

[0035] After completing the construction of the watermark samples, further obtain the original image that can be recognized by the OCR black-box model. Please refer to Figure 2 , where the original image can be an image containing text, and the information in the original image can be recognized by the OCR black-box model. It should be noted that in a pre-trained OCR black-box model, the success rate of using the original image to recognize text by the trained OCR black-box model is generally considered to have a high accuracy. When the recognition rate of the adversarial enhanced image in the OCR black-box model decreases, it can be considered that the attack is successful.

[0036] After obtaining the original image, further combine and superimpose the original image and the adversarial watermark information as Figure 2 shown to generate an adversarial enhanced image. Further, use the OCR black-box model to recognize the text information in the adversarial enhanced image and calculate the correct rate of the text. In the present invention, the text recognition correct rate = 1 - misclassification rate. In the present invention, because it is necessary to detect the influence of the watermark image on the OCR black-box model and calculate the watermark image with the greatest influence through the differential evolution algorithm, the misclassification rate of the OCR black-box model for text is used as the population fitness in the differential evolution algorithm.

[0037] Specifically, the present invention proposes a method for detecting the recognition effect of an OCR black-box model based on an adversarial watermark constraint model, which is used to judge the generation effect of the adversarial watermark. The constraint model of the adversarial watermark includes:

[0038] min f(X + g(Ψ));

[0039]

[0040] where the function f is the adversarial enhanced image with respect to the input original image X where Θ is the character recognition accuracy of Θ = g(Ψ). g is the relationship function between the watermark feature and the watermark image, Ψ is the enhanced adversarial watermark perturbation image of the same size as X, lb i and ub i are the range values of different feature variables of the adversarial watermark respectively. In the present invention, the watermark feature variables can be set to 8, which are respectively the width of a single watermark: ψ1 ∈ [lb1, ub1]; the height of a single watermark: ψ2 ∈ [lb2, ub2]; the x coordinate of the upper left corner point of the initial watermark in the image: ψ3 ∈ [lb3, ub3]; the y coordinate of the upper left corner point of the initial watermark in the image: ψ4 ∈ [lb4, ub4]; the distance between every two watermark widths along the width direction of the initial watermark: ψ5 ∈ [lb5, ub5]; the distance between every two watermark heights along the width direction of the initial watermark: ψ6 ∈ [lb6, ub6]; the rotation angle of all watermarks: ψ7 ∈ [lb7, ub7]; the transparency (gray value) of all watermarks: ψ g ∈ [lb g and ub g . It should be noted that different visual effects are produced on the original image by the watermark image through different value selections of the above watermark feature variables. Through the above constraint function based on the watermark feature variables, the present invention can further construct different adversarial watermark images on the enhanced image. The present invention takes the watermark feature variables in the above adversarial watermark image as the genes of the differential evolution algorithm, and takes the gene combination composed of 8 watermark feature variables as an individual of a population, and further constructs a complete differential evolution algorithm adversarial watermark population including different gene combination individuals. After combining and superimposing each individual in the adversarial watermark population with the original image, the corresponding enhanced image is generated, and the accuracy of character recognition in the original image after the superposition of each individual in the adversarial watermark population is identified through the OCR black box model. And calculate the misclassification rate of the OCR black box model for the enhanced image after the superposition of each adversarial watermark individual as the fitness of the population individual of the differential evolution algorithm. Among them, the higher the misclassification rate of the OCR black box model for the enhanced image, the better the fitness of the corresponding population individual, indicating a better adversarial effect of the adversarial watermark.

[0041] It is worth mentioning that in the present invention, the differential evolution algorithm (DE) adopts real number coding design. 8 watermark feature variable values are randomly generated between the upper and lower boundaries of the 8 watermark feature variables to form an initial individual, and multiple randomly combined individuals are combined into a population. And the corresponding misclassification rate is calculated according to the recognition accuracy of the OCR black box model in the adversarial watermark constraint model as the fitness of the population individual of the adversarial watermark. Further, after sorting according to the fitness values of the population individuals from large to small, the guiding vector operator is calculated:

[0042]

[0043] where μ is the stratification size, μ ∈ [1, 0.5N], and N is the total number of individuals in the population, Q k is an individual in the sorted population. In the present invention, the stratification size is preferably set to μ = 0.2N.

[0044] Further, perform the mutation operation on the individuals in the population according to the differential evolution algorithm, and the method for the mutation operation of the individuals in the population is as follows:

[0045] v i,G+1 = X best,G + F(X r1,G + X r2,G - X r3,G - X r4,G ) + F(g);

[0046] where v i,G+1 is the i-th individual in the next generation, X best,G is the best individual in the current generation, X r1，G , X r2，G , X r3，G , X r4，G are respectively 4 randomly selected individuals in the current generation, F is the scaling factor, and in the present invention, it is preferably set to 0.5. Calculate n times repeatedly to generate n individuals as the mutant offspring in the current iteration.

[0047] The present invention can further perform the crossover selection operation in the differential evolution algorithm, and the crossover selection method includes:

[0048] t i,G+1 = v i,G+1 ; j ∈ {1, 2,..., 8}: randn(0, 1) < CR: t i,G+1 [j] = X i,G [j];

[0049] where t i，G+1 is the i-th individual in the next generation, v i，G+1 is the i-th mutant individual generated, X i,G is the i-th parent in the current generation, CR is the mutation coefficient, and in the present invention, the mutation coefficient is preferably set to 0.7, randn(0, 1) is the standard normal distribution of the population. Calculate n times repeatedly to generate n individuals as the offspring in the current iteration.

[0050] It is worth mentioning that in the present invention, for the countermeasure watermark, the present invention further includes performing a selection operation by using the roulette wheel in the differential evolution algorithm. The selection operation includes: setting a fitness value selection threshold, and selecting the population individuals greater than the fitness threshold as the selected population, and this selected population does not participate in the mutation and crossover operations of the differential evolution algorithm. Among them, a selection loop threshold is set. For example, the present invention sets the selection loop threshold to 100 times. The individual with the best fitness is obtained from the current population as the parent individual, and the other parent individual is obtained from the remaining population by using the roulette wheel algorithm. Compare whether the current two parent individuals are the same. If they are different, calculate the fitness values of the two parent individuals, select the parent individual with a higher fitness value to enter the next selection loop, and record that the current loop count is incremented by 1 until a parent individual with an unchanging fitness is selected or the loop count exceeds the preset loop threshold, or the loop running time exceeds the preset selection time threshold. Select the optimal solution obtained by the differential evolution algorithm and output it as the optimal countermeasure watermark image.

[0051] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the methods of the present application are performed. It should be noted that the computer-readable medium in the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wire segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0052] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0053] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and illustrated in the embodiments. Without departing from the said principles, the embodiments of the present invention may have any variations or modifications.

Claims

1. An adversarial watermark generation method based on an OCR black-box model, characterized in that, The method includes: Obtain a watermark image and randomly generate watermark feature variables; generate adversarial watermark samples according to the watermark feature variables; Generate individual and population of adversarial watermarks composed of adversarial watermark samples with different watermark feature variables; Obtain an original image, and combine the original image and an individual of the adversarial watermark population to form an adversarial enhanced image; Use an OCR black-box model to recognize the adversarial enhanced image, and calculate the misclassification rate of the OCR black-box model as the population fitness value; Generate a guiding vector operator according to the differential evolution algorithm, and perform mutation, crossover, and selection operations on the population individuals until the population meets the condition that the population fitness no longer changes or the algorithm iteration times threshold, and then output the optimal population individual, including: Calculate the fitness value of each individual in the population, where the fitness value is the accuracy value calculated after the OCR black-box model recognition of each adversarial enhanced image. The lower the accuracy calculated after recognition, the higher the corresponding fitness value. After sorting the population individuals from high to low fitness value to obtain a sorted population, calculate the guiding vector operator: where μ is the size of the layer, N is the total number of individuals in the population, Q k is an individual of a certain population, and k is the index of the population individual; Output the optimal solution obtained by the differential evolution algorithm as the optimal adversarial watermark image.

2. The anti-watermark generation method based on the OCR black box model according to claim 1, characterized in that, The method for obtaining the adversarial enhanced image includes: after obtaining the original image and the adversarial watermark image, superimpose the original image and the adversarial watermark image through an adversarial enhanced image constraint model to obtain the adversarial enhanced image.

3. The anti-watermark generation method based on the OCR black-box model according to claim 2, wherein Wherein the adversarial enhanced image constraint model is: minf(X+g(Ψ)); s.t. ψ i ∈ [lb i , ub i , where f is the accuracy function of the original image input into the OCR black-box model, g is the relationship function between the watermark feature variable and the watermark image, Ψ is the watermark feature variable, lb i and ub i are the range values of different watermark feature variables respectively, and X is the input original image.

4. The anti-watermark generation method based on the OCR black-box model according to claim 1, characterized in that The watermark feature variables include: the width of a single watermark; the height of a single watermark; the x coordinate of the upper left corner point of the initial watermark in the image; the y coordinate of the upper left corner point of the initial watermark in the image; the distance between every two watermark widths along the width direction of the initial watermark; the distance between every two watermark heights along the width direction of the initial watermark; the rotation angle of all watermarks; the transparency of all watermarks, where the transparency is a grayscale value.

5. A method for generating adversarial watermark based on OCR black box model according to claim 1, characterized in that, The method for performing population mutation according to the differential evolution algorithm includes: v i，G+1 = X best，G + F(X r1，G + X r2，G - X r3，G - X r4，G ) + F(g); where v i，G+1 is the i-th individual of the next generation, X best，G is the best individual of the current generation, X r1G， X r2G , X r3，G , X r4，G are randomly selected individuals of the current generation respectively, F is the scaling factor, and n calculations are repeated to generate n individuals as the mutant offspring in the current iteration.

6. The anti-watermark generation method based on an OCR black-box model according to claim 1, wherein, Performing a population crossover method according to the differential evolution algorithm includes: The differential evolution algorithm performing the population crossover method includes: T i，G+1 = v i，G+1 ; t i，G+1 [j] = X i，G [j]; where t i，G+1 is the i-th individual of the next generation, v i，G+1 is the i-th mutated individual generated, X i，G is the i-th parent of the current generation, CR is the mutation coefficient, randn(0,1) is the standard normal distribution of the population, and n individuals are generated by repeating the calculation n times as the offspring in the current iteration; t i，G+1 [j] is the j-th parameter value of the i-th generated mutant individual, X i，G [j] is the j-th parameter value of the i-th parental individual, and j is the index of the parameter.

7. A method for generating adversarial watermarks based on an OCR black-box model according to claim 1, characterized in that, The method for generating the adversarial watermark includes: select the individual with the best fitness in the current population as the parent individual, and find another parent individual by roulette method from the remaining population. Search for the parent individuals of two different populations in a loop. If the two parent individuals are different, select the parent individual with a greater fitness to further compare with other parent individuals of different populations, increment the loop count, set the loop round number, and exit when the loop count exceeds the loop round number.

8. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement an adversarial watermark generation method according to any one of claims 1-7 based on an OCR black-box model.

Images