Detection and Mitigation of Cyber Attacks on Binary Image Recognition Systems

Abstract

The computer-implemented method includes receiving binary image data by a computer system, the computer system being configured to detect pixel values ​​within the binary image data and present a non-machine code value associated with the binary image data; determining by the computer system that the binary image data further includes at least one pixel value that has been modified so as to change the non-machine code value associated with the binary image data when read by an image recognition system; and alerting the image recognition system by the computer system to review the binary image data.

Description

Related Applications

[0001] This application claims priority to and is related to U.S. Provisional Patent Application No. 62 / 971,021, filed on February 6, 2020. The entire disclosure of the provisional patent application is incorporated herein by reference for all purposes.

Technical Field

[0002] The present disclosure generally relates to, but is not limited to, the detection and identification of attackable parameters in images related to cyberattacks on imaging systems and certain optimizations thereof.

Background Art

[0003] In recent years, there has been a great deal of interest in understanding the vulnerabilities of artificial intelligence (AI) systems. For example, attacks on image classification models have demonstrated several weaknesses that AI systems need to address. Such attacks distort images in a way that is substantially imperceptible to the human eye and cause conventional image classification systems to misclassify these images. In fact, these vulnerabilities can lead to significant damage, such as widespread financial fraud.

[0004] Considerable effort has been devoted to securing AI classification models for color and grayscale images, but little is known about attacks on models for binary images, particularly those for check scanning. Without knowledge of attacks, there is little that can be done to prevent them. As an example, spoofing attacks are when a malicious party pretends to be another device or user on a network in order to launch an attack on a network host, steal data, bypass access controls, etc. Generally, spoofing attacks distort images in a way that is imperceptible to the human eye and cause conventional models to misclassify these images. Spoofing attacks on image classification models for color and grayscale images rely on introducing small perturbations to the color values of each pixel to hide noise in the distorted image. Because of these known vulnerabilities, conventional methods can be used to defend against those attacks.

[0005] Unlike attacks on color and grayscale images, the search space for attacks on binary images is extremely limited, and noise cannot be hidden with small perturbations of each pixel. Since each pixel of a binary image can only be black or white, optimizing attacks on binary images poses a new fundamental challenge to the attacks.

[0006] It is not possible to fine-tune attacks on color and grayscale images to work on binary images. As described above, in the case of grayscale and color images, when generating attacks, small perturbations can be made to each individual pixel (in order to evaluate what changes are needed and limit the changes to those that are imperceptible to the human eye). These small perturbations are approximately in the range of 1 / 255 to 10 / 255 in magnitude. In the case of binary images, these small perturbations cannot be made to the pixels. Since the pixels are either black or white (e.g., 1 or 0), any change is just a change of 1. This is one order of magnitude larger than the perturbations for attacking color and grayscale images and cannot be transferred to attacks on binary images. Therefore, attacks on binary images are more difficult, and this problem has been scarcely studied. However, the lack of research does not protect image recognition in this area from attacks that exploit weaknesses. Conventional AI systems cannot detect the unknown.

Summary of the Invention

Problems to be Solved by the Invention

[0007] In view of the above, an improved system and method for detecting and mitigating cyberattacks on binary image recognition systems are needed to overcome the above obstacles and deficiencies of conventional cyberattack detection models.

Means for Solving the Problems

[0008] The present disclosure relates to a system and method for detecting the vulnerability of a model for binary image classification.

[0009] According to the first aspect disclosed in this book, a computer-executed method for detecting the vulnerability of a model for binary image classification is disclosed. This method includes receiving binary image data by a computer system, wherein the computer system is configured to detect pixel values in the binary image data and present non-machine language values related to the binary image data, judging by the computer system whether the binary image data further includes at least one pixel value that has been changed so as to change the non-machine language value related to the binary image data when read by an image recognition system, warning by the computer system to prompt the image recognition system to review the binary image data, and including.

[0010] In some embodiments, the step of judging that the binary image data includes a changed pixel value includes judging that the first artificial intelligence model and the second artificial intelligence model of the image recognition system have been attacked simultaneously.

[0011] In some embodiments, the first artificial intelligence model of the image recognition system classifies a portion representing a numerical amount written in numbers in the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the numerical amount written in characters in the binary image data.

[0012] In some embodiments, the step of judging that the two models have been attacked simultaneously includes judging that a non-targeted attack using a shadowed combined attack on the recognition system has been used on at least one of the two models.

[0013] In some embodiments, the method further includes judging whether a targeted version of the shadowed combined attack on the recognition system has been executed twice and attacked both models.

[0014] According to another aspect disclosed in this book, one or more persistent computer-readable media for storing a group of instructions are disclosed. When the group of instructions is executed by a computer system configured to review binary numbers, the computer system is at least receiving binary image data by the computer system, wherein the computer system is configured to detect pixel values in the binary image data and present non-machine language values related to the binary image data, the computer system further determining that the binary image data further includes at least one pixel value changed to change the non-machine language value related to the binary image data when read by an image recognition system, warning the image recognition system by the computer system to review the binary image data, and causing it to be executed.

[0015] In some embodiments, the step of determining that the binary image data includes a changed pixel value includes determining that the first artificial intelligence model and the second artificial intelligence model of the image recognition system have been attacked simultaneously.

[0016] In some embodiments, the first artificial intelligence model of the image recognition system classifies a portion representing a numerical amount written in numbers in the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the numerical amount written in characters in the binary image data.

[0017] In some embodiments, the step of determining that the two models have been attacked simultaneously includes determining that a non-targeted attack using a shadowed combined attack on the recognition system has been used on at least one of the two models, and the method optionally further includes determining whether a targeted version of the shadowed combined attack on the recognition system has been executed twice to attack both models.

[0018] In some embodiments, the binary image data is at least one of an alphanumeric string or a check, and the image recognition system is optionally an optical character recognition system.

[0019] According to another aspect disclosed herein, a computer-executed method for determining the vulnerability of a model for binary image classification is disclosed. This method includes receiving, by a computer system, binary image data, wherein the computer system is configured to test a plurality of sets of pixel values in the binary image data and present a non-machine language value associated with the binary image data in an image recognition system; determining, by the computer system, that the binary image data further includes at least one pixel value changed to change the non-machine language value associated with the binary image data when read by the image recognition system; warning, by the computer system, that the image recognition system is vulnerable to spoofing attacks; and the binary image data is optionally an alphanumeric string or a check.

[0020] In some embodiments, the step of determining that the binary image data includes a changed pixel value includes determining that a first artificial intelligence model and a second artificial intelligence model of the image recognition system are simultaneously attacked.

[0021] In some embodiments, the first artificial intelligence model of the image recognition system classifies a portion representing a numerical amount written in numbers of the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the numerical amount written in characters of the binary image data.

[0022] In some embodiments, the step of determining that the two models are simultaneously attacked includes determining that a non-targeted attack using a shadow combination attack on the recognition system is used on at least one of the two models.

[0023] In some embodiments, this method further includes determining whether a targeted version of a shadow combination attack on the recognition system has been executed twice to attack both models.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 7

Figure 8

Figure 9

[0025] Note that the figures are not drawn to scale, and similar structural or functional elements are generally represented by similar reference numerals throughout all the figures for the purpose of explanation. Also note that the figures are only intended to facilitate the description of the preferred embodiments. The figures do not show all aspects of the embodiments to be described and do not limit the scope of the present disclosure.

Mode for Carrying Out the Invention

[0026] Since currently available cyber attack detection and mitigation systems are not adaptable to attacks on models for binary image classification, a system and method are disclosed for identifying binary images that have been partially modified or damaged so that an image recognition system can record incorrect results based on an image recognition tool training set. This solution advantageously reduces incorrect results that do not trigger a check fraud when a person manually reviews a check.

[0027] The systems and methods disclosed herein advantageously utilize an understanding of how binary image classification models can be bypassed or deceived and allow for the design of machine learning capabilities that can be trained to protect various image recognition systems. An additional solution to this problem is described by a test method for testing, enhancing, and protecting existing image scanning systems.

[0028] This subject matter can help provide additional solutions to this problem by protecting existing systems without a large-scale update of existing image recognition systems. This can be achieved, for example, by inserting a verification process that exists on top of an existing image recognition system as shown in FIG. 1.

[0029] Referring to FIG. 1, a schematic diagram of a model confidentiality protection system 200 for securely deploying an AI model 300 within an AI operating environment 100 is shown. The AI model 300 can include one or more computer-executed mathematical algorithms trained to replicate decisions that an expert would make based on that data and / or expert input when provided with the same information. Representative AI models 300 can include, but are not limited to, expert systems, case-based reasoning, behavior-based artificial intelligence, evolutionary algorithms, classifiers, statistical models, probability models, neural networks, decision trees, hidden Markov models, support vector machines, fuzzy logic, Bayesian classifiers, etc., or any combination thereof.

[0030] The model confidentiality protection system 200 is shown as including a red team engine (or model evaluation engine) 220 and a firewall 240. The red team engine 220 can be configured to identify one or more defects (and / or vulnerabilities) of the AI model 300. In slightly different terms, the red team engine 220 can determine data that can attack the AI model 300. Attacking the AI model 300 can include tricking the AI model 300, for example, spoofing as described above. In slightly different terms, an attack can include tricking the AI model 300 into making an incorrect decision, the AI model recognizing fake data as real data, recognizing synthetic (or forged or tampered) data as real data, and combinations thereof. An attack can include data configured to attack the AI model 300. In one embodiment, the red team engine 220 can output a report summarizing the vulnerabilities of the AI model 300.

[0031] The firewall 240 can protect the AI model 300 from being deceived by the external data 400 based on the defects identified by the red team engine 220. The external data 400 can include any data that would be input into the AI model 300 if the firewall 240 were not established. Put another way, the firewall 240 can patch the loopholes identified by the red team engine 220 and create an additional layer of confidentiality protection between the external data 400 and the AI model 300. In some embodiments, the firewall 240 can generate an alert when it detects an attack within the external data 400.

[0032] In some embodiments, the model confidentiality protection system 200 can be at least partially driven by an application programming interface (API) and inserted into the data supply path of the external data 400 in front of the AI model 300. The model confidentiality protection system 200 can return and / or output clean and unmodified data to the AI model 300. In various embodiments, the AI model 300 may remain as is and / or not be modified. Advantageously, the model confidentiality protection system 200 can protect the AI model 300 without a large-scale update of the AI model 300.

[0033] FIG. 1 shows the red team engine 220 and the firewall 240 as separate units for illustrative purposes only, but the red team engine 220 and the firewall 240 can be at least partially integrated and / or combined without limitation. For example, the red team engine 220 and the firewall 240 can each be implemented on computer hardware, firmware, and / or software. Thus, the red team engine 220 and the firewall 240 can be implemented as encoded instructions stored on one or more computer systems. The encoded instructions related to the red team engine 220 and the firewall 240 can be separate and / or integrated programs, and the red team engine 220 and the firewall 240 are not necessarily implemented on separate hardware.

[0034] Referring to FIG. 2, a representative method 2000 for securely deploying the AI model 300 is shown. One or more defects of the AI model 300 can be identified in step 2010. In various embodiments, the red team engine 220 (shown in FIG. 1) can execute step 2010.

[0035] The AI model 300 can be protected from attacks by the external data 400 based on the identified defects of the AI model 300 (in step 2010) in step 2020. In various embodiments, the firewall 240 (shown in FIG. 1) can execute step 2020. The protection by the firewall 240 is advantageously customized for the AI model 300 and can thus be effective. As described above, the model confidentiality protection system 200 is particularly suitable for detecting and mitigating attacks on binary image classification models.

[0036] In some embodiments, the binary image is defined as a d-dimensional image, and each pixel of the image has an assigned value (e.g., 0 or 1). The pixel is black (e.g., defined as value 0) or white (e.g., defined as value 1). For example, in some embodiments, the system assumes an m-class classifier that maps the binary image to the probability distribution F(x)[0,1] m where F(x) i corresponds to the confidence or probability that the image x belongs to class i. The predicted label y of x is the class with the best confidence. That is, y = arg max i F(x) i . Examples of binary image processing systems include check processing, license plate recognition, receipt processing, insurance document extraction, and legal document text recognition and comparison systems. These binary image processing systems can rely on a model, such as the AI model 300 shown in FIG. 1, to classify binary images.

[0037] In some embodiments, an optical character recognition (OCR) system converts an image of handwritten or printed text into an electronic string. This system has many important applications, including automatic receipt processing, passport recognition, insurance document extraction, and license plate recognition. Typically, some OCR systems, such as Tesseract, perform preprocessing to convert the input into a binary format.

[0038] Binary attack For example, in order to give a certain form to the problem of attacking the AI model 300 of a binary OCR system, a classifier F for OCR whose label is a string is used in this document. Considering a binary image x with label y, the system creates an adversarial example x' that visually resembles x but has a predicted label y' that is substantially different from the expected result y. In other words, y' ≠ y.

[0039] For example, considering an image x of license plate 23FC6A, the system can create a similar image x' that is recognized as a different valid license plate number. The system can then measure the similarity between the original image x of the adversarial image x' using a perceptual metric D x (x'). In the case of binary images, the intrinsic metric is the number of pixels where x and x' are different, which corresponds to the L0 distance between the two images. The L0 distance usually measures the distance between two input images by counting the number of different elements (e.g., pixels) (e.g., represented as a metric) and can be formulated as follows.

[0040]

Equation

[0041] Therefore, finding adversarial examples can be formulated as the following optimization approach.

[0042]

Equation

[0043] Here, k is the maximum amount of difference allowed for the adversarial image x'. In some embodiments, the maximum amount of difference is bounded by k to ensure that the distance between x and x' is not too large. Setting the maximum allowable value k ensures that the adversarial image x' is still close to the original image x in the L0 space. Target label y t For the case of a targeted attack with, the system can maximize F(x') y t as much as possible.

[0044] Check processing system. The check processing system receives as input the binary image x of the check and outputs a confidence score F(x). The confidence score F(x) represents the most likely value of what is written on the check (Cartesian amount recognition (CAR) and Legal amount recognition (LAR)).

[0045] FIG. 3 shows a typical image recognition process 3000 for processing a forged check and determining a false CAR amount and LAR amount, for example. The CAR section 308 designates the numerical amount 304 written in numbers, and the LAR section 306 designates the narrative amount 302 written in characters. The scanning process generates confidence factors for each part of the CAR 308 and LAR 306. Most conventional scanning systems decompose this into a combined amount value 310 and a combined recognition confidence value 312. Here, generally most commercial systems stop the verification investigation.

[0046] The check processing system is a special variant of the image recognition system and uses two independent models that verify each other. The model F C for Cartesian amount recognition (CAR) classifies the amount written in numbers, and another model F L for Legal amount recognition (LAR) classifies the amount written in characters. If the labels predicted by the two models of the input check image do not match, the check is flagged and not further processed. Alternatively, if the two values match, the check is processed. For example, if the CAR 304 of a valid check can be read as "100" and the LAR 302 of the same check can be read as "one hundred", the two values match and the check is processed. One problem with attacking the check processing system with respect to the input x is that F C and FL It is to create an adversarial example x’ that has the same target label for both. Returning to the previous example, a successful adversarial check image might make the CAR read “900” and the LAR read “nine hundred”. Otherwise, the values do not match, so the check is flagged. In the case of this targeted attack, the corresponding optimization problem is

[0047]

Number

[0048] In the case of this targeted attack, the attacker tries to change to a target amount y that is different from the true amount y t and tries to attack F C and F L so that both misclassify x’ as the amount y t . The check processing system also flags checks for which these models have low confidence in their predictions. So the attacker can maximize both probabilities F C (x’) y L and F C (x’) y t and F L (x’) y t . To make x’ look as much like x as possible, the attacker must also limit the number of pixels to be changed to a predetermined number k or less. Many check processing systems are configured to output probabilities only for a limited number of the most likely amounts for F C and F L . This limitation makes it difficult for the attacker to select a target amount different from the true amount. The most likely amounts for each of F C and F L may be disjoint sets.

[0049] Another limitation is that the attacker has no information about the model F being used and can only observe the output of the model. In other words, the attacker only has access to the probability distribution output by the model F for the query x’.

[0050] In FIG. 4, the check forgery attack 4000 consists of a digital image 402 of a check with attack modifications 404 to the numbers and text lines (CAR and LAR). The attack modifications use a modified digital image 406 of the check (having a modified CAR 408 and a modified LAR 410). This changes the digital image 402 of the check in a way that is imperceptible to the human eye, generates a model error for the attacker's benefit, and creates an adversarial model forgery attack that can be detected and mitigated by the model confidentiality protection system 200.

[0051] FIG. 5 shows a representative data flow diagram of a check submission process 5000 that can be used with the systems and methods described herein. Referring to FIG. 5, the check submission process 5000 begins at 5001 when a check is submitted for processing. Next, at 5002, the check is scanned to generate a binary image of the check. Once in binary form, at 5003, an image recognition system (not shown) can process the binary image as described with reference to FIG. 3. For example, in some embodiments, at 5004, a check processing system receives as input a binary image x of the check and outputs a confidence score F(x) representing the most likely values of what is written on the check (cartisian amount recognition (CAR) and legal amount recognition (LAR)). As described above, at 5005, the image recognition system then determines whether the identified CAR and LAR values match.

[0052] In some embodiments, the scanning process generates confidence factors for each part of the CAR 308 and LAR 306 (shown in FIG. 3). The confidence factors are then resolved into a final score. Conventional scanning systems break this down into a combined amount value 310 and a combined recognition confidence value 312 and stop the verification investigation, but the model confidentiality protection system 200 can perform a forgery prevention review at 5006. In other words, at 5006, the model confidentiality protection system 200 blocks any CAR / LAR images that have been tampered with, for example, using the targeted attacks described herein.

[0053] In some embodiments, the anti - forgery review consists of two secondary methods that work in parallel. First, the image - based method involves training a machine - learning model (e.g., AI model 300). The machine - learning model receives raw check images as input and can classify whether the image has been tampered with. Training the model includes a data - generation step and a model - training step. The data - generation step includes the methods described in this document for generating a large number of tampered images. The model - training step includes running one or more computer - vision classification algorithms to train the machine - learning model to classify whether a check is tampered with while training on non - tampered checks and tampered checks generated as described in this document. As an example, the vision - classification algorithm can include one or more convolutional neural networks used to analyze visual images.

[0054] Typically, a black - box attack requires frequently querying the check - processing system. Thus, the firewall 240 monitors the inputs to the check - processing system over time and identifies cases where the input sequence could be part of a hostile attack.

[0055] During each stage of the anti - forgery review 5006 and the determination that the input sequence is part of a hostile attack, a score is generated for each input. The meta - model receives the predictions of two individual models and combines them into a single score. Compared to a single model, these two models have different strengths and weaknesses that complement each other. For example, monitoring queries is particularly suitable for detecting and preventing black - box attacks, and advantageously, the attack can be prevented before adversarial examples are identified. If a fraudster limits the number of queries (e.g., by a forwarding attack), the machine - learning model can better identify the attack.

[0056] When the model - confidentiality - protection system 200 determines that a binary image has been tampered with in the manner described in this document, the tampered check image is prevented from proceeding to the approval process at 5007 and is rejected at 5008.

[0057] Attack binary images As described above, the model confidentiality protection system 200 can detect and mitigate multiple attacks on the image classification model. Representative attacks that can be detected by the model confidentiality protection system 200 are described. Two variations are described for illustrative purposes only, but it is understood that the system can perform and detect any combination of multiple methods that address the problem of hiding noise in binary images and optimizing the number of queries.

[0058] A simplified version of the combined attack on the binary image x classified as the true label y by model F, shown as Algorithm 1, is presented. In each iteration, Algorithm 1 finds the pixel p of the input image x such that inverting x to the opposite color causes the maximum decrease in F(x'). F(x') is the confidence that this perturbed input x' is classified as the true label y. In other words, the system flips the pixel and repeats this process until the classification of the perturbed input is y'≠y or the maximum L0 distance k from the original image is reached. In Algorithm 1 below, x'+e p represents the image x' where the pixel p has been inverted. p The adversarial image x' generated by Algorithm 1 can successfully deceive model 300 and have a small L0 distance from the original image x. However, the noise added to the input may still be visible to the human eye, and the number of queries to the model required to generate the adversarial example is large.

[0059]

Number

[0060] As previously described, in the case of attacks on color or grayscale images, the noise is often imperceptible because it is small relative to the range of colors in which any individual pixel can be changed.

[0061] Hide noise As described above, for attacks on color or grayscale images, the noise is often imperceptible because it is small relative to the range of colors in which any individual pixel can be changed.

[0062] For each pixel in the binary image, any attack can only either invert its color or leave it unchanged. Therefore, small changes to the color of each pixel are not possible, and gradient-based methods cannot be applied. Pixels with noise (i.e., pixels whose color is inverted and different from neighboring pixels) are noticeable because they have a different color from neighboring pixels. Since Algorithm 1 inverts the color of only a small number of pixels, it results in noise with a small L0 distance, but noisy pixels are very noticeable.

[0063] To address this problem, a new constraint is introduced that allows only pixels on the boundary between black and white regions in the image to be changed. A pixel is on the boundary if at least one of its 8 neighboring pixels that are adjacent to it is black (or vice versa). The adversarial examples generated under this constraint have a larger L0 distance from the original image, but the noise is much less noticeable.

[0064] Optimize the number of queries When many queries to the black-box model are required, the attack can be computationally expensive. If the model is a paid service hidden behind an application programming interface (API), performing the attack can also be costly in terms of fees. Several papers have proposed techniques to reduce the number of queries required for a successful attack. Many of the traditional solutions are based on gradient estimation and thus do not apply to binary settings.

[0065] Two optimization techniques are introduced that utilize the correlation relationship between pixels both spatially and repeatedly over time. For each iteration, the gain by inverting pixel p at point x' is defined as follows as the discrete derivative of F in the direction of p.

[0066]

Equation

[0067] If this value is greater than the threshold τ, the pixel has a large gain. That is, if inverting pixel p causes a decrease in the model confidence of label y by an amount greater than τ, then that pixel has a large gain.

[0068] Spatial correlation A first change that can be detected by the model confidentiality protection system 200 is based on the use of the spatial correlation relationship between pixel gains. Pixels within the same spatial region are likely to have similar discrete differentials as shown in FIG. 9. Each time, the attacker will prioritize evaluating the gains of the 8 pixels N(p) adjacent to the pixel p that was changed during the previous iteration of the algorithm. If one of these pixels has a large gain, the attacker will invert it and proceed to the next iteration without evaluating the remaining pixels.

[0069] Temporal correlation A second change that can be detected by the model confidentiality protection system 200 is based on the use of the correlation relationship between gains across different iterations from pixel p. A pixel having a large discrete differential in one iteration is more likely to have a large discrete differential in the next iteration than other pixels.

[0070] Each time, the attacker must first consider the pixels that had large gains during the previous iteration. If one of these pixels continues to generate a large gain in the current iteration, then that pixel is inverted and the system proceeds to the next iteration without evaluating the remaining pixels. This process ignores pixels that have little impact on misclassification over many iterations.

[0071] SCAR A more detailed method for attacking the binary image of algorithm 1 that can be mitigated by the model confidentiality protection system 200 is described in algorithm 2. To improve the number of queries, algorithm 2 prioritizes evaluating the discrete differentials of pixels expected to have large gains according to the above spatial and temporal correlation relationships.

[0072] If one of these pixels has a large gain, that pixel is inverted and the remaining pixels are not evaluated. If none of these pixels has a large gain, the attacker considers all pixels on the boundary B(x) between the black and white regions in the image x. In this case, the pixel with the largest gain is inverted regardless of whether its gain is greater than τ.

[0073] As described above, the standard basis vector in the direction of coordinate i is e i which is represented. The gain of each pixel with vector g is monitored and maintained. In some embodiments, Algorithm 2 represents a shadow combination attack (SCAR) on a recognition system.

[0074]

Number

[0075] Algorithm 2 is an untargeted attack that finds an adversarial example x' that is classified as a label y'≠y by F. The untargeted attack changes the first condition in the above while loop from y = arg max i F(x') i to y t ≠arg max i F(x') i and calculates the gain g p as F(x) y - F(x + e i ) y instead as F(x + e i ) yt - F(x) yt By doing so, it can be easily changed to a targeted attack with the target label y t .

[0076] Simultaneous attack There are two important issues in attacking the check processing system. In the previous section, the issues caused by preprocessing the check image into a binary image were introduced. The second issue is that the check processing system uses two independent models, two models that verify the output of the other model, where F C classifies the amount written in numbers, and F L classifies the amount written in letters. Motivated by this, an algorithm is introduced that addresses the problem of simultaneously attacking two separate OCR systems.

[0077] In some embodiments, the model confidentiality protection system 200 understands when to search for the target amount at the intersection of the amounts that the attack determines are likely to be F C and F L respectively. However, in the case of an unchanged check, the model 300 often has a high confidence in the true amount, and other amounts have a very low probability or do not appear at all as predictions by the model.

[0078] To increase the likelihood of selecting a target amount that would be an adversarial example, non-targeted attacks using SCAR on both F C and F L are initiated, returning an image x where the confidence in the true amount y has decreased. Next, since the goal is to attack both F u and F C and F L both, the target amount is selected to be the amount y with the maximum value min(F C (x u ) i ,F L (x u ) i ). The targeted version of SCAR (T-SCAR) is run twice, performing targeted attacks on both F t and F u with respect to the image x C and F L both. This is formulated as Algorithm 3 below.

[0079]

Number

[0080] Referring to the check in the example shown in FIG. 3, Algorithm 3 can be used to attack a check with, for example, an amount written as $401. As shown in the figure, Algorithm 3 can be executed to misclassify the amount as $701 with a high confidence of 0.909 by using the conventional CAR / LAR recognition process used by many financial institutions. The model confidentiality protection system 200 can determine whether Algorithm 3 was used to modify the binary image of the check.

[0081] Figures 6A-6E illustrate a series of representative attacks on a convolutional neural network (CNN) (e.g., Model 300) trained with various datasets that can be detected by the model confidentiality protection system 200. In each representative figure, the original image is shown on the left end, and various outputs of the attacks described in this document are shown on the right. The attacks shown on the right are not necessarily representative of the progress of the attack, but rather are independent attacks shown for illustrative purposes only.

[0082] Figure 6A illustrates a forgery attack on a number using the different algorithms described above. From left to right (the classification is in parentheses), the original number (2), SCAR (7), VANILLA-SCAR (7), POINTWISE (8), SIMBA (7). The number in parentheses following the type of attack represents the incorrectly determined result of the image recognition process. For example, the second image has been attacked using the SCAR algorithm described above and is forged to have a value of 7.

[0083] Figure 6B illustrates a forgery attack on a character using different algorithms. From left to right (the classification is in parentheses), the original character (8), SCAR (3), VANILLA-SCAR (3), POINTWISE (2), SIMBA (5).

[0084] Figure 6C illustrates a forgery attack on a multi-digit number using different algorithms. From left to right (the classification is in parentheses), the original number (1625), SCAR (15625), SIMBA (1025), VANILLA-SCAR (10625), POINTWISE (1025).

[0085] Figure 6D illustrates the spoofing attack on multi-character words using different algorithms. From left to right, the original word (test), SCAR (fest).

[0086] Figure 6E illustrates the spoofing attack on multi-character words using different algorithms. From left to right, the original word (down), SCAR (dower).

[0087] Four attack methods Four attack methods, SCAR, VANILLA-SCAR, SIMBA, and POINTWISE are compared.

[0088] · SCAR which is Algorithm 2 and has a threshold τ = 0.1 · VANILLA-SCAR which is Algorithm 1. Comparing SCAR with Algorithm 1 shows the importance of hiding noise and optimizing the number of queries.

[0089] · SIMBA which is Algorithm 1 and has ε = 1 in the Cartesian basis. SIMBA is an algorithm for attacking (color) images in a black-box setting using a small number of queries. At each iteration, SIMBA samples a direction q and moves one step towards εq or -εq if one of these increases the target. In the setting where q is sampled from the Cartesian basis and ε = 1, SIMBA corresponds to an L0 attack on binary images and randomly selects and flips pixels repeatedly (if doing so decreases the confidence of the true label).

[0090] POINTWISE first adds random salt-and-pepper noise until the image is misclassified. Then, if the image remains misclassified, POINTWISE gradually returns each modified pixel to its original color.

[0091] Metric To evaluate the effect of each attack A on model F and test set X, three metrics can be used. These metrics also advantageously show the vulnerability of the system / model to such attacks.

[0092] The success rate of A is the proportion of images x (x ∈ X) for which the output image x0 = A(x) is adversarial (i.e., the predicted label y0 of x0 is different from the true label y of x). Only images x that were initially correctly classified by F are attacked.

[0093] The L0 distance is used to measure how similar the image x’ = A(x) is to the original image x, and is the number of pixels for which x and x’ are different.

[0094] The number of queries to model F to obtain the output image x’ = A(x).

[0095] Distance constraint k. Since the image dimension d varies in each experiment, a reasonable method for selecting the maximum L0 distance k is sought. For an image x with label y, the L0 constraint is expressed by the following formula.

[0096]

Equation

[0097] Here, F(x) counts the number of pixels in the foreground of the image, α ∈ |0,1| is a predefined fraction (a fraction between 0 and 1), and |y| is the number of characters in y (e.g., |23FC6A| = 6). In other words, k is the average number of pixels per character in x multiplied by the predefined fraction. In some embodiments, α = 1 / 5.

[0098] Example of a tesseract attack The vulnerabilities of OCR systems are not limited to handwritten characters but also concern printed characters, which one would expect to be more robust. These vulnerabilities are explained in the context of English words, showing that in many cases, a change of a single pixel is sufficient for widely used open-source character recognition systems to misclassify a word as another word with a different semantic meaning within the English dictionary.

[0099] The Tesseract model. Tesseract is an open-source character recognition system designed for printed characters. Tesseract 4 is based on the Long Short-Term Memory (LSTM) model, which is a type of Recurrent Neural Network (RNN) architecture for deep learning. The system takes an image as input, and the image is first divided into images of each line. Tesseract binarizes the input image as part of the preprocessing. Next, each line is processed by the LSTM model, which outputs a string of characters.

[0100] Dataset. Images of single printed English words can be tested using a version of Tesseract trained for English by the system. In some embodiments, English words of length 4 are randomly selected. The accuracy rate for 1000 such images was 0.965, and the average confidence of the correctly classified words was 0.906. Out of the words correctly classified by Tesseract, 100 random words can be selected for attack. In some cases, especially for noisy images, Tesseract fails to recognize any word and rejects the input. Since the goal of these attacks is to misclassify the image as a word with a different meaning from the true word, an attack is considered successful only if the generated adversarial image is classified as a word within the English dictionary.

[0101] Figure 7 shows a graph of the results of the attacks. Success rate versus L0 distance and success rate versus number of queries for a CNN model trained using MNIST, a LeNet5 model trained using EMNIST, an LSTM model for handwritten digits, and a Tesseract model for printed words.

[0102] FIG. 8 is a block diagram showing a software architecture 800 that can be introduced into any one or more of the devices described in this document. FIG. 8 is merely a non-limiting example of a software architecture, and it will be understood that many other architectures can be implemented to enable the functions described in this document. In various embodiments, the software architecture 800 is executed by hardware such as the machine 900 of FIG. 9.

[0103] In this example architecture, the software architecture 800 can be conceptualized as a stack of layers, and each layer may provide a specific function. For example, the software architecture 800 includes layers such as an operating system 804, a library 806, a framework 808, and an application 810. In operation, the application 810 makes an API call 812 through the software stack and receives a message 814 in response to the API call 812. This is the same in some embodiments.

[0104] In various embodiments, the operating system 804 manages hardware resources and provides common services. The operating system 804 includes, for example, a kernel 820, services 822, and drivers 824. The kernel 820 acts as an abstraction layer between the hardware and other software layers. This is the same in some embodiments. For example, the kernel 820 provides memory management, processor management (e.g., scheduling), device management, network connection, security settings, and other functions. The services 822 can provide other common services to other software layers. According to some embodiments, the drivers 824 are responsible for controlling or connecting the underlying hardware. For example, the drivers 824 can include a display driver, a camera driver, a BLUETOOTH or BLUETOOTH low energy driver, a flash memory driver, a serial communication driver (e.g., a universal serial bus (USB) driver), a Wi-Fi driver, an audio driver, a power management driver, etc.

[0105] In some embodiments, library 806 provides a low-level common base that application 810 can utilize. Library 806 may include system library 830 (e.g., C standard library) that can provide functions such as memory allocation function, string manipulation function, mathematical function, etc. Also, library 806 may include API library 832, such as a media library (e.g., a library that supports the representation and manipulation of various media formats, such as Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., OpenGL framework used to draw 2D and 3D in a graphic context on a display), a database library (e.g., SQLite that provides various relational database functions), a web library (e.g., WebKit that provides web browsing function), etc. Library 806 may also include a wide variety of other libraries 834 to provide many other APIs to application 810.

[0106] According to some embodiments, framework 808 provides a high-level common base that application 810 can utilize. For example, framework 808 provides various graphical user interface (GUI) functions, high-level resource management, high-level location services, etc. Framework 808 can provide a wide range of other APIs that application 810 can utilize (some of which are specific to a particular operating system 804 or platform).

[0107] In one embodiment, the application 810 includes a home application 850, a contact application 852, a browsing application 854, an e - book reading application 856, a location application 858, a media application 860, a messaging communication application 862, a game application 864, and a wide variety of other applications, such as a third - party application 866. According to some embodiments, the application 810 is a program that executes functions defined within itself. Various programming languages can be used to create one or more of the applications 810 having various structures. Examples thereof are object - oriented programming languages (e.g., Objective - C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a particular example, the third - party application 866 (e.g., an application developed by a party other than the vendor of a particular platform using an ANDROID (registered trademark) or IOS (trademark) software development kit (SDK)) may be mobile phone software that operates on a mobile phone operating system, such as "IOS", "ANDROID", WINDOWS (registered trademark) Phone, or another mobile phone operating system. In this example, the third - party application 866 can call the API calls 812 provided by the operating system 804 to enable the functions described in this document.

[0108] FIG. 9 shows an overview of a machine 900 in the form of a computer system that executes an instruction set to cause the machine 900 according to the embodiment to execute any one or more of the methods described in this document. Specifically, FIG. 9 shows an overview of a machine 900 in the form of an example of a computer system, in which a set of instructions 916 (e.g., software, program, application, applet, app, or other executable code) for causing the machine 900 to execute any one or more of the methods described in this document can be executed. For example, the set of instructions 916 may cause the machine 900 to execute the method of FIG. 2 or FIG. 5. Additionally or alternatively, the set of instructions 916 may execute any of the functions described with reference to FIGS. 1, 3, and 4. The set of instructions 916 transforms a general-purpose machine 900 that is not programmed into a special-purpose machine 900 programmed to execute the described functions in the described manner. In other embodiments, the machine 900 may operate as a stand-alone device or may be coupled (e.g., network-connected) to other machines. When network-connected, the machine 900 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 900 may be, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a portable device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the set of instructions 916 that specify the operations that the machine 900 performs. Also, although only a single machine 900 is shown, the term "machine" should be considered to include a collection of machines 900 that execute the set of instructions 916 individually or jointly to execute any one or more of the methods described in this document.

[0109] Machine 900 may include a plurality of processors 910, a memory 930, and I / O components 950 configured to communicate with each other via, for example, a bus 902. In one embodiment, the plurality of processors 910 (e.g., central processing unit (CPU), reduced instruction set computing (RISC) processor, complex instruction set computing (CISC) processor, graphics processing unit (GPU), digital signal processor (DSP), application specific integrated circuit (ASIC), radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 912 and a processor 914 that execute instruction groups 916. The term "processor" is intended to include a multi-core processor consisting of two or more independent processors (sometimes called cores) that execute instruction groups 916 simultaneously. FIG. 9 shows a plurality of processors 910, but machine 900 may consist of a single processor 912 with a single core, a single processor 912 with multiple cores (e.g., multi-core processor 912), multiple processors 912, 914 with a single core, multiple processors 912, 914 with multiple cores, or any combination thereof.

[0110] Memory 930 may include a main memory 932, a static memory 934, and a storage unit 936 that are each accessible to processors 910 via bus 902. Main memory 932, static memory 934, and storage unit 936 store instruction groups 916 that embody any one or more of the methods or functions described herein. Instruction groups 916 may be fully or partially present in main memory 932, static memory 934, storage unit 936, within at least one processor 910 (e.g., within the cache memory of the processor), or any suitable combination thereof during execution by machine 900.

[0111] The I / O component 950 may include a variety of components such as receiving an input, providing an output, generating an output, transmitting information, exchanging information, and capturing a measurement value. The specific I / O component 950 included in a particular machine depends on the type of that machine. For example, a portable machine such as a mobile phone may probably include a touch input device or other such input mechanism, while a headless server machine will probably not include such a touch input device. It will be understood that the I / O component 950 may include many other components not shown in FIG. 9. The I / O component 950 is grouped according to its functions for the sake of simplicity in the following description, and this grouping is by no means limiting. In various embodiments, the I / O component 950 may include an output component 952 and an input component 954. The output component 952 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistive mechanism), other signal generators, etc. The input component 954 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or another pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the position and / or force of a touch or touch gesture, or other tactile input components), an audio input component (e.g., a microphone), etc.

[0112] In other embodiments, I / O component 950 may include, among a wide range of other components, a biometric component 956, a motion component 958, an environmental component 960, or a location component 962. For example, biometric component 956 may include components that detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or identification based on electroencephalogram). Motion component 958 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), and the like. Environmental component 960 may include, for example, an illuminance sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects the concentration of harmful gases for safety or to measure pollutants in the atmosphere), or other components that can provide a display, measurement, or signal corresponding to the surrounding physical environment. Location component 962 may include a location sensor component (e.g., a global positioning system (GPS) reception component), an altitude sensor component (e.g., an altimeter or barometer that detects the air pressure from which altitude can be derived), an azimuth sensor component (e.g., a magnetometer), and the like.

[0113] Communication may be implemented using a variety of techniques. The I / O component 950 may include a communication component 964 operable to couple the machine 900 to the network 980 or the device 970 via couplings 982 and 972. For example, the communication component 964 may include a network interface component or another suitable device to interface with the network 980. In a further example, the communication component 964 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a Bluetooth component (e.g., Bluetooth low energy), a Wi-Fi component, and other communication components that provide communication by other means. The device 970 may be either another machine or a variety of peripheral devices (e.g., coupled via USB).

[0114] Also, the communication component 964 may include a component operable to detect an identifier or to detect identifiers. For example, the communication component 964 may include a radio frequency identification (RFID) tag reading component, an NFC smart tag detecting component, an optical reading component (e.g., an optical sensor that detects one-dimensional barcodes such as universal product code (UPC) barcodes, QR code (registered trademark), Aztec code, DATA Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other multi-dimensional barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone that identifies a tagged audio signal). Additionally, the communication component 964 may obtain various information, such as a location based on Internet protocol (IP) geolocation information, a location based on Wi-Fi signal triangulation, a location based on detection of an NFC beacon signal indicating a particular location.

[0115] The various memories (i.e., memories 930, 932, 934 and / or the memory of processor 910) and / or storage unit 936 may store one or more sets of instruction groups 916 and data structures (e.g., software) that embody or are utilized by any one or more of the methods or functions described herein. These instructions (e.g., instruction group 916), when executed by processor 910, cause various operations to implement the disclosed embodiments.

[0116] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” may be used interchangeably. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Thus, these terms are to be understood to include, but are not limited to, solid state memories, and optical and magnetic media, including memories internal or external to a processor. Specific examples of machine storage medium, computer storage medium, and / or device storage medium include, by way of example, non-volatile memories including semiconductor memory devices such as erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), field programmable gate array (FPGA), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” specifically do not include carrier waves, modulated data signals, and other such media (at least some of which are included in the term “signal medium” described below).

[0117] In various embodiments, one or more portions of network 980 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a legacy plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi network, another type of network, or a combination of two or more such networks. For example, network 980 or a portion of network 980 may include a wireless or cellular telephone network, and coupling 982 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection for mobile communications, or another type of cellular or wireless coupling. In this example, coupling 982 may implement any of a variety of types of data transfer technologies such as single carrier radio transmission technology (1xRTT), Evolution-Data Optimized (EVDO) technology, general packet radio service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, 3rd Generation Partnership Project (3GPP®) including 3G, 4th Generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards setting organizations, other long distance protocols, or other data transfer technologies.

[0118] Command group 916 may be transmitted or received over network 980 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 964) and utilizing any one of a plurality of well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, command 916 may be transmitted or received using a transmission medium via coupling 972 (e.g., a peer-to-peer coupling) to device 970. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" include any non-transitory medium that can store, encode, or carry command group 916 for execution by machine 900 and include digital or analog communication signals or other non-transitory media that enable such software communication. Accordingly, the terms "transmission medium" and "signal medium" should be understood to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose one or more characteristics are set or changed so as to encode information in the signal.

[0119] The terms "machine-readable medium", "computer-readable medium", and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Accordingly, these terms include both storage devices / media and carrier waves / modulated data signals.

[0120] Embodiments of this solution are implemented by one or a combination of hardware, firmware, and software. Embodiments may also be implemented as a set of instructions stored in a computer-readable storage device that can be read and executed by at least one processor to perform the operations described herein. The computer-readable storage device may include any non-volatile information in a form readable by a machine (e.g., a computer). For example, the computer-readable storage device may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, cloud servers, or other storage devices and media. Some embodiments may be configured to include one or more processors and have a set of instructions stored in a computer-readable storage device. The following description and the referenced drawings fully describe specific embodiments and enable those skilled in the art to implement them. Other embodiments may include structural, logical, electrical, process, and other changes. Parts and features of some embodiments may be included in or replaced by those of other embodiments. The embodiments recited in the claims include all valid equivalents of those claims.

[0121] It should be understood that the description of one or more method steps does not exclude the existence of additional method steps or intermediate method steps between the explicitly stated steps. Similarly, the description of one or more components in a device or system does not exclude the existence of additional components or intermediate components between the explicitly stated components.

[0122] The above description includes references to the accompanying drawings which form a part of the detailed description. The drawings illustrate specific embodiments in which the invention may be practiced by way of example. These embodiments are also referred to herein as "examples". Such examples may include other elements in addition to those shown or described. However, the inventors also contemplate examples in which only the elements shown or described are provided. The inventors also contemplate examples that use any combination or permutation of the elements (or one or more aspects thereof) shown or described with respect to a particular example (or one or more aspects thereof) or with respect to one or more other examples (or one or more aspects thereof) shown or described in this document.

[0123] It should be understood that the description of one or more method steps does not exclude the existence of additional method steps or intermediate method steps between the explicitly stated steps. Similarly, the description of one or more components within a device or system does not exclude the existence of additional components or intermediate components between the explicitly stated components.

[0124] In some cases, embodiments of the disclosed technology include a system configured to utilize a machine learning algorithm to identify modified binary image data submitted to an image recognition system. In some embodiments, the binarization defense system may utilize machine learning and leverage the interaction / review with a person of a suspicious pattern to help teach the defense algorithm and improve the detection of other defects.

[0125] In the event of any conflicting usage between this document and any document cited, the usage of this document shall prevail.

[0126] In this book, although it is common in patent documents, the English terms "a" or "an" are used to include one or more, independent of any other usage or use of "at least one" or "one or more". In this book, the English term "or" is used to refer to non-exclusive or, and unless otherwise indicated, "A or B" includes "A but not B", "B but not A", and "A and B". In this book, the English terms "including" and "in which" are used as plain English synonyms for the terms "comprising" and "wherein", respectively. Also, in the appended claims, the English terms "including" and "comprising" are open-ended, i.e., systems, devices, articles, compositions, formulations, or processes that include elements in addition to those recited after such terms are considered to be within the scope of the claims. Also, in the appended claims, the terms "first", "second", "third", etc. are used merely as labels and are not intended to introduce numerical requirements for their objects.

[0127] Geometric terms such as "parallel", "perpendicular", "round", or "square" are not intended to require absolute mathematical precision unless the context indicates otherwise. Rather, such geometric terms anticipate variations due to manufacturing or equivalent functionality. For example, if an element is described as round or generally round, elements that are not exactly circular (e.g., slightly elliptical or polygonal) are also included in this description.

[0128] The method examples described in this book can be at least partially executed by a machine or computer. Some embodiments may include a computer-readable medium or machine-readable medium encoded with a set of instructions operable to configure an electronic device to execute the methods described in the above embodiments. Embodiments of such methods may include code such as microcode, assembly language code, high-level language code, etc. Such code may include a set of computer-readable instructions for performing various methods. The code may form part of a computer program product. Further, in one embodiment, the code may be stored, for example, during execution or at other times, on one or more volatile, persistent, or non-volatile tangible computer-readable media. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memory (RAM), read-only memory (ROM), etc.

[0129] The above description is illustrative and is not intended to be limiting. For example, the embodiments described above (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, for example, by those skilled in the art after reviewing the above description. The abstract is provided to comply with 37 CFR 1.72(b) of the United States Patent Practice Rules to allow the reader to quickly find the essence of the technical disclosure. It is submitted with the understanding that the abstract is not to be used to interpret or limit the scope or meaning of the claims. Also, in the above detailed description, various features may have been grouped together to organize the disclosure. This should not be construed as intending that disclosed features not recited in the claims are essential to any claim. Rather, the subject matter of the invention may lie in less than all of the features of a particular disclosed embodiment. Accordingly, the appended claims are recited as examples or embodiments in the detailed description, each claim standing on its own as a separate embodiment, and such embodiments are contemplated to be combinable with each other in various combinations or permutations. The scope of the present invention should be determined with reference to the appended claims along with the full scope of equivalents to which those claims are entitled. Hereinafter, preferred embodiments of the present invention will be described item by item. Embodiment 1 A computer-executed method for detecting the vulnerability of a model for binary image classification, comprising: Receiving binary image data by a computer system, the computer system being configured to detect pixel values in the binary image data and present non-machine language values related to the binary image data; Determining by the computer system that the binary image data further includes at least one pixel value that has been changed so as to change the non-machine language value related to the binary image data when read by an image recognition system; Warning by the computer system to the image recognition system to review the binary image data; A computer-executed method including the above. Embodiment 2 The method according to embodiment 1, wherein the step of determining that the binary image data includes a changed pixel value includes determining that the first artificial intelligence model and the second artificial intelligence model of the image recognition system have been simultaneously attacked. Embodiment 3 The computer-executed method according to embodiment 1 or 2, wherein the first artificial intelligence model of the image recognition system classifies a portion representing a numerical amount written in numbers of the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the numerical amount written in characters of the binary image data. Embodiment 4 The computer-executed method according to any one of embodiments 1 to 3, wherein the step of determining that the two models have been simultaneously attacked includes determining that a non-targeted attack using a shadowed combined attack on the recognition system has been used on at least one of the two models. Embodiment 5 The computer-executed method according to any one of embodiments 1 to 4, further comprising determining whether a targeted version of a shadowed combined attack on the recognition system has been executed twice and has attacked both models. Embodiment 6 One or more persistent computer-readable media storing a set of instructions, which when executed by a computer system configured to review binary numbers, cause the computer system to perform at least Receiving binary image data by the computer system, wherein the computer system is configured to detect pixel values within the binary image data and present non-machine language values associated with the binary image data; Determining by the computer system that the binary image data further includes at least one pixel value that has been changed to change the non-machine language value associated with the binary image data when read by an image recognition system; Warning, by the computer system, the image recognition system to review the binary image data; and A persistent computer-readable medium that causes the above to be executed. Embodiment 7 The step of determining that the binary image data includes a changed pixel value includes determining that a first artificial intelligence model and a second artificial intelligence model of the image recognition system have been attacked simultaneously, according to the persistent computer-readable medium of Embodiment 6. Embodiment 8 The first artificial intelligence model of the image recognition system classifies a portion representing a numerical amount written in numbers in the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the numerical amount written in characters in the binary image data, according to the persistent computer-readable medium of Embodiment 6 or 7. Embodiment 9 The step of determining that the two models have been attacked simultaneously includes determining that a non-targeted attack using a shadowed combined attack on the recognition system has been used on at least one of the two models, and the method optionally further includes determining whether a targeted version of the shadowed combined attack on the recognition system has been executed twice to attack both models, according to the persistent computer-readable medium of any one of Embodiments 6 to 8. Embodiment 10 The binary image data is at least one of an alphanumeric string or a check, and the image recognition system is optionally an optical character recognition system, according to the persistent computer-readable medium of any one of Embodiments 6 to 9. Embodiment 11 A computer-executed method for determining the vulnerability of a model for binary image classification, comprising: A step of receiving binary image data by a computer system, wherein the computer system is configured to test a plurality of pixel value sets within the binary image data and present a non-machine language value associated with the binary image data in an image recognition system, the step; The step of the computer system further determining that the binary image data further includes at least one pixel value that has been changed so as to change the non-machine language value associated with the binary image data when read by the image recognition system; The step of the computer system warning that the image recognition system is vulnerable to spoofing attacks; Including, The binary image data is optionally an alphanumeric string or a check, a computer-executed method. Embodiment 12 The step of determining that the binary image data includes a changed pixel value includes determining that the first artificial intelligence model and the second artificial intelligence model of the image recognition system have been attacked simultaneously, the computer-executed method according to Embodiment 11. Embodiment 13 The first artificial intelligence model of the image recognition system classifies a portion representing a numerical amount written in numbers of the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the numerical amount written in characters of the binary image data, the computer-executed method according to Embodiment 11 or 12. Embodiment 14 The step of determining that the two models have been attacked simultaneously includes determining that a non-targeted attack using a shadowed combined attack on the recognition system has been used on at least one of the two models, the computer-executed method according to any one of Embodiments 11 to 13. Embodiment 15 The computer-executed method according to any one of Embodiments 11 to 14, further including a step of determining whether a targeted version of a shadowed combined attack on the recognition system has been executed twice and attacked both models.

Explanation of Symbols

[0130] 100 AI operation environment 200 Model confidentiality protection system 220 Red group engine (or model evaluation engine) 240 Firewall 302 Narrative amount 304 Numerical amount 306 LAR part 308 CAR part 310 Combined amount value 312 Combined recognition reliability value 400 External Data

Claims

1. A computer-executed method for detecting the vulnerability of a model for binary image classification, comprising: receiving binary image data by a computer system, wherein the computer system is configured to detect pixel values in the binary image data and present values in a readable format related to the binary image data; judging by the computer system that the binary image data further includes at least one pixel value that has been changed so as to change the value in the readable format related to the binary image data when read by an image recognition system; warning by the computer system to prompt the image recognition system to review the binary image data; and a computer-executed method including the above steps.

2. The image recognition system performs recognition using two artificial intelligence models, The step of judging that the binary image data includes changed pixel values includes judging that the first artificial intelligence model and the second artificial intelligence model among the two artificial intelligence models of the image recognition system have been attacked simultaneously. The computer-executed method according to Claim 1.

3. The first artificial intelligence model of the image recognition system classifies a part representing the amount written in numbers in the binary image data, and the second artificial intelligence model of the image recognition system classifies a second part representing the amount written in characters in the binary image data. The computer-executed method according to Claim 2.

4. The step of judging that the first artificial intelligence model and the second artificial intelligence model have been attacked simultaneously includes judging that a non-targeted attack using a shadowed combined attack on the recognition system, which includes repeatedly evaluating the gains of a plurality of pixels in the binary image data based on the spatial and temporal correlation relationships between pixel gains, has been used on at least one of the first artificial intelligence model and the second artificial intelligence model. The computer-executed method according to any one of Claims 2 or 3.

5. The computer-executed method according to any one of Claims 2 to 4 further includes a step of judging whether a targeted version of an attack including repeatedly evaluating the gains of a plurality of pixels in the binary image data based on the spatial and temporal correlation relationships between pixel gains, which is a shadowed combined attack on the recognition system, has been executed twice and attacked both models.

6. One or more persistent computer-readable media storing a set of instructions, which, when executed by a computer system configured to review binary numbers, cause the computer system to perform at least Receive binary image data by the computer system, wherein the computer system is configured to detect pixel values within the binary image data and present a value in a readable format associated with the binary image data; Determine by the computer system that the binary image data further includes at least one pixel value that has been modified to change the value in the readable format associated with the binary image data when read by an image recognition system; Warn by the computer system the image recognition system to review the binary image data; A persistent computer-readable medium that causes the above to be executed. **Claim 7** The image recognition system performs recognition using two artificial intelligence models. The step of determining that the binary image data includes a modified pixel value includes determining that a first artificial intelligence model and a second artificial intelligence model among the two artificial intelligence models of the image recognition system have been attacked simultaneously. The persistent computer-readable medium according to claim 6. **Claim 8** The first artificial intelligence model of the image recognition system classifies a portion representing the amount written in numbers of the binary image data, and the second artificial intelligence model of the image recognition system classifies a second portion representing the amount written in characters of the binary image data. The persistent computer-readable medium according to claim 7. **Claim 9** The step of determining that the first artificial intelligence model and the second artificial intelligence model have been attacked simultaneously includes a non-targeted attack using a shadowed combined attack on the recognition system, which repeatedly evaluates the gains of a plurality of pixels of the binary image data based on the spatial and temporal correlation relationships between pixel gains. Determining that the attack has been used on at least one of the first artificial intelligence model and the second artificial intelligence model, and the set of instructions further causes the computer system to execute a step of determining whether a targeted version of the shadowed combined attack on the recognition system has been executed twice and has attacked both models. The persistent computer-readable medium according to any one of claims 7 or 8. **Claim 10** The binary image data is at least one of an alphanumeric string or a check, and the image recognition system is optionally an optical character recognition system, the persistent computer-readable medium according to any one of claims 7 to 9.

11. A computer-executed method for determining the vulnerability of a model for binary image classification, Receiving binary image data by a computer system, wherein the computer system is configured to test a set of a plurality of pixel values in the binary image data and present a value in a readable format related to the binary image data in the image recognition system, the step; Determining by the computer system that the binary image data further includes at least one pixel value changed to change the value in the readable format related to the binary image data when read by the image recognition system; Warning by the computer system that the image recognition system is vulnerable to spoofing attacks Including, The binary image data is optionally an alphanumeric string or a check, a computer-executed method.

12. The image recognition system performs recognition using two artificial intelligence models, The step of determining that the binary image data includes a changed pixel value includes determining that the first artificial intelligence model and the second artificial intelligence model among the two artificial intelligence models of the image recognition system are simultaneously attacked. The computer-executed method according to claim 11.

13. The first artificial intelligence model of the image recognition system classifies a portion representing the amount written in numbers of the binary image data, and the second artificial intelligence model of the image recognition system represents a second portion representing the amount written in characters of the binary image data. The computer-executed method according to claim 12, which classifies.

14. The step of determining that the first artificial intelligence model and the second artificial intelligence model are simultaneously attacked is a shadow combination attack on the recognition system, based on the spatial and temporal correlation relationships between pixel gains, and a plurality of pixels of the binary image data. The computer-executed method according to any one of claims 12 or 13, including determining that a non-targeted attack using an attack that repeatedly evaluates the gain of is used for at least one of the first artificial intelligence model and the second artificial intelligence model.

15. The computer-executed method according to any one of claims 12 to 14, further comprising the step of determining whether a targeted version of the shadow combination attack on the recognition system has been executed twice to attack both models.

Images