Method and related network and node for protecting pattern classification nodes from malicious requests

Abstract

A method of operating a protection node for protecting a pattern classification node from malicious requests can be provided. The protection node can receive a request from a user node, the request including an original pattern to be classified by a machine learning algorithm executed by the pattern classification node. The protection node can add noise to the original pattern to generate a noisy pattern. The protection node can obtain a first classification of the noisy pattern based on processing of the noisy pattern by a first clone of the machine learning algorithm at the protection node; obtain a second classification of the original pattern based on forwarding a request to process the original pattern by the machine learning algorithm executed at the pattern classification node; and compare the first classification and the second classification to determine whether the first classification and the second classification satisfy a defined similarity rule. The protection node can use the comparison to manage requests from the user node.

Description

Technical Field

[0001] The present disclosure generally relates to protected networks, and more particularly to protecting pattern classification nodes from malicious requests. Background Art

[0002] Machine learning algorithms, especially neural networks, can be vulnerable to certain attacks, where malicious users send carefully designed inputs to confuse the neural network and can force it to produce incorrect predictions / classifications. A well-known example is as Figure 1 shown, where noise can be added to a panda picture and then fed back into a convolutional neural network (CNN, VGG-16) to misclassify the panda picture as a gibbon.

[0003] This problem may generally be harmless and can be improved by retraining the original neural network to correctly classify its inputs. However, in some cases this method can be engineered to cause more serious problems - for example, by adding a small amount of noise to an image of a stop sign, it can be mislabeled as something else, causing an autonomous vehicle to ignore it and potentially leading to an accident. Summary of the Invention

[0004] According to some embodiments of the inventive concept, a method for protecting a pattern classification node from malicious requests, performed by a protection node, may be provided. The protection node may receive a request from a user node, the request including an original pattern to be classified by a machine learning algorithm performed by the pattern classification node. The protection node may add noise to the original pattern to generate a noisy pattern. The protection node may obtain a first classification of the noisy pattern based on processing of the noisy pattern by a first clone of a machine learning algorithm performed by the protection node. The protection node may obtain a second classification of the original pattern based on forwarding the request to process the original pattern by the machine learning algorithm performed by the pattern classification node. Further, the protection node may compare the first classification and the second classification to determine whether the first classification and the second classification satisfy a defined similarity rule. The protection node may use the comparison to manage requests from the user node.

[0005] According to some other embodiments of the inventive concept, a protection node may be provided. The protection node may include at least one processor and at least one memory connected to the at least one processor to perform operations. The operations may include: receiving a request from a user node, the request including an original pattern to be classified by a machine learning algorithm performed by a pattern classification node. The operations may further include adding noise to the original pattern to generate a noisy pattern and obtaining a first classification of the noisy pattern based on processing of the noisy pattern by a first clone of a machine learning algorithm performed by the protection node. The operations may further include obtaining a second classification of the original pattern based on forwarding a request to process the original pattern by a machine learning algorithm performed by the pattern classification node. Additionally, the operations may include comparing the first classification and the second classification to determine whether the first classification and the second classification satisfy defined similarity rules. The operations may further include using the comparison to manage requests from the user node.

[0006] According to some embodiments, a computer program including instructions may be provided, which when executed on at least one processor causes the at least one processor to perform the method performed by the protection node.

[0007] According to some embodiments, a computer program product may be provided, the computer program product including a non-transitory computer-readable medium storing instructions, which when executed on at least one processor causes the at least one processor to perform the method performed by the protection node.

[0008] An operational advantage that one or more embodiments may provide is that by comparing the first classification and the second classification to determine whether the first classification and the second classification satisfy defined similarity rules and managing requests from the user node when the first classification and the second classification do not satisfy the defined similarity rules, the protection node can protect the pattern classification node from malicious requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of the inventive concept. In the drawings:

[0010] Figure 1 is an example showing that noise is added to a panda picture and then fed back to a convolutional neural network (CNN, VGG-16) to misclassify the panda picture as a gibbon;

[0011] Figure 2 is a schematic diagram showing a protection node for protecting a pattern classification node from malicious requests according to some embodiments of the inventive concept.

[0012] Figure 3is a message diagram showing operations / messages of a protection node for protecting a pattern classification node according to some embodiments of the inventive concept;

[0013] Figure 4 is a flowchart showing operations of a protection node according to some embodiments of the inventive concept;

[0014] Figure 5 is a flowchart showing operations of a protection node according to some embodiments of the inventive concept;

[0015] Figure 6 is a flowchart showing operations of a protection node according to some embodiments of the inventive concept;

[0016] Figure 7 is a block diagram showing a protection node according to some embodiments of the inventive concept; and

[0017] Figure 8 is a block diagram showing a pattern classification node according to some embodiments of the inventive concept. Detailed Description

[0018] Some embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. However, other embodiments are included within the scope of the subject matter disclosed herein, and the disclosed subject matter should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0019] In general, all terms used herein should be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is explicitly given and / or is implied from the context in which they are used. All references to an / one / the element, apparatus, component, part, step, etc. should be construed openly as referring to at least one instance of the element, apparatus, component, part, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein need not be performed in the exact order disclosed, unless a step is explicitly described as being after or before another step and / or where it is implied that a step must be after or before another step. Where applicable, any feature of any embodiment disclosed herein may be applied to any other embodiment. Similarly, any advantage of any embodiment may be applied to any other embodiment, and vice versa. Other objectives, features, and advantages of the appended embodiments will be apparent from the following description.

[0020] One way to protect neural networks from malicious attacks can be a generative adversarial network (GAN), where an initial training dataset can be augmented with random noise, then relabeled, and then can be used again to train the original neural network with augmented (“noisy”) inputs, thereby improving the original neural network. A limitation of this method is that it may require generating an entirely new dataset, which may take time, while the initial neural network is still running and remains vulnerable to such attacks. Additionally, this method may not take into account known malicious inputs that may have successfully confused the neural network in the past.

[0021] Certain aspects and embodiments of the present disclosure can provide solutions to these and / or other challenges. According to some embodiments, a method can keep track of such malicious requests sent from user nodes to a machine learning algorithm, can record the actions / intentions of the malicious requests, and can make the malicious user think that it has successfully deceived the machine learning algorithm, block these requests, or monitor these requests for potential future blocking. Since the malicious requests can be recorded, they can be used in the background to effectively improve and strengthen the initial machine learning algorithm being protected without the attacker's knowledge.

[0022] In some embodiments, an enhanced hypertext transfer protocol (http) proxy network node can process machine learning (ml)-based http requests and add noise to the patterns in the requests.

[0023] The maliciously added noise can not be random noise. The maliciously added noise can be a modification to an input that looks like noise but can be a carefully chosen modification to explore weaknesses (e.g., overfitting to training data) of the machine learning algorithm / model to deceive the machine learning model into making incorrect predictions. In some embodiments, random noise can be added to the input containing the malicious modification to disrupt the effect of these modifications, thereby sparing the machine learning algorithm from making the classification desired by these malicious modifications. Adding a small amount of random noise to a generally non-maliciously modified input should not affect the classification result. If the output of running the machine learning algorithm on a user input with and without the added random noise deviates significantly, this can indicate that the input has a malicious modification, or the input can represent an extreme case that may be misinterpreted by the machine learning algorithm. In either case, in some embodiments, the user sending the request containing the input can be recorded and the input from that user can be used to retrain the machine learning algorithm.

[0024] Figure 2FIG. 0 is a schematic diagram of a protection node 212 for protecting a pattern classification node 222 including a machine learning algorithm from malicious requests. The machine learning algorithm may be a machine learning-based algorithm / model that includes an input, a prediction, classification, or decision based on the input, and is expressed as an output by subsequent programming instructions executed by a processor. The machine learning algorithm may include any form of machine learning algorithm, including neural networks. Although the subject matter described herein may be implemented using any suitable components in any suitable type of system, the embodiments disclosed herein are described with respect to a protection node 212 including a pattern classification node 222 including a machine learning algorithm and a honeypot server 216 including a first clone of the machine learning algorithm residing at the pattern classification node 222. The clone of the machine learning algorithm may be executed and modified without directly affecting the machine learning algorithm residing at the pattern classification node 222. The honeypot server 216 may be set as a decoy to lure malicious attackers / user nodes that send malicious requests to the pattern classification node 222. The honeypot server 216 may be a Hypertext Transfer Protocol (http) proxy for the pattern classification node 222. The honeypot server 216 may process machine learning (ml)-based http requests received via the communication network 210. By including a clone of the machine learning algorithm residing at the pattern classification node 222 at the honeypot server 216, the protection node 212 may mimic the pattern classification node 222. By mimicking the pattern classification node 222, the protection node 212 may receive requests sent by the user node 214 to the pattern classification node 222.

[0025] In addition, the honeypot server 216 may protect the pattern classification node 222 from malicious requests sent from the user node 214 to the pattern classification node 222. These requests may attempt to cause the pattern classification node 222 to incorrectly classify the original pattern included in the request from the user node 214. As discussed in more detail below, the honeypot server 216 may execute a first clone of the machine learning algorithm. The protection node 212 may obtain identification information about the user node that sent the request received by the protection node 212, and may provide information about the request from the user node 214 to the training node 224 to train a second clone of the machine learning algorithm residing at the training node 224 in the background. The honeypot server 216 may be collocated at the protection node 212. Alternatively, the honeypot server 216 may be at a separate location on the communication network 210 having a network connection to the protection node 212. In addition, the honeypot server 216 may include a plurality of honeypot servers disposed on the communication network 210 that form a single honeypot server.

[0026] For simplicity, Figure 2Only depicts a pattern classification node 222 that includes a machine learning algorithm. In practice, the communication network 210 may also include any additional units suitable for supporting the machine learning algorithm, such as any other pattern classification node or neural network node.

[0027] For simplicity, Figure 2 Also only depicts a protection node 212 for protecting the pattern classification node 222 from malicious attacks sent to the pattern classification node 222. In practice, any additional units suitable for supporting the protection of the pattern classification node 222 may also be included, such as any other node or device of the neural network. Figure 2 Also only depicts a user node 214. In practice, the communication network 210 may also include additional user nodes. In addition, although Figure 2 Two separate databases 218 and 220 are depicted, but these databases may reside in a single database structure and / or may include additional databases.

[0028] The protection node 212 may be, but is not limited to, a node including a computer, a honeypot server, and a clone of the machine learning algorithm residing at the pattern classification node 222, as further discussed below with reference to Figure 6 The protection node 212 may communicate with the user node 214. The protection node 212 may receive requests sent from the user node 214 to the pattern classification node 222. The protection node 212 may also communicate with the pattern classification node 222 and the training node 224. The communication with the training node 224 may be directly via the communication network 210 or indirectly via the pattern classification node 222. The protection node 212 may also communicate with the database 218 to store and / or collect identifiers associated with the user node 214 into a distrust list in the database 218, as further discussed below. In addition, the protection node 212 may also communicate with the database 220 to store and / or collect information obtained from requests from the user node 214, as further discussed below.

[0029] Now, in the Figure 3 message diagram and Figures 4 - 6 operation flowchart context of, describe these and other related operations that may be performed by the protection node 212.

[0030] Figure 3 Shows an example in which the machine learning algorithm residing at the pattern classification node 222 has no prior information about the intent of requests from the user node and no machine learning model that has been trained in a way to avoid malicious requests. In Figure 3In this case, this is indicated by the white_list output 309 (which is "dont_know") and also by a clone 303 of the machine learning algorithm residing in the honeypot http server 216 (which may be the same as the original machine learning algorithm 301 residing at the pattern classification node 222).

[0031] The HTTP server is an exemplary host for the original machine learning algorithm, which includes but is not limited to TensorFlow Serving or other examples, or machine learning runtimes.

[0032] Continuing to refer to Figure 3 , according to some embodiments, at 301, the author of the machine learning algorithm can deploy the machine learning algorithm on the http server (e.g., deploy(M1)). The http server can be located at, but is not limited to, the network node 222. At 303, a clone of the machine learning algorithm can be deployed at the honeypot http server (e.g., M1_CLONE = copy(M1)). The honeypot http server can be located at the protection node 212.

[0033] At 305, the user node 214 can send a request to the pattern classification node 222, which can be received at the load balancer of the protection node 212 (e.g., request = predict(v1,M1)). The request can include an original pattern to be classified by the machine learning algorithm residing at the pattern classification node 222. The original pattern can include but is not limited to images, video frames, audio samples, and data stream samples (e.g., photos, optical characters, face images, handwriting, speech, fingerprints, text, shapes, etc.). At 307, the load balancer can check the white list of identifiers associated with the user node and / or the user in the database 218 to determine whether the identifier associated with the user node 214 is an unknown user (e.g., check(user)). At 309, the load balancer can determine that the identifier associated with the user node 214 is an unknown user (e.g., don't_know). At 311, the load balancer can send a request to the request scheduler to add noise to the original pattern included in the request from the user node 214 (e.g., n1 = add_noise(request)). The request scheduler can be included in the protection node 212.

[0034] At 313, the request scheduler can add noise to the original pattern to generate a noisy pattern and can send the noisy pattern to the honeypot server 216 (e.g., n1). At 315, based on the processing of the noisy pattern by the first clone of the machine learning algorithm, the honeypot server 216 can perform a first classification of the noisy pattern (e.g., r1).

[0035] At 317, the request scheduler may send a request to a machine learning algorithm residing at the pattern classification node 222 to classify an original pattern included in a request from the user node 214. At 319, based on the processing of the original pattern by the machine learning algorithm performed by the pattern classification node 222, the request scheduler may receive a classification (e.g., a second classification) (e.g., r2) of the original pattern from the machine learning algorithm.

[0036] At 321, the request scheduler may compare the first classification and the second classification to determine whether the first classification and the second classification satisfy a defined similarity rule (e.g., checkResponse(r1,r2)). The defined similarity rule may include rules where the first classification and the second classification match, substantially match, do not match, substantially do not match, or have a defined amount of similarity. At 323, the request scheduler may determine that the first classification and the second classification do not satisfy the defined similarity rule, e.g., because they substantially do not match. When the first classification and the second classification do not satisfy the defined similarity rule, the request scheduler may send an instruction to the database 218 to add an identifier associated with the user node 214 to distrust (e.g., add_user(user,distrust)). The distrust list may be stored, but is not limited to, in the database 218. The database 218 may be directly or indirectly connected to the communication network 210, or may be included in the protection node 212. By adding an identifier associated with the user node 214 to the distrust list, requests sent from the user node 214 to the pattern classification node 222 may be blocked or monitored for potential future blocking.

[0037] At 325, when the first classification and the second classification do not satisfy the defined similarity rule, the request scheduler may send an instruction to store the request in a log request (e.g., request). The log request may be stored, but is not limited to, in the database (220). The database 220 may be directly or indirectly connected to the communication network 210, or may be included in the protection node 212.

[0038] At 327, the request scheduler may send an instruction to the load balancer to send the second classification to the user node 214 because the first classification and the second classification do not satisfy the defined similarity rule (e.g., r2). At 329, the load balancer may send the second classification (e.g., r2) to the user node 214. By sending the second classification to the user node 214, the user node 214 may consider or determine that it has successfully deceived the machine learning algorithm residing at the pattern classification node 214 into incorrectly classifying the original pattern included in the request sent from the user node 214.

[0039] At 331, the request scheduler may send a request to a log request at database 2220 to collect the stored requests from the log request (e.g., collect(M1)). At 333, the log request may send the stored requests to the request scheduler (e.g., noise_requests). At 335, the request scheduler may send a request to training node 224. The training node 224 may include a second clone of the machine learning algorithm residing at pattern classification node 222. The request sent to the training node 224 may include a request to train the second clone of the machine learning algorithm to obtain an incorrect result that the second classification of the original pattern in the requests from user node 214 using the stored requests obtained from database 220 (e.g., M2 = train(M1, noisy_requests)). At 337, the training node 224 may send the result of the training to the machine learning algorithm at pattern classification node 222. The result may include that the machine learning algorithm identifies that the second classification of the original pattern in the requests from user node 214 is incorrect. Since the log request at database 220 may include records of requests from user node 214, the second clone of the machine learning algorithm at training node 224 may use the requests from user node 214 in the background to improve and strengthen the protected machine learning algorithm residing at pattern classification node 222 without the user node 214 knowing about the background training.

[0040] It will be understood that Figure 3 the message / operation sequence shown is for illustrative purposes, and other embodiments may modify the sequence and / or the order of the sequence without departing from the scope of the inventive concept. Additionally, it will be understood that multiple requests from one or more user nodes may be included in these messages / operations.

[0041] Now reference will be made to Figures 4 - 6 the flowchart of to disclose the operations of protection node 700 (e.g., computer 710, etc.) for protecting pattern classification node 222 from malicious requests. For example, protection node 212 may be implemented using the structure of protection node 700 from Figure 7 where computer 710 has a honeypot server 216 in memory 730. The honeypot server 216 includes a first clone of the machine learning algorithm program residing at pattern classification node 222. Protection node 710 has a first clone of the machine learning algorithm in honeypot server 216 residing in memory 730 of protection node 700 such that when the instructions in memory 730 of protection node 700 are executed by processor 720 of protection node 700, processor 720 performs the following with respect to Figures 4 - 6The corresponding operations discussed. Thus, the processor 720 of the protection node 700 can send and / or receive communications to / from one or more nodes / entities / servers (e.g., nodes 214, 222, and 224 and / or databases 218 and 220) of the communication network 210 through the interface 750 of the protection node 600.

[0042] Initially referring Figure 4 , at block 401, the processor 720 of the protection node 700 (e.g., the protection node 212) can receive a request from the user node 214 that includes an original pattern (e.g., Figure 3 305 in

[0043] to be classified by a machine learning algorithm residing at the pattern classification node 222). Figure 3 311 in

[0044] At block 403, the processor 720 of the protection node 700 can add noise to the original pattern to generate a noisy pattern (e.g., Figure 3 311 in Figure 3 ). By sending a request to classify the noisy pattern to a clone of the machine learning algorithm residing in the honeypot server 216, the processor 720 of the protection node 700 can obtain a first classification of the noisy pattern (e.g., Figure 3 313 in

[0045] At block 407, based on forwarding a request to process the original pattern by the machine learning algorithm performed by the pattern classification node 222, the processor 720 of the protection node 700 can obtain a second classification of the original pattern. The processor 720 of the protection node 700 can receive the second classification of the pattern from the machine learning algorithm (e.g., Figure 3 319 in

[0046] At block 409, the processor 720 of the protection node 700 can compare the first classification and the second classification to determine whether the first classification and the second classification satisfy the defined similarity rules (e.g., Figure 3 321 in

[0047] At block 411, the processor 720 of the protection node 700 can use the comparison to manage the request from the user node 214. The processor 720 of the protection node 700 can use the comparison to manage the request from the user node 214 in several ways.

[0048] For example, when the first classification and the second classification do not satisfy the defined similarity rules, the processor 720 of the protection node 700 adds the identifier associated with the user node 214 to the distrust list in the first database (218), where the protection node blocks requests received from identifiers included in the distrust list or monitors requests received from identifiers included in the distrust list for potential future blocking (e.g., Figure 3 323 in Figure 3 . Further, when the first classification and the second classification do not satisfy the defined similarity rules, the processor 720 of the protection node 700 may send the second classification to the user node 214 (e.g.,

[0049] 327 and 329 in Figure 5 ; 509).

[0050] Further, referring to Figure 6, the processor 720 of the protection node 700 may repeat (e.g., 311, 601) adding noise to the original pattern to generate a set of noisy patterns, and each noisy pattern in the set may have different added noise. For each noisy pattern in the set, the processor 720 of the protection node 700 may obtain (e.g., 313, 315, 603) a classification based on the processing by the first clone of the machine learning algorithm executed by the protection node 212. The comparison (e.g., 321) may include performing a comparison of the second classification with each classification for each noisy pattern in the set of noisy patterns to determine whether the second classification satisfies a defined similarity rule compared to each classification for each noisy pattern in the set. The use (e.g., 323, 325, 327, 329, 331, 333, 335, 337, 411) may include using the comparison to manage requests from the user node 214.

[0051] Figure 4 The various operations may be optional with respect to some embodiments. For example, according to some embodiments, the operation of block 413 may be optional. Additionally, Figure 5 and Figure 6 the operations may be optional. Additionally, it will be understood that Figures 4 - 6 the sequence of operations shown is for illustrative purposes, and other embodiments may modify the sequence and / or the order of the sequence without departing from the scope of the inventive concept. Additionally, it will be understood that multiple requests from one or more user nodes may be included in these operations.

[0052] Figure 8 is a block diagram showing a pattern classification node 800 (e.g., computer 810, etc.) that is configured to: deploy a first clone of a machine learning algorithm at the protection node 212; generate a second classification of the original pattern from the user node 214 based on the processing of the original pattern by the machine learning algorithm residing at the pattern classification node 222; and receive the result of further training by the training node 224 to identify that the second classification of the original pattern in the request from the user node 214 is incorrect. For example, the pattern classification node 222 may be implemented using the structure of the pattern classification node 800 from Figure 8 where the computer 810 has a machine learning algorithm in the memory 830 such that when the instructions in the memory 830 of the pattern classification node 800 are executed by the processor 820 of the pattern classification node 800, the processor 820 performs the above with respect to Figure 3The corresponding operations discussed. Accordingly, the processor 820 of the protection node 800 can send and / or receive communications to / from one or more nodes / entities / servers (e.g., nodes 212, 214, and 224 and / or databases 218 and 220) of the communication network 210 via the interface 850 of the pattern classification node 800.

[0053] Although various embodiments have been described in which the machine learning algorithm and the first and second clones of the machine algorithm reside in the memory as software, the machine learning algorithm and the clones may alternatively or additionally be embodied in analog circuits and / or discrete digital circuits (e.g., ASICs).

[0054] In the foregoing description of the various embodiments of the inventive concept, it should be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the inventive concept. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the inventive concept pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art, and will not be understood in an idealized or overly formal sense unless expressly so defined herein.

[0055] When an element is referred to as being "connected," "coupled," "responsive," or a variation thereof to / with another element, it can be directly connected, coupled, or responsive to / with the other element or an intervening element (if any). In contrast, when an element is referred to as being "directly connected," "directly coupled," "directly responsive," or a variation thereof to / with another element, there is no intervening element. The same numerals always refer to the same elements. Further, "coupled," "connected," "responsive," or a variation thereof as used herein can include wireless coupling, connection, or responsiveness. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. For brevity and / or clarity, well-known functions or constructions may not be described in detail. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0056] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments without departing from the teachings of the inventive concept. In the specification, the same reference numerals or reference signs denote the same or similar elements.

[0057] As used herein, the terms "comprising", "including", "having" or variations thereof are open-ended and include one or more of the stated features, integers, elements, steps, components or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Further, as used herein, the general abbreviation "e.g." (from the Latin phrase "exempli gratia") may be used to introduce or specify general examples or a plurality of examples of previously mentioned items and is not intended to limit such items. The general abbreviation "i.e." (from the Latin phrase "id est") may be used to specify a particular item from a more general recitation.

[0058] Exemplary embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It should be understood that the blocks of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to a processor circuit of a general purpose computer circuit, a special purpose computer circuit, and / or other programmable data processing circuits to produce a machine such that the instructions executed via the processor of the computer and / or other programmable data processing means transform and control transistors, values stored in memory locations, and other hardware components within such circuits to implement the functions / actions specified in the block diagrams and / or flowchart blocks, thereby creating means (functions) and / or structures for implementing the functions / actions specified in the block diagrams and / or flowchart blocks.

[0059] These computer program instructions can also be stored in a tangible computer-readable medium, which can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in the block diagrams and / or flowchart blocks. Thus, embodiments of the inventive concept can be embodied in hardware and / or in software (including firmware, resident software, microcode, etc.) running on a processor such as a digital signal processor, which may collectively be referred to as "circuitry", "module" or variations thereof.

[0060] It should also be noted that in some alternative implementations, the functions / actions shown in the blocks may not occur in the order shown in the flowchart. For example, two consecutive blocks shown may actually be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order, depending on the functions / actions involved. Additionally, the functions of a given block in the flowchart and / or block diagram may be divided into multiple blocks and / or the functions of two or more blocks in the flowchart and / or block diagram may be at least partially integrated. Finally, other blocks may be added / inserted between the shown blocks, and / or blocks / operations may be omitted, without departing from the scope of the inventive concept. Further, although some of the schematic diagrams include arrows on the communication paths to indicate the main direction of communication, it should be understood that communication may occur in the direction opposite to that of the shown arrows.

[0061] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the inventive concept. All such variations and modifications are intended to be included within the scope of the inventive concept. Accordingly, the subject matter disclosed above should be considered illustrative and not restrictive, and the examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments that fall within the spirit and scope of the inventive concept. Thus, to the maximum extent permitted by law, the scope of the inventive concept will be determined by the broadest permissible interpretation of this disclosure, including the examples of embodiments and their equivalents, and should not be limited or restricted by the foregoing detailed description.

Claims

1. A method for protecting a pattern classification node from malicious requests, performed by a protection node, the method comprising: Receiving a request from a user node, the request including an original pattern to be classified by a machine learning algorithm performed by the pattern classification node; Adding noise to the original pattern to generate a noisy pattern; Obtaining a first classification of the noisy pattern based on processing of the noisy pattern by a first clone of the machine learning algorithm performed by the protection node; Obtaining a second classification of the original pattern based on forwarding the request to process the original pattern by the machine learning algorithm performed by the pattern classification node; Comparing the first classification and the second classification to determine whether the first classification and the second classification satisfy defined similarity rules; and Using the comparison of the first classification and the second classification to manage the request from the user node.

2. The method according to claim 1, wherein Using the comparison of the first classification and the second classification to manage the request from the user node includes: When the first classification and the second classification do not satisfy the defined similarity rules, adding an identifier associated with the user node to a distrust list in a first database, wherein the protection node blocks requests received from identifiers included in the distrust list or monitors requests received from identifiers included in the distrust list for potential future blocking before forwarding requests received from identifiers included in the distrust list to the pattern classification node.

3. The method according to claim 2, further comprising: Repeating the receiving, the adding of the noise, the obtaining of the first classification, the obtaining of the second classification, the comparison; For each instance where it is determined that the first classification and the second classification obtained for one of the requests do not satisfy the defined similarity rules, storing information obtained from the one of the requests in a set of risk information included in a second database; Obtaining the set of risk information from the second database; And Forwarding the set of risk information to a second clone of the machine learning algorithm at a training node to use the set of risk information to train the second clone of the machine learning algorithm to identify that the second classification of the original pattern in the request is incorrect.

4. The method according to any one of claims 1 to 3, wherein Using the comparison of the first classification and the second classification to manage the request from the user node includes: When the first classification and the second classification do not satisfy the defined similarity rules, sending the second classification to the user node.

5. The method according to any one of claims 1 to 3, wherein The pattern includes one of the following: an image, a video frame, an audio sample, and a data stream sample.

6. The method according to claim 3, wherein The first database and the second database reside in a single database structure.

7. The method according to any one of claims 1 to 3, wherein The machine learning algorithm is a neural network.

8. The method according to any one of claims 1 to 3, wherein The protection node includes a honeypot server that executes the first clone of the machine learning algorithm.

9. The method according to any one of claims 1 to 3, further comprising: Repeatedly add noise to the original pattern to generate a set of noisy patterns, where each of the noisy patterns in the set has different added noise; For each of the noisy patterns in the set, obtain a classification based on the processing by the first clone of the machine learning algorithm performed by the protection node; wherein the comparison includes: performing a comparison of the second classification with each of the classifications for the set of noisy patterns to determine whether the second classification satisfies the defined similarity rule compared to each of the classifications for the set of noisy patterns; wherein the use further includes using the comparison of the second classification with each of the classifications for the set of noisy patterns to manage the request from the user node.

10. A protection node, comprising: at least one processor; at least one memory connected to the at least one processor and storing program code executed by the at least one processor to perform operations, the operations including: receiving a request from a user node, the request including an original pattern to be classified by a machine learning algorithm performed by a pattern classification node; adding noise to the original pattern to generate a noisy pattern; obtaining a first classification of the noisy pattern based on the processing of the noisy pattern by a first clone of the machine learning algorithm performed by the protection node; obtaining a second classification of the original pattern based on forwarding the request to process the original pattern by the machine learning algorithm performed by the pattern classification node; comparing the first classification and the second classification to determine whether the first classification and the second classification satisfy the defined similarity rule; and using the comparison of the first classification and the second classification to manage the request from the user node.

11. The protected node according to claim 10, wherein Using the comparison of the first classification and the second classification to manage the request from the user node includes: when the first classification and the second classification do not satisfy the defined similarity rule, adding the identifier associated with the user node to the distrust list in the first database, wherein before a request received from an identifier included in the distrust list is forwarded to the pattern classification node, the protection node blocks requests received from identifiers included in the distrust list, or monitors requests received from identifiers included in the distrust list for potential future blocking.

12. The protection node according to claim 11, further comprising: repeating the receiving, the adding of noise, the obtaining of the first classification, the obtaining of the second classification, the comparison; for each instance where it is determined that the first classification and the second classification obtained for one of the requests do not satisfy the defined similarity rule, storing the information obtained from the one of the requests in the risk information set included in the second database; obtaining the risk information set from the second database; and Forward the set of risk information to a second clone of the machine learning algorithm at the training node to use the set of risk information to train the second clone of the machine learning algorithm to identify that the second classification of the original pattern in the request is incorrect.

13. The protection node according to any one of claims 10 to 12, wherein Using the comparison of the first classification and the second classification to manage the request from the user node includes: When the first classification and the second classification do not meet the defined similarity rule, send the second classification to the user node.

14. The protected node according to any one of claims 10 to 12, wherein, The pattern includes one of the following: an image, a video frame, an audio sample, and a data stream sample.

15. The protected node according to claim 12, wherein, The first database and the second database reside in a single database structure.

16. The protection node according to any one of claims 10 to 12, wherein The machine learning algorithm is a neural network.

17. The protection node according to any one of claims 10 to 12, wherein, The protection node includes a honeypot server that executes the first clone of the machine learning algorithm.

18. The protection node according to any one of claims 10 to 12, further comprising: Repeatedly add noise to the original pattern to generate a set of noisy patterns, each of the noisy patterns in the set having different added noise; For each of the noisy patterns in the set, obtain a classification based on the processing by the first clone of the machine learning algorithm executed by the protection node; Wherein, the comparison includes: performing a comparison of the second classification with each of the classifications for the set of noisy patterns to determine whether the second classification meets the defined similarity rule compared with each of the classifications for the set of noisy patterns, Wherein, the use further includes using the comparison of the second classification with each of the classifications for the set of noisy patterns to manage the request from the user node.

19. A protection node, wherein, The protection node is adapted to: Receive a request from a user node, the request containing an original pattern to be classified by a machine learning algorithm executed by a pattern classification node; Add noise to the original pattern to generate a noisy pattern; Obtain a first classification of the noisy pattern based on the processing of the noisy pattern by the first clone of the machine learning algorithm executed by the protection node; Obtain a second classification of the original pattern based on forwarding the request to process the original pattern by the machine learning algorithm executed by the pattern classification node; Compare the first classification and the second classification to determine whether the first classification and the second classification meet the defined similarity rule; and Use the comparison of the first classification and the second classification to manage the request from the user node.

20. The protected node according to claim 19, wherein, Using the comparison of the first classification and the second classification to manage the request from the user node includes: When the first classification and the second classification do not satisfy the defined similarity rules, add the identifier associated with the user node to the distrust list in the first database, wherein, before a request received from an identifier included in the distrust list is forwarded to the pattern classification node, the protection node blocks requests received from identifiers included in the distrust list, or monitors requests received from identifiers included in the distrust list for potential future blocking.

21. The protected node according to claim 20, wherein, The protection node is further adapted to: Repeat the receiving, the adding of noise, the obtaining of the first classification, the obtaining of the second classification, the comparison; For each instance where it is determined that the first classification and the second classification obtained for one of the requests do not satisfy the defined similarity rules, store the information obtained from one of the requests in the risk information set included in the second database; Obtain the risk information set from the second database; And At the training node, train a second clone of the machine learning algorithm to obtain a result that the second classification of the original pattern in the request is incorrect using the risk information set obtained from the second database; And Forward the result to the pattern classification node.

22. The protected node according to any one of claims 19 to 21, wherein, Using the comparison of the first classification and the second classification to manage the requests from the user node includes: When the first classification and the second classification do not satisfy the defined similarity rules, send the second classification to the user node.

23. The protected node according to any one of claims 19 to 21, wherein The pattern includes one of the following: an image, a video frame, an audio sample, and a data stream sample.

24. The protection node according to claim 21, wherein, The first database and the second database reside in a single database structure.

25. The protected node according to any one of claims 19 to 21, wherein, The machine learning algorithm is a neural network.

26. The protection node according to any one of claims 19 to 21, wherein The protection node is adapted to include a honeypot server that executes the first clone of the machine learning algorithm.

27. The protection node according to any one of claims 19 to 21, wherein The protection node is further adapted to: Repeatedly add noise to the original pattern to generate a set of noisy patterns, each of the noisy patterns in the set having different added noise; For each of the noisy patterns in the set, obtain a classification based on the processing by the first clone of the machine learning algorithm executed by the protection node; Wherein, the comparison includes: performing a comparison of the second classification with each of the classifications for the set of noisy patterns to determine whether the second classification satisfies the defined similarity rules compared to each of the classifications for the set of noisy patterns, Wherein, the using further includes using the comparison of the second classification with each of the classifications for the set of noisy patterns to manage the requests from the user node.

28. A non - transitory computer - readable medium storing instructions that, when executed on at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 9.

29. A computer program product, comprising: A non-transitory computer-readable medium storing instructions that, when executed on at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 9.

Images