Detecting algorithm attacks against hosted AI systems based on inputs and outputs of hosted AI systems

By using feature classifiers and transformer model detection algorithm attacks in managed artificial intelligence systems, the problem of hosted artificial intelligence systems being vulnerable to algorithm attacks is solved, system security and user privacy protection are improved, and the consumption of detection and remediation resources is reduced. .

CN119013946APending Publication Date: 2024-11-22MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380032942.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-06
Filing Date
2023-03-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Hosted artificial intelligence systems lack effective protection measures and are vulnerable to algorithmic attacks, such as evasion attacks, reverse push attacks and extraction attacks, resulting in system security and user privacy being threatened.

Method used

By detecting algorithm attacks based on inputs and outputs of managed artificial intelligence systems, using feature classifier models and transformer models, classification scores or vectors are generated to determine the possibility or characteristics of attacks, and thus detect and defend against algorithm attacks.

Benefits of technology

It improves the security of managed artificial intelligence systems, reduces the risk of damage, confusion and theft of algorithmic attacks, enhances user privacy protection, reduces the consumption of detection and remediation time and resources, and improves the efficiency and user experience of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013946A_ABST
    Figure CN119013946A_ABST
Patent Text Reader

Abstract

Techniques are described herein that are capable of detecting algorithm attacks against hosted artificial intelligence (AI) systems based on inputs (e.g., queries) and outputs to hosted AI systems. In a first example, a feature-based classifier model is used to generate a classification score based on features derived from a query and an output numerical representation, and to detect an algorithm attack based on the classification score being greater than or equal to a score threshold. In a second example, a transformer-based model is used to generate a vector by providing a multivariable time sequence that is based on the input attribute (s) and the output attribute (s) as an input to the transformer-based model, and detecting an algorithm attack based on a distance between the vector and a point corresponding to the reference vector being less than or equal to a distance threshold.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The widespread adoption of AI has brought significant value to many industries, resulting in an increased motivation for malicious entities to exploit AI. For example, AI systems, especially AI systems hosted in the cloud (i.e., hosted AI systems), can be targeted by algorithmic attacks. Algorithmic attacks are attacks that are performed according to an algorithm (e.g., manually or automatically). For example, algorithmic attacks can be circumvention attacks, inversion attacks, or extraction attacks. An evasion attack is an algorithmic attack configured to damage, confuse, or circumvent an AI system. Inversion attacks are algorithmic attacks configured to use strategically selected queries to discover private training data used to train an AI system. Extraction attacks are algorithmic attacks configured to steal from an AI system. Hosted AI systems have traditionally had little protection against such algorithmic attacks. Summary of the invention

[0002] Various methods are described herein for detecting algorithmic attacks against hosted AI systems, etc., based on (e.g., based at least in part on) the inputs and outputs of the hosted AI systems. Thus, detection can be based on the behavior of the hosted AI systems. AI simulates intelligent behavior in (multiple) processing systems. For example, AI can be configured to perceive or infer information and apply the information to (multiple) adaptive behaviors within an environment or context. An AI system is a system that implements AI. A hosted AI system is an AI system that is hosted in the cloud. Thus, a hosted AI system can be referred to as a cloud-based AI system.

[0003] An algorithmic attack is an attack that is performed according to an algorithm (e.g., manually or automatically). For example, the algorithm can be a finite sequence of well-defined instructions for performing the attack. According to this example, at least some instructions of the algorithm can be iteratively executed to achieve the goal of the attack.

[0004] In a first example method, features associated with a known type of algorithmic attack are derived from a numerical representation of a corresponding query received by a hosted artificial intelligence system and an output resulting from processing of the corresponding query by the hosted artificial intelligence system. The derived features are provided as input to a feature-based classifier model, and the feature-based classifier model is used to generate a classification score indicating a likelihood that at least a portion of the query corresponds to the known type of algorithmic attack. The classification score is compared to a score threshold associated with the known type of algorithmic attack. The algorithmic attack is detected based at least in part on the classification score being greater than or equal to the score threshold, the score threshold associated with the known type of algorithmic attack.

[0005] In a second example method, a multivariate time series is provided as an input to a transformer-based model, the transformer-based model being used to generate a vector summarizing a plurality of queries received by a hosted artificial intelligence system and a plurality of outputs resulting from processing the plurality of corresponding queries by the hosted artificial intelligence system. The multivariate time series includes a plurality of elements. Each element is based at least in part on a pairwise distance between a numerical representation of a corresponding query in the plurality of queries and a numerical representation of a previous query that temporally precedes the corresponding query, and is also based at least in part on one or more attributes of the output resulting from processing the corresponding query. It is determined whether a distance between the vector and a point corresponding to a reference vector is less than or equal to a distance threshold, the reference vector being associated with a known type of algorithmic attack. The algorithmic attack is detected based at least in part on the distance between the vector and the point being less than or equal to the distance threshold.

[0006] The present invention is provided to introduce in a simplified form a selection of concepts that will be further described in the specific implementation below. The present invention is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In addition, it should be noted that the present invention is not limited to the specific embodiments described in the specific implementation and / or other parts of this document. Such embodiments are presented herein only for illustrative purposes. Based on the teachings contained herein, additional embodiments will be apparent to those skilled in the art (s). BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the invention and, together with the description, further serve to explain the principles involved and to enable one skilled in the relevant art(s) to make and use the disclosed technology.

[0008] Figure 1 is a block diagram of an example I / O-based attack detection system according to an embodiment.

[0009] Figure 2 Depicted is a flow diagram of an example method for detecting algorithmic attacks against a hosted artificial intelligence system based on inputs and outputs of the hosted artificial intelligence system, according to an embodiment.

[0010] Figure 3 is a block diagram of an example computing system according to an embodiment.

[0011] Figure 4 , Figure 6-Figure 10 and Fig.12 Depicted is a method for detecting Figure 2 Flowcharts of example methods of the features mentioned in the flowchart.

[0012] Figure 5 , Fig.11 and Fig.13 According to the embodiment Figure 3 A block diagram of an example implementation of feature derivation logic is shown.

[0013] Fig.14 Depicted is a flow chart of another example method for detecting algorithmic attacks against a hosted artificial intelligence system based on inputs and outputs of the hosted artificial intelligence system, in accordance with an embodiment.

[0014] Fig.15 is a block diagram of another example computing system according to an embodiment.

[0015] Fig.16 An example computer is depicted in which embodiments may be implemented.

[0016] Features and advantages of the disclosed technology will become more apparent from the detailed description set forth below in conjunction with the accompanying drawings, in which the same reference numerals identify corresponding elements throughout. In the drawings, the same reference numerals generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference numeral. DETAILED DESCRIPTION I. Introduction

[0017] The following detailed description refers to the accompanying drawings showing exemplary embodiments of the present invention. However, the scope of the present invention is not limited to these embodiments, but is defined by the appended claims. Therefore, embodiments other than the embodiments shown in the accompanying drawings, such as modified versions of the illustrated embodiments, may still be covered by the present invention.

[0018] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of a person skilled in the relevant art(s) to implement such feature, structure, or characteristic in conjunction with other embodiments.

[0019] Descriptors such as "first," "second," "third," etc. are used to reference some elements discussed herein. Such descriptors are used to facilitate discussion of example embodiments and do not indicate a required order of the referenced elements unless an affirmative statement requiring such an order is made herein. II. Example Embodiments

[0020] Example embodiments described herein can detect algorithmic attacks against hosted AI systems based on (e.g., based at least in part on) the inputs and outputs of the hosted AI system. Thus, detection can be based on the behavior of the hosted AI system. AI simulates intelligent behavior in (multiple) processing systems. For example, AI can be configured to sense or infer information and apply the information to (multiple) adaptive behaviors within an environment or context. An AI system is a system that implements AI. A hosted AI system is an AI system that is hosted in the cloud. Thus, a hosted AI system can be referred to as a cloud-based AI system.

[0021] An algorithmic attack is an attack that is performed according to an algorithm (e.g., manually or automatically). For example, the algorithm can be a finite sequence of well-defined instructions for performing the attack. According to this example, at least some instructions of the algorithm can be iteratively executed to achieve the goal of the attack.

[0022] The example techniques described herein have various benefits over conventional techniques for detecting algorithmic attacks. For example, the example techniques may be able to increase the security of a hosted AI system and / or a computing system hosting an AI system. In one aspect, the example techniques may reduce the likelihood that an algorithmic attack against a hosted AI system will damage, confuse, or circumvent the hosted AI system. In another aspect, the example techniques may reduce the likelihood that an algorithmic attack against a hosted AI system will discover private training data used to train the AI ​​system. In yet another aspect, the example techniques may reduce the likelihood that an algorithmic attack against a hosted AI system will steal the hosted AI system.

[0023] The example techniques may increase security for users of hosted AI systems. For example, the example techniques may protect the privacy of users (e.g., by storing numerical representations of queries received from users by the AI ​​system, rather than storing the original queries). Thus, the example techniques may increase the user experience of users.

[0024] Example techniques may reduce the amount of time and / or resources (e.g., processor cycles, memory, network bandwidth) consumed to detect algorithmic attacks against a hosted AI system. For example, example techniques may reduce the amount of data stored in logs about a hosted AI system (e.g., by storing numerical representations of queries received by the hosted AI system in place of the original queries). By using the inputs and outputs of the hosted AI system to detect algorithmic attacks, example techniques may improve the precision, accuracy, speed, and / or efficiency of detection. Thus, additional time and resources that would have been consumed to detect algorithmic attacks may be avoided.

[0025] By increasing the precision, accuracy, speed, and / or efficiency of detection, example techniques may reduce the number of remedial actions that a computing system performs to remediate such attacks. For example, if an algorithmic attack is detected before the algorithmic attack can achieve its objectives, the need to perform remedial actions may be eliminated. Thus, example techniques may be able to reduce the amount of time and / or resources consumed to perform such remedial actions.

[0026] Example techniques may improve the efficiency of a computing system (e.g., a computing system hosting an artificial intelligence system) configured to detect and / or remediate algorithmic attacks by reducing the amount of time and / or resources consumed to detect and / or remediate algorithmic attacks. Example techniques may improve the efficiency of an administrator of a hosted artificial intelligence system. For example, using the inputs and outputs of a hosted artificial intelligence system to detect algorithmic attacks may reduce the amount of time an administrator spends detecting and / or remediating algorithmic attacks.

[0027] Figure 1 is a block diagram of an example I / O-based attack detection system 100 according to an embodiment. In general, the I / O-based attack detection system 100 operates to provide information to a user in response to a request (e.g., a hypertext transfer protocol (HTTP) request) received from the user. The information may include a document (a web page, an image, an audio file, a video file, etc.), the output of an executable file, and / or any other suitable type of information. According to the example embodiment described herein, the I / O-based attack detection system 100 detects algorithmic attacks against a hosted artificial intelligence (AI) system 108 based on (e.g., based at least in part on) the inputs and outputs of the hosted AI system 108. Details regarding techniques for detecting algorithmic attacks against a hosted artificial intelligence system based on the inputs and outputs of the hosted artificial intelligence system are provided in the following discussion.

[0028] like Figure 1 As shown, the I / O-based attack detection system 100 includes a plurality of user devices 102A-102M, a network 104, and a plurality of servers 106A-106N. Communication between the user devices 102A-102M and the servers 106A-106N is implemented through the network 104 using a well-known network communication protocol. The network 104 can be a wide area network (e.g., the Internet), a local area network (LAN), another type of network, or a combination thereof.

[0029] The user devices 102A-102M are processing systems capable of communicating with the servers 106A-106N. An example of a processing system is a system including at least one processor capable of manipulating data according to an instruction set. For example, the processing system may be a computer, a personal digital assistant, etc. The user devices 102A-102M are configured to provide requests (e.g., queries) to the servers 106A-106N for requesting information stored on the servers 106A-106N (or otherwise accessible via the servers 106A-106N). For example, a user may initiate a request to execute a computer program (e.g., an application) using a client (e.g., a web browser, a web crawler, or other type of client) deployed on the user device 102, which is owned by the user or otherwise accessible to the user. According to this example, a user may initiate one or more queries to be processed by the hosted AI system 108. Each query may request a response (also referred to as an output) from the hosted AI system 108. For example, each query may request the hosted AI system 108 to classify the query between a plurality of potential outputs. In response to each query, the user may receive, at the first user device 102A, an output selected from potential outputs by the hosted AI system 108. According to some example embodiments, the user devices 102A-102M are capable of accessing a domain (e.g., a website) hosted by the servers 104A-104N, such that the user devices 102A-102M may access information available via the domain. For example, such a domain may include a web page, which may be provided as a hypertext markup language (HTML) document and objects (e.g., files) linked therein.

[0030] Each of the user devices 102A-102M may include any client-enabled system or device, including, but not limited to, a desktop computer, a laptop computer, a tablet computer, a wearable computer (such as a smart watch or head-mounted computer), a personal digital assistant, a cellular phone, an Internet of Things (IoT) device, etc. It will be appreciated that any one or more of the user devices 102A-102M may communicate with any one or more of the servers 106A-106N.

[0031] Servers 106A-106N are processing systems capable of communicating with user devices 102A-102M. Servers 106A-106N are configured to execute computer programs that provide information to users in response to requests received from users. For example, the information may include documents (web pages, images, audio files, video files, etc.), output of executable files, or any other suitable type of information. Any one or more of the computer programs may be cloud computing services. Cloud computing services are services that are executed at least in part in the cloud. The cloud may be a remote cloud, a local cloud, or a hybrid cloud. It will be appreciated that a local cloud may use remote cloud services. Examples of cloud computing services include, but are not limited to, the Microsoft Azure cloud developed and released by Microsoft Corporation. Google, developed and published by Google Developed and published by Oracle Corporation Amazon Web Services, developed and published by Amazon.com Developed and published by Salesforce.com Developed and published by RackspaceUS According to some example embodiments, servers 106A-106N are configured to host respective websites such that users of I / O-based attack detection system 100 may access the websites.

[0032] For illustrative purposes, the first server(s) 106A are shown as including a hosted AI system 108 and an input / output-based (I / O-based) attack detection logic 110. It can be said that the first server(s) 106A hosts the hosted AI system 108. The hosted AI system 108 implements artificial intelligence, which is configured to generate an output based on a received input (e.g., a query). For example, the hosted AI system 108 is capable of using artificial intelligence to classify queries received from user devices 102A-102M among multiple potential outputs. For example, the hosted AI system 108 can analyze each query received from a user device to determine a potential output from a plurality of potential outputs that most closely corresponds to (e.g., matches) the query. According to this example, the hosted AI system 108 can select, in response to the query, a potential output that most closely corresponds to the query to be provided as an output to the user device.

[0033] It will be appreciated that the hosted AI system 108 uses machine learning to perform at least some of its operations. For example, the hosted AI system 108 may use machine learning to develop and refine classification algorithms that are used by the hosted AI system 108 to classify queries received from user devices 102A-102M among a plurality of potential outputs.

[0034] The hosted AI system 108 may use a neural network to perform machine learning to classify queries among multiple potential outputs. Examples of neural networks include, but are not limited to, feedforward neural networks and long short-term memory (LSTM) neural networks. A feedforward neural network is an artificial neural network in which the connections between units in the neural network do not form a loop. In an example embodiment, the hosted AI system 108 employs a feedforward neural network to train a machine learning model for determining an ML-based confidence. Such ML-based confidences may be used to determine the likelihood that an event will occur. For example, such ML-based confidences may be associated with multiple potential outputs (e.g., potential classifications) for each query received from a user device. According to this example, for each query, the hosted AI system 108 may assign an ML-based confidence to the corresponding potential output to indicate the corresponding likelihood that the query corresponds to the corresponding potential output.

[0035] An LSTM neural network is a recurrent neural network that has a memory and allows data to flow forward and backward in the neural network. An LSTM neural network is able to remember values ​​for short or long periods of time. Thus, an LSTM neural network can prevent stored values ​​from being iteratively diluted over time. In an example, an LSTM neural network may be able to remember relationships between features, such as a query, potential outputs, outputs selected from potential outputs, and ML-based confidences derived therefrom.

[0036] The hosted AI system 108 may include training logic and inference logic. The training logic is configured to train a machine learning algorithm, and the inference logic uses the machine learning algorithm to determine (e.g., infer) ML-based confidence. For example, the training logic may provide sample queries, sample potential outputs, sample outputs selected from potential outputs, and sample confidences as inputs to the algorithm to train the algorithm. The sample data may be labeled. The machine learning algorithm may be configured to derive a relationship between features (e.g., queries, potential outputs, and outputs selected from potential outputs) and the resulting ML-based confidence. The inference logic is configured to utilize the machine learning algorithm trained by the training logic to determine ML-based confidences when features are provided as inputs to the algorithm.

[0037] For illustrative purposes, the hosted AI system 108 is shown as including an application programming interface (API) 112. An API is a connection between one or more computing systems (e.g., computers) and / or one or more computer programs. The API 112 enables communication between the user devices 102A-102M and the hosted AI system 108. For example, the user devices 102A-102M may call the API 112 to provide a query to the hosted AI system 108, and the hosted AI system 108 may respond to the query by providing corresponding output to the user devices 102A-102M from which the query was received using the API 112. The API 112 may be a representational state transfer (REST) ​​API, but the example embodiments are not limited in this regard. Algorithmic attacks that attack the hosted AI system behind the API 112 may be referred to as "closed box" attacks or "black box" attacks.

[0038] The I / O-based attack detection logic 110 is configured to detect algorithmic attacks against the hosted AI system 108 based on the inputs and outputs of the hosted AI system 108. For example, the I / O-based attack detection logic 110 may intercept queries sent from user devices 102A-102M to the hosted AI system 108 and corresponding outputs provided by the hosted AI system 108 to the user devices 102A-102M for the purpose of detecting algorithmic attacks.

[0039] In a first example method, the I / O-based attack detection logic 110 derives features associated with a known type of algorithmic attack from a numerical representation of a corresponding query received by the hosted AI system 108 and an output resulting from processing the corresponding query by the hosted AI system 108. For example, the features may be derived based on a Gaussian projection of the query and the output. In one aspect, the numerical representation of the query and the output may be stored in a log, and the I / O-based attack detection logic 110 may extract the features from the log. By providing the derived features as input to a feature-based classifier model, the I / O-based attack detection logic 110 uses the feature-based classifier model to generate a classification score indicating the likelihood that at least a portion of the query corresponds to a known type of algorithmic attack. The I / O-based attack detection logic 110 compares the classification score to a score threshold associated with the known type of algorithmic attack. The I / O-based attack detection logic 110 detects an algorithmic attack based at least in part on the classification score being greater than or equal to a score threshold associated with the known type of algorithmic attack.

[0040] In a second example method, the I / O-based attack detection logic 110 generates a vector using the transformer-based model by providing a multivariate time series as an input to the transformer-based model, which summarizes the query received by the hosted AI system 108 and the output obtained by the hosted AI system 108 processing the corresponding query. The multivariate time series includes a plurality of elements. Each element is based at least in part on a pairwise distance between a numerical representation of the corresponding query and a numerical representation of a previous query that precedes the corresponding query in time, and is also based at least in part on (multiple) attributes of the output obtained by processing the corresponding query. In one aspect, the numerical representations of the query and the output can be stored in a log, and the I / O-based attack detection logic 110 can generate a multivariate time series from the log. The I / O-based attack detection logic 110 determines whether the distance between the vector and a point corresponding to a reference vector associated with a known type of algorithmic attack is less than or equal to a distance threshold. The I / O-based attack detection logic 110 detects an algorithmic attack based at least in part on the distance between the vector and the point being less than or equal to the distance threshold.

[0041] The I / O-based attack detection logic 110 can be implemented in various ways to detect algorithmic attacks against the hosted AI system 108 based on the inputs and outputs of the hosted AI system 108, including in hardware, software, firmware, or any combination thereof. For example, the I / O-based attack detection logic 110 can be implemented as a computer program code configured to be executed in one or more processors. In another example, at least a portion of the I / O-based attack detection logic 110 can be implemented as hardware logic / circuitry. For example, at least a portion of the I / O-based attack detection logic 110 can be implemented in a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a complex programmable logic device (CPLD), etc. Each SoC may include an integrated circuit chip that includes a processor (microcontroller, microprocessor, digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more of additional circuits and / or embedded firmware to perform its functions.

[0042] I / O-based attack detection logic 110 may be partially or completely incorporated into hosted AI system 108 and / or a cloud computing service, although example embodiments are not limited in this regard. For example, hosted AI system 108 may be a cloud computing service.

[0043] For illustrative purposes, the I / O-based attack detection logic 110 is shown as being incorporated into (multiple) first servers 106A, and is not intended to be limiting. It will be appreciated that the I / O-based attack detection logic 110 (or any (multiple) parts thereof) can be incorporated into any one or more of the user devices 102A-102M. For example, the client-side aspects of the I / O-based attack detection logic 110 can be incorporated into one or more of the user devices 102A-102M (e.g., to record appropriate information and / or ensure the privacy of query data), and the server-side aspects of the I / O-based attack detection logic 110 can be incorporated into (multiple) first servers 106A. For example, the client-side aspects of the I / O-based attack detection logic 110 can be incorporated into (multiple) user devices in the form of (multiple) plug-ins. The plug-in can have any of a variety of forms. For example, the plug-in can be implemented as a Python decorator. In another example, the I / O-based attack detection logic 110 can be distributed in the user devices 102A-102M. In yet another example, the I / O-based attack detection logic 110 may be incorporated into a single user device among the user devices 102A-102M. In another example, the I / O-based attack detection logic 110 may be distributed among (multiple) servers 106A-106N. In yet another example, the I / O-based attack detection logic 110 may be incorporated into a single server among the servers 106A-106N.

[0044] Figure 2 A flowchart 200 is depicted of an example method for detecting an algorithmic attack against a hosted artificial intelligence system based on the input and output of the hosted artificial intelligence system according to an embodiment. For example, the flowchart 200 may be composed of Figure 1 The first server (s) 106A shown executes. For the purpose of illustration, Figure 3 The computing system 300 shown depicts the flowchart 200. Figure 3 is an example implementation of the first server(s) 106A. Figure 3As shown, computing system 300 includes I / O-based attack detection logic 310 and storage 312. I / O-based attack detection logic 310 includes feature derivation logic 314, model usage logic 316, feature-based classifier model 318, comparison logic 320, and detection logic 322. Storage 312 can be any suitable type of storage. One type of storage is a database. For example, storage 312 can be a relational database, an entity-relational database, an object database, an object-relational database, an extensible markup language (XML) database, etc. For non-limiting illustrative purposes, storage 312 is shown as storing score threshold 336. Based on the discussion about flowchart 200, further structural and operational embodiments will be apparent to those skilled in the relevant art(s).

[0045] like Figure 2 As shown, the method of flowchart 200 begins at step 202. In step 202, features are derived from numerical representations of corresponding queries received by the hosted artificial intelligence system and outputs obtained by the hosted artificial intelligence system processing the corresponding queries. In addition to queries and outputs, features may also take into account other factors. Examples of such factors include, but are not limited to, timestamps associated with each query, client identifiers (IDs) indicating the client device or user from which the query is received, and model IDs that uniquely identify the hosted AI system. For example, such factors (as well as numerical representations of queries and outputs) may be stored in logs and retrieved from logs for the purpose of deriving features. These features are associated with known types of algorithmic attacks. For example, features may be derived based at least in part on the association of features with known types of algorithmic attacks. Queries may be associated with a common (e.g., the same) user session. For example, a query may define a user session. A user session is a temporary and interactive exchange of information between a hosted artificial intelligence system and a user device from which the hosted artificial intelligence system receives queries. A user session may start at a first time instance of the first query in time in which a user device initiates a query, and end at a second time instance of the last output in time in which an output is received at the user device. In an example implementation, feature derivation logic 314 derives features 330 associated with known types of algorithmic attacks from numerical representations of corresponding queries 326 received by the hosted artificial intelligence system and outputs 328 resulting from processing the corresponding queries 326 by the hosted artificial intelligence system.

[0046] Deriving features at step 202 can improve the accuracy and / or precision of detecting algorithmic attacks by using both the input (e.g., the numerical representation of the query) and the output of the hosted artificial intelligence system. By using the numerical representation of the query, step 202 can increase the security of the hosted artificial intelligence system and / or increase the privacy of the user who initiated the query. By using features associated with known types of algorithmic attacks, step 202 can reduce the amount of time and / or resources consumed by the computing system to process the features. Deriving features at step 202 can improve the efficiency of the computing system.

[0047] Features may be derived at step 202 in any of a variety of ways. Some features may be specific to known types of algorithmic attacks. Other features may be associated with various types of algorithmic attacks. In an example embodiment, deriving features at step 202 includes spectrally analyzing outputs resulting from processing of the corresponding query by the hosted artificial intelligence system to determine that the outputs exhibit periodicity. According to this embodiment, deriving features at step 202 also includes deriving a first feature based at least in part on the outputs exhibiting periodicity. Further according to this embodiment, the outputs exhibiting periodicity are conducive to increasing the classification score.

[0048] Embodiments in which algorithmic attacks of a known type are inverse attacks are referred to herein as "inverse embodiments." In a first inverse embodiment, deriving features in step 202 includes deriving a first feature based at least in part on the standard deviation of a Gaussian distribution defined by a numerical representation of the corresponding query. The first feature derived according to the first inverse embodiment may be referred to as an "inverse standard deviation test" or a "Kolmogorov-Smirnov test." When the algorithmic attack uses a relatively low standard deviation for model detection, the inverse standard deviation test is relatively large for finite differences. The Kolmogorov-Smirnov test indicates that a Gaussian or uniform perturbation is used for gradient estimation. According to the first inverse embodiment, a relatively low standard deviation is conducive to increasing the classification score, and a relatively high standard deviation is not conducive to increasing the classification score.

[0049] In a second inverse embodiment, deriving the feature at step 202 includes determining an entropy associated with a numerical representation of the corresponding query. According to the second inverse embodiment, deriving the feature at step 202 includes deriving a first feature based at least in part on entropy. For example, the first feature derived according to the second inverse embodiment can be referred to as "input information". According to this example, the input information can indicate entropy under a Gaussian assumption on the query, which can distinguish between human interaction and algorithmic interaction. Further according to the second inverse embodiment, a relatively low entropy is conducive to increasing the classification score, and a relatively high entropy is not conducive to increasing the classification score.

[0050] In the third reverse deduction embodiment, the features obtained in step 202 include Figure 4For example, the features derived using the steps shown in flowchart 400 may also be referred to as "output information". According to this example, the output information may indicate entropy under a Gaussian assumption on the output to distinguish between human interaction and algorithmic interaction. For example, flowchart 400 may be composed of Figure 3 The feature derivation logic 314 is shown to perform. For illustrative purposes, Figure 5 The features shown in FIG. 4 are derived from logic 500 to describe the flowchart 400, Figure 5 is an example implementation of feature derivation logic 314. Figure 5 As shown, feature derivation logic 500 includes confidence determination logic 540, entropy determination logic 542, and feature logic 544. Based on the discussion regarding flowchart 400, further structural and operational embodiments will be apparent to those skilled in the relevant art(s).

[0051] like Figure 4 As shown, the method of flowchart 400 begins at step 402. In step 402, a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query is determined. For example, the confidence associated with the output can be a confidence of the classification of the output (e.g., the confidence that the output will be classified in a specified category). In an example implementation, confidence determination logic 540 determines the confidence associated with the corresponding output 526 obtained by the hosted artificial intelligence system processing the corresponding query. Confidence determination logic 540 can generate confidence information 546 to indicate the confidence.

[0052] At step 404, an entropy associated with the confidence level is determined. In an example implementation, entropy determination logic 542 determines the entropy associated with the confidence level. For example, entropy determination logic 542 may analyze the distribution of confidence levels, as indicated by confidence level information 546, to determine entropy. Entropy determination logic 542 may generate entropy information 548 to indicate entropy.

[0053] At step 406, a first feature is derived based at least in part on entropy. In an example implementation, feature logic 544 derives feature 550 based at least in part on entropy. For example, feature logic 544 may analyze entropy information 548 to determine entropy.

[0054] According to the third inverse deduction embodiment, a relatively low entropy is conducive to increasing the classification score, and a relatively high entropy is not conducive to increasing the classification score.

[0055] In the fourth reverse deduction embodiment, the features obtained in step 202 include Figure 6 For example, the features derived using the steps shown in flowchart 600 may be referred to as “model entropy scores”. For example, flowchart 600 may be composed of Figure 3 The feature derivation logic 314 is shown to execute. For illustrative purposes, Figure 5 The feature derivation logic 500 shown in FIG. 6 depicts a flowchart 600. Based on the discussion regarding flowchart 600, further structural and operational embodiments will be apparent to persons skilled in the relevant art(s).

[0056] like Figure 6 As shown, the method of flowchart 600 begins at step 602. In step 602, a confidence level associated with a corresponding output obtained by processing a corresponding query by a hosted artificial intelligence system is determined. In an example implementation, confidence level determination logic 540 determines a confidence level associated with a corresponding output 526 obtained by processing a corresponding query by a hosted artificial intelligence system. Confidence level determination logic 540 may generate confidence level information 546 to indicate the confidence level.

[0057] At step 604, entropies are determined at respective times corresponding to respective outputs. Each entropy is based on a confidence associated with the respective output and a confidence of each output preceding the respective output. In an example implementation, entropy determination logic 542 determines entropy at respective times corresponding to respective outputs 526. For example, at each time, entropy determination logic 542 may analyze a distribution of confidences associated with the respective output and each output preceding the respective output, as indicated by confidence information 546, to determine the entropy for that time.

[0058] At step 606, it is determined that the trend of entropy decreases over time. In an example implementation, the entropy determination logic 542 determines that the trend of entropy decreases over time. For example, the entropy determination logic 542 may analyze the entropy at each time determined at step 604 to determine that the trend of entropy decreases. The entropy determination logic 542 may generate entropy information 548 to indicate that the trend of entropy decreases over time.

[0059] At step 608, a first feature is derived based at least in part on the trend of entropy decreasing over time. In an example implementation, feature logic 544 derives feature 550 based at least in part on the trend of entropy decreasing over time. For example, feature logic 544 may derive feature 550 based on receiving entropy information (e.g., based on entropy information 548 indicating a trend of entropy decreasing over time).

[0060] According to a fourth inverse embodiment, a trend in entropy decreasing over time is favorable for increasing classification scores. For example, a trend in entropy decreasing over time may indicate an attempt to isolate potential training points for a hosted artificial intelligence system.

[0061] In the fifth reverse implementation, step 608 of flowchart 600 may be used Figure 7For example, the features derived using the steps shown in flowchart 700 may be referred to as "output information gain". According to this example, the output information gain increases at a rate of decrease based on the trend of entropy described above with reference to the fourth inverse embodiment. Figure 7 As shown, the method of flowchart 700 begins at step 702. In step 702, it is determined that the rate at which the trend of entropy decreases over time increases over time. In an example implementation, entropy determination logic 542 determines that the rate at which the trend of entropy decreases over time increases over time. For example, entropy determination logic 542 can analyze the trend to identify the slope of the trend at each time instance. For example, entropy determination logic 542 can determine that the slope of the trend decreases to a greater extent with each consecutive time instance, or that the slope of the trend generally decreases to a greater extent over time. Entropy determination logic 542 can generate entropy information 548 to indicate that the rate at which the trend of entropy decreases over time increases over time.

[0062] At step 704, a first feature is derived based at least in part on the trend of entropy decreasing over time and further at least in part on the rate increasing over time. In an example implementation, feature logic 544 derives feature 550 based at least in part on the trend of entropy decreasing over time and further at least in part on the rate increasing over time. For example, feature logic 544 may derive feature 550 based on receipt of entropy information (e.g., based on entropy information 548 indicating a trend decreasing over time and a rate increasing over time).

[0063] According to the fifth inverse embodiment, the combination of the decreasing trend of entropy over time and the increasing rate over time is beneficial to increasing the classification score.

[0064] Although the reverse inference embodiments discussed above are described with reference to reverse inference attacks, the reverse inference embodiments are not limited to reverse inference attacks. It will be appreciated that the reverse inference embodiments are also applicable to other types of algorithmic attacks in addition to reverse inference attacks. Thus, the known types of algorithmic attacks in the reverse inference embodiments are not necessarily reverse inference attacks; the known types of algorithmic attacks in the reverse inference embodiments may be any suitable type of algorithmic attack (e.g., extraction attacks, evasion attacks, or membership inference attacks).

[0065] Embodiments in which the known type of algorithmic attack is an extraction attack are referred to herein as "extraction embodiments." In the first extraction embodiment, deriving features at step 202 includes the steps described above with reference to the second inversion embodiment. However, according to the first extraction embodiment, relatively high entropy is beneficial to increase the classification score, and relatively low entropy is not beneficial to increase the classification score.

[0066] In the second extraction embodiment, the features obtained in step 202 include the features described above with reference to the second inverse deduction embodiment. Figure 4One or more steps shown in flowchart 400 of . However, according to the second extraction embodiment, a relatively high entropy is conducive to increasing the classification score, and a relatively low entropy is not conducive to increasing the classification score.

[0067] In the third extraction embodiment, the features obtained in step 202 include Figure 8 For example, the features derived using the steps shown in flowchart 800 can be referred to as "model entropy scores". For example, flowchart 800 can be composed of Figure 3 The feature derivation logic 314 is shown to execute. For illustrative purposes, Figure 5 The feature derivation logic 500 shown in FIG. 8 is a flow chart 800. Based on the discussion regarding flow chart 800, further structural and operational embodiments will be apparent to persons skilled in the relevant art(s).

[0068] like Figure 8 As shown, the method of flowchart 800 begins at step 802. In step 802, a confidence level associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query is determined. In an example implementation, confidence determination logic 540 determines a confidence level associated with a corresponding output 526 obtained by the hosted artificial intelligence system processing the corresponding query. Confidence determination logic 540 may generate confidence information 546 to indicate the confidence level.

[0069] At step 804, entropy is determined at a respective time corresponding to a respective output. Each entropy is based on a confidence associated with the respective output and a confidence of each output preceding the respective output. In an example implementation, entropy determination logic 542 determines entropy at a respective time corresponding to a respective output 526.

[0070] At step 806, it is determined that the trend of entropy is increasing over time. In an example implementation, the entropy determination logic 542 determines that the trend of entropy is increasing over time. For example, the entropy determination logic 542 may analyze the entropy at each time determined at step 604 to determine that the trend of entropy is increasing. The entropy determination logic 542 may generate entropy information 548 to indicate that the trend of entropy is increasing over time.

[0071] At step 808, a first feature is derived based at least in part on the trend of entropy increasing over time. In an example implementation, feature logic 544 derives feature 550 based at least in part on the trend of entropy increasing over time. For example, feature logic 544 may derive feature 550 based on receiving entropy information (e.g., based on entropy information 548 indicating that the trend of entropy increases over time).

[0072] According to the third extraction embodiment, the trend of entropy increasing over time is beneficial to increasing the classification score.

[0073] In the fourth extraction embodiment, step 808 of flowchart 800 may be used Fig. 9 For example, the features derived using the steps shown in flowchart 900 may also be referred to as "output information gain". According to this example, the output information gain increases at a rate based on the trend of entropy described above with reference to the third extraction embodiment. Fig. 9 As shown, the method of flowchart 900 begins at step 902. In step 902, it is determined that the rate at which the trend of entropy increases over time increases over time. In an example implementation, entropy determination logic 542 determines that the rate at which the trend of entropy increases over time increases over time. For example, entropy determination logic 542 can analyze the trend to identify the slope of the trend at each time instance. For example, entropy determination logic 542 can determine that the slope of the trend increases to a greater extent with each consecutive time instance, or that the slope of the trend generally increases to a greater extent over time. Entropy determination logic 542 can generate entropy information 548 to indicate the rate at which the trend of entropy decreases over time with reference to time.

[0074] At step 904, a first feature is derived based at least in part on the trend of entropy increasing over time and further at least in part on the rate increasing over time. In an example implementation, feature logic 544 derives feature 550 based at least in part on the trend of entropy increasing over time and further at least in part on the rate increasing over time. For example, feature logic 544 may derive feature 550 based on receipt of entropy information (e.g., based on entropy information 548 indicating a trend increasing over time and a rate increasing over time).

[0075] According to the fourth extraction embodiment, a combination of the trend of entropy increasing over time and the rate increasing over time is beneficial to increasing the classification score.

[0076] Although the above extraction embodiments are described with reference to extraction attacks, the extraction embodiments are not limited to extraction attacks. It will be appreciated that the extraction embodiments are applicable to other types of algorithmic attacks in addition to extraction attacks. Therefore, the known types of algorithmic attacks in the extraction embodiments are not necessarily extraction attacks; the known types of algorithmic attacks in the extraction embodiments can be any suitable type of algorithmic attacks.

[0077] Embodiments where known types of algorithmic attacks are circumvention attacks (e.g., jump attacks or boundary attacks) are referred to herein as "circumvention embodiments." In a first circumvention embodiment, the features derived at step 202 include Fig.10 For example, the features derived using the steps shown in flowchart 1000 may be referred to as “adversarial example scores”. For example, flowchart 1000 may be composed of Figure 3The feature derivation logic 314 is shown to execute. For illustrative purposes, Fig.11 The features shown in FIG. 1 are used to derive logic 1100 to describe flowchart 1000. Fig.11 is an example implementation of feature derivation logic 314. Fig.11 As shown, feature derivation logic 1100 includes input difference logic 1152, output difference logic 1154, ratio determination logic 1156, selection logic 1158, and feature logic 1144. Further structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart 1000.

[0078] like Fig.10 As shown, the method of flowchart 1000 begins at step 1002. At step 1002, a distance between a numerical representation of each query in at least a subset of queries and a numerical representation of a reference query is determined. In an example implementation, input difference logic 1152 determines a distance between a numerical representation of each query in at least a subset of queries 1126 and a numerical representation of a reference query 1166. Input difference logic 1152 may generate input distance information 1170 to indicate a distance between a numerical representation of each query in at least a subset of queries 1126 and a numerical representation of a reference query 1166.

[0079] At step 1004, a difference is determined between an output resulting from processing each query in at least the subset by the hosted artificial intelligence system and a reference output resulting from processing the reference query by the hosted artificial intelligence system. In an example implementation, output difference logic 1154 determines a difference between an output 1128 resulting from processing the corresponding query in at least the subset by the hosted artificial intelligence system and a reference output 1168 resulting from processing the reference query 1166 by the hosted artificial intelligence system. Output difference logic 1154 may generate output difference information 1172 to indicate the difference between output 1128 and reference output 1168.

[0080] At step 1006, a ratio is determined for each query in at least the subset to provide a plurality of ratios. Each ratio is equal to a difference between an output obtained by processing the corresponding query by the hosted artificial intelligence system and a reference output divided by a distance between a numerical representation of the corresponding query and a numerical representation of the reference query. In an example implementation, ratio determination logic 1156 determines a ratio for each query in at least the subset to provide a plurality of ratios. Each ratio is equal to a difference between an output obtained by processing the corresponding query by the hosted artificial intelligence system and a reference output 1168 divided by a distance between a numerical representation of the corresponding query and a numerical representation of the reference query 1166. Ratio determination logic 1156 may generate ratio information 1160 to indicate the plurality of ratios.

[0081] At step 1008, a ratio of the plurality of ratios is selected that is not less than each other ratio of the plurality of ratios. In an example implementation, the selection logic 1158 selects a ratio from the plurality of ratios based on the selected ratio being not less than each other ratio of the plurality of ratios. For example, the selection logic 1158 may analyze the plurality of ratios indicated by the ratio information 1160 to determine that the selected ratio is not less than each other ratio of the plurality of ratios. The selection logic 1158 may generate selection information 1162 to identify the selected ratio.

[0082] At step 1010, a first feature is derived based at least in part on the selected ratio. In an example implementation, feature logic 1144 derives feature 1150 based at least in part on the selected ratio. For example, feature logic 1144 may derive feature 1150 based on receiving selection information 1162 (e.g., based on selection information 1162 identifying the selected ratio).

[0083] According to the first circumvention embodiment, the selected ratio is relatively favorable to increase the classification score, and the selected ratio is relatively low and unfavorable to increase the classification score.

[0084] In the second circumvention embodiment, the features obtained in step 202 include Fig.12 For example, the features derived using the steps shown in flowchart 1200 may be referred to as "intrinsic dimension scores". According to this example, the intrinsic dimension scores may indicate that the sequence of numerical representations of the query is located along the line (e.g., due to an algorithmic attack using a binary method). For example, flowchart 1200 may be composed of Figure 3 The feature derivation logic 314 is shown to execute. For illustrative purposes, Fig.13 The feature derivation logic 1300 shown in FIG. 1 is used to describe the flowchart 1200. Fig.13 is an example implementation of feature derivation logic 314. Fig.13 As shown, feature derivation logic 1300 includes distance determination logic 1374, value determination logic 1376, and feature logic 1344. Based on the discussion regarding flowchart 1200, further structural and operational embodiments will be apparent to those skilled in the relevant art(s).

[0085] like Fig.12As shown, the method of flowchart 1200 begins at step 1202. In step 1202, the distance (e.g., the shortest distance) between each numerical representation of the numerical representation of the corresponding query and the nearest fit line defined by the numerical representation of the corresponding query is determined. In an example implementation, the distance determination logic 1374 determines the distance between each numerical representation of the numerical representation of the corresponding query 1326 and the nearest fit line defined by the numerical representation of the corresponding query 1326. For example, the distance determination logic 1374 can calculate the nearest fit line based on the numerical representation of the query 1326 using any of a variety of techniques (e.g., a least squares fitting technique). The distance determination logic 1374 can compare each numerical representation of the numerical representation of the corresponding query 1326 with the nearest fit line to determine the shortest distance from the corresponding numerical representation to the nearest fit line. The distance determination logic 1374 can generate distance information 1378 to indicate the distance between each numerical representation of the numerical representation of the corresponding query 1326 and the nearest fit line.

[0086] At step 1204, a difference value is determined based at least in part on the distance between the nearest fit line and the respective numerical representation of the respective query. In an example implementation, the value determination logic 1376 determines the difference value 1380 based at least in part on the distance between the nearest fit line and the respective numerical representation of the respective query 1326. For example, the value determination logic 1376 may analyze the distance information to determine the distance between each of the numerical representations of the respective query 1326 and the nearest fit line. The value determination logic 1376 may combine the distances between the numerical representation of the respective query 1326 and the nearest fit line to generate the difference value 1380. For example, the difference value 1380 may be an average or root mean square of the distances.

[0087] At step 1206 , a first feature is derived based at least in part on the difference value. In an example implementation, feature logic 1344 derives feature 1350 based at least in part on difference value 1380 .

[0088] According to the second circumvention embodiment, a relatively low difference value is favorable for increasing the classification score, and a relatively high difference value is unfavorable for increasing the classification score.

[0089] In a third circumvention embodiment, deriving the feature in step 202 includes determining that the numerical representation of the corresponding query defines a Gaussian distribution. For example, the Gaussian distribution can be the result of a Gaussian random number generator used to generate the numerical representation of the corresponding query. According to the third circumvention embodiment, deriving the feature in step 202 further includes deriving the first feature based at least in part on the Gaussian distribution. The first feature derived according to the third circumvention embodiment can be referred to as a "normality test". The normality test indicates the p-value of a hypothesis test sampled from a sequence belonging to a Gaussian distribution. For example, an algorithmic attack can use finite differences to estimate the gradient via samples drawn from a Gaussian. Further according to the third circumvention embodiment, defining the numerical representation of the corresponding query for a Gaussian distribution is beneficial to increase the classification score.

[0090] In the fourth circumvention embodiment, the features obtained in step 202 include the features described above with reference to the third extraction embodiment. Figure 8 One or more steps shown in flowchart 800 of . According to a fourth circumvention embodiment, the trend of entropy increasing over time is favorable to increasing classification scores.

[0091] In the fifth circumvention embodiment, step 808 of flowchart 800 may be replaced by the step described above with reference to the fourth extraction embodiment. Fig. 9 One or more steps shown in flowchart 900 of . According to a fifth circumvention embodiment, a combination of a trend of entropy increasing over time and a rate increasing over time is beneficial for increasing the classification score.

[0092] Although the circumvention embodiments discussed above are described with reference to circumvention attacks, the circumvention embodiments are not limited to circumvention attacks. It will be appreciated that the circumvention embodiments are also applicable to other types of algorithmic attacks in addition to circumvention attacks. Thus, a known type of algorithmic attack in a circumvention embodiment is not necessarily a circumvention attack; a known type of algorithmic attack in a circumvention embodiment may be any suitable type of algorithmic attack.

[0093] Return to reference Figure 2, at step 204, by providing the derived features as input to the feature-based classifier model, the feature-based classifier model is used to generate a classification score. Therefore, the classification score is based at least in part on the derived features. The classification score indicates the likelihood that at least a portion of the query corresponds to an algorithmic attack of a known type. In an example, the classification score can be a multi-category output score, which includes a plurality of sub-scores corresponding to attacks of corresponding types. According to the example, each sub-score indicates the likelihood that at least a portion of the query corresponds to a corresponding attack type. The classification score can be a logit, but the example embodiments are not limited in this regard. The logit function includes a plurality of logits representing the respective probabilities of possible results, and the maximum logit is the predicted result. It will be appreciated that the logit does not necessarily need to sum to 1. The logit can be converted to a probability that sums to one by normalizing the logit (e.g., by applying a sigmoid function to the logit). Examples of feature-based classifier models include, but are not limited to, gradient boosting models, random forest models, support vector machines, and multilayer perceptrons (i.e., deep neural networks).

[0094] In an example implementation, the model-using logic 316 uses the feature-based classifier model 318 to generate a classification score 332 by providing the derived feature 330 as an input to the feature-based classifier model 318, the classification score 332 indicating the likelihood that at least a portion of the query 326 corresponds to a known type of algorithmic attack. For example, the model-using logic 316 may provide the feature 330 to the feature-based classifier model 318, which may trigger the feature-based classifier model 318 to analyze the feature 330. The feature-based classifier model 318 may analyze the feature 330 by comparing the feature 330 (e.g., an attribute of the feature 330) to a criterion associated with an algorithmic attack to determine the extent to which the feature 330 satisfies the criterion. The feature-based classifier model 318 may generate a classification score 332 to indicate the extent to which the feature 330 satisfies the criterion. Examples of attributes of a feature include, but are not limited to, a numerical value of the feature and a confidence level of the numerical value of the feature. Examples of criteria include, but are not limited to, a threshold value, a specified range of values, and an association with one or more other features.

[0095] Using a feature-based classifier model to generate a classification score at step 204 can reduce the amount of time and / or resources consumed to determine whether an algorithmic attack has occurred. Using a feature-based classifier model to generate a classification score at step 204 can improve the efficiency of a computing system used to detect algorithmic attacks.

[0096] At step 206, it is determined whether the classification score is greater than or equal to a score threshold associated with a known type of algorithmic attack. If the classification score is greater than or equal to the score threshold, the flow continues to step 208. Otherwise, the flow continues to step 210. In an example implementation, the comparison logic 320 determines whether the classification score 332 is greater than or equal to a score threshold 336 associated with a known type of algorithmic attack. For example, the comparison logic 320 may retrieve the score threshold 336 from the storage 312. The comparison logic 320 compares the classification score 332 with the score threshold 336 to determine whether the classification score 332 is greater than or equal to the score threshold 336. The comparison logic 320 may generate a comparison result 334 indicating whether the classification score 332 is greater than or equal to the score threshold 336. For example, if the classification score 332 is greater than or equal to the score threshold 336, the comparison result 334 may have a first value (e.g., "1"). If the classification score 332 is less than the score threshold 336, the comparison result 334 may have a second value different from the first value (e.g., "0").

[0097] At step 208, an algorithmic attack is detected. In an example implementation, the detection logic 322 detects an algorithmic attack. For example, the detection logic 322 may detect an algorithmic attack based on (e.g., as a result of) a comparison result 334 indicating that the classification score 332 is greater than or equal to a score threshold 336 (e.g., based on the comparison result 334 having a first value). In one aspect, the detection logic 322 may detect that the algorithmic attack is of a known type based at least in part on the classification score 332 being greater than or equal to the score threshold 336. After completing step 208, the flowchart 200 ends.

[0098] At step 210, an algorithmic attack is not detected. In an example implementation, the detection logic 322 does not detect an algorithmic attack. For example, the detection logic 322 may determine that an algorithmic attack has not occurred or has not occurred. For example, the detection logic 322 may not detect an algorithmic attack based on the comparison result 334 indicating that the classification score 332 is less than the score threshold 336 (e.g., based on the comparison result 334 having the second value). After completing step 210, the flowchart 200 ends.

[0099] In some example embodiments, one or more steps 202, 204, 206, 208 and / or 210 of flowchart 200 may not be performed. In addition, steps in addition to or in place of steps 202, 204, 206, 208 and / or 210 may be performed. For example, in an example embodiment, the numerical representation of the corresponding query is a local sensitive hash of the corresponding query. Local sensitive hashing is the result of applying a local sensitive hashing operation (e.g., a function) to an input (e.g., a query). A local sensitive hashing operation is an operation that hashes similar inputs into the same group (also referred to as a bucket) with a high probability. For example, a local sensitive hashing operation can accept any suitable type of input (e.g., a digital vector, a string, a text file, or an image) and return a single integer (i.e., a group) value so that similar input items return the same group value. It will be appreciated that local sensitive hashing can enable high-dimensional inputs to be simplified to low-dimensional versions while retaining the distance between inputs. In an example, a local sensitive hash can be represented as a series of numbers (e.g., 128 numbers). In another example, a local sensitive hash can be a Gaussian sketch. According to this embodiment, the method of flowchart 200 also includes generating a local sensitive hash based on the corresponding query. It can be said that the query is encoded to provide a corresponding local sensitive hash. It can also be said that the query is converted to a local sensitive hash using a hash operation. In an example implementation, the encoding logic 324 can generate a local sensitive hash of the corresponding query 326. Depending on the type of query represented by the corresponding local sensitive hash, each local sensitive hash in the local sensitive hash can be generated in any of a variety of ways. For example, the process for generating a local sensitive hash of an image may be relatively straightforward, while the process for generating a local sensitive hash of a text may be more complex. For example, if the query includes an image, the encoding logic 324 can generate a local sensitive hash of the image by performing a hash operation on the original image. In another example, if the query includes text, the encoding logic 324 can generate a local sensitive hash of the text by first converting the text into a numerical representation of the text (e.g., by using a Burton codec) and then performing a hash operation on the numerical representation of the text.

[0100] An example implementation using position-sensitive hashing will now be described. Assume that the input is a two-dimensional (2D) vector of numbers, where each element of the vector is between 0.0 and 10.0, such as [2.0, 3.0] or [6.5, 0.4]. The maximum distance of any input vector to the origin at [0, 0] is sqrt(10^2+10^2)=sqrt(200)=14.1421. Assume that three groups are specified: 0, 1, and 2. The position-sensitive hash function can be defined as follows: if the calculated distance is between [0.0, 5.0], group 0 is returned; if it is between [5.0, 10.0], group 1 is returned; if it is between [10.0, 15.0], group 2 is returned. Using this definition, both inputs [0.0, 1.0] and [1.5, 1.5] return group 0. In the example, the locality-sensitive hash function can be described as a clustering algorithm, where the group number is synonymous with the cluster ID.

[0101] It will be appreciated that computing system 300 may not include one or more of I / O-based attack detection logic 310, storage 312, feature derivation logic 314, model usage logic 316, feature-based classifier model 318, comparison logic 320, and / or detection logic 322. Furthermore, computing system 300 may include components in addition to or in place of I / O-based attack detection logic 310, storage 312, feature derivation logic 314, model usage logic 316, feature-based classifier model 318, comparison logic 320, and / or detection logic 322.

[0102] Fig.14 A flowchart 1400 is depicted of another example method for detecting an algorithmic attack against a hosted artificial intelligence system based on the input and output of the hosted artificial intelligence system according to an embodiment. For example, the flowchart 1400 may be composed of Figure 1 The first server (s) 106A shown executes. For the purpose of illustration, Fig.15 The computing system 1500 shown depicts the flowchart 1400. Fig.15 is an example implementation of the first server(s) 106A. Fig.15As shown, computing system 1500 includes I / O-based attack detection logic 1510 and storage 1512. I / O-based attack detection logic 1510 includes model usage logic 1516, comparison logic 1520, detection logic 1522, and transformer-based model 1576. Storage 1512 can be any suitable type of storage. One type of storage is a database. For example, storage 1512 can be a relational database, an entity-relational database, an object database, an object-relational database, an extensible markup language (XML) database, etc. For non-limiting illustrative purposes, storage 1512 is shown as storing reference vector information 1582 and distance threshold 1584. Based on the discussion about flowchart 1400, further structural and operational embodiments will be apparent to those skilled in the relevant art(s).

[0103] like Fig.14 As shown, the method of flowchart 1400 begins at step 1402. In step 1402, by providing a multivariate time series as an input to a transformer-based model, the transformer-based model is used to generate a vector (e.g., a single vector) that summarizes the queries received by the hosted artificial intelligence system and the outputs obtained by the hosted artificial intelligence system processing the corresponding queries. The queries can be associated with a common (e.g., the same) user session. For example, the queries can define a user session. The user session can start at a first time instance of a first query in time when the user device initiates the query and end at a second time instance of the last output in time when the output is received at the user device. The multivariate time series includes multiple elements. Each element is based at least in part on a pairwise distance between a numerical representation of the corresponding query and a numerical representation of a previous query that precedes the corresponding query in time, and is also based at least in part on (multiple) attributes of the output obtained by processing the corresponding query.

[0104] In an example implementation, the model-using logic 1516 generates a vector 1580 using the transformer-based model 1576 by providing the multivariate time series 1578 as an input to the transformer-based model 1576. The vector 1580 summarizes the query 1526 received by the hosted artificial intelligence system and the output 1528 resulting from processing the corresponding query 1526 by the hosted artificial intelligence system. The multivariate time series 1578 includes a plurality of elements. Each element is based at least in part on a pairwise distance between a numerical representation of the corresponding query and a numerical representation of a previous query that precedes the corresponding query in time, and is also based at least in part on an attribute(s) of the output resulting from processing the corresponding query.

[0105] Using a transformer-based model to generate vectors at step 1402 can improve the accuracy and precision of detecting algorithmic attacks by using both the input (e.g., query) and output of the hosted artificial intelligence system. By using summaries of the query and output, step 1402 can increase the security of the hosted artificial intelligence system and / or increase the privacy of the user who initiated the query. If logs are used to store information about queries (e.g., by storing a few numbers for each query instead of multiple megabytes of data), using a transformer-based model to generate vectors at step 1402 can reduce the size of the logs, can reduce the amount of storage consumed by the logs, and can reduce the number of calculations performed on the logs. Therefore, using a transformer-based model to generate vectors at step 1402 can reduce the amount of time and / or resources consumed by the computing system to generate and / or process the logs. Using a transformer-based model to generate vectors at step 1402 can improve the efficiency of the computing system used to detect algorithmic attacks.

[0106] In an example embodiment, the vector includes a plurality of classifications, wherein the outputs are classified, and a plurality of confidence values ​​are associated with the respective classifications. For example, each confidence value may be represented using a respective floating point number. According to this embodiment, each confidence value is based on the confidence of the classification of each output in the respective classification. Further according to this embodiment, the attribute(s) of each output includes the confidence in the respective output (e.g., the confidence that the respective output corresponds to the respective query).

[0107] In another example embodiment, the vector includes a plurality of classifications, wherein the outputs are classified, and a plurality of confidences are associated with the respective classifications. According to this embodiment, the attribute(s) of each output includes an entropy associated with a respective plurality of potential outputs, the respective plurality of potential outputs being associated with a respective query from which the respective output is selected.

[0108] In yet another example embodiment, the property(s) of each output comprises a standard deviation associated with a distribution of a respective plurality of potential outputs associated with a respective query from which the respective output was selected.

[0109] In yet another example embodiment, the numerical representation of each query includes a locality sensitive hash of the corresponding query. For example, each locality sensitive hash may include a Gaussian sketch of the corresponding query.

[0110] In another example embodiment, each element of the multivariate time series is based at least in part on a first pairwise distance between a numerical representation of the corresponding query and a numerical representation of a previous query that most recently preceded the corresponding query in time, and is also based at least in part on a second pairwise distance between the numerical representation of the corresponding query and a numerical representation of a previous query that is the first query in time among the queries in a user session defined by the query.

[0111] At step 1404, it is determined whether the distance (e.g., the shortest distance) between the vector and a point corresponding to a reference vector associated with a known type of algorithmic attack is less than or equal to a distance threshold. For example, cosine similarity may be used to measure the distance. If the distance is less than or equal to the distance threshold, the process continues to step 1406. Otherwise, the process continues to step 1408. In an example implementation, the comparison logic 1520 determines whether the distance between the vector 1580 and a point corresponding to a reference vector associated with a known type of algorithmic attack is less than or equal to a distance threshold 1584. For example, the comparison logic 1520 may retrieve the reference vector information 1582 and the distance threshold 1584 from the storage 1512. The comparison logic 1520 may determine the reference vector based on the reference vector information 1582 indicating the reference vector. The comparison logic 1520 may generate a comparison result 1534 to indicate whether the distance between the vector 1580 and the point is less than or equal to the distance threshold 1584. For example, if the distance between vector 1580 and the point is less than or equal to distance threshold 1584, comparison result 1534 may have a first value (e.g., "1"). If the distance between vector 1580 and the point is greater than distance threshold 1584, comparison result 1534 may have a second value (e.g., "0").

[0112] In an example embodiment, the point corresponds to the centroid of a plurality of reference vectors associated with a known type of algorithmic attack. For example, the point may represent the centroid. In another example, the point may be the centroid. In one aspect of this embodiment, the distance threshold is defined by a radius extending from the centroid to define a circle that includes the reference vector and excludes other vectors generated by the transformer-based model.

[0113] At step 1406, an algorithmic attack is detected. In an example implementation, the detection logic 1522 detects an algorithmic attack. For example, the detection logic 1522 may detect an algorithmic attack based on (e.g., as a result of) the comparison result 1534 indicating that the distance between the vector 1580 and the point is less than or equal to the distance threshold 1584 (e.g., based on the comparison result 1534 having a first value). In one aspect, the detection logic 1522 may detect that the algorithmic attack is of a known type based at least in part on the distance between the vector 1580 and the point being less than or equal to the distance threshold 1584. After completing step 1406, the flowchart 1400 ends.

[0114] At step 1408, no algorithmic attack is detected. In an example implementation, detection logic 1522 does not detect an algorithmic attack. For example, detection logic 1522 may determine that an algorithmic attack has not occurred or has not occurred. For example, detection logic 1522 may not detect an algorithmic attack based on comparison result 1534 indicating that the distance between vector 1580 and the point is greater than distance threshold 1584 (e.g., based on comparison result 1534 having a second value). After completing step 1408, flowchart 1400 ends.

[0115] In some example embodiments, one or more steps 1402, 1404, 1406, and / or 1408 of flowchart 1400 may not be performed. Furthermore, steps in addition to or in place of steps 1402, 1404, 1406, and / or 1408 may be performed.

[0116] It will be appreciated that computing system 1500 may not include one or more of I / O-based attack detection logic 1510, storage 1512, model usage logic 1516, comparison logic 1520, detection logic 1522, and / or transformer-based model 1576. Furthermore, computing system 1500 may include components in addition to or in place of I / O-based attack detection logic 1510, storage 1512, model usage logic 1516, comparison logic 1520, detection logic 1522, and / or transformer-based model 1576.

[0117] Although the operations of some of the disclosed methods are described in a specific, sequential order for ease of presentation, it should be understood that unless a specific ordering is required by the specific language set forth herein, this description encompasses rearrangement. For example, in some cases, the operations described sequentially may be rearranged or performed simultaneously. In addition, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods may be used in conjunction with other methods.

[0118] I / O-based attack detection logic 310, feature derivation logic 314, model use logic 316, feature-based classifier model 318, comparison logic 320, detection logic 322, confidence determination logic 540, entropy determination logic 542, feature logic 544, input difference logic 1152, output difference logic 1154, ratio determination logic 1156, selection logic 1158, feature logic 1144, distance determination logic 1374, value determination logic 1376, Any one or more of the feature logic 1344, I / O-based attack detection logic 1510, model usage logic 1516, comparison logic 1520, detection logic 1522 and / or transformer-based model 1576, flowchart 200, flowchart 400, flowchart 600, flowchart 700, flowchart 800, flowchart 900, flowchart 1000, flowchart 1200 and / or flowchart 1400 may be implemented in hardware, software, firmware, or any combination thereof.

[0119] For example, I / O-based attack detection logic 310, feature derivation logic 314, model use logic 316, feature-based classifier model 318, comparison logic 320, detection logic 322, confidence determination logic 540, entropy determination logic 542, feature logic 544, input difference logic 1152, output difference logic 1154, ratio determination logic 1156, selection logic 1158, feature logic 1144, distance determination logic 1374, value determination logic 1376, feature logic 1377 44. Any one or more of the I / O-based attack detection logic 1510, model usage logic 1516, comparison logic 1520, detection logic 1522 and / or transformer-based model 1576, flowchart 200, flowchart 400, flowchart 600, flowchart 700, flowchart 800, flowchart 900, flowchart 1000, flowchart 1200 and / or flowchart 1400 may be implemented at least in part as computer program code configured to be executed in one or more processors.

[0120] In another example, I / O-based attack detection logic 310, feature derivation logic 314, model use logic 316, feature-based classifier model 318, comparison logic 320, detection logic 322, confidence determination logic 540, entropy determination logic 542, feature logic 544, input difference logic 1152, output difference logic 1154, ratio determination logic 1156, selection logic 1158, feature logic 1144, distance determination logic 1374, value determination logic 1376, feature logic 1344, I / O-based attack detection logic 1510, model usage logic 1516, comparison logic 1520, detection logic 1522 and / or transformer-based model 1576, flowchart 200, flowchart 400, flowchart 600, flowchart 700, flowchart 800, flowchart 900, flowchart 1000, flowchart 1200 and / or flowchart 1400. Any one or more may be implemented at least in part as hardware logic / circuitry. Such hardware logic / circuitry may include one or more hardware logic components. Examples of hardware logic components include, but are not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SoCs), complex programmable logic devices (CPLDs), and the like. For example, a SoC may include an integrated circuit chip that includes a processor (e.g., a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more of additional circuits and / or embedded firmware to perform its functions. III. Further Discussion of Some Example Embodiments

[0121] (A1) A first example system for detecting algorithmic attacks against a hosted artificial intelligence system ( Figure 1 , 102A-102M, 106A-106N; Figure 3 , 300; Fig.16 , 1600) includes a memory ( Fig.16 , 1604, 1608, 1610) and a processing system ( Fig.16 , 1602). The processing system is configured to receive a corresponding query ( Figure 3 , 326) and the output obtained by the managed artificial intelligence system processing the corresponding query ( Figure 3 , 328), we get ( Figure 2 , 202) features associated with known types of algorithmic attacks ( Figure 3 , 330). The processing system is further configured to provide the derived features as input to a feature-based classifier model using ( Figure 2 , 204) Feature-based classifier model ( Figure 3, 318) to generate the classification score ( Figure 3 , 332), the classification score indicates the likelihood that at least a portion of the query corresponds to a known type of algorithmic attack. The processing system is also configured to compare the classification score to a score threshold associated with the known type of algorithmic attack ( Figure 3 , 336) for comparison ( Figure 2 , 206). The processing system is further configured to detect ( Figure 2 , 208) algorithm attack.

[0122] (A2) In the example system of A1, the numerical representation of the corresponding query is a locality sensitive hash of the corresponding query.

[0123] (A3) In the example system of any of items A1 to A2, the query received by the hosted artificial intelligence system includes a first query; wherein the first query includes an image; and wherein the processing system is further configured to: perform a hash operation on the image to generate a first locality sensitive hash, which is included in the locality sensitive hash.

[0124] (A4) In the example system of any of items A1 to A3, the query received by the hosted artificial intelligence system includes a first query; wherein the first query includes text; and wherein the processing system is further configured to: convert the text into a numerical representation of the text; and perform a hash operation on the numerical representation of the text to generate a first locality sensitive hash, which is included in the locality sensitive hash.

[0125] (A5) In the example system of any of items A1 to A4, the processing system is configured to: spectrally analyze the output obtained by the hosted artificial intelligence system processing the corresponding query to determine that the output exhibits periodicity; and derive a first feature based at least in part on the output exhibiting periodicity; and wherein the output exhibiting periodicity is conducive to increasing the classification score.

[0126] (A6) In the example system of any of items A1 to A5, the processing system is configured to: determine a distance between a numerical representation of each query in at least a subset of queries and a numerical representation of a reference query; determine a difference between an output obtained by processing at least the subset of each query by the hosted artificial intelligence system and a reference output obtained by processing the reference query by the hosted artificial intelligence system; determine a ratio for each query in at least the subset to provide a plurality of ratios, each ratio being equal to the difference between the output obtained by processing the corresponding query by the hosted artificial intelligence system and the reference output divided by the distance between the numerical representation of the corresponding query and the numerical representation of the reference query; select a ratio of the plurality of ratios that is not less than each other ratio of the plurality of ratios; and derive a first feature based at least in part on the selected ratio; wherein a relatively high selected ratio is conducive to increasing the classification score; and wherein a relatively low selected ratio is not conducive to increasing the classification score.

[0127] (A7) In the example system of any one of items A1 to A6, the processing system is configured to: determine a distance between each numerical representation of the corresponding query and a nearest fit line defined by the numerical representation of the corresponding query; determine a difference value based at least in part on the distance between the nearest fit line and the corresponding numerical representation of the corresponding query; and derive a first feature based at least in part on the difference value; wherein the difference value is relatively low to facilitate increasing the classification score; and wherein the difference value is relatively high to facilitate not increasing the classification score.

[0128] (A8) In the example system of any one of items A1 to A7, the processing system is configured to: determine that a numerical representation of the corresponding query defines a Gaussian distribution; and derive the first feature based at least in part on the Gaussian distribution; and wherein the numerical representation of the corresponding query that defines the Gaussian distribution is conducive to increasing the classification score.

[0129] (A9) In the example system of any one of items A1 to A8, the processing system is configured to: derive the first feature based at least in part on a standard deviation of a Gaussian distribution; wherein a relatively low standard deviation is conducive to increasing the classification score; and wherein a relatively high standard deviation is not conducive to increasing the classification score.

[0130] (A10) In the example system of any one of items A1 to A9, the processing system is configured to: determine an entropy associated with a numerical representation of a corresponding query; and derive a first feature based at least in part on the entropy; wherein a relatively low entropy is conducive to increasing the classification score; and wherein a relatively high entropy is not conducive to increasing the classification score.

[0131] (A11) In the example system of any of items A1 to A10, the processing system is configured to: determine a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query; determine an entropy associated with the confidence; and derive a first feature based at least in part on the entropy; wherein a relatively low entropy is conducive to increasing the classification score; and wherein a relatively high entropy weight is not conducive to increasing the classification score.

[0132] (A12) In the example system of any of items A1 to A11, the processing system is configured to: determine a confidence associated with a corresponding output resulting from processing the corresponding query by the hosted artificial intelligence system; determine an entropy at a corresponding time corresponding to the corresponding output, each entropy being based on the confidence associated with the corresponding output and the confidence of each output preceding the corresponding output; determine a trend of the entropy decreasing over time; and derive a first feature based at least in part on the trend of the entropy decreasing over time; and wherein the trend of the entropy decreasing over time is conducive to increasing the classification score.

[0133] (A13) In the example system of any of items A1 to A12, the processing system is configured to: determine an entropy associated with a numerical representation of a corresponding query; and derive a first feature based at least in part on the entropy; wherein a relatively high entropy is conducive to increasing the classification score; and wherein a relatively low entropy is not conducive to increasing the classification score.

[0134] (A14) In the example system of any of items A1 to A13, the processing system is configured to: determine a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query; determine an entropy associated with the confidence; and derive a first feature based at least in part on the entropy; wherein a relatively high entropy is conducive to increasing the classification score; and wherein a relatively low entropy is disadvantageous to not increasing the classification score.

[0135] (A15) In the example system of any one of items A1 to A14, the processing system is configured to: determine a confidence associated with a corresponding output resulting from processing the corresponding query by the hosted artificial intelligence system; determine an entropy at a corresponding time corresponding to the corresponding output, each entropy being based on the confidence associated with the corresponding output and the confidence of each output preceding the corresponding output; determine a trend of entropy increasing over time; and derive a first feature based at least in part on the trend of entropy increasing over time; and wherein the trend of entropy increasing over time is conducive to increasing the classification score.

[0136] (A16) In an example system of any one of A1 to A15, the processing system is configured to: determine a confidence associated with a corresponding output resulting from processing the corresponding query by the hosted artificial intelligence system; determine an entropy at a corresponding time corresponding to the corresponding output, each entropy based on the confidence associated with the corresponding output and the confidence of each output preceding the corresponding output; determine a trend of the entropy increasing or decreasing over time; determine that the rate at which the trend of the entropy increasing or decreasing over time increases over time; and derive a first feature based at least in part on the trend of the entropy increasing or decreasing over time and also at least in part on the rate increasing over time; and wherein the combination of the trend of the entropy increasing or decreasing over time and the rate increasing over time is conducive to increasing the classification score.

[0137] (A17) In the example system of any of A1 to A16, the processing system is configured to: detect that the algorithmic attack is a known type based at least in part on the classification score being greater than or equal to a score threshold associated with the known type of algorithmic attack.

[0138] (B1) A second example system for detecting algorithmic attacks against a hosted artificial intelligence system ( Figure 1 , 102A-102M, 106A-106N; Fig.15 , 1500; Fig.16 , 1600) includes a memory ( Fig.16 , 1604, 1608, 1610) and a processing system ( Fig.16 , 1602). The processing system is configured to provide a multivariate time series ( Fig.15 , 1578) as the input of the transformer-based model, using ( Fig.14 , 1402) Transformer-based model ( Fig.15 , 1576) to generate the vector ( Fig.15 , 1580), which summarizes multiple queries received by the managed AI system ( Fig.15 , 1526) and multiple outputs obtained by processing the multiple corresponding queries by the hosted artificial intelligence system ( Fig.15 , 1528). The multivariate time series includes a plurality of elements. Each element is based at least in part on a pairwise distance between a numerical representation of a corresponding query in the plurality of queries and a numerical representation of a previous query that precedes the corresponding query in time, and is also based at least in part on one or more attributes of an output obtained by processing the corresponding query. The processing system is also configured to determine ( Fig.14 , 1404) is the distance between the vector and the point corresponding to the reference vector associated with a known type of algorithmic attack less than or equal to a distance threshold ( Fig.15 , 1584). The processing system is further configured to detect ( Fig.14 , 1406) algorithm attack.

[0139] (B2) In the example system of B1, wherein the vector includes multiple categories in which the output is classified and multiple confidence values ​​associated with the multiple corresponding categories, wherein each confidence value is based on the confidence of the classification of each output in the corresponding category; and wherein one or more attributes of each output include the confidence in the corresponding output.

[0140] (B3) In the example system of any of items B1 to B2, wherein the vector includes multiple classifications, wherein the outputs are classified and multiple confidences are associated with the multiple corresponding classifications; and wherein one or more attributes of each output include entropy associated with a corresponding multiple potential outputs, the corresponding multiple potential outputs are associated with a corresponding query, and the corresponding output is selected from the corresponding query.

[0141] (B4) In the example system of any of items B1 to B3, the one or more attributes of each output include a standard deviation associated with a distribution of a corresponding plurality of potential outputs, the plurality of potential outputs being associated with a corresponding query from which the corresponding output is selected.

[0142] (B5) In the example system of any of B1 to B4, the numerical representation of each query in the plurality of queries comprises a locality sensitive hash of the corresponding query.

[0143] (B6) In an example system of any of items B1 to B5, each element is based at least in part on a first pairwise distance between a numerical representation of a corresponding query in a plurality of queries and a numerical representation of a previous query that most recently preceded the corresponding query in time, and is also based at least in part on a second pairwise distance between the numerical representation of the corresponding query and a numerical representation of a previous query that is the first query in time in a user session defined by the plurality of queries.

[0144] (B7) In the example system of any of items B1 to B6, the processing system is configured to: determine whether the distance between the vector and a point is less than or equal to a distance threshold, the point corresponding to the centroid of multiple reference vectors associated with a known type of algorithmic attack.

[0145] (B8) In the example system of any of items B1 to B7, wherein the distance threshold is defined by a radius extending from the centroid to define a circle that includes the plurality of reference vectors and excludes other vectors generated by the transformer-based model.

[0146] (B9) In an example system of any of items B1 to B8, the processing system is configured to: detect that the algorithmic attack is of a known type based at least in part on a distance between the vector and a point being less than or equal to a distance threshold, the point corresponding to a reference vector associated with the known type of algorithmic attack.

[0147] (C1) A first example method for detecting an algorithmic attack against a hosted artificial intelligence system. The method comprises a computing system ( Figure 1 , 102A-102M, 106A-106N; Figure 3 , 300; Fig.16 , 1600). The method includes receiving a corresponding query ( Figure 3 , 326) and the output obtained by the managed artificial intelligence system processing the corresponding query ( Figure 3 , 328), we get ( Figure 2 , 202) features associated with known types of algorithmic attacks ( Figure 3 , 330). The method further includes providing the derived features as input to a feature-based classifier model, using ( Figure 2 , 204) Feature-based classifier model ( Figure 3 , 318) to generate the classification score ( Figure 3 , 332), the classification score indicates the likelihood that at least a portion of the query corresponds to a known type of algorithmic attack. The method also includes comparing the classification score to a score threshold associated with the known type of algorithmic attack ( Figure 3 , 336) for comparison ( Figure 2 , 206). The method also includes detecting ( Figure 2 , 208) algorithm attack.

[0148] (C2) In the method of C1, the numerical representation of the corresponding query is a locality sensitive hash of the corresponding query.

[0149] (C3) In the method of any one of items C1 to C2, wherein the query received by the hosted artificial intelligence system includes a first query; wherein the first query includes an image; and wherein the method further includes: performing a hash operation on the image to generate a first locality sensitive hash, the first locality sensitive hash being included in the locality sensitive hash.

[0150] (C4) In the method of any one of items C1 to C3, wherein the query received by the hosted artificial intelligence system includes a first query; wherein the first query includes text; and wherein the method further includes: converting the text into a numerical representation of the text; and performing a hash operation on the numerical representation of the text to generate a first locality sensitive hash, which is included in the locality sensitive hash.

[0151] (C5) In the method of any one of items C1 to C4, deriving the feature includes: spectrally analyzing the output obtained by the hosted artificial intelligence system processing the corresponding query to determine that the output exhibits periodicity; and deriving the first feature based at least in part on the output exhibiting periodicity; and wherein the output exhibiting periodicity is conducive to increasing the classification score.

[0152] (C6) In the method of any one of items C1 to C5, wherein deriving the feature includes: determining a distance between a numerical representation of each query in at least a subset of queries and a numerical representation of a reference query; determining a difference between an output obtained by processing at least a subset of each query by the hosted artificial intelligence system and a reference output obtained by processing the reference query by the hosted artificial intelligence system; determining a ratio for each query in at least a subset to provide a plurality of ratios, each ratio being equal to a difference between an output obtained by processing the corresponding query by the hosted artificial intelligence system and the reference output divided by a distance between the numerical representation of the corresponding query and the numerical representation of the reference query; selecting a ratio of the plurality of ratios that is not less than each other ratio of the plurality of ratios; and deriving a first feature based at least in part on the selected ratio; wherein a relatively high ratio selected is conducive to increasing the classification score; and wherein a relatively low ratio selected is not conducive to increasing the classification score.

[0153] (C7) In the method of any one of items C1 to C6, wherein deriving the feature includes: determining a distance between each numerical representation of the corresponding query and a nearest fit line defined by the numerical representation of the corresponding query; determining a difference value based at least in part on the distance between the nearest fit line and the corresponding numerical representation of the corresponding query; and deriving the first feature based at least in part on the difference value; wherein the difference value is relatively low to facilitate increasing the classification score; and wherein the difference value is relatively high to facilitate not increasing the classification score.

[0154] (C8) In the method of any one of items C1 to C7, wherein deriving the feature includes: determining that the numerical representation of the corresponding query defines a Gaussian distribution; and deriving the first feature based at least in part on the Gaussian distribution; and wherein the numerical representation of the corresponding query that defines the Gaussian distribution is beneficial to increasing the classification score.

[0155] (C9) In the method of any one of items C1 to C8, deriving the first feature includes: deriving the first feature based at least in part on the standard deviation of a Gaussian distribution; wherein a relatively low standard deviation is conducive to increasing the classification score; and wherein a relatively high standard deviation is not conducive to increasing the classification score.

[0156] (C10) In the method of any one of items C1 to C9, wherein deriving the feature includes: determining an entropy associated with a numerical representation of the corresponding query; and deriving a first feature based at least in part on the entropy; wherein a relatively low entropy is conducive to increasing the classification score; and wherein a relatively high entropy is not conducive to increasing the classification score.

[0157] (C11) In the method of any one of items C1 to C10, deriving the feature includes: determining a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query; determining an entropy associated with the confidence; and deriving the first feature based at least in part on the entropy; wherein a relatively low entropy is conducive to increasing the classification score; and wherein a relatively high entropy weight is not conducive to increasing the classification score.

[0158] (C12) In the method of any one of items C1 to C11, wherein deriving the feature includes: determining a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query; determining an entropy at a corresponding time corresponding to the corresponding output, each entropy being based on the confidence associated with the corresponding output and the confidence of each output before the corresponding output; determining a trend of the entropy decreasing over time; and deriving the first feature based at least in part on the trend of the entropy decreasing over time; and wherein the trend of the entropy decreasing over time is conducive to increasing the classification score.

[0159] (C13) In the method of any one of items C1 to C12, wherein deriving the feature includes: determining an entropy associated with a numerical representation of the corresponding query; and deriving a first feature based at least in part on the entropy; wherein a relatively high entropy is conducive to increasing the classification score; and wherein a relatively low entropy is not conducive to increasing the classification score.

[0160] (C14) In the method of any one of items C1 to C13, wherein deriving the feature includes: determining a confidence associated with a corresponding output obtained by processing the corresponding query by the hosted artificial intelligence system; determining an entropy associated with the confidence; and deriving the first feature based at least in part on the entropy; wherein a relatively high entropy is conducive to increasing the classification score; and wherein a relatively low entropy is not conducive to not increasing the classification score.

[0161] (C15) In the method of any one of items C1 to C14, wherein deriving the feature includes: determining a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query; determining an entropy at a corresponding time corresponding to the corresponding output, each entropy being based on the confidence associated with the corresponding output and the confidence of each output before the corresponding output; determining a trend of entropy increasing over time; and deriving the first feature based at least in part on the trend of entropy increasing over time; and wherein the trend of entropy increasing over time is conducive to increasing the classification score.

[0162] (C16) In the method of any one of items C1 to C15, wherein deriving a feature includes: determining a confidence associated with a corresponding output obtained by the hosted artificial intelligence system processing the corresponding query; determining an entropy at a corresponding time corresponding to the corresponding output, each entropy being based on the confidence associated with the corresponding output and the confidence of each output before the corresponding output; determining a trend of the entropy increasing or decreasing over time; determining that the rate at which the trend of the entropy increasing or decreasing over time increases over time; and deriving a first feature based at least in part on the trend of the entropy increasing or decreasing over time and also at least in part on the rate increasing over time; and wherein the combination of the trend of the entropy increasing or decreasing over time and the rate increasing over time is conducive to increasing the classification score.

[0163] (C17) In the method of any of C1 to C16, detecting that the algorithmic attack is of a known type is based at least in part on the classification score being greater than or equal to a score threshold associated with an algorithmic attack of a known type.

[0164] (D1) A second example method for detecting an algorithmic attack against a hosted artificial intelligence system. The method comprises a computing system ( Figure 1 , 102A-102M, 106A-106N; Fig.15 , 1500; Fig.16 , 1600). The method includes providing a multivariate time series ( Fig.15 , 1578) as the input of the transformer-based model, using ( Fig.14 , 1402) Transformer-based model ( Fig.15 , 1576) to generate the vector ( Fig.15 , 1580), which summarizes multiple queries received by the managed AI system ( Fig.15 , 1526) and multiple outputs obtained by processing the multiple corresponding queries by the hosted artificial intelligence system ( Fig.15, 1528), the multivariate time series includes a plurality of elements, each element based at least in part on a pairwise distance between a numerical representation of a corresponding query in the plurality of queries and a numerical representation of a previous query that precedes the corresponding query in time, and also based at least in part on one or more attributes of an output obtained by processing the corresponding query. The method also includes determining ( Fig.14 , 1404) is the distance between the vector and the point corresponding to the reference vector associated with a known type of algorithmic attack less than or equal to a distance threshold ( Fig.15 , 1584). The method includes detecting (based at least in part on a distance between the vector and the point being less than or equal to a distance threshold) Fig.14 , 1406) algorithm attack.

[0165] (D2) In the method of D1, wherein the vector includes multiple categories in which the output is classified and multiple confidence values ​​associated with the multiple corresponding categories, wherein each confidence value is based on the confidence of the classification of each output in the corresponding category; and wherein one or more attributes of each output include the confidence in the corresponding output.

[0166] (D3) In the method of any one of D1 to D2, wherein the vector comprises multiple classifications, wherein the outputs are classified and multiple confidences are associated with the multiple corresponding classifications; and wherein one or more attributes of each output include entropy associated with a corresponding multiple potential outputs, the corresponding multiple potential outputs are associated with a corresponding query, and the corresponding output is selected from the corresponding query.

[0167] (D4) In the method of any of D1 to D3, wherein the one or more attributes of each output include a standard deviation associated with a distribution of a corresponding plurality of potential outputs, the plurality of potential outputs being associated with a corresponding query from which the corresponding output is selected.

[0168] (D5) In the method of any one of D1 to D4, wherein the numerical representation of each query in the plurality of queries comprises a locality sensitive hash of the corresponding query.

[0169] (D6) In the method of any of items D1 to D5, each element is based at least in part on a first pairwise distance between a numerical representation of a corresponding query in a plurality of queries and a numerical representation of a previous query that most recently preceded the corresponding query in time, and is also based at least in part on a second pairwise distance between the numerical representation of the corresponding query and a numerical representation of a previous query that is the first query in time in a user session defined by the plurality of queries.

[0170] (D7) In the method of any one of D1 to D6, determining whether the distance between a vector and a point is less than or equal to a distance threshold includes: determining whether the distance between the vector and a point is less than or equal to the distance threshold, the point corresponding to the centroid of multiple reference vectors associated with a known type of algorithmic attack.

[0171] (D8) In the method of any of D1 to D7, wherein the distance threshold is defined by a radius extending from the centroid to define a circle that includes the plurality of reference vectors and excludes other vectors generated by the transformer-based model.

[0172] (D9) In the method of any one of items D1 to D8, detecting an algorithmic attack includes: detecting that the algorithmic attack is of a known type based at least in part on a distance between a vector and a point being less than or equal to a distance threshold, the point corresponding to a reference vector associated with the known type of algorithmic attack.

[0173] (E1) First example computer program product ( Fig.16 , 1618, 1622), including a computer-readable storage medium having recorded thereon instructions for causing a processor-based system ( Figure 1 , 102A-102M, 106A-106N; Figure 3 , 300; Fig.16 , 1600) can detect algorithmic attacks against a hosted artificial intelligence system by performing operations. The operations include receiving a corresponding query ( Figure 3 , 326) and the output obtained by the managed artificial intelligence system processing the corresponding query ( Figure 3 , 328), we get ( Figure 2 , 202) features associated with known types of algorithmic attacks ( Figure 3 , 330). The operation also includes providing the derived features as input to a feature-based classifier model, using ( Figure 2 , 204) Feature-based classifier model ( Figure 3 , 318) to generate the classification score ( Figure 3 , 332), the classification score indicates a likelihood that at least a portion of the query corresponds to a known type of algorithmic attack. The operation also includes comparing the classification score to a score threshold associated with the known type of algorithmic attack ( Figure 3 , 336) for comparison ( Figure 2 , 206). The operations also include detecting ( Figure 2 , 208) algorithm attack.

[0174] (F1) Second example computer program product ( Fig.16, 1618, 1622), including a computer-readable storage medium having recorded thereon instructions for causing a processor-based system ( Figure 1 , 102A-102M, 106A-106N; Fig.15 , 1500; Fig.16 , 1600) is capable of detecting algorithmic attacks against a hosted artificial intelligence system by performing operations. The operations include providing a multivariate time series ( Fig.15 , 1578) as the input of the transformer-based model, using ( Fig.14 , 1402) Transformer-based model ( Fig.15 , 1576) to generate the vector ( Fig.15 , 1580), which summarizes multiple queries received by the hosted AI system ( Fig.15 , 1526) and multiple outputs obtained by the hosted artificial intelligence system processing multiple corresponding queries ( Fig.15 , 1528). The multivariate time series includes a plurality of elements. Each element is based at least in part on a pairwise distance between a numerical representation of a corresponding query in the plurality of queries and a numerical representation of a previous query that precedes the corresponding query in time, and is also based at least in part on one or more attributes of an output obtained by processing the corresponding query. The operation also includes determining ( Fig.14 , 1404) is the distance between the vector and the point corresponding to the reference vector associated with a known type of algorithmic attack less than or equal to a distance threshold ( Fig.15 , 1584). The operation also includes detecting ( Fig.14 , 1406) algorithm attack. IV. Example Computer System

[0175] Fig.16 An example computer 1600 is depicted in which embodiments may be implemented. Figure 1 Any one or more of the user devices 102A-102M and / or any one or more of the servers 106A-106N shown in Figure 3 The computing system 300 shown in FIG. 1 and / or Fig.15 The computing system 1500 shown in the can be implemented using a computer 1600, including one or more features and / or alternative features of the computer 1600. The computer 1600 can be a general-purpose computing device in the form of, for example, a conventional personal computer, a mobile computer, or a workstation, or the computer 1600 can be a special-purpose computing device. The description of the computer 1600 provided herein is provided for illustrative purposes and is not intended to be limiting. As known to those skilled in the relevant art(s), the embodiments can be implemented in other types of computer systems.

[0176] like Fig.16 As shown, computer 1600 includes a processing unit 1602, a system memory 1604, and a bus 1606 that couples various system components including the system memory 1604 to the processing unit 1602. Bus 1606 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. System memory 1604 includes read-only memory (ROM) 1608 and random access memory (RAM) 1610. Basic input / output system 1612 (BIOS) is stored in ROM 1608.

[0177] The computer 1600 also has one or more of the following drives: a hard disk drive 1614 for reading and writing a hard disk, a disk drive 1616 for reading and writing a removable disk 1618, and an optical drive 1620 for reading and writing a removable optical disk 1622 such as a CD ROM, DVD ROM or other optical media. The hard disk drive 1614, the disk drive 1616, and the optical drive 1620 are connected to the bus 1606 via a hard disk drive interface 1624, a disk drive interface 1626, and an optical drive interface 1628, respectively. The drives and their associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer. Although a hard disk, a removable disk, and a removable optical disk are described, other types of computer-readable storage media may be used to store data, such as flash memory cards, digital video disks, random access memory (RAM), read-only memory (ROM), etc.

[0178] A plurality of program modules may be stored in a hard disk, a disk, an optical disk, a ROM, or a RAM, and these programs include an operating system 1630, one or more application programs 1632, other program modules 1634, and program data 1636. The application program 1632 or the program module 1634 may include, for example, computer program logic for implementing any one or more of the following: (e.g., at least a portion thereof) I / O-based attack detection logic 310, feature derivation logic 314, model use logic 316, feature-based classifier model 318, comparison logic 320, detection logic 322, confidence determination logic 540, entropy determination logic 542, feature logic 544, input difference logic 1152, output difference logic 1154, ratio determination logic 1156, selection logic 1158, feature logic 1144, distance determination logic 1374, value determination logic 1376, feature logic 1344, I / O-based attack detection logic 1510, model use logic 1376, feature logic 1344, Using logic 1516, comparison logic 1520, detection logic 1522 and / or transformer-based model 1576, flowchart 200 (including any step of flowchart 200), flowchart 400 (including any step of flowchart 400), flowchart 600 (including any step of flowchart 600), flowchart 700 (including any step of flowchart 700), flowchart 800 (including any step of flowchart 800), flowchart 900 (including any step of flowchart 900), flowchart 1000 (including any step of flowchart 1000), flowchart 1200 (including any step of flowchart 1200) and / or flowchart 1400 (including any step of flowchart 1400), as described herein.

[0179] A user may enter commands and information into the computer 1600 through input devices such as a keyboard 1638 and pointing device 1640. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, touch screen, camera, accelerometers, gyroscopes, etc. These and other input devices are typically connected to the processing unit 1602 through a serial port interface 1642 coupled to the bus 1606, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB).

[0180] A display device 1644 (eg, a monitor) is also connected to bus 1606 via an interface, such as a video adapter 1646. In addition to display device 1644, computer 1600 may also include other peripheral output devices (not shown), such as speakers and printers.

[0181] The computer 1600 is connected to a network 1648 (eg, the Internet) through a network interface or adapter 1650, a modem 1652, or other means for establishing communications over the network. The modem 1652 (which may be internal or external) is connected to the bus 1606 via the serial port interface 1642.

[0182] As used herein, the terms "computer program medium" and "computer readable storage medium" are generally used to refer to media (e.g., non-transitory media) such as a hard disk associated with hard disk drive 1614, removable disk 1618, removable optical disk 1622, and other media such as flash memory cards, digital video disks, random access memory (RAM), read-only memory (ROM), etc. A computer-readable storage medium is not a signal, such as a carrier signal or a propagating signal. For example, a computer-readable storage medium may not include a signal. Therefore, a computer-readable storage medium itself does not constitute a signal. Such computer-readable storage media are distinguished from and do not overlap with communication media (excluding communication media). Communication media embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave. The term "modulated data signal" refers to a signal having one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared and other wireless media and wired media. Example embodiments also relate to such communication media.

[0183] As described above, computer programs and modules (including application programs 1632 and other program modules 1634) can be stored in a hard disk, disk, optical disk, ROM or RAM, and such computer programs can also be received via the network interface 1650 or the serial port interface 1642. Such computer programs, when executed or loaded by the application, enable the computer 1600 to implement the features of the embodiments discussed herein. Therefore, such computer programs represent controllers of the computer 1600.

[0184] Example embodiments also relate to a computer program product comprising software (e.g., computer-readable instructions) stored on any computer-usable medium. When executed in one or more data processing devices, such software causes (multiple) data processing devices to operate as described herein. Embodiments may employ any computer-usable or computer-readable medium known now or in the future. Examples of computer-readable media include, but are not limited to, storage devices such as RAM, hard disk drives, floppy disks, CD ROMs, DVD ROMs, zip disks, magnetic tapes, magnetic storage devices, optical storage devices, MEMS-based storage devices, nanotechnology-based storage devices, and the like.

[0185] It will be appreciated that the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure. V. Conclusion

[0186] Although the subject matter has been described with language specific to structural features and / or actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to fall within the scope of the claims.

Claims

1. A system for detecting algorithmic attacks against a hosted artificial intelligence system, the system comprising: Memory; as well as a processing system coupled to the memory, the processing system being configured to: deriving features associated with known types of algorithmic attacks from numerical representations of corresponding queries received by the hosted AI system and outputs resulting from processing of the corresponding queries by the hosted AI system; generating a classification score using the feature-based classifier model by providing the derived features as input to a feature-based classifier model, the classification score indicating a likelihood that at least a portion of the query corresponds to the known type of algorithmic attack; comparing the classification score to a score threshold associated with the known type of algorithmic attack; and The algorithmic attack is detected based at least in part on the classification score being greater than or equal to the score threshold, the score threshold associated with the known type of algorithmic attack.

2. The system of claim 1, wherein the processing system is configured to: spectrally analyzing the output resulting from the hosted artificial intelligence system's processing of the corresponding query to determine that the output exhibits periodicity; and deriving a first characteristic based at least in part on the output exhibiting a periodicity; and The output exhibiting periodicity is beneficial to increasing the classification score.

3. The system according to any one of claims 1 to 2, wherein the processing system is configured to: determining a distance between the numerical representation of each query of at least a subset of the queries and a numerical representation of a reference query; determining a difference between an output resulting from processing at least each query in the subset by the hosted AI system and a reference output resulting from processing the reference query by the hosted AI system; determining a ratio for each query in at least the subset to provide a plurality of ratios, each ratio being equal to the difference between the output resulting from processing the corresponding query by the hosted artificial intelligence system and the reference output divided by the distance between the numerical representation of the corresponding query and the numerical representation of the reference query; selecting a ratio of the plurality of ratios that is not less than every other ratio of the plurality of ratios; and deriving a first characteristic based at least in part on the selected ratio; wherein the selected ratio is relatively high in favor of increasing the classification score; and The selected ratio is relatively low which is not conducive to increasing the classification score.

4. The system according to any one of claims 1 to 3, wherein the processing system is configured to: determining a distance between each of the numerical representations of the corresponding query and a closest fit line defined by the numerical representations of the corresponding query; determining a difference value based at least in part on the distance between the closest fit line and the corresponding numerical representation of the corresponding query; as well as deriving a first feature based at least in part on the difference; wherein the difference is relatively low and is conducive to increasing the classification score; and Wherein the difference is relatively high, it is not conducive to increasing the classification score.

5. The system according to any one of claims 1 to 4, wherein the processing system is configured to: determining that the numerical representation of the corresponding query defines a Gaussian distribution; and deriving a first feature based at least in part on the Gaussian distribution; and The numerical representation of the corresponding query wherein the Gaussian distribution is defined is beneficial to increasing the classification score.

6. The system according to any one of claims 1 to 5, wherein the processing system is configured to: deriving the first feature based at least in part on a standard deviation of the Gaussian distribution; wherein a relatively low standard deviation is beneficial to increasing the classification score; and Wherein a relatively high standard deviation is not conducive to increasing the classification score.

7. The system according to any one of claims 1 to 6, wherein the processing system is configured to: determining an entropy associated with the numerical representation of the corresponding query; and deriving a first feature based at least in part on the entropy; and Any of the following: A relatively low entropy is conducive to increasing the classification score, and a relatively high entropy is not conducive to increasing the classification score, or A relatively high entropy is conducive to increasing the classification score, and a relatively low entropy is not conducive to increasing the classification score.

8. The system according to any one of claims 1 to 7, wherein the processing system is configured to: determining a confidence level associated with the corresponding output resulting from processing the corresponding query by the hosted artificial intelligence system; determining an entropy associated with the confidence level; and deriving a first feature based at least in part on the entropy; Among them are the following Any one of: A relatively low entropy is conducive to increasing the classification score, and a relatively high entropy is not conducive to increasing the classification score, or A relatively high entropy is conducive to increasing the classification score, and a relatively low entropy is not conducive to increasing the classification score.

9. The system according to any one of claims 1 to 8, wherein the processing system is configured to: determining a confidence level associated with the corresponding output resulting from processing the corresponding query by the hosted artificial intelligence system; determining entropies at respective times corresponding to the respective outputs, each entropy being based on the confidence associated with the respective output and the confidence of each output preceding the respective output; determining a trend of said entropy increasing or decreasing over time; as well as deriving a first characteristic based at least in part on the tendency of the entropy to increase or decrease over time; and Wherein the trend of the entropy increasing or decreasing over time is beneficial to increasing the classification score.

10. The system according to any one of claims 1 to 9, wherein the processing system is configured to: determining a confidence level associated with the corresponding output resulting from processing the corresponding query by the hosted artificial intelligence system; determining entropies at respective times corresponding to the respective outputs, each entropy being based on the confidence associated with the respective output and the confidence of each output preceding the respective output; determining a trend of said entropy increasing or decreasing over time; determining that the rate at which the tendency of the entropy to increase or decrease over time increases over time; and deriving a first characteristic based at least in part on the tendency of the entropy to increase or decrease over time and also based at least in part on the rate at which the entropy increases over time; and The combination of the trend of the entropy increasing or decreasing over time and the rate increasing over time is beneficial to increasing the classification score.

11. A method for detecting an algorithmic attack against a hosted artificial intelligence system, the method being implemented by a computing system, the method comprising: generating a vector using the transformer-based model by providing a multivariate time series as an input to the transformer-based model, the vector summarizing a plurality of queries received by the hosted artificial intelligence system and a plurality of outputs resulting from processing the plurality of corresponding queries by the hosted artificial intelligence system, the multivariate time series comprising a plurality of elements, each element based at least in part on a pairwise distance between a numerical representation of a corresponding query in the plurality of queries and a numerical representation of a previous query that temporally preceded the corresponding query, and further based at least in part on one or more attributes of the output resulting from processing the corresponding query; determining whether a distance between the vector and a point corresponding to a reference vector is less than or equal to a distance threshold, the reference vector being associated with a known type of algorithmic attack; as well as The algorithmic attack is detected based at least in part on the distance between the vector and the point being less than or equal to the distance threshold.

12. The method of claim 11, wherein the one or more attributes of each output include a standard deviation associated with a distribution of a corresponding plurality of potential outputs associated with the corresponding query from which the corresponding output was selected.

13. A method according to any one of claims 11 to 12, wherein each element is based at least in part on a first pairwise distance between the numerical representation of the corresponding query in the multiple queries and a numerical representation of a previous query that most recently preceded the corresponding query in time, and is also based at least in part on a second pairwise distance between the numerical representation of the corresponding query and a numerical representation of a previous query that is the first query in time in a user session defined by the multiple queries.

14. The method according to any one of claims 11 to 13, wherein determining whether the distance between the vector and the point is less than or equal to the distance threshold comprises: A determination is made whether the distance between the vector and the point is less than or equal to a distance threshold, the point corresponding to a centroid of a plurality of reference vectors associated with the known type of algorithmic attack.

15. A computer program product comprising a computer-readable storage medium having recorded thereon instructions for enabling a processor-based system to detect an algorithmic attack against a hosted artificial intelligence system by performing operations comprising: generating a vector using the transformer-based model by providing a multivariate time series as an input to the transformer-based model, the vector summarizing a plurality of queries received by the hosted artificial intelligence system and a plurality of outputs resulting from processing the plurality of corresponding queries by the hosted artificial intelligence system, the multivariate time series comprising a plurality of elements, each element based at least in part on a pairwise distance between a numerical representation of a corresponding query in the plurality of queries and a numerical representation of a previous query that temporally preceded the corresponding query, and further based at least in part on one or more attributes of the output resulting from processing the corresponding query; determining whether a distance between the vector and a point corresponding to a reference vector is less than or equal to a distance threshold, the reference vector being associated with a known type of algorithmic attack; as well as The algorithmic attack is detected based at least in part on the distance between the vector and the point being less than or equal to the distance threshold.