Visual AI agent detection method, apparatus, and electronic device

Abstract

This invention provides a visual AI agent detection method, apparatus, and electronic device, relating to the field of network security technology. The visual AI agent detection method includes: acquiring time information and / or coordinate information of interaction events on a user's interactive terminal within a preset time period; and predicting whether the interaction event was executed by a visual AI agent based on the time information and / or coordinate information. This invention can accurately and efficiently identify the automated access behavior of visual AI agents.

Description

Technical Field

[0001] This invention relates to the field of computer network security technology, and in particular to a visual AI agent detection method, device, and electronic device. Background Technology

[0002] In the field of cybersecurity, bot detection and automated access identification technologies are crucial. They effectively prevent the abuse of network resources by automated tools and ensure the secure and stable operation of business systems. Currently, existing bot detection and automated access identification technologies are mainly based on browser-side automation traces or network protocol characteristics. These technologies identify and block traditional automated tools based on script-driven or DOM parsing by detecting automation flags in the browser environment (such as navigator.webdriver), global objects or variables injected by automated frameworks, abnormal characteristics of the script execution environment, and abnormal HTTP header fields, access frequency, and path patterns on the request side.

[0003] However, with the continuous development of technology, vision-based AI agents have emerged and been applied. These next-generation vision-based agents operate in a completely different way from traditional automation tools. They do not rely on parsing HTML / DOM, nor do they require injecting script execution control logic into the page. Instead, they acquire screenshots, perform semantic understanding of image content, and execute interactive operations such as clicking, dragging, and input based on coordinates or pixel-level positioning. Their operation process is highly consistent with that of a human user. Because vision-based agents often run in real browsers and real rendering environments, their network request sequences, session persistence methods, page resource loading behavior, and TLS / HTTP fingerprints are extremely similar to those of normal user access. This makes traditional detection methods based on "script control / DOM-driven traces" risk failure, making it difficult to build stable and generalizable discrimination criteria. Meanwhile, detection methods that rely solely on traffic-side statistical features (such as frequency thresholds and path rules) are easily circumvented by visual agents through methods such as "humanized rhythm control, interaction delay perturbation, and behavior trajectory fitting," leading to problems such as high false positive rates, high false negative rates, and significant impact on business operations. This fails to meet the need for security protection against the abuse of visual agents.

[0004] Therefore, how to accurately and efficiently identify the automated access behavior of visual AI agents is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a visual AI agent detection method, apparatus, and electronic device to accurately and efficiently identify the automated access behavior of visual AI agents.

[0006] This invention provides a visual AI agent detection method, comprising the following steps.

[0007] Obtain time information and / or coordinate information of interactive events on a user's interactive terminal within a preset time period; predict whether the interactive events will be executed by a visual AI agent based on the time information and / or coordinate information of the interactive events.

[0008] According to a visual AI agent detection method provided by the present invention, the step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: Based on the time information of the interactive events within the preset time period, it is determined whether the interactive events exhibit inference waiting characteristics of a visual AI agent; if the interactive events exhibit inference waiting characteristics of a visual AI agent, it is predicted that the interactive events will be executed by a visual AI agent; wherein, the inference waiting characteristics of the visual AI agent include at least one of the following: multiple repetitive static windows exist within the preset time period; wherein, the density of interactive events occurring within the repetitive static windows is lower than a preset density threshold, the duration difference between the repetitive static windows is less than a preset difference threshold; and the interval time between the interactive events is within a preset duration range.

[0009] According to a visual AI agent detection method provided by the present invention, the interaction event includes multiple click operations performed by a mouse, and the coordinate information includes the click position coordinates of the mouse. The step of predicting whether the interaction event is performed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: determining whether the click operation has mathematically precise click characteristics and / or grid-aligned click characteristics based on the offset distribution between the click position coordinates of each click operation and the geometric center of the clicked target object; if the click operation has mathematically precise click characteristics and / or grid-aligned click characteristics, predicting that the interaction event is performed by a visual AI agent; wherein, the mathematically precise click characteristics include: the offset between the click position coordinates and the preset position coordinates of the clicked target object is less than a preset distance threshold; wherein, the grid-aligned click characteristics include: the click position coordinates of the multiple click operations exhibit a distribution according to a fixed spacing pattern in the horizontal and vertical directions.

[0010] According to a visual AI agent detection method provided by the present invention, the time information and / or coordinate information of the interaction event includes the time information and coordinate information of each pixel on the cursor path formed by mouse movement during the click operation; the step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: Based on the time and coordinate information of each pixel on the cursor path, it is determined whether the click operation has the trajectory characteristics of a visual AI agent; if it is determined that the click operation has the trajectory characteristics of a visual AI agent, it is predicted that the interaction event will be executed by the visual AI agent; wherein, the trajectory characteristics of the visual AI agent include at least one of the following characteristics: the coordinates of each pixel on the cursor path form a smooth straight line or curve; the cursor path does not have a curved path formed by mouse operation overshoot and callback; the cursor path speed is constant, or the cursor path acceleration is constant.

[0011] According to a visual AI agent detection method provided by the present invention, the step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: determining that the proportion of a first interaction event in the interaction event is less than a first preset proportion threshold, and / or the proportion of a second interaction event is greater than a second preset proportion threshold, and predicting that the interaction event is executed by a visual AI agent; wherein, the first interaction event includes scrolling, dragging, or infinite scrolling operations on the display interface of the user interaction terminal; the second interaction event includes key stepping and page-turning positioning operations on the display interface of the user interaction terminal; and the second preset proportion threshold is greater than the first preset proportion threshold.

[0012] According to a visual AI agent detection method provided by the present invention, the interaction event is an editing operation on a form object, and the time information and / or coordinate information of the interaction event includes the coordinate information and editing time information of each field of the form object; the step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: determining, based on the coordinate information and editing time information of each field of the form object, that the editing operation is performed in a preset order, and / or that there is no backtracking modification operation in the editing operation, predicting that the interaction event is executed by a visual AI agent.

[0013] According to a visual AI agent detection method provided by the present invention, the method further includes: Based on the time and / or coordinate information of the interaction event, if the probability that the interaction event is executed by the visual AI agent is greater than a preset probability threshold, at least one of the following operations is performed: a) Embed a guidance carrier invisible to the human interaction subject in the display interface of the user interaction terminal; if the guidance carrier is accessed, it is determined that the interaction event is executed by the visual AI agent; b) Embed a guidance instruction in the accessibility attribute of the form control in the display interface, so as to prompt the human interaction subject to input preset text content in the form control using the guidance instruction; set a verification path that the human interaction subject cannot execute corresponding to the guidance instruction; if it is determined that the verification path is executed and the content submitted by the form control is the preset text, it is determined that the interaction event is executed by the visual AI agent; c) Set a form field that is invisible or inaccessible to the human interaction subject, and set a placeholder in the associated position of the form field, and use the placeholder to prompt the execution of the access operation for the form field; if it is determined that the form field is accessed, it is determined that the interaction event is executed by the visual AI agent.

[0014] According to a visual AI agent detection method provided by the present invention, the method further includes: determining, through at least one of the following methods, whether the interaction event is executed by a visual AI agent: determining whether the source network corresponding to the interaction event belongs to a known cloud data center autonomous system or an abnormal proxy network; if the source network belongs to a known cloud data center autonomous system or an abnormal proxy network, determining that the interaction event is executed by a visual AI agent; detecting whether the browser function set and version used by the user interaction terminal are consistent; if the browser function set and version used by the user interaction terminal are inconsistent, determining that the interaction event is executed by a visual AI agent; detecting whether the renderer information of the browser used by the user interaction terminal exhibits software rasterizer characteristics; if the renderer information exhibits software rasterizer characteristics, further determining that the interaction event is executed by a visual AI agent.

[0015] The present invention also provides a visual AI agent detection device, comprising: The acquisition module is used to acquire time information and / or coordinate information of interactive events on the user's interactive terminal within a preset time period; the prediction module is used to predict whether the interactive event is executed by a visual AI agent based on the time information and / or coordinate information of the interactive event.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual AI agent detection method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual AI agent detection method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the visual AI agent detection method as described above.

[0019] The visual AI agent detection method, apparatus, and electronic device provided by this invention acquire time and / or coordinate information of interaction events on a user's interactive terminal within a preset time period, and predict whether the interaction events are executed by a visual AI agent based on this information. This invention can accurately and efficiently identify automated access behavior of visual AI agents from the temporal and spatial dimensions of the interaction process, without relying on or with minimal reliance on traditional script automation traces and network fingerprints. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the visual AI agent detection method provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the visual AI agent verification method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of the visual AI agent detection device provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The following is combined Figures 1-2 The present invention describes a visual AI agent detection method.

[0027] Figure 1 This is a flowchart illustrating the visual AI agent detection method provided by the present invention.

[0028] This method can be executed by visual AI agent detection software, which can be deployed in the same operating environment as the website server accessed by the user's interactive terminal. The visual AI agent detection software is deployed separately as a software package and performs the following operations, including initialization log auditing and evidence chain storage.

[0029] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain the time information and / or coordinate information of the interactive events of the user's interactive terminal within a preset time period.

[0030] User interaction terminals are devices used by users to interact with applications, such as computers and mobile phones.

[0031] Interaction events are various actions that occur when a user interacts with an application, such as mouse movement, clicking, scrolling, or keyboard input and focus switching.

[0032] In the specific implementation process, data collection scripts or plugins can be deployed on the user interaction terminal; all interactive events performed by the user interaction terminal within a preset time period can be captured and recorded in real time; the timestamp of each interactive event can be extracted as time information; for interactive events involving coordinates (such as clicks, drags, etc.), their coordinate values ​​can be extracted as coordinate information.

[0033] Step 102: Based on the time information and / or coordinate information of the interaction event, predict whether the interaction event will be executed by the visual AI agent.

[0034] Visual AI agents are software agents that can simulate human visual interaction and complete specific tasks in an automated manner.

[0035] In practice, various methods can be used to predict whether an interactive event will be executed by a visual AI agent based on the time and / or coordinate information of the interactive event, without being limited by the description in this manual.

[0036] In the "screenshot-analysis-decision-execution" cycle, visual AI agents, due to reasoning wait, will form a characteristic of highly consistent interaction intervals between repetitive static windows and interactive events.

[0037] In response to the above characteristics, in some embodiments, it can be determined whether the interaction event has the inference waiting feature of the visual AI agent based on the time information of the interaction event within a preset time period; if the interaction event has the inference waiting feature of the visual AI agent, it is predicted that the interaction event will be executed by the visual AI agent.

[0038] The inference waiting characteristics of visual AI agents include at least one of the following: There are multiple repetitive static windows within a preset time period; the density of interactive events occurring within the repetitive static windows is lower than a preset density threshold, and the duration difference between the repetitive static windows is less than a preset difference threshold.

[0039] In the specific implementation process, by traversing all interactive events within a preset time period, the time period in which the density of interactive events occurs is lower than a preset density threshold can be identified as a static window; the duration difference between each static window can be calculated; if there are multiple static windows and the duration difference between them is less than a preset difference threshold, it is determined that there are duplicate static windows.

[0040] The intervals between interactive events are all within the preset duration range.

[0041] In the specific implementation process, the time interval between adjacent interactive events can be calculated to determine whether these time intervals are all within the preset duration range; if they are all within the preset range, it is determined that the interactive events have a consistent interval time.

[0042] In the specific implementation process, an interval histogram can be constructed based on the time interval between each adjacent interactive event; the coefficient of variation can be calculated based on the interval histogram. If the coefficient of variation is less than a preset threshold, it can be determined that there is a consistent interval time between interactive events; if the interval distribution of the interval histogram shows multiple significant peaks falling within a preset time range, it can be determined that there are multiple repeating static windows within the preset time period.

[0043] Visual AI agents are typically designed to automatically perform precise tasks, such as identifying target objects and calculating click locations. These tasks require visual AI agents to possess a high degree of automation and accuracy to simulate or surpass the behavior of human users.

[0044] Therefore, in some embodiments, the interaction event includes multiple click operations performed by the mouse, and the coordinate information includes the mouse click position coordinates. In this embodiment, it can be determined whether the click operation has mathematically precise click characteristics and / or grid-aligned click characteristics based on the offset distribution of the click position coordinates of each click operation from the geometric center of the clicked target object; if the click operation has mathematically precise click characteristics and / or grid-aligned click characteristics, the interaction event is predicted to be executed by a visual AI agent.

[0045] Mathematical precision click features include: the offset between the click location coordinates and the preset location coordinates of the clicked target object is less than a preset distance threshold.

[0046] In the specific implementation process, the following can be done: obtain the click position coordinates and the geometric center coordinates of the clicked target object for each click operation; calculate the offset distribution between each click position coordinate and the geometric center coordinates; based on the offset distribution, determine whether the offset between each click position coordinate and the preset position coordinates of the clicked target object is less than a preset distance threshold; if it is less, it is determined to have mathematically precise click characteristics.

[0047] Mathematically precise clicks typically mean that there is a very small, or even no, offset between the click location and the geometric center or a specific mathematical point (such as a vertex or midpoint) of the target object. This precision exceeds the range of natural hand tremors in humans, thus indicating that the click operation may be precisely controlled by a program or algorithm. If multiple click operations exhibit the same mathematical precision—that is, each click accurately lands on the same mathematical point of the target object—this regularity further enhances the possibility that the click operation is executed by a visual AI agent.

[0048] The preset position coordinates may include, but are not limited to: the geometric center coordinates of the target object, the top left corner coordinates of the target object, and the bottom left corner coordinates.

[0049] The grid-aligned click feature includes the following: the click position coordinates of multiple click operations are distributed in the horizontal and vertical directions according to a fixed spacing pattern.

[0050] In the specific implementation process, the click position coordinates of multiple click operations can be collected; the distribution of each click position coordinate in the horizontal and vertical directions can be analyzed; it can be determined whether there is a feature of distribution according to a fixed spacing pattern; if so, it can be determined that it has grid-aligned click characteristics.

[0051] Grid-aligned clicks refer to clicks that align with virtual or physical grid lines. In graphical user interface (GUI) design, grids are commonly used for layout and element alignment to ensure a clean and consistent interface. If clicks frequently align with these grid lines, it may indicate that the clicks are performed by a visual AI agent that follows grid layout rules. Grid-aligned clicks not only require the click location to align with grid lines but may also involve systematic patterns such as click order and intervals. These patterns contrast sharply with the random or intuitive clicking behavior of human users, further supporting the hypothesis that visual AI agents perform the clicks.

[0052] In some embodiments, the time and / or coordinate information of the interaction event includes the time and coordinate information of each pixel on the cursor path formed by the mouse movement during the click operation. In this embodiment, it can be determined whether the click operation has the trajectory characteristics of a visual AI agent based on the time and coordinate information of each pixel on the cursor path; if it is determined that the click operation has the trajectory characteristics of a visual AI agent, it is predicted that the interaction event will be executed by the visual AI agent.

[0053] The trajectory features of the visual AI agent include at least one of the following features: The coordinates of each pixel along the cursor path form a smooth straight line or curve.

[0054] In the specific implementation process, the coordinates of each pixel on the cursor path are collected, and curve fitting or straight line detection algorithms are used to analyze whether these coordinate points form a smooth straight line or curve; if so, the click operation is determined to have the trajectory characteristics of a visual AI agent.

[0055] The cursor path does not contain a curved path formed by mouse overshoot and callback.

[0056] When humans use input devices such as mice to control cursor movement, due to the inherent delay and inertia in hand muscle control, it is difficult to precisely stop the cursor at the target position in one go. Often, the cursor will move beyond the target position, a phenomenon known as overshoot. For example, when quickly moving the mouse to click a button, the cursor might overshoot the button and reach a position to the right of it.

[0057] After an overshoot occurs, humans use visual feedback and fine motor control to correct the cursor, returning it to its target position. This correction process is often not instantaneous but requires time and multiple minor adjustments. During this process, the cursor's trajectory will exhibit noticeable curvature and fluctuations, causing the pixel coordinates along the cursor path to deviate from the ideal straight-line interval defined by the start and end points.

[0058] Visual AI agents control cursor movement based on pre-defined algorithms and programs. They can accurately calculate the position and movement path from the starting point to the destination, and through precise control commands, make the cursor move along a predetermined trajectory, avoiding the overshoot phenomenon caused by hand inertia in human operation. For example, a visual AI agent can accurately calculate the required distance and direction of movement based on the target position and the current cursor position, and send corresponding control signals to make the cursor directly and accurately reach the target position.

[0059] Since the visual AI agent can precisely control cursor movement and there is no overshoot problem, there is no need to perform callback operations.

[0060] The cursor path velocity is constant, or the cursor path acceleration is constant.

[0061] Human hand muscle control is imprecise. When moving the cursor with a mouse, the force output of the hand fluctuates due to the contraction and relaxation of muscles. Therefore, the speed and acceleration of the cursor path formed by human operation are usually not constant. In contrast, visual AI agents control cursor movement based on preset algorithms. They can accurately calculate the distance and time the cursor needs to move and move the cursor according to a fixed speed command.

[0062] In some embodiments, it can be determined that the proportion of the first interactive event in the interactive events is less than a first preset proportion threshold, and / or the proportion of the second interactive event is greater than a second preset proportion threshold, and the interactive events are predicted to be executed by a visual AI agent.

[0063] The first interactive event includes scrolling, dragging, or infinite scrolling operations on the display interface of the user interactive terminal; the second interactive event includes button stepping and page-turning positioning operations on the display interface of the user interactive terminal; the second preset ratio threshold is greater than the first preset ratio threshold.

[0064] In the specific implementation process, the proportion of each of the first and second interactive events in the total interactive events can be calculated by counting the number of the first interactive events and the second interactive events in the total interactive events; it can be determined whether the proportion of the first interactive event is less than the first preset proportion threshold, and / or whether the proportion of the second interactive event is greater than the second preset proportion threshold; if the conditions are met, the interactive event is predicted to be executed by the visual AI agent.

[0065] When browsing a display interface, human users naturally and frequently use scrolling and dragging to view all the content. For example, when browsing web pages or documents, they use the mouse wheel or drag the scroll bar to view different parts of the content; for interfaces that support infinite scrolling, such as social media feeds, users will also continuously scroll down to load more content. This operation method conforms to the intuitive visual browsing habits of humans, allowing for a smooth acquisition of information.

[0066] Due to the issue of discontinuous screenshots, visual AI agents tend to avoid scrolling, dragging, and infinite scrolling of the interface during operation. These operations require continuous screenshots to accurately identify the position and changes of interface elements, and discontinuous screenshots make it difficult for the agent to accurately judge the interface state after scrolling or dragging, easily leading to operational errors. Therefore, the proportion of such first interaction events performed by the visual AI agent is relatively low. When statistics show that the proportion of first interaction events is less than a first preset threshold, it suggests that the interaction event may not have been performed by a human user, but rather by the visual AI agent.

[0067] When filling out forms, human users often have a high degree of randomness in the order in which they type. Users may choose the order based on their habits, what they are thinking about, or their familiarity with certain fields.

[0068] Visual AI agents perform form editing operations based on preset algorithms and programs. To complete form filling tasks efficiently and accurately, it operates according to the preset order of form fields, typically filling them in a top-down, systematic order. This order conforms to the logic of the form design and facilitates the agent processing data according to a fixed process.

[0069] Therefore, in some embodiments, the interaction event is an editing operation on a form object, and the time and / or coordinate information of the interaction event includes the coordinate information of each field of the form object and the editing time information. In this embodiment, based on the coordinate information and editing time information of each field of the form object, it can be determined that the editing operation is performed in a preset order, and / or that there is no backtracking modification operation in the editing operation, and the interaction event is predicted to be executed by a visual AI agent.

[0070] In the specific implementation process, the coordinate information and editing time information of each field of the form object can be collected; based on the coordinate information and editing time information of each field of the form object, the editing operation is determined to be performed in a preset order, such as editing from top to bottom; at the same time, it is checked whether there is a backtracking modification operation during the editing process (i.e., editing the later fields first and then editing the earlier fields); if the editing operation is performed in a preset order and there is no backtracking modification operation, the interactive event is predicted to be triggered by the AI ​​agent.

[0071] In practical implementation, one or more combinations of the above embodiments can be executed to predict whether the interaction event is executed by a visual AI agent. For each embodiment, a risk score is performed based on the prediction result, and the final risk score is obtained by weighted summation of the risk scores of all embodiments. Subsequently, corresponding operations are performed based on the final risk score, including: allowing access, secondary verification, rate limiting, blocking, or honeypot redirection, etc.

[0072] If, based on the time and / or coordinate information of the interaction event, the probability that the interaction event is executed by a visual AI agent is greater than a preset probability threshold, then execution can proceed. Figure 2 The steps shown are executed by the visual AI agent to verify the interaction events.

[0073] Figure 2 This is a flowchart illustrating the visual AI agent verification method provided by the present invention.

[0074] like Figure 2 As shown, the method includes the following steps: Step 201: Verify whether the interactive event is executed by the visual AI agent through the inducement command.

[0075] Inducement commands are special commands designed to detect visual AI agents. They are invisible to human users or do not trigger normal interaction paths, but can be read or executed by visual AI agents.

[0076] In some embodiments, an inductive carrier that is not visible to the human interaction subject can be embedded in the display interface of the user interaction terminal; if the inductive carrier is accessed, it is determined that the interaction event is executed by a visual AI agent.

[0077] In the specific implementation process, techniques such as CSS hiding, zero opacity, or off-screen positioning are used to embed invisible inducement carriers, such as inducement text or links, into the display interface of the user interaction terminal; whether these inducement carriers are accessed is monitored; if they are accessed, it is determined that the interaction event is executed by the visual AI agent.

[0078] In some embodiments, a prompting instruction is embedded in the accessibility attribute of a form control on the display interface to prompt the user to enter preset text content in the form control; a verification path that is not executable by the human interaction subject corresponding to the prompting instruction is set; if it is determined that the verification path is executed and the content submitted by the form control is preset text, it is determined that the interaction event is executed by a visual AI agent.

[0079] In practice, you can embed prompting instructions to guide the visual AI agent to enter preset text content in the form control by modifying the accessibility properties of the form control (e.g., ARIA label); set a verification path that cannot be triggered by human users (e.g., a hidden button or link); monitor whether this verification path is triggered and check whether the content submitted by the form control is the preset text; if the verification path is triggered and the submitted content is the preset text, then determine that the interaction event is executed by the visual AI agent.

[0080] In some embodiments, form fields that are invisible or inaccessible to the human interaction subject are set, and placeholders are set in the associated positions of the form fields. The placeholders are used to prompt the execution of the access operation for the form fields. If it is determined that the form field is accessed, the interaction event is determined to be executed by the visual AI agent.

[0081] In practice, a form field that is invisible or inaccessible to human users can be created; a placeholder with a prompt message can be set in the associated position of the field to prompt the visual AI agent to perform an access operation on the form field; the access of the form field can be monitored; if it is accessed, it can be determined that the interaction event was performed by the visual AI agent.

[0082] Step 202: Verify whether the interaction event is executed by the visual AI agent through environmental fingerprint inconsistency detection.

[0083] Environmental fingerprints refer to characteristic information that can identify the environment in which a user's interactive terminal is located, such as network address and browser version.

[0084] Environmental fingerprint inconsistency detection verifies whether an interactive event is executed by a visual AI agent by comparing the actual detected environmental fingerprint with the expected environmental fingerprint.

[0085] In the specific implementation process, one or more of the following detection methods can be used to verify whether the interactive event is executed by a visual AI agent.

[0086] Determine whether the source network corresponding to the interaction event belongs to a known cloud data center autonomous system or an abnormal proxy network; if the source network belongs to a known cloud data center autonomous system or an abnormal proxy network, further determine that the interaction event is executed by a visual AI agent.

[0087] In the specific implementation process, the source network address of the interaction event can be extracted; it can be queried whether this address belongs to a known cloud data center autonomous system or abnormal proxy network; if it does, it can be further indicated that the interaction event is executed by a visual AI agent.

[0088] A cloud data center autonomous system (AS) is typically an area where a large number of servers and network devices are centrally managed, and visual AI agents can run on cloud servers. If the source network of an interaction event belongs to a known cloud data center AS, the likelihood that the interaction event was triggered by a visual AI agent increases significantly. For example, some cloud service providers' data centers have specific AS numbers, and when an interaction event is detected to originate from one of these ASs, there is reason to suspect that it was initiated by a visual AI agent deployed in the cloud.

[0089] Anomaly proxy networks can be used to conceal the true source of an action. Visual AI agents can leverage proxy networks to bypass security checks or perform large-scale automated operations. If the source network of an interaction event is an anomaly proxy network, it suggests that the interaction event was not initiated directly by a normal human user through the user's terminal, but rather carried out by a visual AI agent using the proxy network.

[0090] Check whether the browser's feature set and version used by the user interaction terminal are consistent.

[0091] The browsers used by real users are usually mainstream and widely used versions, which have relatively stable and standardized sets of features. For example, common browsers such as Chrome, Firefox, and Safari follow a certain pattern of feature evolution during normal updates, with new versions gradually adding new features while maintaining compatibility with older features.

[0092] To achieve specific automated tasks or simulate human interaction, visual AI agents may use non-mainstream or specially modified browser environments. These environments may lack a complete set of features or have version information that differs from commonly used browsers. For example, some visual AI agents may implement only the basic rendering functions of a browser to save resources or improve efficiency, while ignoring advanced features that are irrelevant to the automated task, such as specific Web API support or complex graphics processing capabilities.

[0093] Therefore, by collecting the function set and version information of the browser used by the user's interactive terminal, and checking whether this information is consistent or as expected, if it is inconsistent or as expected, it can be further determined that the interaction event was executed by a visual AI agent.

[0094] Real users typically use the latest stable versions of mainstream browsers (such as Chrome, Firefox, Safari, etc.), which have built-in hardware-accelerated rasterizers that fully utilize the GPU for graphics rendering, improving performance and efficiency. Therefore, the browser renderer information of real users usually does not reflect the characteristics of a software rasterizer.

[0095] To simulate human interaction or perform specific tasks, visual AI agents may use unconventional or specially modified browser environments. These environments may lack full hardware acceleration capabilities, or, for the sake of realism, may intentionally use software rasterizers (such as Microsoft Basic Render Driver, Mesa / llvmpipe, etc.) for graphics rendering. Therefore, the browser renderer information of a visual AI agent may exhibit characteristics of a software rasterizer.

[0096] Therefore, it is possible to detect whether the renderer information of the browser used by the user's interactive terminal exhibits software rasterizer characteristics (such as a specific renderer name or version number); if the renderer information exhibits software rasterizer characteristics, it can be further determined that the interactive event is executed by a visual AI agent.

[0097] The following describes specific implementation methods. Figure 1 and Figure 2 The execution process of each step: In the specific implementation process, the sequence of interactive events and page / element state information can be collected from the target page or client of the user interaction terminal. The interactive events include at least mousemove, click, wheel / scroll, keydown / keyup, input, focus / blur, and drag, and the corresponding timestamps, coordinates, scroll amount, and focus state are recorded. At the same time, the state information such as the bounding box of the interactive element and its geometric center coordinates, window size and scaling ratio are obtained. According to the preset time window and the main identifier, these interactive event information are aggregated to form an interactive session, and noise reduction, time alignment and normalization processing are performed.

[0098] Calculate the interval between adjacent events based on interactive sessions. Construct an interaction interval histogram and calculate the coefficient of variation. .

[0099] When the interaction intervals show high consistency (e.g., CV < 0.2) or the interval distribution shows repeated static peaks consistent with visual reasoning wait (e.g., multiple static peaks in the range of 1-5 seconds), the session is marked as temporally suspicious; at the same time, the frequency of mouse movement events is monitored in real time, and patterns in which the frequency of mouse movement in the static window is close to zero and there is a significant break with the active phase are identified, and temporally suspicious features are output.

[0100] Calculate the interval between adjacent interaction events Construct interval histograms and calculate the coefficient of variation: in, Time interval Standard deviation; Time interval The average value.

[0101] A time series is marked as "suspected" if any of the following conditions are met: ;or The interval distribution shows a significant peak falling on And it appears repeatedly within the session (which matches the inference / API wait characteristics).

[0102] Obtain the coordinates of the click event and the geometric center coordinates of the target element, calculate the click offset distance distribution, and identify a large number of clicks falling on the center of the element; reconstruct the mouse movement trajectory and extract curvature, velocity / acceleration profiles and smoothness features to identify straight lines or overly smooth synthetic trajectories with approximately constant velocity; detect overshoot and fine-tuning behavior when the mouse approaches the clicked target object. If there is a lack of natural correction actions such as retraction, fine-tuning, and overshoot re-correction, and the proportion of direct arrival at the calculated coordinate point is abnormal, it is marked as a trajectory and accuracy questionable.

[0103] Center click offset detection: For each click, obtain the geometric center of the target element. With click points Calculate the offset distance : ; in, These are the x-axis and y-axis coordinates of the geometric center of the target element, respectively. These are the x-axis and y-axis coordinates of the clicked point, respectively.

[0104] Statistical offset distance Distribution characteristics (including center hit rate, variance, kurtosis, grid alignment rate, etc.). Patterns such as "abnormally concentrated offsets" or "a large number of perfect center / grid point hits" are marked as suspicious.

[0105] This study statistically analyzes the frequency, duration, displacement / amplitude, and proportion of page scrolling and dragging operations in a session. It also analyzes the proportion of Tab navigation and PageUp / PageDown discrete jumps to identify avoidance of scrolling / dragging and over-reliance on discrete navigation. Furthermore, it models the form input sequence according to the position of each field on the page, calculates top-down sequential consistency and backtracking modification probability, identifies systematic sequential filling patterns lacking backtracking modification, and outputs interaction preferences and suspicious features of form patterns.

[0106] By integrating the multidimensional features output from the above embodiments, a risk score is generated: Suspicious features in time series, suspicious features in trajectory and accuracy, and suspicious features in interaction preferences and form patterns are weighted and integrated according to preset fusion rules to obtain the risk score.

[0107] If the risk score is low, the release operation can be performed; if the risk score is high, the following methods can be used to set up induced instruction traps to further verify whether the interactive events are executed by visual AI agents.

[0108] (1) Visually invisible inducement carriers: Inducement text or links that are invisible to humans are embedded through CSS hiding, zero opacity or off-screen positioning.

[0109] (2) Semantic trap carrier: embed inducement instructions in ARIA tags or accessibility attributes, and set verification paths that should not be triggered by normal users.

[0110] (3) Field decoys: Set invisible or unreachable form fields and placeholders.

[0111] Set up monitoring and verification points. Once an invisible field is focused / entered / submitted, a hidden link is accessed, or a specific action is caused by an instruction whose semantic attributes are visible only, strong evidence will be output and the session will be marked as high risk.

[0112] In addition, network fingerprints can be used to further verify high-risk interaction events.

[0113] Detect whether the source network belongs to a cloud data center autonomous system or an abnormal proxy network; detect whether the browser's function set and version are consistent (e.g., missing required APIs); detect whether the rendering features exhibit software rasterizer characteristics (such as Microsoft Basic Render Driver, Mesa / llvmpipe, etc.), and whether the output environment consistency is suspicious.

[0114] If the interaction event is verified to be executed by a visual AI agent through the above methods, handling strategies such as rate limiting, blocking, or honeypot diversion can be implemented.

[0115] The visual AI agent detection device provided by the present invention is described below. The visual AI agent detection device described below can be referred to in correspondence with the visual AI agent detection method described above.

[0116] Figure 3 This is a schematic diagram of the structure of the visual AI agent detection device provided by the present invention. Figure 3 As shown, the visual AI agent detection device 300 includes the following modules.

[0117] The acquisition module 310 is used to acquire the time information and / or coordinate information of interactive events of the user interaction terminal within a preset time period.

[0118] The prediction module 320 is used to predict whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event.

[0119] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a visual AI agent detection method, which includes: acquiring time information and / or coordinate information of interaction events of a user interaction terminal within a preset time period; and predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event.

[0120] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual AI agent detection method provided by the above methods. The method includes: acquiring time information and / or coordinate information of an interaction event of a user interaction terminal within a preset time period; and predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the visual AI agent detection method provided by the above methods. The method includes: acquiring time information and / or coordinate information of an interaction event of a user interaction terminal within a preset time period; and predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual AI agent detection method, characterized in that, include: Obtain time information and / or coordinate information of interactive events on the user's interactive terminal within a preset time period; Based on the time and / or coordinate information of the interaction event, predict whether the interaction event is executed by a visual AI agent.

2. The visual AI agent detection method according to claim 1, characterized in that, The step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: Based on the time information of the interactive events within the preset time period, determine whether the interactive events have inference waiting characteristics of visual AI agents; If the interaction event exhibits inference waiting characteristics of a visual AI agent, it is predicted that the interaction event will be executed by the visual AI agent. The reasoning waiting characteristics of the visual AI agent include at least one of the following: Within the preset time period, there are multiple repetitive static windows; wherein, the density of interactive events occurring within the repetitive static windows is lower than a preset density threshold, and the duration difference between the repetitive static windows is less than a preset difference threshold. The intervals between the interactive events are all within a preset time range.

3. The visual AI agent detection method according to claim 1, characterized in that, The interactive events include multiple click operations performed by the mouse, and the coordinate information includes the coordinates of the mouse click position; The step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: Based on the offset distribution between the click position coordinates of each click operation and the geometric center of the clicked target object, determine whether the click operation has mathematically precise click characteristics and / or grid-aligned click characteristics; If the click operation has mathematically precise click characteristics and / or grid-aligned click characteristics, the interaction event is predicted to be executed by a visual AI agent; The mathematically precise click feature includes: the offset between the click position coordinates and the preset position coordinates of the clicked target object is less than a preset distance threshold; The grid-aligned click feature includes the following: the click position coordinates of the multiple click operations are distributed in the horizontal and vertical directions according to a fixed spacing pattern.

4. The visual AI agent detection method according to claim 3, characterized in that, The time information and / or coordinate information of the interactive event includes the time information and coordinate information of each pixel on the cursor path formed by the mouse movement during the click operation. The step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: Based on the time and coordinate information of each pixel on the cursor path, determine whether the click operation has trajectory features of a visual AI agent; If it is determined that the click operation has trajectory characteristics of a visual AI agent, it is predicted that the interaction event will be executed by the visual AI agent; The trajectory features of the visual AI agent include at least one of the following features: The coordinates of each pixel on the cursor path form a smooth straight line or curve; The cursor path does not contain a curved path formed by mouse operation overshoot and callback; The cursor path velocity is constant, or the cursor path acceleration is constant.

5. The visual AI agent detection method according to claim 1, characterized in that, The step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: If the proportion of the first interactive event in the interactive events is less than a first preset proportion threshold, and / or the proportion of the second interactive event is greater than a second preset proportion threshold, it is predicted that the interactive events will be executed by a visual AI agent. The first interactive event includes scrolling, dragging, or infinite scrolling operations on the display interface of the user interaction terminal; the second interactive event includes button stepping and page-turning positioning operations on the display interface of the user interaction terminal; and the second preset ratio threshold is greater than the first preset ratio threshold.

6. The visual AI agent detection method according to claim 1, characterized in that, The interactive event is an editing operation on a form object, and the time information and / or coordinate information of the interactive event includes the coordinate information and editing time information of each field of the form object; The step of predicting whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event includes: Based on the coordinate information and editing time information of each field of the form object, it is determined that the editing operation is performed in a preset order, and / or there is no backtracking modification operation in the editing operation, and it is predicted that the interaction event will be executed by a visual AI agent.

7. The visual AI agent detection method according to claim 1, characterized in that, The method further includes: Based on the time information and / or coordinate information of the interaction event, if the probability that the interaction event is executed by a visual AI agent is greater than a preset probability threshold, at least one of the following operations is performed: a: Embed a guiding carrier that is invisible to the human interaction subject in the display interface of the user interaction terminal; If the inducing carrier is accessed, it is determined that the interaction event was executed by a visual AI agent; b: Embed prompting instructions in the accessibility properties of the form controls on the display interface, so as to prompt the user to enter preset text content in the form controls using the prompting instructions; Set a verification path that the human interaction entity corresponding to the inducement instruction cannot execute; If it is determined that the verification path is executed and the content submitted by the form control is the preset text, it is determined that the interaction event is executed by the visual AI agent. c: Set form fields that are invisible or inaccessible to the human interaction subject, and set placeholders in the associated positions of the form fields, and use the placeholders to prompt the execution of access operations for the form fields; If it is determined that the form field is accessed, it is determined that the interaction event is executed by the visual AI agent.

8. The visual AI agent detection method according to claim 1, characterized in that, The method further includes: The interaction event is further determined to be executed by a visual AI agent by at least one of the following methods: Determine whether the source network corresponding to the interaction event belongs to a known cloud data center autonomous system or an abnormal proxy network; If the source network belongs to a known cloud data center autonomous system or an abnormal proxy network, the interaction event is determined to be executed by the visual AI agent. Detect whether the browser function set and version used by the user interaction terminal are consistent; If the browser function set and version used by the user interaction terminal are inconsistent, it is determined that the interaction event is executed by the visual AI agent; Detect whether the renderer information of the browser used by the user interaction terminal exhibits software rasterizer characteristics; If the renderer information exhibits software rasterizer characteristics, it is further determined that the interactive event is executed by a visual AI agent.

9. A visual AI agent detection device, characterized in that, include: The acquisition module is used to acquire time information and / or coordinate information of interactive events on the user's interactive terminal within a preset time period; The prediction module is used to predict whether the interaction event is executed by a visual AI agent based on the time information and / or coordinate information of the interaction event.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the visual AI agent detection method as described in any one of claims 1 to 8.

Images