Crawler behavior detection method and device based on multiple time dimensions and electronic equipment

Abstract

The embodiment of the invention provides a crawler behavior detection method and device based on multiple time dimensions and electronic equipment. The method comprises the steps of obtaining access information of a current access behavior in response to the current access behavior of a user; in a first time window where the time of the current access behavior is located, determining a first access frequency of accessing the interface address by the user; if it is determined that the first access frequency is smaller than a first threshold value, whether the current access behavior is a crawler behavior or not is determined in at least one second time window where the time of the current access behavior is located; wherein the first threshold value is used for indicating the maximum access frequency of allowing the user to access the interface address in the first time window; the time dimension of the second time window is greater than that of the first time window. The method is used for achieving the effect of improving the detection accuracy of crawler behaviors.

Description

Technical Field

[0001] This application relates to the field of information security technology, and in particular, to a method, device, and electronic device for detecting crawler behavior based on multiple time dimensions. Background Art

[0002] With the continuous development of Internet technology, the data on websites and applications is becoming increasingly rich. Some network data is threatened by crawler behavior. Crawler behavior will obtain a large amount of network data on the website or attack the website through crawler behavior. Therefore, the detection of crawler behavior is very important.

[0003] In some technologies, by detecting the user's access behavior, if it is detected that the user's access frequency exceeds a preset threshold, it is determined that the user's access behavior is abnormal crawler behavior. In the above technologies, the accuracy of monitoring and identifying crawler behavior is low.

[0004] Therefore, there is an urgent need for a solution that can more accurately determine crawler behavior. Summary of the Invention

[0005] The detection method, device, and electronic device for crawler behavior based on multiple time dimensions provided by the embodiments of this application are used to achieve the effect of improving the detection accuracy for crawler behavior.

[0006] In a first aspect, an embodiment of this application provides a method for detecting crawler behavior based on multiple time dimensions, including:

[0007] In response to the user's current access behavior, obtain the access information of the current access behavior; where the access information includes the interface address of the current access behavior and the time of the current access behavior;

[0008] Determine the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located;

[0009] If it is determined that the first access count is less than the first threshold, then within at least one second time window where the time of the current access behavior is located, determine whether the current access behavior is crawler behavior; where the first threshold is used to indicate the maximum access count allowed for the user to access the interface address within the first time window; the time dimension of the second time window is greater than the time dimension of the first time window.

[0010] In a possible implementation, the second time window includes: a minute - dimension time window and a month - dimension time window;

[0011] Within at least one second time window where the time of the current access behavior is located, determine whether the current access behavior is crawler behavior, including:

[0012] Within the minute - dimension time window where the time of the current access behavior is located, determine the second access count of the user accessing the interface address;

[0013] If it is determined that the second access count is greater than or equal to the second threshold, determine that the current access behavior is a crawler behavior; where the second threshold is used to indicate the maximum number of accesses allowed for the user to access the interface address within the minute - dimension time window;

[0014] If it is determined that the second access count is less than the second threshold, within the month - dimension time window where the time of the current access behavior is located, determine whether the current access behavior is a crawler behavior.

[0015] In a possible implementation manner, within the month - dimension time window where the time of the current access behavior is located, determining whether the current access behavior is a crawler behavior includes:

[0016] Within the month - dimension time window where the time of the current access behavior is located, determine the third access count of the user accessing the interface address;

[0017] If it is determined that the third access count is greater than or equal to the third threshold, determine that the current access behavior is a crawler behavior; where the third threshold is used to indicate the maximum number of accesses allowed for the user to access the interface address within the month - dimension time window;

[0018] If it is determined that the third access count is less than the third threshold, determine that the current access behavior is not a crawler behavior.

[0019] In a possible implementation manner, before obtaining the access information of the current access behavior in response to the user's current access behavior, the method further includes:

[0020] If it is determined that the detection flag for crawler behavior is enabled, perform the step of obtaining the access information of the current access behavior in response to the user's current access behavior.

[0021] In a possible implementation manner, after obtaining the access information of the current access behavior, the method further includes:

[0022] If it is determined that the interface address of the current access behavior belongs to the preset anti - crawler interface address list, perform the step of determining the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located.

[0023] In a possible implementation manner, the method further includes:

[0024] If it is determined that the first access count is greater than or equal to the first threshold, determine that the current access behavior is a crawler behavior.

[0025] In a possible implementation manner, the method further includes:

[0026] If it is determined that the current access behavior is a crawler behavior, intercept the current access behavior and record the access information of the current access behavior in the exception log.

[0027] In a possible implementation, after intercepting the current access behavior, the method further includes:

[0028] Intercept all subsequent accesses of the user within a preset time period after the current access behavior; wherein, the interface address of the subsequent access is the same as the interface address of the current access behavior.

[0029] In a second aspect, an embodiment of the present application provides a detection device for crawler behavior based on multiple time dimensions, including:

[0030] An acquisition module, configured to acquire access information of the current access behavior in response to the user's current access behavior; wherein, the access information includes the interface address of the current access behavior and the time of the current access behavior;

[0031] A processing module, configured to determine a first access count of the user accessing the interface address within a first time window where the time of the current access behavior is located;

[0032] The processing module is further configured to, if it is determined that the first access count is less than a first threshold, determine whether the current access behavior is a crawler behavior within at least one second time window where the time of the current access behavior is located; wherein, the first threshold is used to indicate the maximum access count allowed for the user to access the interface address within the first time window; the time dimension of the second time window is greater than the time dimension of the first time window.

[0033] In a possible implementation, the second time window includes: a minute - dimension time window and a month - dimension time window;

[0034] Within at least one second time window where the time of the current access behavior is located, to determine whether the current access behavior is a crawler behavior, the processing module is used for:

[0035] Determine a second access count of the user accessing the interface address within the minute - dimension time window where the time of the current access behavior is located;

[0036] If it is determined that the second access count is greater than or equal to a second threshold, determine that the current access behavior is a crawler behavior; wherein, the second threshold is used to indicate the maximum access count allowed for the user to access the interface address within the minute - dimension time window;

[0037] If it is determined that the second access count is less than the second threshold, determine whether the current access behavior is a crawler behavior within the month - dimension time window where the time of the current access behavior is located.

[0038] In a possible implementation, within the monthly time window where the time of the current access behavior is located, determine whether the current access behavior is a crawler behavior. The processing module is used for:

[0039] Within the monthly time window where the time of the current access behavior is located, determine the third access count of the user accessing the interface address;

[0040] If it is determined that the third access count is greater than or equal to the third threshold, determine that the current access behavior is a crawler behavior; where the third threshold is used to indicate the maximum access count that allows the user to access the interface address within the monthly time window;

[0041] If it is determined that the third access count is less than the third threshold, determine that the current access behavior is not a crawler behavior.

[0042] In a possible implementation, before obtaining the access information of the current access behavior in response to the user's current access behavior, the obtaining module is further used for:

[0043] If it is determined that the detection flag for crawler behavior is enabled, perform the step of obtaining the access information of the current access behavior in response to the user's current access behavior.

[0044] In a possible implementation, after obtaining the access information of the current access behavior, the processing module is further used for:

[0045] If it is determined that the interface address of the current access behavior belongs to the preset anti-crawler interface address list, perform the step of determining the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located.

[0046] In a possible implementation, the processing module is further used for:

[0047] If it is determined that the first access count is greater than or equal to the first threshold, determine that the current access behavior is a crawler behavior.

[0048] In a possible implementation, the processing module is further used for:

[0049] If it is determined that the current access behavior is a crawler behavior, intercept the current access behavior and record the access information of the current access behavior in the exception log.

[0050] In a possible implementation, after intercepting the current access behavior, the processing module is further used for:

[0051] Intercept all subsequent accesses of the user within the preset time period after the current access behavior; where the interface address of the subsequent access is the same as the interface address of the current access behavior.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementation manners of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementation manners of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the above first aspect and / or various possible implementation manners of the first aspect.

[0057] The detection method, device and electronic device for crawler behavior based on multiple time dimensions provided by the embodiments of the present application obtain the time and interface address of the current access behavior in response to the user's current access behavior; determine whether it is a crawler behavior within the first time window where the time of the current access behavior is located. If it is determined that it is not a crawler behavior within the first time window, then further determine whether the current access behavior is a crawler behavior in a time window with a larger time dimension. Detecting crawler behavior for the user's access behavior based on time windows with multiple different sizes can accurately detect crawler behaviors with low access frequencies, thereby improving the overall detection accuracy of crawler behaviors. Description of the Drawings

[0058] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0059] Figure 1 It is a flowchart illustration of the detection method for crawler behavior based on multiple time dimensions provided by the present application Figure 1 ;

[0060] Figure 2 It is a flowchart illustration of the detection method for crawler behavior based on multiple time dimensions provided by the present application Figure 2 ;

[0061] Figure 3 It is a flowchart illustration of the detection method for crawler behavior based on multiple time dimensions provided by the present application Figure 3 ;

[0062] Figure 4Schematic structural diagram of the crawler behavior detection device based on multiple time dimensions provided by this application;

[0063] Figure 5 Schematic structural diagram of the electronic device provided by this application.

[0064] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Specific Embodiments

[0065] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0066] First, the terms involved in this application are explained:

[0067] Web crawler: Also known as a web spider or crawler robot; it refers to a program or script that automatically grabs Internet information according to certain rules. Crawler behavior refers to the behavior of a user accessing a target website and using a web crawler to perform a large number of data grabbing operations on the accessed target website.

[0068] Time dimension: It refers to the dimension of the time window; for example, the time dimension of the time window can include, in ascending order: seconds, minutes, hours, natural days, natural months.

[0069] With the continuous development of Internet technology, the data on websites and applications is becoming increasingly rich. For example, network data includes, but is not limited to: text data, picture data, audio data, video data, and personal information data. Some network data is threatened by crawler behavior. Crawler behavior can obtain a large amount of network data on a website or attack the website through crawler behavior. This will lead to a decline in the security of network data, and the proliferation of crawler behavior may cause the system to crash.

[0070] In particular, on websites related to banking services, business data and user information are stored. These banking service-related data need to ensure their security and privacy.

[0071] Therefore, the detection of crawler behavior is very important.

[0072] In some embodiments, the crawling behavior is intercepted by detecting whether the user's Internet Protocol Address (IP address for short) is in a preset interception list.

[0073] In some embodiments, automated crawling behavior is prevented by using verification codes and human-machine verification.

[0074] In some embodiments, by monitoring the request frequency from the same IP address or the same user, if the request frequency exceeds a preset frequency threshold, it is determined that the access belongs to a crawling behavior and is intercepted.

[0075] In the above embodiments, the accuracy rate of detecting crawling behavior is relatively low. Especially for crawling behavior with a low access frequency, accurate detection and recognition cannot be achieved.

[0076] The detection method of crawling behavior based on multiple time dimensions provided by this application, by responding to the user's current access behavior, obtaining the time and interface address of the current access behavior; judging whether it is a crawling behavior within the first time window where the time of the current access behavior is located. If it is judged not to be a crawling behavior within the first time window, then further judging whether the current access behavior is a crawling behavior in a time window with a larger time dimension, solves the technical problem of low accuracy rate in detecting crawling behavior.

[0077] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0078] Figure 1 Schematic flow of the detection method of crawling behavior based on multiple time dimensions provided by this application Figure 1 , as Figure 1 shown, the method includes:

[0079] Step 101. Respond to the user's current access behavior and obtain the access information of the current access behavior.

[0080] Among them, the access information includes the interface address of the current access behavior and the time of the current access behavior.

[0081] Exemplarily, when a user accesses or logs in to a website, it can be regarded as an access behavior. Respond to the user's current access behavior and obtain the access information of the current access behavior.

[0082] Among them, the access information of the current access behavior may include the interface address of the current access behavior and the time of the current access behavior. The interface address may refer to the Uniform Resource Locator (URL) accessed by the user, and the URL accessed by the user refers to the address of the data resource accessed by the user. The time of the current access behavior refers to the specific time when the user accesses or logs in to the website.

[0083] Step 102. Determine the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located.

[0084] Exemplarily, according to the time of the current access behavior, it can be determined that within the first time window where this time is located, the first access count of the user accessing this interface address. Specifically, the time dimension of the first time window is in seconds. It can be understood that the first access count refers to the number of times the user accesses this interface address within the second when the time of the current access behavior is located.

[0085] For example, the time when user A accesses interface address X1 is 12:24:30 on March 1, 2025; then the first access count represents the number of times user A accesses interface address X1 from 12:24:30 on March 1, 2025 to 12:24:31 on March 1, 2025.

[0086] The current access behavior of the user can be stored in the database according to the data structure of an ordered queue. In the ordered queue, the user name, interface address, and second identifier are used as the key, and the time when the user with this user name accesses this interface address is used as the score.

[0087] It can be understood that in the ordered queue in the database, the access information of each access behavior of different users accessing different interface addresses will be stored. According to the time of the current access behavior, the first time window for this judgment is determined. Discard the records in the ordered queue whose scores do not belong to this first time window. Count the remaining data quantity in the ordered queue at this time, and the first access count can be obtained.

[0088] Step 103. If it is determined that the first access count is less than the first threshold, determine whether the current access behavior is a crawler behavior within at least one second time window where the time of the current access behavior is located.

[0089] Among them, the first threshold is used to indicate the maximum number of times a user is allowed to access the interface address within the first time window; the time dimension of the second time window is greater than the time dimension of the first time window.

[0090] Exemplarily, the first access count is compared with a first threshold, where the first threshold indicates the maximum number of accesses allowed for a user to access an interface address within a first time window. It can be understood that if the first access count is greater than or equal to the first threshold, it indicates that the access count of the user within the first time window reaches the maximum number of accesses; correspondingly, if the first access count is less than the first threshold, it indicates that the access count of the user within the first time window does not reach the maximum number of accesses.

[0091] If it is determined that the first access count is less than the first threshold, then further within at least one second time window where the time of the current access behavior is located, it is determined whether the current access behavior is a crawler behavior. Among them, the time dimension of the second time window is greater than the time dimension of the first time window. For example, if the time dimension of the first time window is in seconds, the time dimension of the second time window can be in minutes, hours, natural days, natural months, etc.

[0092] This is because, according to the foregoing example, the time dimension of the first time window can be in seconds; if the access count of the crawler behavior is not high within the time dimension of seconds but is high within a larger time dimension, it is necessary to further determine the access count of the time window with a larger time dimension, so as to improve the more comprehensive detection of crawler behaviors with low access frequencies.

[0093] The detection method of crawler behavior based on multiple time dimensions provided by the embodiments of the present application, by responding to the current access behavior of a user, obtains the time of the current access behavior and the interface address; determines whether it is a crawler behavior within the first time window where the time of the current access behavior is located. If it is determined that it is not a crawler behavior within the first time window, then further determines whether the current access behavior is a crawler behavior in the time window with a larger time dimension. By respectively determining whether it belongs to a crawler behavior in time windows with different time dimensions, it can accurately detect crawler behaviors with low access frequencies. Thus, the effect of improving the detection accuracy of overall crawler behaviors is achieved.

[0094] Figure 2 This is a flowchart of the detection method of crawler behavior based on multiple time dimensions provided by the present application Figure 2 , this embodiment Figure 1 On the basis of the embodiment, step 103 is described in detail.

[0095] In one example, the second time window includes: a minute-dimension time window and a month-dimension time window.

[0096] Exemplarily, since the time dimension of the second time window should be greater than the time dimension of the first time window. Therefore, when the time dimension of the first time window is in seconds, the time dimension of the second time window should be in minutes, or hours, or natural days, or natural months.

[0097] It can be understood that if the time dimension of the first time window is in minutes, the time dimension of the second time window should be in hours, or natural days, or natural months.

[0098] Furthermore, as Figure 2 shown, the steps of determining the crawler behavior within at least one second time window specifically include:

[0099] Step 201. Determine the second access count of the user accessing the interface address within the minute - dimension time window where the time of the current access behavior is located.

[0100] Exemplarily, according to the time of the current access behavior, it can be determined that within the minute - dimension time window where this time is located, the second access count of the user accessing this interface address. It can be understood that the second access count refers to the number of times the user accesses this interface address within the minute where the time of the current access behavior is located.

[0101] For example, the time when user A accesses interface address X1 is 12:24:30 on March 1, 2025; then the second access count represents the number of times user A accesses interface address X1 within the period from 12:24:00 to 12:25:00 on March 1, 2025.

[0102] The current access behavior of the user can be stored in the database in the data structure of an ordered queue. In the ordered queue, the user name, interface address, and minute identifier (minit) are used as the key, and the time when the user accesses this interface address is used as the score.

[0103] It can be understood that in the ordered queue in the database, the access information of each access behavior of different users accessing different interface addresses will be stored. According to the time of the current access behavior, the minute - dimension time window for this judgment is determined. The records whose scores in the ordered queue represent times that do not belong to this minute - dimension time window are discarded. By counting the remaining data quantity in the ordered queue at this time, the second access count can be obtained.

[0104] Step 202. If it is determined that the second access count is greater than or equal to the second threshold, then determine that the current access behavior is a crawler behavior.

[0105] Among them, the second threshold is used to indicate the maximum number of times a user is allowed to access the interface address within the minute - dimension time window.

[0106] Exemplarily, the second access count is compared with the second threshold. The second threshold indicates the maximum number of accesses allowed for a user to access the interface address within a time window in the minute dimension. It can be understood that if the second access count is greater than or equal to the second threshold, it indicates that the number of accesses by the user within the time window in the minute dimension has reached the maximum number of accesses. Therefore, the current access behavior can be determined as a crawler behavior.

[0107] Step 203. If it is determined that the second access count is less than the second threshold, determine whether the current access behavior is a crawler behavior within the time window in the month dimension where the time of the current access behavior is located.

[0108] Exemplarily, if the second access count is less than the second threshold, it indicates that the number of accesses by the user within the time window in the minute dimension has not reached the maximum number of accesses.

[0109] If it is determined that the second access count is less than the second threshold, then further determine whether the current access behavior is a crawler behavior within the time window in the month dimension where the time of the current access behavior is located.

[0110] It should be noted that the above steps 202 and 203 can be executed simultaneously or sequentially, and the execution order is not limited.

[0111] In the above example, for access behaviors that are not crawler behaviors within the first time window, further detection of crawler behaviors is performed based on a second time window with a larger time dimension. For example, the time dimension of the first time window can be in seconds. Based on the determination that the current access behavior does not belong to a crawler behavior within the first time window in the second dimension, further determination is continued within the time window in the minute dimension. Crawler behaviors with low access frequencies can be detected comprehensively and accurately.

[0112] Specifically, Figure 3 is a schematic flowchart of the detection method for crawler behaviors based on multiple time dimensions provided by this application Figure 3 In one example, as Figure 3 shown, for step 203, the steps of determining crawler behaviors within the time window in the month dimension specifically include:

[0113] Step 301. Determine the third access count of the user accessing the interface address within the time window in the month dimension where the time of the current access behavior is located.

[0114] Exemplarily, according to the time of the current access behavior, the third access count of the user accessing the interface address within the time window in the month dimension where this time is located can be determined. It can be understood that the third access count refers to the number of times the user accesses the interface address within the natural month where the time of the current access behavior is located.

[0115] For example, the time when user A accesses interface address X1 is 12:24:30 on March 1, 2025; then the third access count represents the number of times user A accesses interface address X1 from 00:00:00 on March 1, 2025 to 00:00:00 on March 31, 2025.

[0116] A new data object can be constructed in the database for the user's current access behavior. In this data object, the user name, interface address, and month identifier are used as the key, and the third access count is used as the score. Among them, the month identifier is used to represent the specific value of the month of the natural month in which the time of the current access behavior is located.

[0117] Specifically, if the user's current access behavior is not a crawler behavior within the first time window with the second as the time dimension and is not a crawler behavior within the minute dimension time window, then check whether there is a data object in the database corresponding to the month in which the time of the current access behavior is located. For example, the time when user A accesses interface address X1 is 12:24:30 on March 1, 2025, and this access is not a crawler behavior within the first time window in the second dimension and within the month dimension time window. Then, check whether there is a data object with user A, interface address X1, and month March as the key in the database.

[0118] If the data object does not exist in the database, create a data object with user A, interface address X1, and month March as the key, and set the score in this data object to 1.

[0119] Optionally, when creating a data object with user A, interface address X1, and month March as the key, the expiration time of this data object can be set. For example, set the expiration time of this data object to 31 days. Then, on the 32nd day after creating this data object, the system automatically clears this data object. This can prevent the accumulation of data objects in the database, resulting in redundant content in the database.

[0120] If the data object exists in the database, then increment by 1 the score corresponding to the data object with user A, interface address X1, and month March as the key. This indicates that the number of times user A accesses interface address X1 in March is incremented by 1.

[0121] Step 302. If it is determined that the third access count is greater than or equal to the third threshold, then determine that the current access behavior is a crawler behavior.

[0122] Among them, the third threshold is used to indicate the maximum number of times a user is allowed to access the interface address within the month dimension time window.

[0123] Exemplarily, according to the score of the data object in the foregoing example, that is, the third access count is compared with the third threshold. Wherein, the third threshold indicates the maximum access count that allows a user to access the interface address within the monthly time window. It can be understood that if the third access count is greater than or equal to the third threshold, it indicates that the access count of the user within the monthly time window reaches the maximum access count. Therefore, the current access behavior can be determined as a crawler behavior.

[0124] Step 303. If it is determined that the third access count is less than the third threshold, determine that the current access behavior is not a crawler behavior.

[0125] Exemplarily, if the third access count is less than the third threshold, it indicates that the access count of the user within the monthly time window does not reach the maximum access count. Therefore, determine that the current access behavior is not a crawler behavior.

[0126] It can be understood that taking the time dimension of the first time window as seconds as an example, combined with the foregoing example, if an access behavior is determined not to be a crawler behavior in the second time window, it is further judged in the minute time window; if it is determined not to be a crawler behavior in the minute time window, it is further judged in the monthly time window.

[0127] It should be noted that the above steps 302 and 303 can be executed simultaneously or sequentially, and the execution order is not limited.

[0128] In the above example, by detecting crawler behaviors sequentially in multiple time windows from small to large in the time dimension, the detection accuracy of low-frequency crawler behaviors can be effectively guaranteed.

[0129] Based on the foregoing embodiments, the detection method of crawler behavior provided by the present application may have a startup condition, and when the startup condition is met, the method provided by the present application is started to be executed.

[0130] In one example, before obtaining the access information of the current access behavior in response to the current access behavior of the user, the method further includes:

[0131] If it is determined that the detection flag of the crawler behavior is turned on, execute the step of obtaining the access information of the current access behavior in response to the current access behavior of the user.

[0132] Exemplarily, the detection method of crawler behavior provided in this embodiment can be integrated into a virtual module or a virtual tool. The virtual module or virtual tool is provided with a detection flag for crawler behavior, which is used to represent the working state of the virtual module or virtual tool, and the state of the detection flag can be turned on or off.

[0133] If it is determined that the detection flag for crawler behavior is characterized as enabled, it indicates that crawler behavior detection is now required. Therefore, if it is determined that the detection flag for crawler behavior is characterized as enabled, the step of obtaining access information for the current access behavior in response to the user's current access behavior is performed.

[0134] Correspondingly, if it is determined that the detection flag for crawler behavior is characterized as disabled, the subsequent steps are not executed, that is, crawler behavior detection is not required currently.

[0135] In the above example, whether to execute the method provided in the foregoing embodiments of the present application can be controlled by the detection flag of the crawler behavior of the virtual module or virtual tool, which can meet various different development requirements. Improve the flexibility of crawler behavior detection.

[0136] Furthermore, in practical applications, there may be some websites or interface addresses that do not require crawler behavior detection.

[0137] In one example, after obtaining the access information of the current access behavior, the method further includes:

[0138] If it is determined that the interface address of the current access behavior belongs to a preset anti-crawler interface address list, the step of determining the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located is performed.

[0139] Exemplarily, a preset anti-crawler interface address list can be stored in the database. In the anti-crawler interface address list, at least one interface address that requires crawler behavior detection is stored.

[0140] After obtaining the access information of the current access behavior, if it is determined that the interface address of the current access behavior belongs to a preset anti-crawler interface address list, it indicates that crawler behavior detection needs to be performed on the current access behavior. That is, if it is determined that the interface address of the current access behavior belongs to a preset anti-crawler interface address list, the first access count is determined within the first time window where the time of the current access behavior is located.

[0141] Correspondingly, if it is determined that the interface address of the current access behavior does not belong to a preset anti-crawler interface address list, the step of determining the first access count is not executed.

[0142] In the above example, whether to perform further crawler behavior detection is judged according to whether the interface address of the current access behavior belongs to a preset anti-crawler interface address list. It can meet the requirements for websites with different security levels. Further improve the flexibility of crawler behavior detection.

[0143] Based on the foregoing embodiments, during the process of determining crawler behavior within the first time window, if it is detected that the first access count within the first time window has exceeded the threshold, it can be directly determined that the access behavior is a crawler behavior.

[0144] In one example, if it is determined that the first access count is greater than or equal to the first threshold, it is determined that the current access behavior is a crawler behavior.

[0145] Exemplarily, in combination with the foregoing embodiments, the first threshold indicates the maximum number of accesses allowed for a user to access the interface address within the first time window. If within the first time window with a time dimension of seconds, the first access count reaches the maximum access count, it can be directly determined that the current access behavior is a crawler behavior.

[0146] In the above example, different thresholds are set based on time windows with different time dimensions, and crawler behaviors in different time dimensions can be accurately detected.

[0147] Based on any of the foregoing embodiments, if it is detected that the access behavior is a crawler behavior, it can be intercepted.

[0148] In one example, if it is determined that the current access behavior is a crawler behavior, the current access behavior is intercepted, and the access information of the current access behavior is recorded in the exception log.

[0149] Exemplarily, if it is determined that the current access behavior is a crawler behavior, the current access behavior is intercepted. Specifically, the current access behavior can be intercepted in any one or a combination of the following ways:

[0150] (1) Directly for the crawler behavior, reject its request and prohibit it from obtaining the data resources on the interface address;

[0151] (2) Add interactive verification to the crawler behavior. Among them, common practices of interactive verification can include, but are not limited to: verification codes, behavior detection (detecting mouse movement and click patterns, etc.), and dynamic questions.

[0152] (3) For the access behavior determined to be a crawler behavior, add noise to the data it obtains so that the data obtained by the crawler behavior is not real data.

[0153] In addition, the access information corresponding to the current access behavior that has been determined to be a crawler behavior is recorded in the exception log. It can be understood that in the exception log, all access information determined to be crawler behaviors is recorded.

[0154] Furthermore, after intercepting the current access behavior, the method further includes:

[0155] Intercept all subsequent accesses of the user within a preset time period after the current access behavior.

[0156] Among them, the interface address of the subsequent access is the same as the interface address of the current access behavior.

[0157] Exemplarily, for the current access behavior that has been determined to be a crawler behavior, intercept all subsequent accesses of the user within a preset time period after the current access behavior.

[0158] Among them, the interface address of the subsequent access is the same as the interface address of the current access behavior.

[0159] It can be understood that for subsequent access behaviors of the same user that have been determined to be crawler behaviors, most of them are likely to be crawler behaviors. Then, within a preset time period after the current access behavior, intercept all subsequent access behaviors of the user to the same interface address.

[0160] Optionally, the preset time period can be flexibly adjusted according to the number of crawler behaviors.

[0161] In the above example, for the access behavior determined to be a crawler behavior, intercept and record the exception log; and for all subsequent accesses of the user within a preset time period after the current access behavior, intercept them. On the basis of accurately detecting crawler behaviors, it is possible to accurately intercept crawler behaviors, thereby improving the security and reliability of the data resources of the website or application.

[0162] The detection method of crawler behavior based on multiple time dimensions provided by the embodiments of the present application, by responding to the current access behavior of the user, obtains the time and interface address of the current access behavior; determines whether it is a crawler behavior within the first time window where the time of the current access behavior is located. If it is determined that it is not a crawler behavior within the first time window, further determine whether the current access behavior is a crawler behavior in a time window of a larger time dimension. Judging whether it belongs to a crawler behavior in time windows with different time dimensions can accurately detect crawler behaviors with a low access frequency. Thus, the effect of improving the overall detection accuracy of crawler behaviors is achieved.

[0163] Figure 4 This is a schematic structural diagram of the detection device for crawler behavior based on multiple time dimensions provided by the present application, as Figure 4 shown, the detection device 40 for crawler behavior based on multiple time dimensions provided in this embodiment includes:

[0164] An acquisition module 401, configured to obtain access information of the current access behavior in response to the current access behavior of the user; among them, the access information includes the interface address of the current access behavior and the time of the current access behavior;

[0165] A processing module 402 is configured to determine a first access count of a user accessing an interface address within a first time window where the time of the current access behavior is located;

[0166] The processing module 402 is further configured to, if it is determined that the first access count is less than a first threshold, determine whether the current access behavior is a crawler behavior within at least one second time window where the time of the current access behavior is located; wherein, the first threshold is used to indicate the maximum number of times a user is allowed to access the interface address within the first time window; the time dimension of the second time window is greater than the time dimension of the first time window.

[0167] In a possible implementation manner, the second time window includes: a minute - dimension time window and a month - dimension time window;

[0168] Within at least one second time window where the time of the current access behavior is located, to determine whether the current access behavior is a crawler behavior, the processing module 402 is configured to:

[0169] Within the minute - dimension time window where the time of the current access behavior is located, determine a second access count of the user accessing the interface address;

[0170] If it is determined that the second access count is greater than or equal to a second threshold, determine that the current access behavior is a crawler behavior; wherein, the second threshold is used to indicate the maximum number of times a user is allowed to access the interface address within the minute - dimension time window;

[0171] If it is determined that the second access count is less than the second threshold, within the month - dimension time window where the time of the current access behavior is located, determine whether the current access behavior is a crawler behavior.

[0172] In a possible implementation manner, within the month - dimension time window where the time of the current access behavior is located, to determine whether the current access behavior is a crawler behavior, the processing module 402 is configured to:

[0173] Within the month - dimension time window where the time of the current access behavior is located, determine a third access count of the user accessing the interface address;

[0174] If it is determined that the third access count is greater than or equal to a third threshold, determine that the current access behavior is a crawler behavior; wherein, the third threshold is used to indicate the maximum number of times a user is allowed to access the interface address within the month - dimension time window;

[0175] If it is determined that the third access count is less than the third threshold, determine that the current access behavior is not a crawler behavior.

[0176] In a possible implementation manner, before obtaining the access information of the current access behavior in response to the user's current access behavior, the obtaining module 401 is further configured to:

[0177] If it is determined that the detection flag for crawler behavior is enabled, then perform the step of obtaining the access information of the current access behavior in response to the user's current access behavior.

[0178] In a possible implementation manner, after obtaining the access information of the current access behavior, the processing module 402 is further configured to:

[0179] If it is determined that the interface address of the current access behavior belongs to a preset list of anti-crawler interface addresses, then perform the step of determining the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located.

[0180] In a possible implementation manner, the processing module 402 is further configured to:

[0181] If it is determined that the first access count is greater than or equal to the first threshold, then determine that the current access behavior is a crawler behavior.

[0182] In a possible implementation manner, the processing module 402 is further configured to:

[0183] If it is determined that the current access behavior is a crawler behavior, then intercept the current access behavior and record the access information of the current access behavior in the exception log.

[0184] In a possible implementation manner, after intercepting the current access behavior, the processing module 402 is further configured to:

[0185] Intercept all subsequent accesses of the user within a preset time period after the current access behavior; wherein, the interface address of the subsequent access is the same as the interface address of the current access behavior.

[0186] The detection device for crawler behavior based on multiple time dimensions provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here in this embodiment.

[0187] Figure 5 It is a schematic structural diagram of the electronic device provided in this application. As Figure 5 shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. Among them, the processor 501, the memory 502, and the communication component 503 are connected through a bus 504.

[0188] In a specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0189] For the specific implementation process of the processor 501, reference may be made to the foregoing method embodiments. Their implementation principles and technical effects are similar, and will not be elaborated herein.

[0190] In the foregoing embodiments, it should be understood that the processor may be a central processing unit (CPU for short), or other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.

[0191] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0192] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0193] This application also provides a computer program product, including a computer program which, when executed by a processor, implements the foregoing method.

[0194] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the foregoing method.

[0195] The above-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk. The readable storage medium may be any available medium accessible by a general-purpose or special-purpose computer.

[0196] An exemplary readable storage medium is coupled to the processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium may also be a component of the processor. The processor and the readable storage medium may be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium may also exist as discrete components in a device.

[0197] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other may be indirect couplings or communication connections through some interfaces, devices, or units, and may be in electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, the functional units in each embodiment of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit.

[0200] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, and other various media that can store program codes.

[0201] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When this program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: ROMs, RAMs, magnetic disks, or optical discs, and other various media that can store program codes.

[0202] Finally, it should be noted that: After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily think of other implementation manners of the present invention. The present invention is intended to cover any variations, uses, or adaptations of the present invention. These variations, uses, or adaptations follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A detection method for crawler behavior based on multiple time dimensions, characterized in that, Including: In response to the user's current access behavior, obtain the access information of the current access behavior; wherein, the access information includes the interface address of the current access behavior and the time of the current access behavior; Within the first time window where the time of the current access behavior is located, determine the first access count of the user accessing the interface address; If it is determined that the first access count is less than the first threshold, then within at least one second time window where the time of the current access behavior is located, determine whether the current access behavior is a crawler behavior; wherein, the first threshold is used to indicate the maximum access count allowed for the user to access the interface address within the first time window; the time dimension of the second time window is greater than the time dimension of the first time window.

2. The method according to claim 1, wherein The second time window includes: a minute - dimension time window and a month - dimension time window; Determining whether the current access behavior is a crawler behavior within at least one second time window where the time of the current access behavior is located includes: Within the minute - dimension time window where the time of the current access behavior is located, determine the second access count of the user accessing the interface address; If it is determined that the second access count is greater than or equal to the second threshold, then determine that the current access behavior is a crawler behavior; wherein, the second threshold is used to indicate the maximum access count allowed for the user to access the interface address within the minute - dimension time window; If it is determined that the second access count is less than the second threshold, then within the month - dimension time window where the time of the current access behavior is located, determine whether the current access behavior is a crawler behavior.

3. The method according to claim 2, characterized in that, Determining whether the current access behavior is a crawler behavior within the month - dimension time window where the time of the current access behavior is located includes: Within the month - dimension time window where the time of the current access behavior is located, determine the third access count of the user accessing the interface address; If it is determined that the third access count is greater than or equal to the third threshold, then determine that the current access behavior is a crawler behavior; wherein, the third threshold is used to indicate the maximum access count allowed for the user to access the interface address within the month - dimension time window; If it is determined that the third access count is less than the third threshold, determine that the current access behavior is not a crawler behavior.

4. The method according to claim 1, wherein Before obtaining the access information of the current access behavior in response to the user's current access behavior, the method further includes: If it is determined that the detection flag for crawler behavior is enabled, then execute the step of obtaining the access information of the current access behavior in response to the user's current access behavior.

5. The method according to claim 4, characterized in that, After obtaining the access information of the current access behavior, the method further includes: If it is determined that the interface address of the current access behavior belongs to a preset anti - crawler interface address list, then execute the step of determining the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located.

6. The method according to claim 1, characterized in that, The method further includes: If it is determined that the first access count is greater than or equal to the first threshold, then determine that the current access behavior is a crawler behavior.

7. The method according to any one of claims 1 to 6, characterized in that The method further includes: If it is determined that the current access behavior is a crawler behavior, intercept the current access behavior and record the access information of the current access behavior in the exception log.

8. The method according to claim 7, characterized in that After intercepting the current access behavior, the method further includes: Intercepting all subsequent accesses of the user within a preset time period after the current access behavior; wherein the interface address of the subsequent access is the same as the interface address of the current access behavior.

9. A detection device for crawler behavior based on multiple time dimensions, characterized in that, It includes: An acquisition module, configured to acquire the access information of the current access behavior in response to the user's current access behavior; wherein the access information includes the interface address of the current access behavior and the time of the current access behavior. A processing module, configured to determine the first access count of the user accessing the interface address within the first time window where the time of the current access behavior is located. The processing module is further configured to, if it is determined that the first access count is less than the first threshold, determine whether the current access behavior is a crawler behavior within at least one second time window where the time of the current access behavior is located; wherein the first threshold is used to indicate the maximum number of accesses allowed for the user to access the interface address within the first time window; the time dimension of the second time window is greater than the time dimension of the first time window.

10. An electronic device, characterized in that, It includes: A memory, a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of claims 1-8.

Images