Anti-crawler method, device and equipment based on BS architecture, and storage medium

Abstract

The application discloses a BS architecture-based anti-crawler method and device, equipment and a storage medium. By obtaining access behavior information and browser user information, and determining access cost data and access benefit data of an access request, the actual benefit and actual cost of the access request to a server can be quantified. According to the access cost data and the access benefit data, target access behavior index information meeting a preset access constraint condition is determined, so that the actual cost and the actual benefit are used to intelligently determine the target access behavior index information of the access request, effectively avoiding the problem of misjudgment caused by subjectivity of artificial setting. Finally, according to the target access behavior index information, target anti-crawler decision information corresponding to the access request is determined, so that the target anti-crawler decision information is used to make a blocking decision or a passing decision on the access request, reduces the misjudgment rate of anti-crawler identification, and improves the accuracy of anti-crawler identification.

Description

Technical Field

[0001] This application relates to the field of information security technology, and in particular to an anti-crawler method, apparatus, device and storage medium based on a B / S architecture. Background Technology

[0002] A web crawler is a program or script that retrieves information from the World Wide Web according to predetermined rules, effectively extracting and utilizing this information. However, malicious web crawlers can increase the workload of web servers, severely impacting their performance and making it difficult for them to provide services to legitimate users. Therefore, anti-crawler technology is used to detect and identify malicious web crawlers, thereby providing defense against them.

[0003] Currently, some technologies use manually set detection criteria. When a browser's access to a web server meets a certain criterion, the access behavior is judged as web crawling. For example, if a browser accesses a web server a preset number of times within a unit of time, it is judged as web crawling. However, normal browser users may also continuously refresh the access request for various reasons, and such normal browser users should not actually be judged as web crawlers. It is evident that the current anti-web crawling identification results have a high false positive rate. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for anti-crawler identification based on a B / S architecture, in order to solve the technical problem that the current anti-crawler identification results have a high false positive rate.

[0005] To address the aforementioned technical problems, embodiments of this application provide an anti-crawler method based on a B / S architecture, comprising:

[0006] Obtain access behavior information and browser user information. Access behavior information refers to the behavior of the browser when sending access requests to the server.

[0007] Based on access behavior information and browser user information, determine the access cost data and access benefit data of the access request;

[0008] Based on access cost data and access benefit data, determine the target access behavior indicators that meet the preset access constraints.

[0009] Based on the target access behavior indicator information, the target anti-crawler decision information corresponding to the access request is determined. The target anti-crawler decision information is used to make decisions to block or allow the access request.

[0010] This embodiment acquires access behavior information and browser user information, and determines the access cost data and access benefit data of the access request to quantify the actual benefits and costs brought to the server by the access request. Based on the access cost data and access benefit data, it determines the target access behavior indicator information that meets preset access constraints. This allows for the intelligent determination of the target access behavior indicator information of the access request using actual costs and benefits, effectively avoiding the problem of misjudgment caused by subjectivity in manual settings. Finally, based on the target access behavior indicator information, it determines the target anti-crawler decision information corresponding to the access request, and uses this target anti-crawler decision information to make blocking or allowing decisions on the access request, reducing the misjudgment rate of anti-crawler identification and improving the accuracy of anti-crawler identification.

[0011] In one embodiment, based on access behavior information and browser user information, the access cost data and access benefit data of the access request are determined, including:

[0012] Based on browser user information, determine the cost parameters and benefit parameters for the browser user;

[0013] Based on access behavior information and cost parameters, determine the access cost data for browser users;

[0014] Based on access behavior information and benefit parameters, determine the access benefit data for browser users.

[0015] In one embodiment, based on access cost data and access benefit data, target access behavior indicator information that meets preset access constraints is determined, including:

[0016] Based on the access cost data and access benefit data, determine the access constraint function for the access request;

[0017] Using a variety of linear algorithms, based on the access constraint function, multiple initial access behavior indicators that satisfy the preset access constraint conditions are generated;

[0018] Using a genetic algorithm, multiple initial access behavior indicators are combined and iterated until a preset iteration termination condition is met, thus obtaining the target access behavior indicator information.

[0019] In one embodiment, multiple linear algorithms are used to generate multiple initial behavioral indicator information that satisfy preset access constraint conditions based on the access constraint function, including:

[0020] Using the simplex method, we determine multiple behavioral indicators that satisfy preset access constraints.

[0021] Using behavioral index information as a feasible basis for the simplex method, the access constraint function is debased to obtain the simplex matrix.

[0022] If the test number of the simplex matrix is ​​not non-negative, the basic and non-basic variables of the feasible basis are replaced, and the access constraint function is debased until the test number of the simplex matrix is ​​non-negative, thus obtaining the initial access behavior index information.

[0023] In one embodiment, multiple linear algorithms are used to generate multiple initial access behavior indicator information that satisfy preset access constraints based on the access constraint function, including:

[0024] The access constraint function is iterated using the interior point method to obtain the iteration result;

[0025] Based on the iteration results, the parameters of the access constraint function are updated until the iteration results meet the preset access constraint conditions, and the initial access behavior index information is obtained.

[0026] In one embodiment, a genetic algorithm is used to combine and iterate multiple initial access behavior indicator information until a preset iteration termination condition is reached to obtain target access behavior indicator information, including:

[0027] Multiple initial visit behavior indicators are used as chromosomes for the genetic population;

[0028] Genetic algorithms are used to iterate the evolution of the genetic population.

[0029] Calculate the target fitness of all chromosomes in the genetic population during evolutionary iterations based on the access constraint function.

[0030] When the number of evolutionary iterations reaches the preset number of evolutionary iterations, the iteration stops, and the initial access behavior index information corresponding to the chromosome with the highest target fitness is used as the target access behavior index information.

[0031] In one embodiment, determining the target anti-crawler decision information corresponding to the access request based on target access behavior indicator information includes:

[0032] Match access behavior information with target access behavior metrics information;

[0033] If the access behavior information matches the target access behavior indicator information, it is determined that the target anti-crawler decision information is used to make a decision on whether to allow the access request.

[0034] If the access behavior information does not match the target access behavior indicator information, it is determined that the target anti-crawler decision information is used to make a decision to block the access request.

[0035] Secondly, embodiments of this application provide an anti-crawler device based on a B / S architecture, comprising:

[0036] The acquisition module is used to acquire access behavior information and browser user information. The access behavior information is the behavior information of the browser when sending access requests to the server.

[0037] The first determining module is used to determine the access cost data and access benefit data of the access request based on access behavior information and browser user information.

[0038] The second determining module is used to determine the target access behavior indicator information that meets the preset access constraints based on access cost data and access benefit data.

[0039] The third determination module is used to determine the target anti-crawler decision information corresponding to the access request based on the target access behavior indicator information. The target anti-crawler decision information is used to make blocking or allowing decisions on the access request.

[0040] Thirdly, embodiments of this application provide a computer device, characterized in that it includes a processor and a memory, the memory being used to store a computer program, and the computer program, when executed by the processor, implements the anti-crawler method based on the B / S architecture as described in the first aspect.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the anti-crawler method based on a B / S architecture as described in the first aspect.

[0042] It should be noted that the beneficial effects of the second to fourth aspects mentioned above are described in the relevant description of the first aspect, and will not be repeated here. Attached Figure Description

[0043] Figure 1 A flowchart illustrating the anti-crawler method based on a B / S architecture provided in this application embodiment;

[0044] Figure 2 A schematic diagram of the structure of the anti-crawler device based on the BS architecture provided in the embodiments of this application;

[0045] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] As documented in relevant technical records, by manually setting detection indicators, a browser's access to a web server is considered a web crawler if it meets certain criteria. For example, if a browser accesses a web server a preset number of times within a given time period, it is considered a web crawler. However, normal browser users may continuously refresh the access request for various reasons, and such normal browser users should not actually be considered web crawlers. Therefore, current anti-crawler identification methods have a high false positive rate.

[0048] To address this, embodiments of this application provide an anti-crawler method, apparatus, device, and storage medium based on a B / S architecture. By acquiring access behavior information and browser user information, and determining the access cost data and access benefit data of the access request, the actual benefits and costs brought to the server by the access request can be quantified. Furthermore, based on the access cost data and access benefit data, target access behavior indicator information that meets preset access constraints is determined. This allows for the intelligent determination of target access behavior indicator information for access requests using actual costs and benefits, effectively avoiding the problem of misjudgment caused by subjectivity in manual settings. Finally, based on the target access behavior indicator information, target anti-crawler decision information corresponding to the access request is determined, enabling the use of this target anti-crawler decision information to make blocking or allowing decisions for the access request, reducing the misjudgment rate of anti-crawler identification and improving the accuracy of anti-crawler identification.

[0049] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a B / S architecture-based anti-crawler method provided in an embodiment of this application. The B / S architecture-based anti-crawler method of this embodiment can be applied to computer devices, such as laptops, desktop computers, physical servers, and cloud servers. The B / S architecture-based anti-crawler method of this embodiment includes steps S101 to S104, detailed below:

[0050] Step S101: Obtain access behavior information and browser user information, wherein the access behavior information is the behavior information of the browser when sending an access request to the server.

[0051] In this step, access behavior information includes, but is not limited to, access duration, number of accesses, access frequency, and accessed pages. Browser user information includes, but is not limited to, IP address, operating system and version, CPU type, and browser type and version.

[0052] Step S102: Determine the access cost data and access benefit data of the access request based on the access behavior information and the browser user information.

[0053] In this step, access cost data represents the total cost incurred by the server when a browser user accesses the server. Access benefit data represents the total benefit the server can obtain when a browser user accesses the server. Optionally, access behavior information and browser user information are quantified using preset values, and access cost data and access benefit data are calculated based on the quantified values.

[0054] Step S103: Based on the access cost data and the access benefit data, determine the target access behavior indicator information that meets the preset access constraints.

[0055] In this step, the preset access constraints are constraints on behavioral metrics, such as the number of accesses, access duration, and access frequency. Therefore, the preset access constraints would be the maximum number of accesses, the minimum access duration, the maximum access frequency, and the threshold of the access constraint function. For example, the access constraint function = P(x i ,c i )-P(x i ,r i If x > 100, the number of visits x1 < 10 times, the visit duration x2 > 10 seconds, and the visit frequency x3 < 10 times / min.

[0056] Understandably, current technology involves manually setting uniform behavioral indicator thresholds for access requests, which introduces significant subjectivity. This embodiment, however, combines access cost data and access benefit data to redetermine the applicable behavioral indicator thresholds for the specific user within a certain range, making anti-crawler identification more accurate and targeted.

[0057] Step S104: Based on the target access behavior indicator information, determine the target anti-crawler decision information corresponding to the access request. The target anti-crawler decision information is used to make a blocking decision or a allowing decision on the access request.

[0058] In this step, the target anti-crawler decision information is used to make blocking or allowing decisions on the access request, thereby reducing the false positive rate of anti-crawler identification and improving the accuracy of anti-crawler identification.

[0059] In one embodiment, in Figure 1 Based on the illustrated embodiment, step S102 above includes:

[0060] Based on the browser user information, determine the cost parameters and benefit parameters of the browser user;

[0061] Based on the access behavior information and the cost parameters, determine the access cost data of the browser user;

[0062] Based on the access behavior information and the benefit parameters, determine the access benefit data of the browser user.

[0063] In this embodiment, the cost parameter evaluates the various costs incurred by the server when a browser user accesses the server. For example, the access behavior increases the system load, and a cost value is preset based on the amount of load increase. The benefit parameter evaluates the various benefits that the server can obtain when a browser user accesses the server. For example, browser user access to the server can increase system activity, and a corresponding benefit value is preset based on the amount of increase in system activity.

[0064] Optionally, determining the browser user's access cost function based on the access behavior information and the cost parameters includes: determining multiple behavioral features in the access behavior information; and determining the browser user's access cost function based on the multiple behavioral features and the cost parameters according to a preset access cost calculation formula, wherein the access cost calculation formula is:

[0065]

[0066] Where P(x) i ,c i Let x represent the access cost function for the browser user. i c represents the i-th behavior indicator in the access behavior information. i This represents the cost parameter corresponding to the i-th behavioral indicator.

[0067] Optionally, determining the browser user's access benefit function based on the access behavior information and the benefit parameters includes: determining the browser user's access benefit function based on multiple behavioral features in the access behavior information and the benefit parameters according to a preset access benefit calculation formula, wherein the access benefit calculation formula is:

[0068]

[0069] Where P(x) i ,r i ) represents the access benefit function for the browser user, x i This represents the i-th behavior indicator in the access behavior information, r i This represents the benefit parameter corresponding to the i-th behavioral indicator.

[0070] In one embodiment, in Figure 1 Based on the illustrated embodiment, step S103 above includes:

[0071] Based on the access cost data and the access benefit data, determine the access constraint function for the access request;

[0072] Using a variety of linear algorithms, based on the access constraint function, multiple initial access behavior indicator information that satisfy preset access constraint conditions are generated;

[0073] Using a genetic algorithm, multiple initial access behavior indicators are combined and iterated until a preset iteration termination condition is met, thereby obtaining the target access behavior indicator information.

[0074] In this embodiment, the access constraint function = access cost data - access benefit data, that is, the access constraint function Z = P(x i ,c i )-P(x i ,r i Linear algorithms can be simplex methods, interior-point methods, or linear autoregressive methods, etc. For example, the access constraint function = P(x) i ,c i )-P(x i ,r i Given that x1 > 100, access count x1 < 10 times, access duration x2 > 10s, and access frequency x3 < 10 times / min, the initial access behavior indicators are calculated as follows: maximum access count = 5, minimum access duration = 12s, and maximum access frequency = 3 times / min.

[0075] It is understandable that this initial access behavior indicator information only corresponds to the access request of this browser user. For access requests from other browser users, the initial access behavior indicator information may be different.

[0076] Optionally, the step of using multiple linear algorithms to generate multiple initial behavioral indicator information that satisfy preset access constraint conditions according to the access constraint function includes:

[0077] Using the simplex method, we determine multiple behavioral indicators that satisfy preset access constraints.

[0078] Using the behavioral index information as a feasible basis for the simplex method, the access constraint function is debased to obtain the simplex matrix.

[0079] If the test number of the simplex matrix is ​​not non-negative, then the basic and non-basic variables of the feasible basis are replaced, and the access constraint function is debased until the test number of the simplex matrix is ​​non-negative, thus obtaining the initial access behavior index information.

[0080] In this optional embodiment, the simplex method obtains feasible solutions by setting different basis vectors and performing linear transformations of the matrix. It then determines whether the feasible solution is the optimal feasible solution. Otherwise, it continues to set another set of basis vectors and repeats the above steps until the optimal feasible solution is found.

[0081] Specifically, various behavioral indicators are combined to determine multiple behavioral indicator information (this behavioral indicator information is a combination of behavioral indicators) that satisfy the preset access constraints. One behavioral indicator information is used as a feasible basis of the simplex method, and a feasible basis is specified to perform a non-basicization operation on the access constraint function to obtain a simplex matrix (simplex table). It is determined whether all test numbers in the test rows of the simplex matrix are non-negative. If not, the basic and non-basic variables of the feasible basis are replaced, that is, the behavioral feature values ​​in the behavioral indicator information are replaced. For example, the maximum number of accesses of 10 is replaced with the maximum number of accesses of 5. The access constraint function is re-non-basicized using the new feasible basis to obtain a new simplex matrix. If all test numbers in the test rows of the new simplex matrix are not non-negative, the basic and non-basic variables of the feasible basis are replaced again until all test numbers in the test rows of the simplex matrix are non-negative. The optimal solution of the simplex method is obtained, and the optimal solution is used as the initial access behavioral indicator information.

[0082] Optionally, the step of using multiple linear algorithms to generate multiple initial access behavior indicator information that satisfy preset access constraint conditions according to the access constraint function includes:

[0083] The access constraint function is iterated using the interior point method to obtain the iteration result;

[0084] Based on the iteration results, the parameters of the access constraint function are updated until the iteration results satisfy the preset access constraint conditions, thereby obtaining the initial access behavior index information.

[0085] In this optional embodiment, the interior point method defines the penalty function within the feasible region and finds the extreme point of the penalty function within the feasible region. That is, the exploration point when solving an unconstrained problem is always within the feasible region, and the solutions to the series of unconstrained optimization problems obtained are always feasible solutions, thereby gradually approaching the optimal solution of the original constrained optimization problem within the feasible region.

[0086] Optionally, the step of using a genetic algorithm to combine and iterate multiple initial access behavior indicator information until a preset iteration termination condition is met to obtain the target access behavior indicator information includes:

[0087] The initial access behavior indicators are used as chromosomes of the genetic population.

[0088] The genetic algorithm is used to perform evolutionary iterations on the genetic population;

[0089] Calculate the target fitness of all chromosomes in the genetic population during evolutionary iteration based on the access constraint function;

[0090] When the number of evolutionary iterations reaches the preset number of evolutionary iterations, the iteration stops, and the initial access behavior index information corresponding to the chromosome with the highest target fitness is used as the target access behavior index information.

[0091] In this optional embodiment, the initial access behavior index information is used as the initial chromosome of the genetic population to initialize the genetic algorithm; the behavior index in the initial access behavior index information is encoded as a real number; then the genetic population undergoes evolutionary iteration according to the access constraint function Z = P(x i ,c i )-P(x i ,r i The fitness of each chromosome in the genetic population is calculated. Since the access constraint function Z is the final reward value of the access behavior, and the larger the reward value, the more the access behavior meets the requirements of normal behavior, the initial access behavior index information corresponding to the chromosome with the largest Z is used as the target access behavior index information.

[0092] The genetic algorithm includes selection, crossover, and mutation operations. Specifically, based on the access constraint function, the total fitness of the genetic population and the individual fitness of each chromosome in the genetic population are determined; based on the total fitness and the individual fitness, the relative fitness of each chromosome is determined; based on the relative fitness, a new chromosome is generated; a crossover operation is performed on all chromosomes in the genetic population to generate offspring chromosomes; and a mutation operation is performed on the offspring chromosomes to obtain mutated chromosomes.

[0093] Genetic algorithms can effectively avoid the situation where linear algorithms get stuck in local optima and fail to obtain the global optimum, thus compensating for the nonlinear space solutions processed by linear programming and improving the reliability of target access behavior index information.

[0094] In one embodiment, determining the target anti-crawler decision information corresponding to the access request based on the target access behavior indicator information includes:

[0095] The access behavior information is matched with the target access behavior indicator information;

[0096] If the access behavior information matches the target access behavior indicator information, then the target anti-crawler decision information is used to make a decision to allow the access request.

[0097] If the access behavior information does not match the target access behavior indicator information, then the target anti-crawler decision information is used to make a blocking decision on the access request.

[0098] In this embodiment, the target access behavior indicator information includes multiple behavior indicator thresholds, such as maximum access count = 5, minimum access duration = 12s and maximum access frequency = 3 times / min. The access count, access duration and access frequency in the access behavior information are compared with each behavior indicator threshold. If they match, it means that the access request conforms to normal access behavior and is allowed; otherwise, it is blocked.

[0099] To implement the anti-crawler method based on a B / S architecture corresponding to the above method embodiments, and to achieve the corresponding functions and technical effects. See also Figure 2 , Figure 2 This diagram illustrates a structural block diagram of a B / S architecture-based anti-crawler device according to an embodiment of this application. For ease of explanation, only the parts relevant to this embodiment are shown. The B / S architecture-based anti-crawler device provided in this embodiment includes:

[0100] The acquisition module 201 is used to acquire access behavior information and browser user information, wherein the access behavior information is the behavior information of the browser when sending an access request to the server;

[0101] The first determining module 202 is used to determine the access cost data and access benefit data of the access request based on the access behavior information and the browser user information.

[0102] The second determining module 203 is used to determine target access behavior indicator information that meets preset access constraints based on the access cost data and the access benefit data.

[0103] The third determining module 204 is used to determine the target anti-crawler decision information corresponding to the access request based on the target access behavior indicator information. The target anti-crawler decision information is used to make a blocking decision or a allowing decision on the access request.

[0104] In one embodiment, the first determining module 202 includes:

[0105] The first determining unit is used to determine the cost parameters and benefit parameters of the browser user based on the browser user information.

[0106] The second determining unit is used to determine the access cost data of the browser user based on the access behavior information and the cost parameters;

[0107] The third determining unit is used to determine the access benefit data of the browser user based on the access behavior information and the benefit parameters.

[0108] In one embodiment, the second determining module 203 includes:

[0109] The fourth determining unit is used to determine the access constraint function of the access request based on the access cost data and the access benefit data.

[0110] The generation unit is used to generate multiple initial access behavior indicator information that satisfy preset access constraint conditions based on the access constraint function using various linear algorithms.

[0111] An iterative unit is used to combine and iterate multiple initial access behavior indicator information using a genetic algorithm until a preset iteration termination condition is reached to obtain the target access behavior indicator information.

[0112] In one embodiment, the generation unit includes:

[0113] The first determining sub-unit is used to determine multiple behavioral index information when the preset access constraints are met using the simplex method.

[0114] The operation subunit is used to use the behavior index information as a feasible basis of the simplex method, and to perform a non-basicization operation on the access constraint function to obtain the simplex matrix.

[0115] The sub-unit is replaced if the test number of the simplex matrix is ​​not non-negative. In this case, the basic and non-basic variables of the feasible basis are replaced, and the access constraint function is debased until the test number of the simplex matrix is ​​non-negative, thereby obtaining the initial access behavior index information.

[0116] In one embodiment, the generation unit includes:

[0117] The first iterative subunit is used to iterate the access constraint function using the interior point method to obtain the iterative result;

[0118] An update subunit is used to update the parameters of the access constraint function based on the iteration results until the iteration results satisfy the preset access constraint conditions, thereby obtaining the initial access behavior index information.

[0119] In one embodiment, the iterative unit includes:

[0120] As a subunit, it is used to treat multiple initial access behavior index information as chromosomes of a genetic population;

[0121] The second iteration subunit is used to perform evolutionary iteration on the genetic population using the genetic algorithm;

[0122] The computational subunit is used to calculate the target fitness of all chromosomes in the genetic population during evolutionary iteration, based on the access constraint function.

[0123] The stopping subunit is used to stop the iteration when the number of evolutionary iterations reaches a preset number of evolutionary iterations, and to use the initial access behavior index information corresponding to the chromosome with the highest target fitness as the target access behavior index information.

[0124] In one embodiment, the third determining module 204 includes:

[0125] A matching unit is used to match the access behavior information with the target access behavior indicator information;

[0126] The first determination unit is used to determine that if the access behavior information matches the target access behavior indicator information, the target anti-crawler decision information is used to make a decision to allow the access request.

[0127] The second determination unit is used to determine that if the access behavior information does not match the target access behavior indicator information, the target anti-crawler decision information is used to make a blocking decision on the access request.

[0128] The aforementioned anti-crawler device based on a B / S architecture can implement the anti-crawler method based on a B / S architecture described in the above method embodiments. The options in the above method embodiments are also applicable to this embodiment, and will not be detailed here. The remaining content of this application's embodiments can be referred to the content of the above method embodiments, and will not be repeated in this embodiment.

[0129] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3 As shown, the computer device 3 of this embodiment includes: at least one processor 30 ( Figure 3 (Only one is shown in the diagram) a processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 executes the computer program 32 to implement the steps in any of the above method embodiments.

[0130] The computer device 3 may be a computing device such as a laptop computer, desktop computer, or cloud server. This computer device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0131] The processor 30 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0132] In some embodiments, the memory 31 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 31 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 31 may include both internal and external storage units of the computer device 3. The memory 31 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0133] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0134] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0135] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0136] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A method for anti-crawler based on BS architecture, characterized in that, The method comprises the following steps: acquiring access behavior information and browser user information, the access behavior information being behavior information when a browser sends an access request to a server; determining access cost data and access benefit data of the access request according to the access behavior information and the browser user information; determining target access behavior index information meeting a preset access constraint condition according to the access cost data and the access benefit data, specifically, determining an access constraint function of the access request according to the access cost data and the access benefit data; generating a plurality of initial access behavior index information meeting the preset access constraint condition according to the access constraint function by using a plurality of linear algorithms; combining and iterating the plurality of initial access behavior index information by using a genetic algorithm until a preset iteration termination condition is reached to obtain the target access behavior index information; determining target anti-crawler decision information corresponding to the access request according to the target access behavior index information, the target anti-crawler decision information being used for making a blocking decision or a passing decision on the access request.

2. The anti-crawler method based on BS architecture according to claim 1, wherein, The method of determining the access cost data and the access benefit data of the access request according to the access behavior information and the browser user information comprises the following steps: determining cost parameters and benefit parameters of a browser user according to the browser user information; determining access cost data of the browser user according to the access behavior information and the cost parameters; determining access benefit data of the browser user according to the access behavior information and the benefit parameters. 3.The anti-crawler method based on BS architecture of claim 1, wherein, The method of generating a plurality of initial behavior index information meeting the preset access constraint condition according to the access constraint function by using a plurality of linear algorithms comprises the following steps: determining a plurality of behavior index information meeting the preset access constraint condition by using a simplex method; performing non-basic operation on the access constraint function by taking the behavior index information as a feasible basis of the simplex method to obtain a simplex matrix; if the test number of the simplex matrix is not a non-negative number, replacing a basic variable and a non-basic variable of the feasible basis and performing non-basic operation on the access constraint function until the test number of the simplex matrix is a non-negative number to obtain the initial access behavior index information.

4. The anti-crawler method based on BS architecture according to claim 1, wherein, The method of generating a plurality of initial access behavior index information meeting the preset access constraint condition according to the access constraint function by using a plurality of linear algorithms comprises the following steps: performing iteration on the access constraint function by using an interior point method to obtain an iteration result; updating parameters of the access constraint function based on the iteration result until the iteration result meets the preset access constraint condition to obtain the initial access behavior index information.

5. The anti-crawler method based on BS architecture according to claim 1, wherein, The method of combining and iterating the plurality of initial access behavior index information by using a genetic algorithm until a preset iteration termination condition is reached to obtain the target access behavior index information comprises the following steps: taking the plurality of initial access behavior index information as chromosomes of a genetic population; performing evolutionary iteration on the genetic population by using the genetic algorithm; calculating target fitness of all chromosomes in the genetic population during the evolutionary iteration according to the access constraint function; When the evolution iteration number reaches a preset evolution iteration number, the iteration is stopped, and the initial access behavior index information corresponding to the chromosome with the maximum target fitness is taken as the target access behavior index information.

6. The anti-crawler method based on BS architecture according to claim 1, wherein, The target anti-crawler decision information corresponding to the access request is determined according to the target access behavior index information, including: The access behavior information is matched with the target access behavior index information; If the access behavior information matches the target access behavior index information, it is determined that the target anti-crawler decision information is used for pass decision of the access request; If the access behavior information does not match the target access behavior index information, it is determined that the target anti-crawler decision information is used for blocking decision of the access request. 7.A device for anti-crawler based on BS architecture, characterized in that, Including: An acquisition module is configured to acquire access behavior information and browser user information, the access behavior information being behavior information when a browser sends an access request to a server; A first determination module is configured to determine access cost data and access benefit data of the access request according to the access behavior information and the browser user information; A second determination module is configured to determine target access behavior index information meeting a preset access constraint condition according to the access cost data and the access benefit data; A third determination module is configured to determine target anti-crawler decision information corresponding to the access request according to the target access behavior index information, the target anti-crawler decision information being used for blocking decision or pass decision of the access request; The second determination module includes: A fourth determination unit is configured to determine an access constraint function of the access request according to the access cost data and the access benefit data; A generation unit is configured to generate a plurality of initial access behavior index information meeting the preset access constraint condition according to the access constraint function by using a plurality of linear algorithms; An iteration unit is configured to combine and iterate the plurality of initial access behavior index information by using a genetic algorithm until a preset iteration termination condition is reached, so as to obtain the target access behavior index information.

8. A computer device, comprising: A processor and a memory are included, the memory is configured to store a computer program, and the computer program is executed by the processor to implement the anti-crawler method based on the BS architecture according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and executed by the processor to implement the anti-crawler method based on the BS architecture according to any one of claims 1 to 6.

Images