Anti-crawler data valuation method and system based on data incompatibility and out-of-package estimation

Abstract

The present invention proposes an anti-crawler data valuation method and system based on data incompatibility and out-of-package estimation, which relates to the field of network security technology. The technical points of the present invention include: utilizing the data incompatibility between noise data and clean data, dividing the data set into finite and non-overlapping incompatible data subsets by calculating the self-expanding error of the server access behavior data set; utilizing the compatible properties of the non-overlapping data subsets, identifying the clean data subsets from the incompatible data subsets through multiple rounds of voting; performing out-of-package estimation on the training data based on the clean data subset, and then performing anti-crawler data valuation. The present invention can dynamically identify and remove noise data present in the training data set, avoiding the adverse effects of noise data on out-of-package estimation, thereby effectively improving the robustness of the anti-crawler data valuation method based on out-of-package estimation to noise. Compared with traditional methods, the present invention shows significant superiority in processing anti-crawler data valuation.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an anti-crawler data valuation method and system based on data incompatibility and out-of-packet estimation. Background Art

[0002] With the rapid development of web crawler technology, a large amount of unauthorized data collection poses a threat to data owners. To address this, machine learning-based anti-crawler methods are gaining attention. These methods typically utilize behavioral analysis and feature extraction, combined with classification models, to detect and block crawler traffic.

[0003] Current anti-crawler systems primarily rely on the following technologies: 1) Rule-matching anti-crawler: This filters crawler traffic by setting rules such as access frequency and IP blacklists, but this method is easily circumvented. 2) Behavioral feature analysis: This uses machine learning models to detect crawlers based on user behavior features such as access patterns and clickstreams. However, this method is very sensitive to noisy data and is prone to false positives and false negatives. 3) Deep learning-enhanced anti-crawler: This uses deep learning methods to automatically extract features and detect complex crawler behaviors, but this requires high-quality data support, and incorrectly labeled or noisy data will directly affect the model's effectiveness.

[0004] However, since training data may contain noise or bias, such as normal user behavior being mislabeled as crawler behavior, the presence of noisy data can significantly reduce the effectiveness of anti-crawler models. Therefore, how to accurately evaluate the value of training data in anti-crawler tasks becomes a key issue in optimizing anti-crawler models. Summary of the Invention

[0005] In view of the above problems, the present invention proposes an anti-crawler data valuation method and system based on data incompatibility and out-of-package estimation, in an effort to solve or alleviate one or more of the above problems.

[0006] According to one aspect of the present invention, an anti-crawler data valuation method based on data incompatibility and out-of-package estimation is proposed, the method comprising:

[0007] Acquire a server access behavior data set, wherein the server access behavior data set includes crawler behavior data and normal access behavior data;

[0008] The server access behavior dataset is divided into mutually disjoint and incompatible data subsets based on self-expansion error;

[0009] Based on the compatibility theorem of disjoint data subsets, clean data subsets are identified from incompatible data subsets;

[0010] The access behavior data in the server access behavior dataset is valued based on out-of-pack estimation of the clean data subset.

[0011] Furthermore, the access behavior data in the server access behavior data set includes a behavior feature vector, and the behavior feature vector includes traffic features, path features, timing features, interaction features, and comprehensive features.

[0012] Furthermore, dividing the server access behavior dataset into mutually disjoint and incompatible data subsets based on the self-expansion error includes:

[0013] The server access behavior dataset is sampled in multiple rounds, and the sampling size is reduced in each round, that is, the next round of sampling dataset D m+1 is the initial sampling data set D for this round m Subtract the incompatible data subset D′ obtained in this round of sampling m ; Until the sampling size is 0, the server access behavior dataset is divided into multiple incompatible data subsets D1',...,D based on the self-expansion error M '; Among them, for each round of cyclic sampling, the sampling rules are:

[0014]

[0015] Where S represents the candidate incompatible data subset of this round; D m represents the initial data set of this round; S t-1 represents the data subset sampled in the t-1th cycle of this round; |S t-1 | represents the data subset S t-1 The amount of data, β∈(0,1) represents a fixed-size threshold, β|S t-1 | represents the sampling size; x j Indicates the jth behavior data, y j Indicates the label of the j-th behavior data; e t-1 (x j ,y j ) represents the self-expansion error of the t-1th cycle of this round;

[0016] The data subset S obtained by sampling the tth cycle in this round is t Train the classifier f t , and use the classifier f t The loss function is updated from the expanded error e t (x j ,y j );

[0017] When each round of cyclic sampling reaches the termination condition, the current cycle is stopped and the incompatible data subset D′ of this round is obtained. m .

[0018] Furthermore, the self-expanding error et (x j ,y j ) is updated as follows:

[0019] e t (x j ,y j )=η·e t-1 (x j ,y j )+(1-η)·l(y j ,f t (x j ))

[0020] Among them, f t (x j ) represents the classifier f t For the jth behavior data x j The predicted label, η∈(0,1) represents the weight coefficient of the soft update; l(·,·) represents the classifier f t The loss function of e0(x j ,y j )=0.

[0021] Furthermore, the conditions for terminating each round of cyclic sampling are:

[0022] |S T |≤(1-β)·|D action |

[0023] Among them, S T represents the data subset obtained by sampling in the Tth cycle; D action Represents a server access behavior dataset.

[0024] Furthermore, the method of identifying a clean data subset from incompatible data subsets based on the compatibility theorem of disjoint data subsets includes:

[0025] Initialize the clean data subset to an empty set; define the empirical risk as:

[0026]

[0027] Where D′ represents the tested data set; g(f(x j )≠y j ) represents the indicator function. When the classifier f is used for the behavioral data x j The category prediction f(x j ) is wrong, then g(f(x j )≠y j )=1, otherwise g(f(x j )≠y j )=0;

[0028] Using incompatible data subsets D1',...,D M 'Train classifiers f1',...,f M ';For i,j∈{1,...,M}, if the classifier f i ′ for incompatible data subset D j If the empirical risk of ′ is less than one-half, then the data subset D is incompatible. j ' will be D i 'vote:

[0029]

[0030] Where g(.) represents the indicator function; M represents the total number of incompatible data subsets;

[0031] When voting When , the corresponding incompatible data subset is merged into the clean data subset to obtain the clean data subset.

[0032] Furthermore, the estimating the access behavior data in the server access behavior dataset based on the out-of-packet estimation of the clean data subset includes:

[0033] Perform multiple sampling with replacement on the clean data subset to obtain B data subsets: G1,...,G b ,...,G B ; Data subset G1,...,G b ,...,G B Used to train weak classifiers: f1^,...,f b ^,...,f B ^; For the training data point (x i ,y i )∈D action , out-of-bag estimation is performed, and the estimated value is regarded as a behavioral data point (x i ,y i ) value; the formula for out-of-pocket estimation is:

[0034]

[0035] Where g(.) represents the indicator function.

[0036] According to another aspect of the present invention, an anti-crawler data valuation system based on data incompatibility and out-of-package estimation is proposed, the system comprising:

[0037] a data acquisition module configured to acquire a server access behavior data set, wherein the server access behavior data set includes crawler behavior data and normal access behavior data;

[0038] an incompatible data partitioning module configured to partition the server access behavior dataset into mutually disjoint incompatible data subsets based on a self-expanding error;

[0039] a data identification module configured to identify a clean data subset from the incompatible data subsets based on a compatibility theorem of disjoint data subsets;

[0040] The data valuation module is configured to estimate the access behavior data in the server access behavior dataset based on the out-of-packet estimation of the clean data subset.

[0041] Furthermore, the incompatible data partitioning module partitions the server access behavior dataset into mutually disjoint incompatible data subsets based on the self-expansion error, including:

[0042] The server access behavior dataset is sampled in multiple rounds, and the sampling size is reduced in each round, that is, the next round of sampling dataset D m+1 is the initial sampling data set D for this round m Subtract the incompatible data subset D′ obtained in this round of sampling m ; Until the sampling size is 0, the server access behavior dataset is divided into multiple incompatible data subsets D1',...,D based on the self-expansion error M '; Among them, for each round of cyclic sampling, the sampling rules are:

[0043]

[0044] Where S represents the candidate incompatible data subset of this round; D m represents the initial data set of this round; S t-1 represents the data subset sampled in the t-1th cycle of this round; |S t-1 | represents the data subset S t-1 The amount of data, β∈(0,1) represents a fixed-size threshold, β|S t-1 | represents the sampling size; x j Indicates the jth behavior data, y j Indicates the label of the j-th behavior data; e t-1 (x j ,y j ) represents the self-expansion error of the t-1th cycle of this round;

[0045] The data subset S obtained by sampling the tth cycle in this round is t Train the classifier f t , and use the classifier f t The loss function is updated from the expanded error e t (x j ,y j ); the self-expansion error et (x j ,y j ) is updated as follows:

[0046] e t (x j ,y j )=η·e t-1 (x j ,y j )+(1-η)·l(y j ,f t (x j ))

[0047] Among them, f t (x j ) represents the classifier f t For the jth behavior data x j The predicted label, η∈(0,1) represents the weight coefficient of the soft update; l(·,·) represents the classifier f t The loss function of e0(x j ,y j )=0;

[0048] When each round of cyclic sampling reaches the termination condition, the current cycle is stopped and the incompatible data subset D′ of this round is obtained. m .

[0049] Furthermore, the data identification module, based on the compatibility theorem of disjoint data subsets, identifies a clean data subset from an incompatible data subset, including:

[0050] Initialize the clean data subset to an empty set; define the empirical risk as:

[0051]

[0052] Where D′ represents the tested data set; g(f(x j )≠y j ) represents the indicator function. When the classifier f is used for the behavioral data x j The category prediction f(x j ) is wrong, then g(f(x j )≠y j )=1, otherwise g(f(x j )≠y j )=0;

[0053] Using incompatible data subsets D1',...,D M 'Train classifiers f1',...,f M ';For i,j∈{1,...,M}, if the classifier f i ′ for incompatible data subset Dj If the empirical risk of ′ is less than one-half, then the data subset D is incompatible. j ' will be D i 'vote:

[0054]

[0055] Where g(.) represents the indicator function; M represents the total number of incompatible data subsets;

[0056] When voting When , the corresponding incompatible data subset is merged into the clean data subset to obtain the clean data subset.

[0057] Furthermore, the data valuation module estimates the access behavior data in the server access behavior dataset based on the out-of-packet estimation of the clean data subset, including:

[0058] Perform multiple sampling with replacement on the clean data subset to obtain B data subsets: G1,...,G b ,...,G B ; Data subset G1,...,G b ,...,G B Used to train weak classifiers: f1^,...,f b ^,...,f B ^; For the training data point (x i ,y i )∈D action , out-of-bag estimation is performed, and the estimated value is regarded as a behavioral data point (x i ,y i ) value; the formula for out-of-pocket estimation is:

[0059]

[0060] Where g(.) represents the indicator function.

[0061] The beneficial technical effects of the present invention are:

[0062] The present invention proposes an anti-crawler data valuation method and system based on data incompatibility and out-of-package estimation, aiming to accurately evaluate the value of training data in anti-crawler tasks. The present invention utilizes the incompatibility between noisy data and clean data, and divides the training data into finite and non-overlapping incompatible data subsets by calculating the self-expanding error of the server access behavior data set; through a multi-round voting mechanism, a clean data subset is identified from the incompatible data subset; finally, an out-of-package estimation is performed based on these clean data subsets, and anti-crawler data valuation is performed accordingly. The present invention can dynamically identify and exclude noise data, reduce its adverse effects on out-of-package estimation, thereby enhancing the robustness of the anti-crawler data valuation method to noise and effectively improving the accuracy of the valuation. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0064] Figure 1 Schematic diagram of the flow of the anti-crawler data valuation method based on data incompatibility and out-of-package estimation according to an embodiment of the present invention;

[0065] Figure 2 This is an overall framework diagram of the anti-crawler data valuation method based on data incompatibility and out-of-package estimation according to an embodiment of the present invention;

[0066] Figure 3 This is a structural diagram of an anti-crawler data valuation system based on data incompatibility and out-of-package estimation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0068] It is known to those skilled in the art that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. It should be understood that any number of elements in the accompanying drawings is for illustration and not limitation, and any nomenclature is for distinction only and does not have any limiting meaning.

[0069] Previous data valuation methods are easily affected by noise data when performing anti-crawler data valuation, resulting in inaccurate data valuation. The embodiment of the present invention proposes an anti-crawler data valuation method based on data incompatibility and out-of-package estimation, such as Figures 1-2 As shown, the method includes the following steps:

[0070] S1. Obtain a server access behavior data set, where the server access behavior data set includes crawler behavior data and normal access behavior data;

[0071] S2, dividing the server access behavior dataset into disjoint and incompatible data subsets based on self-expansion error;

[0072] S3. Based on the compatibility theorem of disjoint data subsets, identify clean data subsets from incompatible data subsets;

[0073] S4. Estimate the access behavior data in the server access behavior dataset based on out-of-packet estimation of the clean data subset.

[0074] The method starts at S1. In S1, a server access behavior dataset is obtained, wherein the server access behavior dataset includes crawler behavior data and normal access behavior data.

[0075] According to an embodiment of the present invention, crawling behavior is detected and labeled from the server side to obtain a labeled server access behavior dataset, thereby providing a preliminary screening basis for subsequent noise data detection and valuation. Specifically, the server access behavior dataset is composed of server access behavior data, and the data is manually labeled and divided into crawler behavior data and normal access behavior data. Crawler behavior data refers to data generated by web crawlers; normal access behavior data refers to data generated by ordinary users' normal web browsing activities through browsers or other client software. The server access behavior dataset can be described from the following two aspects:

[0076] 1) Each piece of data in the dataset consists of a behavioral feature vector, which includes the following features:

[0077] Traffic characteristics: request rate, peak traffic, number of requests per second, etc.

[0078] Path characteristics: access path length distribution, depth distribution, repetition frequency, etc.

[0079] Time series characteristics: access interval distribution, time series characteristics of requests;

[0080] Interaction characteristics: whether it contains abnormal user agent strings (User-Agent), referer distribution patterns, etc.

[0081] Comprehensive features: Combine traffic and access logs to calculate comprehensive features such as the proportion of repeated behavior and request distribution.

[0082] 2) For the entire data set, it is manually labeled according to the access behavior and divided into two categories: crawler behavior data and normal access behavior data.

[0083] Then, S2 is executed, in which the server access behavior dataset is divided into mutually disjoint and incompatible data subsets based on the self-expanding error.

[0084] According to an embodiment of the present invention, for a server access behavior dataset: D action ={(x i ,y i )|i=1,...,n}, where x i Represents the i-th behavior data, y i ∈{0,1} K Denotes the label of the i-th behavior data, and K denotes the number of categories of behavior data. The server access behavior dataset is sampled in multiple rounds, and the sampling size is reduced in each round, that is, the next round of sampling dataset D m+1 is the initial sampling data set D for this round m Subtract the incompatible data subset D′ obtained in this round of sampling m ; Until the sampling size is 0, the server access behavior dataset is divided into multiple incompatible data subsets D1',...,D based on the self-expansion error M Specifically, initialize the sampling data set: D1 = D action ; Initialize the self-expansion error of the current data set: e0(x j ,y j )=0,(x i ,y i )∈D1; For the mth round, m∈{1,...,M}, assuming that t-1 rounds of sampling have been performed, in the tth round of sampling, the sampling rule is:

[0085]

[0086] Among them, S represents the candidate incompatible data subset of this round; D m represents the initial data set of this round; S t-1 represents the data subset sampled in the t-1th cycle of this round; |S t-1 | represents the data subset S t-1 The same is true for |S|; β∈(0,1) represents a fixed-size threshold; therefore, β|S t-1 | represents the sampling size. As the loop progresses, the sampling size will gradually decrease.t-1 (x j ,y j ) represents the self-expansion error of the t-1th cycle of this round;

[0087] Data subset S t is used to train the classifier f t ; then the self-expanding error will be updated according to the loss function l(·,·):

[0088] e t (x j ,y j )=η·e t-1 (x j ,y j )+(1-η)·l(y j ,f t (x j ))

[0089] Among them, f t (x j ) represents the classifier f t For the training data x j The predicted label, η∈(0,1) represents the weight coefficient of the soft update. Assume that in the Tth time, the sampled data subset S T The following conditions are met:

[0090] |S T |≤(1-β)·|D action |

[0091] At this time, the data subset S T It can be considered as a possible data subset of the current data set, a data subset that only generalizes itself; that is: D m '=S T is the incompatible data subset in the current dataset;

[0092] At this point, the next round of sampling data set will be updated as: D m+1 =D m \D m '; that is: D m+1 =D m -D m ';

[0093] when When the server access behavior dataset is divided into finite incompatible data subsets; for the convenience of representation, let D1',...,D M ' are incompatible data subsets based on self-expanding error partitioning, which do not intersect with each other.

[0094] Then S3 is performed, in which a clean data subset is identified from the incompatible data subset based on the compatibility theorem of disjoint data subsets.

[0095] According to an embodiment of the present invention, through D1',...,D M 'Train classifier f1',...,f M ';At the same time, define the empirical risk as:

[0096]

[0097] Where D′ represents the tested data set; g(f(x j )≠y j ) represents the indicator function. When the classifier f is used for the behavioral data x j The category prediction f(x j ) is wrong, then g(f(x j )≠y j )=1, otherwise g(f(x j )≠y j )=0; that is:

[0098]

[0099] The disjoint subset compatibility theorem states that a classifier trained on a clean data subset has a higher recognition accuracy on a disjoint clean data subset than a On the contrary, the recognition accuracy of the classifier trained on the clean data subset is lower than that on the disjoint noise data subset. Using the compatibility theorem for disjoint data subsets, if the empirical risk of the classifier for incompatible data subsets is lower than The data subset used to train the classifier will vote for the corresponding incompatible data subset:

[0100]

[0101] Initialize a clean data subset: when When the corresponding incompatible data subset D j ' is merged into the clean data subset, that is: c_data=c_data∪D j ',

[0102] Then, S4 is executed, in which the access behavior data in the server access behavior dataset is valued based on the out-of-pack estimation of the clean data subset.

[0103] According to an embodiment of the present invention, multiple samplings with replacement are performed on the clean data subset to obtain B data subsets: G1,...,G b ,...,GB ; Data subset G1,...,G b ,...,G B Used to train weak classifiers: f1^,...,f b ^,...,f B ^; For the training data point (x i ,y i )∈D action , out-of-bag estimation is performed, and the estimated value is regarded as a behavioral data point (x i ,y i )’s value:

[0104]

[0105] Right now: Based on the clean subset G1,...,G b ,...,G B The out-of-package estimation is also the behavior data point (x i ,y i )’s value.

[0106] Another embodiment of the present invention proposes an anti-crawler data valuation system based on data incompatibility and out-of-package estimation, such as Figure 3 As shown, the system includes:

[0107] A data acquisition module 310 is configured to acquire a server access behavior data set, wherein the server access behavior data set includes crawler behavior data and normal access behavior data;

[0108] an incompatible data partitioning module 320 configured to partition the server access behavior dataset into mutually disjoint incompatible data subsets based on a self-expanding error;

[0109] a data identification module 330 configured to identify a clean data subset from the incompatible data subsets based on a compatibility theorem for disjoint data subsets;

[0110] The data valuation module 340 is configured to estimate the access behavior data in the server access behavior dataset based on the out-of-packet estimation of the clean data subset.

[0111] In this embodiment, optionally, the step of dividing the server access behavior dataset into mutually disjoint incompatible data subsets based on the self-expansion error in the incompatible data dividing module 320 includes:

[0112] The server access behavior dataset is sampled in multiple rounds, and the sampling size is reduced in each round, that is, the next round of sampling dataset D m+1 is the initial sampling data set D for this round mSubtract the incompatible data subset D′ obtained in this round of sampling m ; Until the sampling size is 0, the server access behavior dataset is divided into multiple incompatible data subsets D1',...,D based on the self-expansion error M '; Among them, for each round of cyclic sampling, the sampling rules are:

[0113]

[0114] Where S represents the candidate incompatible data subset of this round; D m represents the initial data set of this round; S t-1 represents the data subset sampled in the t-1th cycle of this round; |S t-1 | represents the data subset S t-1 The amount of data, β∈(0,1) represents a fixed-size threshold, β|S t-1 | represents the sampling size; x j Indicates the jth behavior data, y j Indicates the label of the j-th behavior data; e t-1 (x j ,y j ) represents the self-expansion error of the t-1th cycle of this round;

[0115] The data subset S obtained by sampling the tth cycle in this round is t Train the classifier f t , and use the classifier f t The loss function is updated from the expanded error e t (x j ,y j ); the self-expansion error e t (x j ,y j ) is updated as follows:

[0116] e t (x j ,y j )=η·e t-1 (x j ,y j )+(1-η)·l(y j ,f t (x j ))

[0117] Among them, f t (x j ) represents the classifier f t For the jth behavior data x j The predicted label, η∈(0,1) represents the weight coefficient of the soft update; l(·,·) represents the classifier f tThe loss function of e0(x j ,y j )=0;

[0118] When each round of cyclic sampling reaches the termination condition, the current cycle is stopped and the incompatible data subset D′ of this round is obtained. m .

[0119] In this embodiment, optionally, the data identification module 330 identifies a clean data subset from incompatible data subsets based on the compatibility theorem of disjoint data subsets, including:

[0120] Initialize the clean data subset to an empty set; define the empirical risk as:

[0121]

[0122] Where D′ represents the tested data set; g(f(x j )≠y j ) represents the indicator function. When the classifier f is used for the behavioral data x j The category prediction f(x j ) is wrong, then g(f(x j )≠y j )=1, otherwise g(f(x j )≠y j )=0;

[0123] Using incompatible data subsets D1',...,D M 'Train classifiers f1',...,f M ';For i,j∈{1,...,M}, if the classifier f i ′ for incompatible data subset D j If the empirical risk of ′ is less than one-half, then the data subset D is incompatible. j ' will be D i 'vote:

[0124]

[0125] Where g(.) represents the indicator function; M represents the total number of incompatible data subsets;

[0126] When voting When , the corresponding incompatible data subset is merged into the clean data subset to obtain the clean data subset.

[0127] In this embodiment, optionally, the valuing of the access behavior data in the server access behavior dataset based on the out-of-packet estimation of the clean data subset in the data valuation module 340 includes:

[0128] Perform multiple sampling with replacement on the clean data subset to obtain B data subsets: G1,...,G b ,...,G B ; Data subset G1,...,G b ,...,G B Used to train weak classifiers: f1^,...,f b ^,...,f B ^; For the training data point (x i ,y i )∈D action , out-of-bag estimation is performed, and the estimated value is regarded as a behavioral data point (x i ,y i ) value; the formula for out-of-pocket estimation is:

[0129]

[0130] Where g(.) represents the indicator function.

[0131] The function of an anti-crawler data valuation system based on data incompatibility and out-of-package estimation proposed in an embodiment of the present invention can be described by the aforementioned anti-crawler data valuation method based on data incompatibility and out-of-package estimation. Therefore, for the parts not described in detail in the system embodiment, please refer to the above method embodiment and will not be repeated here.

[0132] It should be noted that although several units, modules, or submodules are mentioned in the detailed description above, such division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above may be embodied in one module. Conversely, the features and functions of one module described above may be further divided and embodied by multiple modules.

[0133] Furthermore, although the operations of the method of the present invention are described in a particular order in the accompanying drawings, this does not require or imply that these operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0134] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features of these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. Anti-crawler data valuation method based on data incompatibility and out-of-package estimation, characterized by: include: Acquire a server access behavior data set, wherein the server access behavior data set includes crawler behavior data and normal access behavior data; Based on the self-expansion error, the server access behavior dataset is divided into mutually disjoint and incompatible data subsets, including: performing multiple rounds of cyclic sampling on the server access behavior dataset, reducing the sampling size in each round, that is, the next round of sampling dataset D m+1 is the initial sampling data set D for this round m Subtract the incompatible data subset D' obtained in this round of sampling m ; Until the sampling size is 0, the server access behavior dataset is divided into multiple incompatible data subsets D1',...,D based on the self-expansion error M '; Among them, for each round of cyclic sampling, the sampling rules are: Where S represents the candidate incompatible data subset of this round; D m represents the initial data set of this round; S t-1 represents the data subset sampled in the t-1th cycle of this round; |S t-1 | represents the data subset S t-1 The amount of data, β∈(0,1) represents a fixed-size threshold, β|S t-1 | represents the sampling size; x j Indicates the jth behavior data, y j Indicates the label of the j-th behavior data; e t-1 (x j ,y j ) represents the self-expansion error of the t-1th cycle of this round; The data subset S obtained by sampling the tth cycle in this round is t Train the classifier f t , and use the classifier f t The loss function is updated from the expanded error e t (x j ,y j ); the self-expansion error e t (x j ,y j ) is updated as follows: e t (x j ,y j )=η·e t-1 (x j ,y j )+(1-η)·l(y j ,f t (x j )) Among them, f t (x j ) represents the classifier f t For the jth behavior data x j The predicted label, η∈(0,1) represents the weight coefficient of the soft update; l(·,·) represents the classifier f t The loss function of e0(x j ,y j )=0; When each round of cyclic sampling reaches the termination condition, the current cycle is stopped and the incompatible data subset D' of this round is obtained. m ; Based on the compatibility theorem of disjoint data subsets, clean data subsets are identified from incompatible data subsets; The access behavior data in the server access behavior dataset is valued based on out-of-pack estimation of the clean data subset.

2. The anti-crawler data valuation method based on data incompatibility and out-of-package estimation according to claim 1, characterized in that: The access behavior data in the server access behavior data set includes a behavior feature vector, and the behavior feature vector includes a flow feature, a path feature, a timing feature, an interaction feature, and a comprehensive feature.

3. The anti-crawler data valuation method based on data incompatibility and out-of-package estimation according to claim 1, characterized in that: The conditions for terminating each round of cyclic sampling are: |S T |≤(1-β)·|D action | Among them, S T represents the data subset obtained by sampling in the Tth cycle; D action Represents a server access behavior dataset.

4. The anti-crawler data valuation method based on data incompatibility and out-of-package estimation according to claim 3, characterized in that: The compatibility theorem based on disjoint data subsets, identifying a clean data subset from incompatible data subsets includes: Initialize the clean data subset to an empty set; define the empirical risk as: Where D' represents the tested data set; g(f(x j )≠y j ) represents the indicator function. When the classifier f is used for the behavioral data x j The category prediction f(x j ) is wrong, then g(f(x j )≠y j )=1, otherwise g(f(x j )≠y j )=0; Using incompatible data subsets D1',...,D M 'Train classifiers f1',...,f M ';For i,j∈{1,...,M}, if the classifier f i 'For incompatible data subset D j If the empirical risk is less than one-half, then the data subset D is incompatible j 'Will be D i 'vote: Where g(.) represents the indicator function; M represents the total number of incompatible data subsets; When voting When , the corresponding incompatible data subset is merged into the clean data subset to obtain the clean data subset.

5. The anti-crawler data valuation method based on data incompatibility and out-of-package estimation according to claim 4, characterized in that: The out-of-packet estimation based on the clean data subset to estimate the access behavior data in the server access behavior dataset includes: Perform multiple sampling with replacement on the clean data subset to obtain B data subsets: G1,...,G b ,...,G B ; Data subset G1,...,G b ,...,G B Used to train weak classifiers: f1^,...,f b ^,...,f B ^; For the training data point (x i ,y i )∈D action , out-of-bag estimation is performed, and the estimated value is regarded as a behavioral data point (x i ,y i ) value; the formula for out-of-pocket estimation is: Where g(.) represents the indicator function. 6.Anti-crawler data valuation system based on data incompatibility and out-of-package estimation, characterized by: include: a data acquisition module configured to acquire a server access behavior data set, wherein the server access behavior data set includes crawler behavior data and normal access behavior data; An incompatible data partitioning module is configured to partition the server access behavior dataset into mutually disjoint incompatible data subsets based on a self-expanding error, comprising: The server access behavior dataset is sampled in multiple rounds, and the sampling size is reduced in each round, that is, the next round of sampling dataset D m+1 is the initial sampling data set D for this round m Subtract the incompatible data subset D' obtained in this round of sampling m ; Until the sampling size is 0, the server access behavior dataset is divided into multiple incompatible data subsets D1',...,D based on the self-expansion error M '; Among them, for each round of cyclic sampling, the sampling rules are: Where S represents the candidate incompatible data subset of this round; D m represents the initial data set of this round; S t-1 represents the data subset sampled in the t-1th cycle of this round; |S t-1 | represents the data subset S t-1 The amount of data, β∈(0,1) represents a fixed-size threshold, β|S t-1 | represents the sampling size; x j Indicates the jth behavior data, y j Indicates the label of the j-th behavior data; e t-1 (x j ,y j ) represents the self-expansion error of the t-1th cycle of this round; The data subset S obtained by sampling the tth cycle in this round is t Train the classifier f t , and use the classifier f t The loss function is updated from the expanded error e t (x j ,y j ); the self-expansion error e t (x j ,y j ) is updated as follows: e t (x j ,y j )=η·e t-1 (x j ,y j )+(1-η)·l(y j ,f t (x j )) Among them, f t (x j ) represents the classifier f t For the jth behavior data x j The predicted label, η∈(0,1) represents the weight coefficient of the soft update; l(·,·) represents the classifier f t The loss function of e0(x j ,y j )=0; When each round of cyclic sampling reaches the termination condition, the current cycle is stopped and the incompatible data subset D' of this round is obtained. m ; a data identification module configured to identify a clean data subset from the incompatible data subsets based on a compatibility theorem of disjoint data subsets; The data valuation module is configured to estimate the access behavior data in the server access behavior dataset based on the out-of-packet estimation of the clean data subset.

7. The anti-crawler data valuation system based on data incompatibility and out-of-package estimation according to claim 6, characterized in that: The data identification module, based on the compatibility theorem of disjoint data subsets, identifies a clean data subset from incompatible data subsets, including: Initialize the clean data subset to an empty set; define the empirical risk as: Where D' represents the tested data set; g(f(x j )≠y j ) represents the indicator function. When the classifier f is used for the behavioral data x j The category prediction f(x j ) is wrong, then g(f(x j )≠x j )=1, otherwise g(f(x j )≠y j )=0; Using incompatible data subsets D1',...,D M 'Train classifiers f1',...,f M ';For i,j∈{1,...,M}, if the classifier f i 'For incompatible data subset D j If the empirical risk is less than one-half, then the data subset D is incompatible j 'Will be D i 'vote: Where g(.) represents the indicator function; M represents the total number of incompatible data subsets; When voting When , the corresponding incompatible data subset is merged into the clean data subset to obtain the clean data subset; The data valuation module, in which the out-of-packet estimation based on the clean data subset is used to estimate the access behavior data in the server access behavior dataset, comprises: Perform multiple sampling with replacement on the clean data subset to obtain B data subsets: G1,...,G b ,...,G B ; Data subset G1,...,G b ,...,G B Used to train weak classifiers: f1^,...,f b ^,...,f B ^; For the training data point (x i ,y i )∈D action , out-of-bag estimation is performed, and the estimated value is regarded as a behavioral data point (x i ,y i ) value; the formula for out-of-pocket estimation is: Where g(.) represents the indicator function.

Images